# Release Notes

> Release notes from the previous versions of Flexter.

---

Site [llms.txt](/flexter/v2.1/llms.txt)

---

### FLEXTER 2.14.2

_New features_

* **merge2er**
  * Declared data type changes for selected columns:
    * `--in-column-type [[SCHEMA.]TABLE.]COLUMN=DATA_TYPE` — Enforces the data type of the matched columns when the tables are loaded, instead of the inferred one (e.g. csv/tsv/psv columns are strings by default). The path is a glob: `**` matches anything (crossing dots), `*` and `?` match within a single segment. Can be repeated. Ex: `"**.??code=DECIMAL(38,18)"`, `"tab.col=BIGINT"`.
    * `--in-column-type-pattern [[SCHEMA.]TABLE.]COLUMN=DATA_TYPE` — Same as `--in-column-type`, but the path is a regular expression that must match the whole column path.
    * `--out-column-type [[SCHEMA.]TABLE.]COLUMN=DATA_TYPE` — Converts the matched columns to the data type after the tables are loaded and before they are written. Same path and glob rules as `--in-column-type`. Can be repeated. Ex: `"**.amount=DECIMAL(38,18)"`.
    * `--out-column-type-pattern [[SCHEMA.]TABLE.]COLUMN=DATA_TYPE` — Same as `--out-column-type`, but the path is a regular expression that must match the whole column path.

_Improvements_

* **common**
  * Improved the Map and Key-Value parsing for `--rename`, `--in-opt` and `--out-opt`.
* **docker**
  * **flexter-cmd** switched to [spark](https://hub.docker.com/_/spark) from [apache/spark](https://hub.docker.com/r/apache/spark) to get recent fixes and updates at base image level.

_Bug Fixes_

* **merge2er**
  * Fixed partition detection for partitioned input folders: when a partition level contained multiple directories (e.g. `year=2024/month=1` and `year=2024/month=2`), the same partition key was detected once per sibling directory, resulting in duplicated partition columns. The partition keys are now detected along a single branch.


### FLEXTER 2.14.1

_New features_

* **xml2er / json2er**
  * `--validate-empty-as-null` — Treat empty values as `NULL` for `NOT NULL` validation. Uses the configured tolerance mode for warnings or interruptions.
* **merge2er**
  * `--tolerance-level LEVEL` — configurable error handling with three modes:
    * `zero` — interrupt the job at the first table that fails to be written.
    * `document` — skip the tables that fail to be written and keep processing the other tables.
    * `graceful` — same as `document` for merge2er (existing default behaviour). The job fails when no table could be written.

_Improvements_

* **flexter-ui**
  * Moved the Import Model button above the pagination controls.

_Bug Fixes_

* **xml2er / json2er**
  * Fixed an issue where invalid conversion errors were incorrectly reported as `doc_stat.empty_notnull_qty`.
* **xml2er / json2er / xsd2er**
  * Fixed an issue where numeric precision and scale aggregations resulted in reduced precision and scale values.
* **flexter-ui**
  * Fixed data type status reporting issues during job consolidation.
* **merge2er**
  * Fixed an issue where write errors were skipped without interrupting the entire batch.
* **common**
  * Fixed an issue where Snowflake file-based private keys were not detected as reliably as inline keys.
  * Resolved Snowflake column ordering issues by linking `--remap-tables` and `--remap-table` with `--out-opt column_mapping=name`, allowing column mapping by name to be enabled on a per-table basis.


### FLEXTER 2.14.0

_Improvements_

* **flexter-ui**
  * Upgraded to Spring Boot 4.1.1, resolving all CVEs related to older versions.

_Bug Fixes_

* **xml2er / json2er**
  * Fixed an issue where the sequence formula did not increment `rowId` consistently as precision increased.
* **common**
  * Reverted Snowflake writes to avoid using `column_mapping=name` by default, as it prevents new tables from being created. Related to [#377](https://github.com/snowflakedb/spark-snowflake/issues/377).

_Dependency Updates_

* Apache Spark (Default Installation): **3.5.6 → 3.5.9**
* MariaDB JDBC: **3.5.7 → 3.5.10**
* MongoDB Driver Sync (JDBC): **5.6.4 → 5.1.4** (Spark Connector Compatible)
* MS SQL Server JDBC: **13.2.1 → 13.6.0**
* Oracle JDBC: **23.26.1.0.0 → 23.26.3.0.0**
* PostgreSQL JDBC: **42.7.10 → 42.7.13**
* Snowflake JDBC: **3.28.0 → 4.0.2**
* Teradata JDBC: **20.00.00.54 → 20.00.00.60**
* BigQuery Spark Connector: **0.44.0 → 0.45.0**
* MongoDB Spark Connector: **10.6.0 → 10.7.0**
* Snowflake Spark Connector: **3.1.7 → 3.2.2**
* Iceberg Spark Connector: **1.10.1 → 1.11.0**

### FLEXTER 2.13.9

_New features_

* **common**
  * Snowflake key-pair authentication:
    * `--in-key`, `--in-key-file`, `--in-key-password` — configure Snowflake key-pair authentication for input sources using inline private keys or key files (with optional passphrase).
    * `--out-key`, `--out-key-file`, `--out-key-password` — configure Snowflake key-pair authentication for output targets using inline private keys or key files (with optional passphrase).

_Improvements_

* **flexter-ui**
   * Improved workflow loading performance and made job counts and duration values more accurate.
   * Removed hidden date filters from the document statistics page.
   * Updated backend security dependencies.
   * Expired Azure AD sessions now redirect users to the sign-in page.
   * Improved minor UI labels for consistency. 
* **common**
  * Updated Snowflake key-pair encryption compatibility to support the latest Snowflake JDBC driver releases.
  * Snowflake private key parameters (`--in-key`, `--out-key`) are now masked in command logs and previews to protect credentials.
  * Added password-protected decryption support for inline plain-text Snowflake private keys, matching existing file-based key behaviour.

_Bug Fixes_

* **flexter-ui**
  * Fixed an issue where diagrams did not refresh correctly.
  * Fixed DDL file extension handling. 
* **xml2er**
  * Fixed an issue where namespace-qualified XML tags failed to match namespace-less schema mappings.
* **common**
  * Fixed Snowflake column matching when target table column order changes by setting the `spark-snowflake` output option `column_mapping=name` by default.  
 

### FLEXTER 2.13.8

_Improvements_

* **flexter-ui**
  * Improved UI SQL queries performance.
* **flexchma** 
  * Improved database metadata index performance.

_Bug Fixes_

* **xml2er / json2er**
  * Fixed document name generation when using multiple input paths, preventing incorrect or empty id_doc values for files from subsequent paths.

* **flexter-ui**
   * Fixed enforced `flexmeta` schema usage, so other schema names now work correctly.

* **flexchma** 
  * `calcmap`
    * Fixed mandatory NOT NULL columns handling


### FLEXTER 2.13.7

_Improvements_

* **xml2er / json2er**
  * Improved timestamp handling in XML and JSON imports, including support for microseconds.

_Bug Fixes_

* **flexchma**
  * Fixed an issue where very large “Referenced Namespaces” values were not handled correctly.
  * `migration.py`
    * Fixed an export issue where backslashes were not escaped correctly, which could cause schema imports to fail.

### FLEXTER 2.13.6

_Bug Fixes_

* **xsd2er**
  * Fixed stats aggregation where DECIMAL combined with DOUBLE PRECISION was recognized as TIME, and TIME combined with any numeric type was promoted to VARCHAR. 

### FLEXTER 2.13.5

_Bug Fixes_

* **flexchma**
  * `migration.py` 
    * Fixed `FLEXTER_MIGRATION_LOG` being ignored in migration subprocesses when set externally.
* **flexter-ui**
  * Fixed an RxJS subscription leak in AppComponent route-title handling that accumulated subscriptions and re-issued workflow-name lookups on every navigation.
  * Fixed the multi-second delay when navigating to auth-guarded routes.
  * Fixed an auto-refresh interval not being persisted on the execution log page.

### FLEXTER 2.13.4

_New features_

* **common**
  * `--tz TIMEZONE` - specify the timezone to use when parsing dates and timestamps. Defaults to the JVM timezone.

_Bug Fixes_

* **merge2er**
  * Fixed incorrect inference of numeric types as double in all cases for SAS data files (`.sas7bdat`)
  * Fixed an issue where setting `spark.sql.timestampType=TIMESTAMP_NTZ` had no effect.
  * Fixed issues when processing multiple paths passed via the INPUT or `--manifest` parameter.

### FLEXTER 2.13.3

_New features_

* **xml2er / json2er / merge2er**
  * `--comment-len N` — cap or disable the column comments Flexter writes when creating tables. Fixes write failures on **AWS Glue / Hive**, which reject column comments longer than 255 characters.
    * For Glue / Hive, use `--comment-len 254` (or `0` to suppress comments entirely).
    * Default (`-1`) defers to each JDBC dialect's own limit. Hive is now standardised at **255** characters — previously **256**, which was the exact mismatch that caused the failures this release fixes.
    * Applies to both JDBC and Hive/Iceberg outputs.
    * Configuration key: `output.commentLen`.


### FLEXTER 2.13.2

_New features_

* **xml2er / json2er / merge2er**
  * `--manifest PATH` — specify input paths via a CSV manifest file. Supports 1–3 column formats (File Path; Bucket/Path; Bucket/Path/Version). Paths inside a manifest are resolved relative to the manifest's own filesystem location, and multiple `--manifest` arguments can be combined with direct input paths.

_Bug Fixes_

* **common**
  * Fixed a JSON parameter serialisation error that caused job-parameter records to fail; column-value quoting now escapes special characters correctly.
  * Fixed a crash during Spark job initialisation when configuration values such as `jars` or `files` were left empty.
  * Added a validation check to ensure the required `--output` (`-o`) parameter is provided before processing begins, preventing silent failures.

_Improvements_

* **common**
  * Cloud storage detection (S3 and other cloud schemes) now validates *all* input paths, including those listed inside manifests, ensuring credentials are configured correctly for multi-path jobs.


### FLEXTER 2.13.1

_Bug Fixes_

* **xml2er**
  * Fixed an "integer out of range" failure that occurred when writing reference-count arrays for very large schemas. The affected columns have been widened so this no longer happens.
* **xsd2er**
  * ComplexType elements that extend or restrict `simpleContent` are no longer wrongly flagged as mixed content. Classification now triggers only when an element truly contains both text and child elements.
* **common**
  * Fixed a class-not-found error on the `jdbcx` data source when running parallel JDBC writes with `--tables-at-once != 1`. Parallel writes now behave as expected.
  * Iceberg writes automatically detect whether the target table exists and switch between *create* and *append* modes accordingly, preventing task failures on first-time writes.


### FLEXTER 2.13.0

_New features_

* **merge2er**
  * Native SAS datafile support — `.sas7bdat` files can be loaded as individual tables (`--file-table` / `-q`) or merged from a directory into a single dataframe.
* **xml2er / json2er / merge2er**
  * Apache Iceberg write integration — output operations now use the `DataFrameWriterV2` API for atomic commits. Catalog setup is centralised and resolves catalog names through `--in-schema` or `--table` parameters.
  * `--round-mode MODE` — control rounding behaviour (e.g. `HALF_UP`, `HALF_EVEN`) during precision or scale reduction. Default remains `HALF_UP`.
  * `--in-schema` and `--has-header` parameters are now active across all three pipelines. `--meta-schema` added for configuring the metadata output schema.
* **utility**
  * `download.sh` extended with new flags:
    * `-j` / `-J <VERSION>` — download core Iceberg dependencies.
    * `-m <MODULE>` — include specific Iceberg modules.
    * `-n` — retrieve SAS7BDAT format assets.

_Bug Fixes_

* **xsd2er**
  * Stats merging now handles same-type elements correctly when one variant contains only text and the other has child elements (previously produced an inconsistent result).
* **common**
  * Iceberg jobs now read the input schema from `--in-schema` (rather than incorrectly defaulting to `--table`), so the catalog resolves to the right target.
  * The API batch size no longer appears in the run summary unless it has actually been configured.
  * Run summary line order corrected — `xpaths`, `documents`, and overall status are now printed in the expected sequence.
* **flexchma**
  * Removed duplicate rows in consolidated-stats reports that appeared when the same XPath came from multiple source schemas.

_Flexter UI/API_

* **flexter-ui/api**
  * DocStat filtering — per-job results are no longer mixed with *all jobs* results, and the filter slider correctly retains its maximum handle when switching between views.
  * Safer datatype-change detection — changes are no longer flagged as breaking when the new column type still fits the existing data, allowing safe demotions to pass through.

_Dependency Updates_

* Oracle JDBC: **23.9.0.25.07 → 23.26.1.0.0**
* PostgreSQL JDBC: **42.7.7 → 42.7.10**
* MariaDB JDBC: **3.5.5 → 3.5.7**
* MS SQL Server JDBC: **13.2.0 → 13.2.1**
* MongoDB Driver Sync (JDBC): **5.5.1 → 5.6.4**
* Snowflake JDBC: **3.24.2 → 3.28.0**
* Teradata JDBC: **20.00.00.49 → 20.00.00.54**
* Snowflake Spark Connector: **3.1.4 → 3.1.7**
* BigQuery Spark Connector: **0.42.4 → 0.44.0**
* MongoDB Spark Connector: **10.5.0 → 10.6.0**
* **New**: `spark-sas7bdat 3.0.0-s_2.12` (enables native SAS ingestion).
* **New**: `iceberg-spark-runtime-3.5 1.10.1` (enables the new Iceberg output pipeline).
* **Downgrade**: GCS Connector **2.2.28 → 2.2.14** for Iceberg/Hadoop classpath alignment. Please re-verify if you pin a higher GCS version externally.




### FLEXTER 2.12.7

_New features_

* **xml2er / json2er**
  * `--stop-mixed` — immediately terminates a job when mixed content is detected, without extracting data or writing statistics. Useful as a quick validation gate before a long extraction.
  * `--tolerance-level LEVEL` — configurable error handling with three modes:
    * `zero` — abort the entire batch if any document fails.
    * `document` — isolate failures by rejecting only the affected document.
    * `graceful` — attempt to extract as much as possible (existing default behaviour).
* **xml2er / json2er / merge2er**
  * `--legacy-dialect-version N` — opt in to the updated Hive and Impala type mappings. Version `1` (default) preserves existing behaviour; version `2` enables the newer mappings for small-integer and date types.
* **flexchma**
  * Database migration v2.12.7 refines `get_mapping` and the consolidated-stats functions to ensure XPaths from historical schemas appear correctly in reporting.

_Bug Fixes_

* **common**
  * Corrected a casing typo in the help text for the `--tolerance-level` parameter.

_Flexter UI/API_

* **flexter-ui/api**
  * Integrated document statistics — a dedicated doc-stats page linked to job navigation, initially filtered by the selected job, with the option to broaden the view to aggregate across all jobs.
  * Enhanced mapping visibility — the Mapping Stats view now includes column data lengths so truncation issues can be spotted visually. The *Show Impact* checkbox narrows the list to inconsistent entries only — datatype mismatches, truncations, cardinality changes, and new XPaths.
  * Added a *Doc Stats* shortcut on the execution-log page and a scrollable container for long-running processes.


### FLEXTER 2.12.6

_New features_

* **flexter-ui/api**
  * Schema-mapping export endpoint — returns a schema mapping as **JSON** (default) or **CSV**, enabling automated downloading and external scripting without using the interface. A mapping-comparison endpoint is reserved for a future release.

_Bug Fixes_

* **xml2er**
  * Mixed-content warnings no longer repeat for nodes already classified as mixed content in the logical schema. Warnings are now raised only the first time mixed content is genuinely discovered.


### FLEXTER 2.12.5

_Bug Fixes_

* **flexchma / json2er**
  * `calcmap` no longer generates table or column names containing literal backslash characters that previously caused Snowflake write failures from json2er jobs.
* **xml2er**
  * Text-only elements are no longer incorrectly identified as mixed content. Type information is now preserved during statistics collection, so these elements are classified correctly without the erroneous string-length cap.


### FLEXTER 2.12.4

_Bug Fixes_

* **xml2er / json2er**
  * `--extra-column FILENAME|FILEPATH` no longer produces duplicate entries on root tables that already have native `filepath` or `filename` fields.
  * Trailing zeros in decimal values no longer trigger false precision-loss warnings — they are stripped before the precision check runs.
* **xml2er**
  * XML attribute nodes are no longer wrongly flagged as mixed content; attributes are now skipped during mixed-content classification.
* **merge2er**
  * Cross-filesystem merges no longer fail when the input and output directories reside on different storage platforms. Temporary merge files now use the appropriate filesystem.
* **common**
  * Resolved a stack overflow in the numeric-statistics path caused by an infinite-recursion bug.

_Improvements_

* **common**
  * Help text for `--truncate-on-overflow` now indicates that it accepts a `VALUE` argument.

_Flexter UI/API_

* **flexter-ui/api**
  * Stats comparison no longer reports false-positive "new XPath" notifications, and optional-cardinality columns are no longer flagged as breaking changes.


### FLEXTER 2.12.3

_New features_

* **xml2er / json2er**
  * `--truncate-on-overflow` — explicit control over data that exceeds a column's length limit. Truncation remains the default to keep behaviour consistent with earlier versions; setting this to `false` causes the job to fail on overflow instead of silently truncating, preventing unintended data loss.

_Bug Fixes_

* **common**
  * `--default-varchar-len` no longer overrides columns that already have a length defined. The default now applies only to string columns with no length set, leaving explicitly-defined lengths untouched.
  * Fixed an inconsistency where custom output-format properties were ignored for certain targets. Property application is now consistent across all supported output formats.


### FLEXTER 2.12.2

_Bug Fixes_

* **flexter-docker / flexter-ui/api**
  * Schema import/export buttons no longer fail on default `docker compose up -d` deployments. The backend now correctly resolves the datasource configuration for out-of-the-box containerised setups.


### FLEXTER 2.12.1

_New features_

* **flexchma**
  * Compressed schema export (`--zip`) — exports the metadata directory as a zip archive (also triggered by a `.zip` extension on the output path) and removes the uncompressed source files.
  * Automated archive ingestion — schema imports now recognise and extract `.zip` files, with the temporary directory cleared once the operation finishes.
* **xsd2er**
  * `--default-varchar-len` can now be stored as part of the global settings (`-g`). It is applied as a database update step after schema calculation, assigning lengths to string columns that have none defined.

_Bug Fixes_

* **flexchma**
  * Schema import no longer fails when the statistics file is unavailable — a warning is logged and the remaining steps proceed.

_Improvements_

* **utility**
  * `download.sh` — Hadoop, Hive, and Spark downloads now use Apache's dynamic mirror redirect service for higher throughput.
  * `download.sh` — all component download URLs standardised to HTTPS.

_Database_

* **flexchma**
  * `compare_mappings` enhanced to handle duplicate foreign keys. It now uses a **Full Outer Join** between source and target schema mappings for better delta detection, improved statistics-presence tracking, and consistent result ordering.

_Flexter UI/API_

* **flexter-ui/api**
  * Optional Azure AD sign-in — **opt-in** Azure AD authentication. When no Azure AD configuration is provided, the interface remains accessible without authentication, exactly as before.
  * Visual schema management — the new zip-based import and export capabilities are now available directly from the web dashboard.
  * Credential masking — passwords supplied to job runs are obscured in command previews and job-detail views, and are filtered out of any command saved to the metadata database.
  * Frontend modernisation — application stack upgraded from **Angular 5 to Angular 18**, with refreshed assets and browser identity.
  * *Show Impact* toggle fix — highlighted mismatches now render correctly in the mapping views.


### FLEXTER 2.12.0

_New features_

* **xsd2er**
  * Comprehensive XSD constraint support — `xsd2er` now extracts and aggregates `minLength`, `maxLength`, `totalDigits`, `fractionDigits`, and length attributes from union and list types, ensuring full metadata coverage.
* **xml2er / xsd2er / json2er / merge2er**
  * `--store-credentials` — API, input, and output credentials are no longer included in serialised job parameters by default. Pass `--store-credentials` to opt back in to the previous behaviour.
* **common**
  * `--api-conn-timeout` and `--api-read-timeout` — connection and read timeouts for the Flexter API can now be configured from the command line.

_Bug Fixes_

* **json2er**
  * Fixed a formatting error in mixed-mode JSON parsing where arrays were incorrectly terminated with curly braces instead of square brackets.
  * Field labels are no longer omitted for values nested inside JSON arrays during ingestion.
* **xml2er / json2er**
  * Decimal-statistics accuracy — corrected a calculation error that produced incorrect precision and scale metrics for decimal fields.
  * `--default-float-type max` now applies maximum precision consistently across all scales when operating in `max` mode.
* **common**
  * Snowflake authentication no longer passes `null` when the private-key password is omitted — it now correctly falls back to the primary connection password.
  * Restored the missing schema parameters (`origin`, `logical`, `job`, `org`) to the generated job execution strings.

_Flexter UI/API_

* **flexter-ui/api**
  * The backend now applies `stringtype=unspecified` for PostgreSQL automatically, so it no longer needs to be appended to the connection URL.

_Dependency Updates_

* Apache Spark (Default Installation): **3.5.2 → 3.5.6**
* PostgreSQL JDBC: **42.7.3 → 42.7.7**
* Snowflake JDBC: **3.17.0 → 3.24.2**
* Oracle JDBC: **23.4.0.24.05 → 23.9.0.25.07**
* MariaDB JDBC: **3.4.0 → 3.5.5**
* MS SQL Server JDBC: **12.6.3 → 13.2.0**
* MongoDB Driver: **4.8.2 → 5.5.1**


### FLEXTER 2.11
_New features_

* **Java**
  * Extending Java support between 8 and 17 versions
* **Spark**
  * Extending Spark support between 3.1.x and 3.5.x versions 
  * Default Spark upgraded to 3.5.2 version
* **xsd2er**
  * Included `--xpath-phase <unit|def|all>` parameter to help to print out XPaths in different XPath build phases
* **xml2er / json2er**
  * Included `--ignore-mixed-content` parameter to avoid parsing xml tags or json values with mixed content
  * Included `--parallel-sequence` parameter to enforce sequences for parallel environments like clusters. 
* **xml2er**
  * Included `--parse-lib <LIB>` enabling different SAX parser implementations beyond the SAX parser shipped with Java JDK
* **flexter-ui / rest-api**
  * Included endpoint to terminate active jobs

_Improvements_

* **xml2er / json2er**
  * Performance improvements while processing statistics
  * Performance improvements while processing mixed content
  * Memory consumption improvements between parsing and caching data 
* **flexchma / calcmap**
  * The order of the table's columns is now also determined by the numeric suffixes of field names
* **flexchma / migration.py**
  * The export utility consolidates the specified schema and all its parent schemas, ensuring that only the requested model is included
* **flexter-ui**
  * Keeping track of the processed IDs, enabling a check on whether the processes are still alive
* **xsd2er**
  * Improved dependency issues messages, testing all possible issues and compiling a list of issues
  * Included warning messages when root tags match between stats and XSDs, when one side has empty namespace
    
_Fixes_

* **xml2er / json2er / merge2er**
  * Enhanced the maximum decimal precision and scale for BigQuery from 19,9 to 38,38 (maximum accepted by spark) based on recent BigQuery improvements
  * Included `quoteIdentifier` parameter for `jdbc:postgresql://` URI enabling optionally double-quotes in identifiers
  * Oracle CLOB columns weren't accepting text above 4000 characters. 
* **xml2er / json2er**
  * Removed `--tables-at-once` restriction policy to run only in java 8
  * Truncating text content bigger than 16mb for Snowflake, avoiding process crashes
  * Fixed `--id-column <name> extra` and `--extra-column <name> id` parameter cases, which injects an extra column based on input's ID value
  * Numeric data types statistics detection was allowing precision and scale beyond limits.
  * Fixed sequences in Databricks environments where the Spark Cluster is kept alive across multiple jobs. 
* **json2er**
  * Fix for single and empty arrays, they weren't detected as 1:N
* **xsd2er**
  * Fixed false positive mixed content assumption, for abstract xsd types without content 
  * Fix for bi-directional file reference cases, which was causing some failed links and performance loss
  * Fix to handle prefixes larger than 30 characters, now it supports up to 255 characters
  * Fix to bring all historical stats linked with the schema origin, not only the ones informed by --use-stats parameter
  * Fix combined stats with already processed data flow/mapping were causing failures
* **flexter-ui / merge2er**
  * Making new merge2er jobs listed in the flexter-ui jobs list
* **flexter-ui**
  * Removing jobs list Spark History Server button if it isn't set
  * Fixed populating multiple input parameters in job table
* **flexchma / migration.py**
  * Export schemas wasn't including id_du in the exported du_stat tables.
* **flexchma / calcstruct**
  * Stats consolidation should not preserve data_units from historical schemas
* **log4j**
  * File appender implemented for log4j2 (spark 3.3-3.5) as it was for log4j1 (spark 3.1-3.2)
  * Fixed print out commands parameter `-c` for log4j2 (spark 3.3-3.5)
  * Fixed parameter `-L` for log4j2 environments Spark 3.3 and higher 
  * Including the starting log events in log files that use job id in the name.



### FLEXTER 2.10
_New features_

* **Spark**
  * Default Spark upgraded to 3.3.4 version.
* **download.sh**
  * Included option to download _spark-hadoop-cloud_ dependency
  * Included option to download Hadoop Aliyun Cloud dependencies
  * Included option to download Hadoop Tencent OSS Cloud dependencies
  * Included option to download Hadoop Open Stack Cloud dependencies
  * Included option to download Hadoop Huawei Cloud OBS dependencies
  * Separated Google Cloud Storage and Google Big Query options
  * Include option to download separated hive packages
* **xml2er / json2er**
  * Included `--remap-tables` and `--remap-table <TABLE,...>` parameters to reorder table columns based on the tables found in the output

_Fixes_

* **xml2er / json2er / merge2er**
  * Fix in the HDFS blocksize, which Flexter was enforcing always 0 after Spark 3.3 changes.
* **xml2er / json2er**
  * Fix the Spark application name which was appearing with only (...) instead of xml2er (...) or json2er (...).
* **xml2er**
  * Fix performance issues for large files containing mixed content, ex: HTML, formatted text and other tags mixed with text cases.

### FLEXTER 2.9
_New features_

* **flexchma / migration.py** 
  * By default, the export utility consolidates schema's data before exporting it. This behaviour can be disabled passing the optional `--export-full` parameter
  * Bulk copy when importing previously extracted schemas
  * Logging improvements, collecting the full mapping with the list of new schema ids and their original ones
* **flexchma / db**
  * New ad-hoc function for stats consolidation.
  * New accessory function (compact_stats) to consolidate stats and persisting them, with an optional switch to purge obsolete entries
* **flexter-ui / rest-api**
  * Included endpoint to call import/export metadata.

### FLEXTER 2.8
_Fixes_

* **xml2er / xsd2er**
  * Fix in _xsi:type_ cases with missing default types.
* **flexter-ui**
  * Masking passwords sent by parameter.
* **xml2er**
  * Fix attributes of _xsi:type_ tags in the statistics process.


### FLEXTER 2.7
_New features_

* **Spark**
  * Extending Spark support between 3.1.x and 3.3.x versions.
* **Docker**
  * Support to Kubernetes environments.

_Improvements_

* **Databricks**
  * Support delta tables merge schema feature.
    
_Fixes_

* **Spark**
  * Fix dynamic log settings loading for Spark 3.3.
  * Fix loading `--conf` parameters from the application.
* **flexchma**
  * Fix generating schema from previous could cause false positive mixed content cases.
* **Databricks**
  * Job error status detection
* **Cloudera**
  * Fix kyro serialization
* **xml2er / json2er**
  * Fix date formats with 3 character Months like Jan, Feb, Mar... 


### FLEXTER 2.6
_New features_

* **Spark**
  * Extending Spark support between 3.1.x and 3.2.x versions.
* **xml2er**
  * Including the support for _xsi:type_ stats only cases.
    
_Improvements_

* **xml2er / xsd2er**
  * Pattern matching between XML stats and XSDs.
* **docker**
  * Changes to support non-root users for kubernetes.
* **databricks**
  * Support to load flexter and log4j settings.

_Fixes_

* **xml2er**
  * Detecting recursive tables generated by reuse optimisation algorithm.
* **Snowflake**
  * Including truncation for 16+ kilobytes text data.
  * JDBC - _VARIANT_ switched to _VARCHAR_ to avoid 16+ kilobytes issues.
* **json2er**
  * Fixed null values being treated without regard to the column nullable definition.
* **Cloudera**
  * Removing Spark 3.1 and 3.2 verbose logging.
* **Spark**
  * Fixed spark dependencies downloads.
* **xml2er / json2er**
  * Generating a data flow without informing an input or `-x` parameter: ex `xml2er -a123 -g1`.
  * XPaths, tables and columns can be disabled in the metadata database.
  * The parameter `--default-varchar-len` doesn't take effect
  * Calling with `-a <id>` and `-g1` inserts 1 extra metadata schema
* **xsd2er**
  * Possibility to set _mixed=false_ in an inherited _mixed=true_ type.
* **xml2er / json2er / xsd2er**
  * Duplicated table names due to case sensitivity
* **download.sh**
  * Root and non-root users are accessing the same download directory
    

### FLEXTER 2.5
_New features_
* **xml2er**
  * Support casting XML tags using _xsi:type_ attribute.
* **xsd2er**
  * Support associating _xsi:type_ statistics with XSDs statistics.
* **download.sh**
  * Support downloading dependencies: aws, azure, gcloud, snowflake and custom packages.

_Fixes_
* **xml2er / json2er**
  * Numeric values with precision/scale higher than 38 were truncated or not supported by the output resource, now are considered as `VARCHAR`.
  * Making possible to change the level log particular parsing messages in the _log4j.properties_.
* **xml2er**
  * Namespace-less tag names with _type_ were being ignored by xml2er.


### FLEXTER 2.4
_New features_

* **xml2er / json2er**
  * Customizable _integer/float_ output data types.
  * Preventing table/column names that use Oracle reserved words
  * Ignoring jdbc input when the table isn't informed and `-g,--map` parameter called.
  * Including the number of tables to be written in the log
  * Improved the _FK_ref_as_ algorithm to filling it with relative XPaths
	
_Fixes_
* **Spark**
  * Multiple flexter.conf files cause spark crashes
  * JDBC debug is failing after spark 3 upgrade
  * JDBC dialects always being written as _CLOB_ after Spark 3 upgrade
  * PostgreSQL JDBC dialect inserts quotes except for comments
  * Hive annoying WARN messages with Spark 3
* **flexter-ui / rest-api**
  * Rest api doesn't support json2er and merge2er modules
* **flexchma**
  * Calcmap reuse optimisation causes table column loss
* **xml2er / json2er / xsd2er**
  * Omitted `--name-max-len` is truncating generated names bigger than 30
* **xml2er / json2er**
  * `--sequence-type` isn't enforced to reduce the numeric precision
  * failing to parse `-R "columnName=2001-01-01 00:00:00"`
* **xml2er**
  * XML tags with _xsi:nil="true"_ is detected as text tag
  * xml2er doesn't load `-i` parameter from a job
* **json2er**
  * json2er isn't working with mongodb
  * json2er isn't generating correct schema name with select clause
* **xsd2er**
  * xsd2er isn't locating the correct path in docker
  * xsd2er `--stop-policy +0` produce same results as `+1`


### FLEXTER 2.3	
_New features_

* **json2er**
  * Support to MongoDB as input source
* **xml2er / json2er**
  * Support to MongoDB as output target
	
_Fixes_
* **xml2er / json2er / xsd2er**
  * Numeric scale was stored as _null_ in some cases
* **xml2er / json2er**
  * Extra columns with regular expressions

### FLEXTER 2.2
_New features_

* **Spark**
  * Spark baseline version migrated to 3.1.x. It's no longer compatible with Spark 2.x 
  * Default Spark upgraded to 3.1.2 version.

### FLEXTER 2.1	
_New features_

* **xml2er / json2er**
  * Included `--sequence-type <SQLTYPE>` accepting both _VARCHAR_ and _NUMERIC(precision, scale)_ and other numeric variants (_integer_, _long_...)
  * Included `--console <s|p>` shortcut to call flexter application as console functions: ex `xml2er()`
* **Spark**
  * Default Spark upgraded to 2.4.7 version.

_Fixes_
* **flexchma**
  * Calcmap - Preventing generating table/column names starting with _ and numbers

### FLEXTER 2.0
_New features_

* **xml2er / xsd2er**
  * More accurate namespace + xpath analysis
* **flexchma**
  * Flexter Schema now can be installed and upgraded by command-line
	
_Fixes_

* **xml2er**
  * Extra hidden characters are being filtered in XML documents
* **xsd2er**
  * Detecting _xs:nil_ tags as data column without sample
	
### FLEXTER 1.10	
_New features_

* **Yellowbrick**
  * Support to Yellowbrick data warehouse
* **Google Cloud**
  * Support to Google BigQuery data warehouse and Google Storage
* **xml2er / json2er / xsd2er**
  * Included `--default-varchar-len` parameter
  * Changed `--use-stats` parameter now has the `-a` shortcut 
	
### FLEXTER 1.9
_New features_

* **Spark**
  * More compatibility between hive tables and other hive-based tools for orc formats.
* **xml2er / json2er**
  * Included `--namemode` to enforce lower, upper or camel case table/column names.
* **json2er**
  * Accepting json fields with spaces, slashes and other special chars.
* **xml2er / json2er / xsd2er / merge2er**
  * Logging into files throughout log4j
  * Included `--license` parameter to load it externally
* **AWS**
  * Support to Redshift JDBC connections
  * Experimental support for Redshift Spark Connector
	
### FLEXTER 1.8	
_New features_

* **xml2er / json2er**
  * Included --extra-column and --partition-column replacing --partition-by and --partition-by-name parameters.
  * Improved --column and --id-column to accept expressions, casting and aliases.
  * Included --prefix and --suffix at output table names.
  * Included --rename to be able to rename output table names.
* **merge2er**
  * Enabling copying --constraints from jdbc to jdbc tables in the merge2er as an experimental feature.
	
### FLEXTER 1.7
_New features_

* **Spark**
  * Default Spark upgraded to 2.4.3 version.
  * Extending Spark support between 2.1.x and 2.4.x versions.
  * Included other spark-submit parameters into bash script launcher for kerberos authentication (`--queue`, `--principal`, `--keytab`)
* **xml2er / json2er**
  * Printing the output tables' DDL out to the console for Hive or Jdbc dialect
  * Writing tables' schema directly to Hive or Jdbc database
  * Included two further options to disable writing document statistics and xpath statistics
* **xsd2er**          
  * The xsd2er module will keep track of stats used for schema generation
* **merge2er**
  * The merge2er module now works in skip mode and is capable to import SQL query result.
  * Added support for jdbc input/output targets

_Fixes_

* **xml2er / json2er / merge2er**
  * Removed table exists SQL checks from the logs when using the `-c` parameter
  * Disabled quotation for generic JDBC driver, while enabled for specific ones
* **xml2er / json2er**
  * Processing dates with different formats (eg. Day-Month-Year, Month-Day-Year ...)
* **xml2er**
  * Mixed flag is ignored if not explicitly set in stats
* **merge2er**
  * Reviewed merge2er log messages and default output format