Rest API
Enabling it
The change must be done at /usr/share/flexter/flexter-ui/conf/backend.properties or ~/share/flexter/flexter-ui/conf/backend.properties for non root installations.
Please set the cmd.enabled value as below:
cmd.enabled = true
And after that, please restart the flexterui-backend service:
systemctl restart flexterui-backend
Example job command:
Here an example of generating schema from xml, note that I use jq at the end to format the output.
curl -H 'Content-Type: application/json' -XPOST -d '{
"executions":[
{
"app_name":"xml2er",
"source":{
"file":{
"command":"/usr/share/flexter/samples/donut.xml.gz",
"enabled":true
}
},
"target":{
"generation":[
{
"command":"-g",
"enabled":"true",
"value":"3"
}
]
}
}
]
}' http://<FLEXTER_UI_SERVER>:<PORT>/v2/run/cmd | jq
And the related output:
{
"error": false,
"logPath": null,
"cmds": [
"xml2er -j 7"
],
"jobs": [
7
]
}
Tracking the job status
You can also track status of the job calling:
curl http://<FLEXTER_UI_SERVER>:<PORT>/v2/executions/get/7
or
curl http://<FLEXTER_UI_SERVER>:<PORT>/v2/executions/get/7 | jq
And get as output:
{
"params": "/usr/share/flexter/samples/donut.xml.gz -g 3",
"status": "C",
"host": "<HOSTNAME>/<IP_ADDRESS>",
"parm_path_input": null,
"id_org": 1,
"base_path": "/",
"id_app": "local-1674054248306",
"id_lschma": 4,
"start_date": 1674054245996,
"os_user": "root",
"id_sch": 11,
"job_type": "Parse",
"parm_path_output": null,
"params_json": "{\"spark\": {\"master\": \"local[*]\"}, \"action\": {\"parsemode\": \"s\"}}",
"msg_proc_qty": 0,
"log_file": null,
"id_parent_job": null,
"end_date": 1674054257301,
"id_job": 7
}
Example of a more complex job command
curl -H 'Content-Type: application/json' -XPOST -d '{
"executions":[
{
"app_name":"xml2er",
"source":{
"file":{
"command":"/usr/share/flexter/samples/donut.xml.gz",
"enabled":true
},
"origin_id":{
"command":"-x",
"enabled":true,
"value":"11"
}
},
"target":{
"path":{
"command":"-o",
"enabled":true,
"value":"snowflake://<SNOWFLAKE_ADDRESS>?db=<DB>&schema=<SCHEMA>&warehouse=<WAREHOUSE>"
},
"format":[
{
"command":"-u",
"enabled":"true",
"value":"<USER>"
},
{
"command":"-p",
"enabled":"true",
"value":"<PASSWORD>"
}
],
"writing_mode":[
{
"command":"-S",
"enabled":"true",
"value":"o"
},
{
"command":"-e",
"enabled":"true",
"value":"D"
}
]
}
}
]
}' http://<FLEXTER_UI_SERVER>:<PORT>/v2/run/cmd | jq
Schema migration through Flexter API
Schema migration tool Migration.py from flexchma package is now available through Flexter API urls:
- /v2/schema/export
- /v2/schema/import
Configuration
To use schema migration tools 2 configuration must be set up properly:
flexter.migration.binary - path to the migration.py binary
flexter.migration.azcopy_binary - path to the azcopy utility (in case of using Azure Storage Account for import/export)
Default values are:
flexter.migration.binary=/usr/share/flexter/flexchma/schema_migration/migration.py
flexter.migration.azcopy_binary=azcopy
Default flexmeta connection parameters passed to the migration script (username and password) are used from the backend.properties file,
you must specify hostname and port (in case of non-default port 5432) in the requests.
Export schema
Request format
Example command through cURL
$ curl -X POST -H 'Content-Type: application/json' -d '{
"origin_id": "origin_id of the schema you want to export",
"output_path": "path on the local file system where API is deployed, to which schema will be exported",
"az_output_path": "https://<storage_account>.dfs.core.windows.net/<container>/<path>/?<SAS_TOKEN>",
"flexmeta_host": "localhost",
"flexmeta_port": "5432",
"flexmeta_user": "flex2er",
"flexmeta_db": "x2er"
}' http://<FLEXTER_UI_SERVER>:<PORT>/v2/schema/export
Note that in case of exporting schema to Azure Storage Account - az_output_path should end with trailing slash, before passing SAS Token
Also flexmeta_* parameters are optional, but must be submitted with the request if flexter metadata db is on the different host/container
You can also submit flexmeta_password parameter or control it through PGPASSWORD environment variable.
Response format
{
"error": false,
"az_output_path": "https://<storage_account>.dfs.core.windows.net/<container>/<path>/"
}
error parameter in the response can be either:
true- which indicates some error happened during the exporting, full detailed log is printed in the API console, but not available over the response.false- schema exported successfully
Import schema
Request format
Example command through cURL
$ curl -X POST -H 'Content-Type: application/json' -d '{
"input_path": "temporary path to store schema downloaded from adls",
"az_input_path": "https://<storage_account>.dfs.core.windows.net/<container>/<path>/?<SAS_TOKEN>",
"flexmeta_host": "localhost",
"flexmeta_port": "5432",
"flexmeta_user": "flex2er",
"flexmeta_db": "x2er"
}' http://<FLEXTER_UI_SERVER>:<PORT>/v2/schema/import
Note that in case of importing from Azure Storage Account - az_input_path should end with trailing slash, before passing SAS Token
Also flexmeta_* parameters are optional, but must be submitted with the request if flexter metadata db is on the different host/container
You can also submit flexmeta_password parameter or control it through PGPASSWORD environment variable.
Response format
{
"error": false,
"origin_ids": "710, 711"
}
error parameter in the response can be either:
true- which indicates some error happened during the importing, full detailed log is printed in the API console, but not available over the response.false- schema exported successfully
origin_ids parameter contains origin ids under which new schema has been i