+
+See examples of messages that
+
+
+
+- Now that the database is up, go back to the virtual desktop and launch `Teradata Studio Express`
+
+
+
+- When you first start it you will be offered a tour. Once you close the tour, you will see a wizard window to add a new connection. Select `Teradata`:
+
+
+
+- On the next screen, connect to the database on your localhost using `dbc` for the username and password:
+
+
diff --git a/quickstarts/_partials/running-sample-queries.mdx b/quickstarts/_partials/running-sample-queries.mdx
new file mode 100644
index 0000000000..6471ea1690
--- /dev/null
+++ b/quickstarts/_partials/running-sample-queries.mdx
@@ -0,0 +1,9 @@
+import GettingStartedQueries from './getting-started-queries.mdx';
+
+## Run sample queries
+1. We will now run some queries in the VM. To avoid copy/paste issues between the host and the VM, we will open this quick start in the VM. Go to the virtual desktop, start Firefox and point it to this quick start.
+2. Once in Teradata Studio Express, go to `Query Development` perspective (go to the top menu and select `Window` -> `Query Development`).
+3. Connect using the previously created connection profile by double-clicking on `Database Connections` -> `New Teradata`.
+4. Using `dbc` user, we will create a new database called `HR`. Copy/paste this query and run it by hitting the run query button (image:run.query.button.png[Run Query Button, 24, 24]) or pressing [F5] key:
+
+See examples of messages that pdestate returns when the database is still initializing.
+
++PDE state is DOWN/HARDSTOP. + +PDE state is START/NETCONFIG. + +PDE state is START/GDOSYNC. + +PDE state is START/TVSASTART. + +PDE state is START/READY. +PDE state is RUN/STARTED. + +DBS state is 1/1: DBS Startup - Initializing DBS Vprocs +PDE state is RUN/STARTED. + +DBS state is 1/5: DBS Startup - Voting for Transaction Recovery +PDE state is RUN/STARTED. + +DBS state is 1/4: DBS Startup - Starting PE Partitions +PDE state is RUN/STARTED. ++ +
+
+
+Now copy the Bearer token from the ModelOps user interface (Left Menu -> Your Account -> Session Details) and set it here as an environment varibale using the following command:
+
+``` bash , id="Bearer token", role="content-editable emits-gtm-events"
+export BEARER_TOKEN='your_token_here'
+```
+
+Now you can execute the previously created config file, which will create a new ini file inside config folder containing all the required parameters which will be used in the DAG creation step.
+
+``` python , id="Create config ini", role="content-editable emits-gtm-events"
+python3 createConfig.py
+```
+
+## Create a Airflow DAG containing full ModelOps Lifecycle
+
+Now you can create a DAG using the following python code. Add this python code file inside dags folder. This DAG contains 5 tasks of ModelOps lifecycle (i.e., Train, Evaluate, Approve, Deploy and Retire)
+
+.Click to reveal the Python code
+[%collapsible]
+====
+``` python , id="DAG Code", role="content-editable emits-gtm-events"
+import base64
+from datetime import datetime, timedelta, date
+import json
+import os
+import time
+
+from airflow import DAG
+from airflow.operators.python import PythonOperator
+
+import requests
+
+from configparser import ConfigParser
+
+# Read from Config file
+config = ConfigParser()
+config.read('config/modelOpsConfig.ini')
+
+config_main = config["MAIN"]
+config_hyper_params = config["HYPERPARAMETERS"]
+config_resources = config["RESOURCES"]
+config_model = config["MODEL"]
+
+# Default args for DAG
+default_args = {
+ 'owner': 'Tayyaba',
+ 'retries': 5,
+ 'retry_delay': timedelta(minutes=2)
+}
+
+def get_job_status(job_id):
+
+ # Use the fetched Job ID to check Job Status
+ headers_for_status = {
+ 'AOA-PROJECT-ID': config_main['projectid'],
+ 'Authorization': 'Bearer ' + config_main['bearertoken'],
+ }
+
+ status_response = requests.get('https://airflow-u9usja4twtauvt3s.env.clearscape.teradata.com:8443/modelops/core/api/jobs/' + job_id + '?projection=expandJob', headers=headers_for_status)
+ status_json = status_response.json()
+ job_status = status_json.get('status')
+ return job_status
+
+
+def train_model(ti):
+
+ headers = {
+ 'AOA-Project-ID': config_main['projectid'],
+ 'Accept': 'application/json, text/plain, */*',
+ 'Accept-Language': 'en-US,en;q=0.9',
+ 'Authorization': 'Bearer ' + config_main['bearertoken'],
+ 'Content-Type': 'application/json',
+ }
+
+ json_data = {
+ 'datasetId': config_main['trainDatasetId'],
+ 'datasetConnectionId': config_main['datasetConnectionId'],
+ 'modelConfigurationOverrides': {
+ 'hyperParameters': {
+ 'eta': config_hyper_params['eta'],
+ 'max_depth': config_hyper_params['max_depth'],
+ },
+ },
+ 'automationOverrides': {
+ 'resources': {
+ 'memory': config_resources['memory'],
+ 'cpu': config_resources['cpu'],
+ },
+ 'dockerImage': config_model['dockerImage'],
+ },
+ }
+
+
+ response = requests.post('https://airflow-u9usja4twtauvt3s.env.clearscape.teradata.com:8443/modelops/core/api/models/' + config_model['modelid'] + '/train', headers=headers, json=json_data)
+
+ json_data = response.json()
+
+ # Get the Training Job ID
+ job_id = json_data.get('id')
+ ti.xcom_push(key='train_job_id', value=job_id)
+
+ job_status = get_job_status(job_id)
+ print("Started - Training Job - Status: ", job_status)
+
+ while job_status != "COMPLETED":
+ if job_status=="ERROR":
+ print("The training job is terminated due to an Error")
+ ti.xcom_push(key='trained_model_id', value='NONE') # Setting the Trained Model Id to None here and check in next step (Evaluate)
+ break
+ elif job_status=="CANCELLED":
+ ti.xcom_push(key='trained_model_id', value='NONE')
+ print("The training job is Cancelled !!")
+ break
+ print("Job is not completed yet. Current status", job_status)
+ time.sleep(5) #wait 5s
+ job_status = get_job_status(job_id)
+
+ # Checking Job status at the end to push the correct trained_model_id
+ if(job_status == "COMPLETED"):
+ train_model_id = json_data['metadata']['trainedModel']['id']
+ ti.xcom_push(key='trained_model_id', value=train_model_id)
+ print('Model Trained Successfully! Job ID is : ', job_id, 'Trained Model Id : ', train_model_id, ' Status : ', job_status)
+ else:
+ ti.xcom_push(key='trained_model_id', value='NONE')
+ print("Training Job is terminated !!")
+
+
+def evaluate_model(ti):
+
+ trained_model_id = ti.xcom_pull(task_ids = 'task_train_model', key = 'trained_model_id')
+
+ headers = {
+ 'AOA-Project-ID': config_main['projectid'],
+ 'Accept': 'application/json, text/plain, */*',
+ 'Accept-Language': 'en-US,en;q=0.9',
+ 'Authorization': 'Bearer ' + config_main['bearertoken'],
+ 'Content-Type': 'application/json',
+ }
+
+ json_data = {
+ 'datasetId': config_main['evaluatedatasetid'],
+ 'datasetConnectionId': config_main['datasetConnectionId'],
+ 'modelConfigurationOverrides': {
+ 'hyperParameters': {
+ 'eta': config_hyper_params['eta'],
+ 'max_depth': config_hyper_params['max_depth'],
+ },
+ },
+ 'automationOverrides': {
+ 'resources': {
+ 'memory': config_resources['memory'],
+ 'cpu': config_resources['cpu'],
+ },
+ 'dockerImage': config_model['dockerImage'],
+ },
+ }
+
+ if trained_model_id == 'NONE':
+ ti.xcom_push(key='evaluated_model_status', value='FALIED')
+ print("Evaluation cannot be done as the Training Job was terminated !!")
+ else:
+ response = requests.post('https://airflow-u9usja4twtauvt3s.env.clearscape.teradata.com:8443/modelops/core/api/trainedModels/' + trained_model_id + '/evaluate', headers=headers, json=json_data)
+ json_data = response.json()
+
+ # Get the Evaluation Job ID
+ eval_job_id = json_data.get('id')
+ ti.xcom_push(key='evaluate_job_id', value=eval_job_id)
+
+ job_status = get_job_status(eval_job_id)
+ print("Started - Job - Status: ", job_status)
+
+ while job_status != "COMPLETED":
+ if job_status=="ERROR":
+ print("The evaluation job is terminated due to an Error")
+ # Set the Trained Model Id to None here and check in next step (Evaluate)
+ break
+ elif job_status=="CANCELLED":
+ print("The evaluation job is Cancelled !!")
+ break
+ print("Job is not completed yet. Current status", job_status)
+ time.sleep(5) # wait 5s
+ job_status = get_job_status(eval_job_id)
+
+ # Checking Job status at the end to push the correct evaluate_job_id
+ if(job_status == "COMPLETED"):
+ ti.xcom_push(key='evaluated_model_status', value='EVALUATED')
+ print('Model Evaluated Successfully! Job ID is : ', eval_job_id, ' Status : ', job_status)
+ else:
+ ti.xcom_push(key='evaluated_model_status', value='FAILED')
+ print("Evaluation Job is terminated !!")
+
+
+def approve_model(ti):
+
+ evaluated_model_status = ti.xcom_pull(task_ids = 'task_evaluate_model', key = 'evaluated_model_status')
+
+ if evaluated_model_status == 'FAILED':
+ ti.xcom_push(key='approve_model_status', value='FALIED')
+ print("Approval cannot be done as the Evaluation was failed !!")
+ else:
+ trained_model_id = ti.xcom_pull(task_ids = 'task_train_model', key = 'trained_model_id')
+
+ headers = {
+ 'AOA-Project-ID': config_main['projectid'],
+ 'Accept': 'application/json, text/plain, */*',
+ 'Accept-Language': 'en-US,en;q=0.9',
+ 'Authorization': 'Bearer ' + config_main['bearertoken'],
+ 'Content-Type': 'application/json',
+ }
+
+ json_data = {
+ "comments": (base64.b64encode(config_model['approvalComments'].encode()).decode())
+ }
+
+ response = requests.post('https://airflow-u9usja4twtauvt3s.env.clearscape.teradata.com:8443/modelops/core/api/trainedModels/' + trained_model_id + '/approve' , headers=headers, json=json_data)
+ response_json = response.json()
+ approval_status = response_json['status']
+ if(approval_status == 'APPROVED'):
+ ti.xcom_push(key='approve_model_status', value='EVALUATED')
+ print('Model Approved Successfully! Status: ', approval_status)
+ else:
+ ti.xcom_push(key='approve_model_status', value='FAILED')
+ print('Model not approved! Status: ', approval_status)
+
+
+def deploy_model(ti):
+
+ approve_model_status = ti.xcom_pull(task_ids = 'task_approve_model', key = 'approve_model_status')
+
+ headers = {
+ 'AOA-Project-ID': config_main['projectid'],
+ 'Accept': 'application/json, text/plain, */*',
+ 'Accept-Language': 'en-US,en;q=0.9',
+ 'Authorization': 'Bearer ' + config_main['bearertoken'],
+ 'Content-Type': 'application/json',
+ }
+
+
+ json_data = {
+ 'engineType': config_model['engineType'],
+ 'engineTypeConfig': {
+ 'dockerImage': config_model['dockerImage'],
+ 'engine': "python-batch",
+ 'resources': {
+ 'memory': config_resources['memory'],
+ 'cpu': config_resources['cpu'],
+ }
+ },
+ 'language':"python",
+ 'datasetConnectionId': config_main['datasetConnectionId'],
+ 'datasetTemplateId': config_main['datasetTemplateId'],
+ 'cron': config_model['cron'],
+ 'publishOnly': "false",
+ 'args':{}
+ }
+
+ if approve_model_status == 'FAILED':
+ ti.xcom_push(key='deploy_model_status', value='FALIED')
+ print("Deployment cannot be done as the model is not approved !!")
+ else:
+ trained_model_id = ti.xcom_pull(task_ids = 'task_train_model', key = 'trained_model_id')
+
+ response = requests.post('https://airflow-u9usja4twtauvt3s.env.clearscape.teradata.com:8443/modelops/core/api/trainedModels/' + trained_model_id + '/deploy', headers=headers, json=json_data)
+ json_data = response.json()
+
+ # Get the Deployment Job ID
+ deploy_job_id = json_data.get('id')
+ ti.xcom_push(key='deploy_job_id', value=deploy_job_id)
+
+ # deployed_model_id = json_data['metadata']['deployedModel']['id']
+
+ job_status = get_job_status(deploy_job_id)
+ print("Started - Deployment Job - Status: ", job_status)
+
+ while job_status != "COMPLETED":
+ if job_status=="ERROR":
+ ti.xcom_push(key='deploy_model_status', value='FAILED')
+ print("The deployment job is terminated due to an Error")
+ break
+ elif job_status=="CANCELLED":
+ ti.xcom_push(key='deploy_model_status', value='FAILED')
+ print("The deployment job is Cancelled !!")
+ break
+ print("Job is not completed yet. Current status", job_status)
+ time.sleep(5) # wait 5s
+ job_status = get_job_status(deploy_job_id)
+
+ # Checking Job status at the end to push the correct deploy_model_status
+ if(job_status == "COMPLETED"):
+ ti.xcom_push(key='deploy_model_status', value='DEPLOYED')
+ print('Model Deployed Successfully! Job ID is : ', deploy_job_id, ' Status : ', job_status)
+ else:
+ ti.xcom_push(key='deploy_model_status', value='FAILED')
+ print("Deployment Job is terminated !!")
+
+
+
+def retire_model(ti):
+
+ deployed_model_status = ti.xcom_pull(task_ids = 'task_deploy_model', key = 'deploy_model_status')
+
+ if deployed_model_status == 'FAILED':
+ ti.xcom_push(key='retire_model_status', value='FALIED')
+ print("Retirement cannot be done as the model is not deployed !!")
+ else:
+ trained_model_id = ti.xcom_pull(task_ids = 'task_train_model', key = 'trained_model_id')
+
+ headers = {
+ 'AOA-Project-ID': config_main['projectid'],
+ 'Accept': 'application/json, text/plain, */*',
+ 'Accept-Language': 'en-US,en;q=0.9',
+ 'Authorization': 'Bearer ' + config_main['bearertoken'],
+ 'Content-Type': 'application/json',
+ }
+
+ # Identifying the deployment ID
+ get_deployment_id_response = requests.get('https://airflow-u9usja4twtauvt3s.env.clearscape.teradata.com:8443/modelops/core/api/deployments/search/findByStatusAndTrainedModelId?projection=expandDeployment&status=DEPLOYED&trainedModelId=' + trained_model_id , headers=headers)
+
+ get_deployment_id_json = get_deployment_id_response.json()
+ deployment_id = get_deployment_id_json['_embedded']['deployments'][0]['id']
+
+ json_data = {
+ "deploymentId": deployment_id
+ }
+
+ # Retire the specific deployment
+ retire_model_response = requests.post('https://airflow-u9usja4twtauvt3s.env.clearscape.teradata.com:8443/modelops/core/api/trainedModels/' + trained_model_id + '/retire', headers=headers, json=json_data)
+ retire_model_response_json = retire_model_response.json()
+
+ # Get the Evaluation Job ID
+ retire_job_id = retire_model_response_json.get('id')
+ ti.xcom_push(key='retire_job_id', value=retire_job_id)
+
+ job_status = get_job_status(retire_job_id)
+ print("Started - Job - Status: ", job_status)
+
+ while job_status != "COMPLETED":
+ if job_status=="ERROR":
+ print("The Retire job is terminated due to an Error")
+ # Set the Trained Model Id to None here and check in next step (Evaluate)
+ break
+ elif job_status=="CANCELLED":
+ print("The Retire job is Cancelled !!")
+ break
+ print("Job is not completed yet. Current status", job_status)
+ time.sleep(5) # wait 5s
+ job_status = get_job_status(retire_job_id)
+
+ # Checking Job status at the end to push the correct evaluate_job_id
+ if(job_status == "COMPLETED"):
+ ti.xcom_push(key='retire_model_status', value='RETIRED')
+ print('Model Retired Successfully! Job ID is : ', retire_job_id, ' Status : ', job_status)
+ else:
+ ti.xcom_push(key='retire_model_status', value='FAILED')
+ print("Retire Job is terminated !!")
+
+
+
+with DAG(
+ dag_id = 'ModelOps_Accelerator_v1',
+ default_args=default_args,
+ description = 'ModelOps lifecycle accelerator for Python Diabetes Prediction model',
+ start_date=datetime.now(), # Set the start_date as per requirement
+ schedule_interval='@daily'
+) as dag:
+ task1 = PythonOperator(
+ task_id='task_train_model',
+ python_callable=train_model
+ )
+ task2 = PythonOperator(
+ task_id='task_evaluate_model',
+ python_callable=evaluate_model
+ )
+ task3 = PythonOperator(
+ task_id='task_approve_model',
+ python_callable=approve_model
+ )
+ task4 = PythonOperator(
+ task_id='task_deploy_model',
+ python_callable=deploy_model
+ )
+ task5 = PythonOperator(
+ task_id='task_retire_model',
+ python_callable=retire_model
+ )
+
+
+task1.set_downstream(task2)
+task2.set_downstream(task3)
+task3.set_downstream(task4)
+task4.set_downstream(task5)
+```
+====
+
+## Initialize Airflow in Docker Compose
+
+While initializing Airflow services like the internal Airflow database, for operating systems other than Linux, you may get a warning that AIRFLOW_UID is not set, but you can safely ignore it. by setting its environment variable using the following command.
+
+``` bash , id="UID Airflow variable", role="content-editable emits-gtm-events"
+echo -e "AIRFLOW_UID=5000" > .env
+```
+
+To run internal database migrations and create the first user account, initialize the database using this command:
+
+``` bash , id="", role="content-editable emits-gtm-events"
+docker compose up airflow-init
+```
+
+After initialization is complete, you should see a message like this:
+
+``` bash , id="Check Airflow init", role="content-editable emits-gtm-events"
+ airflow-init_1 | Upgrades done
+ airflow-init_1 | Admin user airflow created
+ airflow-init_1 | 2.8.2
+ start_airflow-init_1 exited with code 0
+```
+
+## Clean up Airflow demo environment
+
+You can clean up the environment which will remove the preloaded example DAGs using this command:
+
+``` bash , id="Docker compose down", role="content-editable emits-gtm-events"
+docker-compose down -v
+```
+
+Then update this parameter in docker-compose.yaml file as given below:
+
+``` bash , id="Docker compose yaml", role="content-editable emits-gtm-events"
+AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
+```
+
+## Launch Airflow with Model Factory Solution Accelerator
+
+Launch Airflow using this command:
+
+``` bash , id="Docker compose up", role="content-editable emits-gtm-events"
+docker-compose up -d
+```
+
+
+## Run Airflow DAG of Model Factory Solution with ModelOps
+
+* Now you can access Airflow UI uisng the following http://localhost:8080/
+
+
+
+* Login with Usename: airflow and Password: airflow. In the DAGs menu you will be able to see your created DAGs.
+
+
+
+* Select your latest created DAG and the graph will look like this:
+
+
+
+* Now you can trigger the DAG using the play icon on the top right side.
+
+* You can check the logs by selecting any task and then click on the logs menu:
+
+* On the ClearScape Analytics ModelOps - Jobs section you can see that the jobs have started running:
+
+
+
+* Now you can see that all the tasks are successfully executed.
+
+
+
+## Summary
+
+This tutorial aimed at providing a hands on exercise on how to install an Airflow environment on a Linux server and how to use Airflow to interact with ClearScape Analytics ModelOps and Teradata Vantage database. An additional example is provided on how to integrate Airflow and the data modelling and maintenance tool dbt to create and load a Teradata Vantage database.
+
+## Further reading
+* [ModelOps documentatioN](https://docs.teradata.com/search/documents?query=ModelOps&sort=last_update&virtual-field=title_only&content-lang=).
diff --git a/quickstarts/analyze-data/integrate-teradata-jupyter-extensions-with-google-vertex-ai.md b/quickstarts/analyze-data/integrate-teradata-jupyter-extensions-with-google-vertex-ai.md
new file mode 100644
index 0000000000..d29dca6768
--- /dev/null
+++ b/quickstarts/analyze-data/integrate-teradata-jupyter-extensions-with-google-vertex-ai.md
@@ -0,0 +1,180 @@
+---
+sidebar_position: 11
+author: Hailing Jiang
+email: Hailing.Jiang@teradata.com
+page_last_update: June 28th, 2022
+description: Integrate Teradata Jupyter extensions with Google Vertex AI
+keywords: [data warehouses, compute storage separation, teradata, vantage, cloud data platform, business intelligence, enterprise analytics, jupyter, teradatasql, ipython-sql, teradatasqlalchemy]
+---
+import ClearscapeDocsNote from '../_partials/vantage_clearscape_analytics.mdx'
+import JupyterClearscapeNote from '../_partials/jupyter_notebook_clearscape_analytics_note.mdx';
+
+# Integrate Teradata Jupyter extensions with Google Vertex AI
+
+
+Click to reveal the Python code
+ +
+``` python , id="Model Factory Solution Config File", role="content-editable emits-gtm-events"
+from configparser import ConfigParser
+import os
+
+config = ConfigParser()
+
+config['MAIN'] = {
+ "projectId": "23e1df4b-b630-47a1-ab80-7ad5385fcd8d",
+ "bearerToken": os.environ['BEARER_TOKEN'],
+ "trainDatasetId": "ba39e766-2fdf-426f-ba5c-4ca3e90955fc",
+ "evaluateDatasetId": "74489d62-2af5-4402-b264-715e151a420a",
+ "datasetConnectionId" : "151abf05-1914-4d38-a90d-272d850f212c",
+ "datasetTemplateId": "d8a35d98-21ce-47d0-b9f2-00d355777de1"
+}
+
+config['HYPERPARAMETERS'] = {
+ "eta": 0.2,
+ "max_depth": 6
+}
+
+config['RESOURCES'] = {
+ "memory": "500m",
+ "cpu": "0.5"
+}
+
+config['MODEL'] = {
+ "modelId": "f937b5d8-02c6-5150-80c7-1e4ff07fea31",
+ "approvalComments": "Approving this model!",
+ "cron": "@once",
+ "engineType": "DOCKER_BATCH",
+ "engine": "python-batch",
+ "dockerImage": "artifacts.td.teradata.com/tdproduct-docker-snapshot/avmo/aoa-python-base:3.9.13-1"
+}
+
+
+with open('./config/modelOpsConfig.ini', 'w') as f:
+ config.write(f)
+```
+
+"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "from aoa import ModelContext, DatasetInfo\n",
+ "from teradataml import configure\n",
+ "\n",
+ "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
+ "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
+ "\n",
+ "# define the training dataset \n",
+ "sql = \"\"\"\n",
+ "SELECT \n",
+ " F.*, D.hasdiabetes\n",
+ "FROM PIMA_PATIENT_FEATURES F \n",
+ "JOIN PIMA_PATIENT_DIAGNOSES D\n",
+ "ON F.patientid = D.patientid\n",
+ " WHERE D.patientid MOD 5 <> 0\n",
+ "\"\"\"\n",
+ "\n",
+ "feature_metadata = {\n",
+ " \"database\": database,\n",
+ " \"table\": \"aoa_feature_metadata\"\n",
+ "}\n",
+ "hyperparams = {\"max_depth\": 5, \"eta\": 0.2}\n",
+ "\n",
+ "entity_key = \"PatientId\"\n",
+ "target_names = [\"HasDiabetes\"]\n",
+ "feature_names = [\"NumTimesPrg\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\", \"Age\"]\n",
+ " \n",
+ "dataset_info = DatasetInfo(sql=sql,\n",
+ " entity_key=entity_key,\n",
+ " feature_names=feature_names,\n",
+ " target_names=target_names,\n",
+ " feature_metadata=feature_metadata)\n",
+ "\n",
+ "\n",
+ "ctx = ModelContext(hyperparams=hyperparams,\n",
+ " dataset_info=dataset_info,\n",
+ " artifact_output_path=\"/tmp\",\n",
+ " model_version=\"v1\",\n",
+ " model_table=\"aoa_model_v1\")\n",
+ "\n",
+ "train(context=ctx)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Define Evaluation Function\n",
+ "\n",
+ "The evaluation function takes the following shape\n",
+ "\n",
+ "```python\n",
+ "def evaluate(context: ModelContext, **kwargs):\n",
+ " aoa_create_context()\n",
+ "\n",
+ " # read your model\n",
+ " model = joblib.load(f\"{context.artifact_input_path}/model.joblib\")\n",
+ " \n",
+ " # your evaluation logic\n",
+ " \n",
+ " record_evaluation_stats(...)\n",
+ "```\n",
+ "\n",
+ "You can execute this from the CLI or directly within the notebook as shown."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# %%writefile ../model_modules/evaluation.py\n",
+ "\n",
+ "from sklearn import metrics\n",
+ "from teradataml import DataFrame, copy_to_sql\n",
+ "from aoa import (\n",
+ " record_evaluation_stats,\n",
+ " save_plot,\n",
+ " aoa_create_context,\n",
+ " ModelContext\n",
+ ")\n",
+ "\n",
+ "import joblib\n",
+ "import json\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "\n",
+ "\n",
+ "def evaluate(context: ModelContext, **kwargs):\n",
+ "\n",
+ " aoa_create_context()\n",
+ "\n",
+ " model = joblib.load(f\"{context.artifact_input_path}/model.joblib\")\n",
+ "\n",
+ " feature_names = context.dataset_info.feature_names\n",
+ " target_name = context.dataset_info.target_names[0]\n",
+ "\n",
+ " test_df = DataFrame.from_query(context.dataset_info.sql)\n",
+ " test_pdf = test_df.to_pandas(all_rows=True)\n",
+ "\n",
+ " X_test = test_pdf[feature_names]\n",
+ " y_test = test_pdf[target_name]\n",
+ "\n",
+ " print(\"Scoring\")\n",
+ " y_pred = model.predict(X_test)\n",
+ "\n",
+ " y_pred_tdf = pd.DataFrame(y_pred, columns=[target_name])\n",
+ " y_pred_tdf[\"PatientId\"] = test_pdf[\"PatientId\"].values\n",
+ "\n",
+ " evaluation = {\n",
+ " 'Accuracy': '{:.2f}'.format(metrics.accuracy_score(y_test, y_pred)),\n",
+ " 'Recall': '{:.2f}'.format(metrics.recall_score(y_test, y_pred)),\n",
+ " 'Precision': '{:.2f}'.format(metrics.precision_score(y_test, y_pred)),\n",
+ " 'f1-score': '{:.2f}'.format(metrics.f1_score(y_test, y_pred))\n",
+ " }\n",
+ "\n",
+ " with open(f\"{context.artifact_output_path}/metrics.json\", \"w+\") as f:\n",
+ " json.dump(evaluation, f)\n",
+ "\n",
+ " metrics.plot_confusion_matrix(model, X_test, y_test)\n",
+ " save_plot('Confusion Matrix', context=context)\n",
+ "\n",
+ " metrics.plot_roc_curve(model, X_test, y_test)\n",
+ " save_plot('ROC Curve', context=context)\n",
+ "\n",
+ " # xgboost has its own feature importance plot support but lets use shap as explainability example\n",
+ " import shap\n",
+ "\n",
+ " shap_explainer = shap.TreeExplainer(model['xgb'])\n",
+ " shap_values = shap_explainer.shap_values(X_test)\n",
+ "\n",
+ " shap.summary_plot(shap_values, X_test, feature_names=feature_names,\n",
+ " show=False, plot_size=(12, 8), plot_type='bar')\n",
+ " save_plot('SHAP Feature Importance', context=context)\n",
+ "\n",
+ " feature_importance = pd.DataFrame(list(zip(feature_names, np.abs(shap_values).mean(0))),\n",
+ " columns=['col_name', 'feature_importance_vals'])\n",
+ " feature_importance = feature_importance.set_index(\"col_name\").T.to_dict(orient='records'](0]\n",
+ "\n",
+ " predictions_table = \"predictions_tmp\"\n",
+ " copy_to_sql(df=y_pred_tdf, table_name=predictions_table, index=False, if_exists=\"replace\", temporary=True)\n",
+ "\n",
+ " record_evaluation_stats(features_df=test_df,\n",
+ " predicted_df=DataFrame.from_query(f\"SELECT * FROM {predictions_table}\"),\n",
+ " importance=feature_importance,\n",
+ " context=context)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:aoa.util.connections:teradataml context already exists. Skipping create_context.\n",
+ "Scoring\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+ "ntree_limit is deprecated, use `iteration_range` or model slicing instead.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:aoa.stats.stats:Computing evaluation dataset statistics\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "
"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "
"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
+ "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
+ "\n",
+ "# define the evaluation dataset \n",
+ "sql = \"\"\"\n",
+ "SELECT \n",
+ " F.*, D.hasdiabetes \n",
+ "FROM PIMA_PATIENT_FEATURES F \n",
+ "JOIN PIMA_PATIENT_DIAGNOSES D\n",
+ "ON F.patientid = D.patientid\n",
+ " WHERE D.patientid MOD 5 = 0\n",
+ "\"\"\"\n",
+ "\n",
+ "dataset_info = DatasetInfo(sql=sql,\n",
+ " entity_key=entity_key,\n",
+ " feature_names=feature_names,\n",
+ " target_names=target_names,\n",
+ " feature_metadata=feature_metadata)\n",
+ "\n",
+ "ctx = ModelContext(hyperparams=hyperparams,\n",
+ " dataset_info=dataset_info,\n",
+ " artifact_output_path=\"/tmp\",\n",
+ " artifact_input_path=\"/tmp\",\n",
+ " model_version=\"v1\",\n",
+ " model_table=\"aoa_model_v1\")\n",
+ "\n",
+ "evaluate(context=ctx)\n",
+ "\n",
+ "# view evaluation results\n",
+ "with open(f\"{ctx.artifact_output_path}/metrics.json\") as f:\n",
+ " print(json.load(f))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Define Scoring Function\n",
+ "\n",
+ "The scoring function takes the following shape\n",
+ "\n",
+ "```python\n",
+ "def score(context: ModelContext, **kwargs):\n",
+ " aoa_create_context()\n",
+ "\n",
+ " # read your model\n",
+ " model = joblib.load(f\"{context.artifact_input_path}/model.joblib\")\n",
+ " \n",
+ " # your evaluation logic\n",
+ " \n",
+ " record_scoring_stats(...)\n",
+ "```\n",
+ "\n",
+ "You can execute this from the CLI or directly within the notebook as shown."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# %%writefile ../model_modules/scoring.py\n",
+ "\n",
+ "from teradataml import copy_to_sql, DataFrame\n",
+ "from aoa import (\n",
+ " record_scoring_stats,\n",
+ " aoa_create_context,\n",
+ " ModelContext\n",
+ ")\n",
+ "\n",
+ "import joblib\n",
+ "import pandas as pd\n",
+ "\n",
+ "\n",
+ "def score(context: ModelContext, **kwargs):\n",
+ "\n",
+ " aoa_create_context()\n",
+ "\n",
+ " model = joblib.load(f\"{context.artifact_input_path}/model.joblib\")\n",
+ "\n",
+ " feature_names = context.dataset_info.feature_names\n",
+ " target_name = context.dataset_info.target_names[0]\n",
+ " entity_key = context.dataset_info.entity_key\n",
+ "\n",
+ " features_tdf = DataFrame.from_query(context.dataset_info.sql)\n",
+ " features_pdf = features_tdf.to_pandas(all_rows=True)\n",
+ "\n",
+ " print(\"Scoring\")\n",
+ " predictions_pdf = model.predict(features_pdf[feature_names])\n",
+ "\n",
+ " print(\"Finished Scoring\")\n",
+ "\n",
+ " # store the predictions\n",
+ " predictions_pdf = pd.DataFrame(predictions_pdf, columns=[target_name])\n",
+ " predictions_pdf[entity_key] = features_pdf.index.values\n",
+ " # add job_id column so we know which execution this is from if appended to predictions table\n",
+ " predictions_pdf[\"job_id\"] = context.job_id\n",
+ "\n",
+ " # teradataml doesn't match column names on append.. and so to match / use same table schema as for byom predict\n",
+ " # example (see README.md), we must add empty json_report column and change column order manually (v17.0.0.4)\n",
+ " # CREATE MULTISET TABLE pima_patient_predictions\n",
+ " # (\n",
+ " # job_id VARCHAR(255), -- comes from airflow on job execution\n",
+ " # PatientId BIGINT, -- entity key as it is in the source data\n",
+ " # HasDiabetes BIGINT, -- if model automatically extracts target\n",
+ " # json_report CLOB(1048544000) CHARACTER SET UNICODE -- output of\n",
+ " # )\n",
+ " # PRIMARY INDEX ( job_id );\n",
+ " predictions_pdf[\"json_report\"] = \"\"\n",
+ " predictions_pdf = predictions_pdf[[\"job_id\", entity_key, target_name, \"json_report\"]]\n",
+ "\n",
+ " copy_to_sql(df=predictions_pdf,\n",
+ " schema_name=context.dataset_info.predictions_database,\n",
+ " table_name=context.dataset_info.predictions_table,\n",
+ " index=False,\n",
+ " if_exists=\"append\")\n",
+ " \n",
+ " print(\"Saved predictions in Teradata\")\n",
+ "\n",
+ " # calculate stats\n",
+ " predictions_df = DataFrame.from_query(f\"\"\"\n",
+ " SELECT \n",
+ " * \n",
+ " FROM {context.dataset_info.get_predictions_metadata_fqtn()} \n",
+ " WHERE job_id = '{context.job_id}'\n",
+ " \"\"\")\n",
+ "\n",
+ " record_scoring_stats(features_df=features_tdf, predicted_df=predictions_df, context=context)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:aoa.util.connections:teradataml context already exists. Skipping create_context.\n",
+ "Scoring\n",
+ "Finished Scoring\n",
+ "Saved predictions in Teradata\n",
+ "INFO:aoa.stats.stats:Computing scoring dataset statistics\n",
+ "WARNING:aoa.stats.metrics:Publishing scoring metrics is not enabled\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
+ "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
+ "\n",
+ "# define the scoring dataset \n",
+ "\n",
+ "sql = \"\"\"\n",
+ "SELECT \n",
+ " F.*\n",
+ "FROM PIMA_PATIENT_FEATURES F \n",
+ " WHERE F.patientid MOD 5 = 0\n",
+ "\"\"\"\n",
+ "\n",
+ "# where to store predictions\n",
+ "predictions = {\n",
+ " \"database\": database,\n",
+ " \"table\": \"pima_patient_predictions_tmp\"\n",
+ "}\n",
+ "\n",
+ "import uuid\n",
+ "job_id=str(uuid.uuid4())\n",
+ "\n",
+ "dataset_info = DatasetInfo(sql=sql,\n",
+ " entity_key=entity_key,\n",
+ " feature_names=feature_names,\n",
+ " target_names=target_names,\n",
+ " feature_metadata=feature_metadata,\n",
+ " predictions=predictions)\n",
+ "\n",
+ "ctx = ModelContext(hyperparams=hyperparams,\n",
+ " dataset_info=dataset_info,\n",
+ " artifact_output_path=\"/tmp\",\n",
+ " artifact_input_path=\"/tmp\",\n",
+ " model_version=\"v1\",\n",
+ " model_table=\"aoa_model_v1\",\n",
+ " job_id=job_id)\n",
+ "\n",
+ "score(context=ctx)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "\t
"
+ ],
+ "text/plain": [
+ " job_id PatientId HasDiabetes json_report\n",
+ "0 2d16fcf4-78e3-4801-a052-a9c224814b85 360 1 \n",
+ "1 2d16fcf4-78e3-4801-a052-a9c224814b85 45 1 \n",
+ "2 2d16fcf4-78e3-4801-a052-a9c224814b85 490 0 \n",
+ "3 2d16fcf4-78e3-4801-a052-a9c224814b85 660 1 \n",
+ "4 2d16fcf4-78e3-4801-a052-a9c224814b85 30 1 \n",
+ "5 2d16fcf4-78e3-4801-a052-a9c224814b85 220 1 \n",
+ "6 2d16fcf4-78e3-4801-a052-a9c224814b85 355 1 \n",
+ "7 2d16fcf4-78e3-4801-a052-a9c224814b85 560 0 \n",
+ "8 2d16fcf4-78e3-4801-a052-a9c224814b85 460 0 \n",
+ "9 2d16fcf4-78e3-4801-a052-a9c224814b85 325 0 "
+ ]
+ },
+ "execution_count": 11,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "DataFrame.from_query(f\"SELECT * FROM {database}.pima_patient_predictions_tmp\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Define Model Metadata\n",
+ "\n",
+ "Finally, create the configuration files.\n",
+ "\n",
+ "Requirements file with the dependencies and versions\n",
+ "\n",
+ "```\n",
+ "%%writefile ../model_modules/requirements.txt\n",
+ "xgboost==0.90\n",
+ "scikit-learn==0.24.2\n",
+ "shap==0.36.0\n",
+ "matplotlib==3.3.1\n",
+ "teradataml==17.0.0.4\n",
+ "nyoka==4.3.0\n",
+ "aoa==6.0.0\n",
+ "```\n",
+ "\n",
+ "The hyper parameter configuration (defaults)\n",
+ "```\n",
+ "%%writefile ../config.json\n",
+ "{\n",
+ " \"hyperParameters\": {\n",
+ " \"eta\": 0.2,\n",
+ " \"max_depth\": 6\n",
+ " }\n",
+ "}\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python [conda env:py39]",
+ "language": "python",
+ "name": "conda-env-py39-py"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.12"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/quickstarts/modelops/attachments/ModelOps_Operationalize_v7.ipynb b/quickstarts/modelops/attachments/ModelOps_Operationalize_v7.ipynb
new file mode 100755
index 0000000000..0447e1182f
--- /dev/null
+++ b/quickstarts/modelops/attachments/ModelOps_Operationalize_v7.ipynb
@@ -0,0 +1,732 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Overview\n",
+ "\n",
+ "Once we have finished experiementation and found a good model, we want to operationalize it. \n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Host: tdprd.td.teradata.com\n",
+ "Username: wf250003\n",
+ "Password: ········\n"
+ ]
+ }
+ ],
+ "source": [
+ "from teradataml import create_context\n",
+ "import getpass\n",
+ "\n",
+ "host = input(\"Host: \")\n",
+ "username = input(\"Username: \")\n",
+ "password = getpass.getpass(\"Password: \")\n",
+ "val_db = input(\"VAL DB: \")\n",
+ "byom_db = input(\"BYOM DB: \")\n",
+ "\n",
+ "# configure byom/val installation\n",
+ "configure.val_install_location = val_db\n",
+ "configure.byom_install_location = byom_db\n",
+ "\n",
+ "# by default we assume your are using your user database. change as required\n",
+ "database = username\n",
+ "\n",
+ "create_context(host=host, username=username, password=password, logmech=\"TDNEGO\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Define Training Function\n",
+ "\n",
+ "The training function takes the following shape\n",
+ "\n",
+ "```python\n",
+ "def train(context: ModelContext, **kwargs):\n",
+ " aoa_create_context()\n",
+ " \n",
+ " # your training code\n",
+ " \n",
+ " # save your model\n",
+ " joblib.dump(model, f\"{context.artifact_output_path}/model.joblib\")\n",
+ " \n",
+ " record_training_stats(...)\n",
+ "```\n",
+ "\n",
+ "You can execute this from the CLI or directly within the notebook as shown."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# %%writefile ../model_modules/training.py\n",
+ "\n",
+ "from xgboost import XGBClassifier\n",
+ "from sklearn.preprocessing import MinMaxScaler\n",
+ "from sklearn.pipeline import Pipeline\n",
+ "from nyoka import xgboost_to_pmml\n",
+ "from teradataml import DataFrame\n",
+ "from aoa import (\n",
+ " record_training_stats,\n",
+ " save_plot,\n",
+ " aoa_create_context,\n",
+ " ModelContext\n",
+ ")\n",
+ "\n",
+ "import joblib\n",
+ "\n",
+ "\n",
+ "def train(context: ModelContext, **kwargs):\n",
+ " aoa_create_context()\n",
+ "\n",
+ " feature_names = context.dataset_info.feature_names\n",
+ " target_name = context.dataset_info.target_names[0]\n",
+ "\n",
+ " # read training dataset from Teradata and convert to pandas\n",
+ " train_df = DataFrame.from_query(context.dataset_info.sql)\n",
+ " train_pdf = train_df.to_pandas(all_rows=True)\n",
+ "\n",
+ " # split data into X and y\n",
+ " X_train = train_pdf[feature_names]\n",
+ " y_train = train_pdf[target_name]\n",
+ "\n",
+ " print(\"Starting training...\")\n",
+ "\n",
+ " # fit model to training data\n",
+ " model = Pipeline([('scaler', MinMaxScaler()),\n",
+ " ('xgb', XGBClassifier(eta=context.hyperparams[\"eta\"],\n",
+ " max_depth=context.hyperparams[\"max_depth\"]))])\n",
+ "\n",
+ " model.fit(X_train, y_train)\n",
+ "\n",
+ " print(\"Finished training\")\n",
+ "\n",
+ " # export model artefacts\n",
+ " joblib.dump(model, f\"{context.artifact_output_path}/model.joblib\")\n",
+ "\n",
+ " # we can also save as pmml so it can be used for In-Vantage scoring etc.\n",
+ " xgboost_to_pmml(pipeline=model, col_names=feature_names, target_name=target_name,\n",
+ " pmml_f_name=f\"{context.artifact_output_path}/model.pmml\")\n",
+ "\n",
+ " print(\"Saved trained model\")\n",
+ "\n",
+ " from xgboost import plot_importance\n",
+ " model[\"xgb\"].get_booster().feature_names = feature_names\n",
+ " plot_importance(model[\"xgb\"].get_booster(), max_num_features=10)\n",
+ " save_plot(\"feature_importance.png\", context=context)\n",
+ "\n",
+ " feature_importance = model[\"xgb\"].get_booster().get_score(importance_type=\"weight\")\n",
+ "\n",
+ " record_training_stats(train_df,\n",
+ " features=feature_names,\n",
+ " predictors=[target_name],\n",
+ " categorical=[target_name],\n",
+ " importance=feature_importance,\n",
+ " context=context)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:aoa.util.connections:teradataml context already exists. Skipping create_context.\n",
+ "Starting training...\n",
+ "Finished training\n",
+ "Saved trained model\n",
+ "INFO:aoa.stats.stats:Computing training dataset statistics\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "
| job_id | \n", + "\t\tPatientId | \n", + "\t\tHasDiabetes | \n", + "\t\tjson_report | \n", + "\t
|---|---|---|---|
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t360 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t45 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t490 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t660 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t30 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t220 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t355 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t560 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t460 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t325 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "from aoa import ModelContext, DatasetInfo\n",
+ "from teradataml import configure\n",
+ "\n",
+ "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
+ "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
+ "\n",
+ "# define the training dataset \n",
+ "sql = \"\"\"\n",
+ "SELECT \n",
+ " F.*, D.hasdiabetes\n",
+ "FROM PIMA_PATIENT_FEATURES F \n",
+ "JOIN PIMA_PATIENT_DIAGNOSES D\n",
+ "ON F.patientid = D.patientid\n",
+ " WHERE D.patientid MOD 5 <> 0\n",
+ "\"\"\"\n",
+ "\n",
+ "feature_metadata = {\n",
+ " \"database\": database,\n",
+ " \"table\": \"aoa_feature_metadata\"\n",
+ "}\n",
+ "hyperparams = {\"max_depth\": 5, \"eta\": 0.2}\n",
+ "\n",
+ "entity_key = \"PatientId\"\n",
+ "target_names = [\"HasDiabetes\"]\n",
+ "feature_names = [\"NumTimesPrg\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\", \"Age\"]\n",
+ " \n",
+ "dataset_info = DatasetInfo(sql=sql,\n",
+ " entity_key=entity_key,\n",
+ " feature_names=feature_names,\n",
+ " target_names=target_names,\n",
+ " feature_metadata=feature_metadata)\n",
+ "\n",
+ "\n",
+ "ctx = ModelContext(hyperparams=hyperparams,\n",
+ " dataset_info=dataset_info,\n",
+ " artifact_output_path=\"/tmp\",\n",
+ " model_version=\"v1\",\n",
+ " model_table=\"aoa_model_v1\")\n",
+ "\n",
+ "train(context=ctx)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Define Evaluation Function\n",
+ "\n",
+ "The evaluation function takes the following shape\n",
+ "\n",
+ "```python\n",
+ "def evaluate(context: ModelContext, **kwargs):\n",
+ " aoa_create_context()\n",
+ "\n",
+ " # read your model\n",
+ " model = joblib.load(f\"{context.artifact_input_path}/model.joblib\")\n",
+ " \n",
+ " # your evaluation logic\n",
+ " \n",
+ " record_evaluation_stats(...)\n",
+ "```\n",
+ "\n",
+ "You can execute this from the CLI or directly within the notebook as shown."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# %%writefile ../model_modules/evaluation.py\n",
+ "\n",
+ "from sklearn import metrics\n",
+ "from teradataml import DataFrame, copy_to_sql\n",
+ "from aoa import (\n",
+ " record_evaluation_stats,\n",
+ " save_plot,\n",
+ " aoa_create_context,\n",
+ " ModelContext\n",
+ ")\n",
+ "\n",
+ "import joblib\n",
+ "import json\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "\n",
+ "\n",
+ "def evaluate(context: ModelContext, **kwargs):\n",
+ "\n",
+ " aoa_create_context()\n",
+ "\n",
+ " model = joblib.load(f\"{context.artifact_input_path}/model.joblib\")\n",
+ "\n",
+ " feature_names = context.dataset_info.feature_names\n",
+ " target_name = context.dataset_info.target_names[0]\n",
+ "\n",
+ " test_df = DataFrame.from_query(context.dataset_info.sql)\n",
+ " test_pdf = test_df.to_pandas(all_rows=True)\n",
+ "\n",
+ " X_test = test_pdf[feature_names]\n",
+ " y_test = test_pdf[target_name]\n",
+ "\n",
+ " print(\"Scoring\")\n",
+ " y_pred = model.predict(X_test)\n",
+ "\n",
+ " y_pred_tdf = pd.DataFrame(y_pred, columns=[target_name])\n",
+ " y_pred_tdf[\"PatientId\"] = test_pdf[\"PatientId\"].values\n",
+ "\n",
+ " evaluation = {\n",
+ " 'Accuracy': '{:.2f}'.format(metrics.accuracy_score(y_test, y_pred)),\n",
+ " 'Recall': '{:.2f}'.format(metrics.recall_score(y_test, y_pred)),\n",
+ " 'Precision': '{:.2f}'.format(metrics.precision_score(y_test, y_pred)),\n",
+ " 'f1-score': '{:.2f}'.format(metrics.f1_score(y_test, y_pred))\n",
+ " }\n",
+ "\n",
+ " with open(f\"{context.artifact_output_path}/metrics.json\", \"w+\") as f:\n",
+ " json.dump(evaluation, f)\n",
+ "\n",
+ " metrics.plot_confusion_matrix(model, X_test, y_test)\n",
+ " save_plot('Confusion Matrix', context=context)\n",
+ "\n",
+ " metrics.plot_roc_curve(model, X_test, y_test)\n",
+ " save_plot('ROC Curve', context=context)\n",
+ "\n",
+ " # xgboost has its own feature importance plot support but lets use shap as explainability example\n",
+ " import shap\n",
+ "\n",
+ " shap_explainer = shap.TreeExplainer(model['xgb'])\n",
+ " shap_values = shap_explainer.shap_values(X_test)\n",
+ "\n",
+ " shap.summary_plot(shap_values, X_test, feature_names=feature_names,\n",
+ " show=False, plot_size=(12, 8), plot_type='bar')\n",
+ " save_plot('SHAP Feature Importance', context=context)\n",
+ "\n",
+ " feature_importance = pd.DataFrame(list(zip(feature_names, np.abs(shap_values).mean(0))),\n",
+ " columns=['col_name', 'feature_importance_vals'])\n",
+ " feature_importance = feature_importance.set_index(\"col_name\").T.to_dict(orient='records'](0]\n",
+ "\n",
+ " predictions_table = \"predictions_tmp\"\n",
+ " copy_to_sql(df=y_pred_tdf, table_name=predictions_table, index=False, if_exists=\"replace\", temporary=True)\n",
+ "\n",
+ " record_evaluation_stats(features_df=test_df,\n",
+ " predicted_df=DataFrame.from_query(f\"SELECT * FROM {predictions_table}\"),\n",
+ " importance=feature_importance,\n",
+ " context=context)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:aoa.util.connections:teradataml context already exists. Skipping create_context.\n",
+ "Scoring\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+ "ntree_limit is deprecated, use `iteration_range` or model slicing instead.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:aoa.stats.stats:Computing evaluation dataset statistics\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "
"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "
"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
+ "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
+ "\n",
+ "# define the evaluation dataset \n",
+ "sql = \"\"\"\n",
+ "SELECT \n",
+ " F.*, D.hasdiabetes \n",
+ "FROM PIMA_PATIENT_FEATURES F \n",
+ "JOIN PIMA_PATIENT_DIAGNOSES D\n",
+ "ON F.patientid = D.patientid\n",
+ " WHERE D.patientid MOD 5 = 0\n",
+ "\"\"\"\n",
+ "\n",
+ "dataset_info = DatasetInfo(sql=sql,\n",
+ " entity_key=entity_key,\n",
+ " feature_names=feature_names,\n",
+ " target_names=target_names,\n",
+ " feature_metadata=feature_metadata)\n",
+ "\n",
+ "ctx = ModelContext(hyperparams=hyperparams,\n",
+ " dataset_info=dataset_info,\n",
+ " artifact_output_path=\"/tmp\",\n",
+ " artifact_input_path=\"/tmp\",\n",
+ " model_version=\"v1\",\n",
+ " model_table=\"aoa_model_v1\")\n",
+ "\n",
+ "evaluate(context=ctx)\n",
+ "\n",
+ "# view evaluation results\n",
+ "with open(f\"{ctx.artifact_output_path}/metrics.json\") as f:\n",
+ " print(json.load(f))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Define Scoring Function\n",
+ "\n",
+ "The scoring function takes the following shape\n",
+ "\n",
+ "```python\n",
+ "def score(context: ModelContext, **kwargs):\n",
+ " aoa_create_context()\n",
+ "\n",
+ " # read your model\n",
+ " model = joblib.load(f\"{context.artifact_input_path}/model.joblib\")\n",
+ " \n",
+ " # your evaluation logic\n",
+ " \n",
+ " record_scoring_stats(...)\n",
+ "```\n",
+ "\n",
+ "You can execute this from the CLI or directly within the notebook as shown."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# %%writefile ../model_modules/scoring.py\n",
+ "\n",
+ "from teradataml import copy_to_sql, DataFrame\n",
+ "from aoa import (\n",
+ " record_scoring_stats,\n",
+ " aoa_create_context,\n",
+ " ModelContext\n",
+ ")\n",
+ "\n",
+ "import joblib\n",
+ "import pandas as pd\n",
+ "\n",
+ "\n",
+ "def score(context: ModelContext, **kwargs):\n",
+ "\n",
+ " aoa_create_context()\n",
+ "\n",
+ " model = joblib.load(f\"{context.artifact_input_path}/model.joblib\")\n",
+ "\n",
+ " feature_names = context.dataset_info.feature_names\n",
+ " target_name = context.dataset_info.target_names[0]\n",
+ " entity_key = context.dataset_info.entity_key\n",
+ "\n",
+ " features_tdf = DataFrame.from_query(context.dataset_info.sql)\n",
+ " features_pdf = features_tdf.to_pandas(all_rows=True)\n",
+ "\n",
+ " print(\"Scoring\")\n",
+ " predictions_pdf = model.predict(features_pdf[feature_names])\n",
+ "\n",
+ " print(\"Finished Scoring\")\n",
+ "\n",
+ " # store the predictions\n",
+ " predictions_pdf = pd.DataFrame(predictions_pdf, columns=[target_name])\n",
+ " predictions_pdf[entity_key] = features_pdf.index.values\n",
+ " # add job_id column so we know which execution this is from if appended to predictions table\n",
+ " predictions_pdf[\"job_id\"] = context.job_id\n",
+ "\n",
+ " # teradataml doesn't match column names on append.. and so to match / use same table schema as for byom predict\n",
+ " # example (see README.md), we must add empty json_report column and change column order manually (v17.0.0.4)\n",
+ " # CREATE MULTISET TABLE pima_patient_predictions\n",
+ " # (\n",
+ " # job_id VARCHAR(255), -- comes from airflow on job execution\n",
+ " # PatientId BIGINT, -- entity key as it is in the source data\n",
+ " # HasDiabetes BIGINT, -- if model automatically extracts target\n",
+ " # json_report CLOB(1048544000) CHARACTER SET UNICODE -- output of\n",
+ " # )\n",
+ " # PRIMARY INDEX ( job_id );\n",
+ " predictions_pdf[\"json_report\"] = \"\"\n",
+ " predictions_pdf = predictions_pdf[[\"job_id\", entity_key, target_name, \"json_report\"]]\n",
+ "\n",
+ " copy_to_sql(df=predictions_pdf,\n",
+ " schema_name=context.dataset_info.predictions_database,\n",
+ " table_name=context.dataset_info.predictions_table,\n",
+ " index=False,\n",
+ " if_exists=\"append\")\n",
+ " \n",
+ " print(\"Saved predictions in Teradata\")\n",
+ "\n",
+ " # calculate stats\n",
+ " predictions_df = DataFrame.from_query(f\"\"\"\n",
+ " SELECT \n",
+ " * \n",
+ " FROM {context.dataset_info.get_predictions_metadata_fqtn()} \n",
+ " WHERE job_id = '{context.job_id}'\n",
+ " \"\"\")\n",
+ "\n",
+ " record_scoring_stats(features_df=features_tdf, predicted_df=predictions_df, context=context)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:aoa.util.connections:teradataml context already exists. Skipping create_context.\n",
+ "Scoring\n",
+ "Finished Scoring\n",
+ "Saved predictions in Teradata\n",
+ "INFO:aoa.stats.stats:Computing scoring dataset statistics\n",
+ "WARNING:aoa.stats.metrics:Publishing scoring metrics is not enabled\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
+ "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
+ "\n",
+ "# define the scoring dataset \n",
+ "\n",
+ "sql = \"\"\"\n",
+ "SELECT \n",
+ " F.*\n",
+ "FROM PIMA_PATIENT_FEATURES F \n",
+ " WHERE F.patientid MOD 5 = 0\n",
+ "\"\"\"\n",
+ "\n",
+ "# where to store predictions\n",
+ "predictions = {\n",
+ " \"database\": database,\n",
+ " \"table\": \"pima_patient_predictions_tmp\"\n",
+ "}\n",
+ "\n",
+ "import uuid\n",
+ "job_id=str(uuid.uuid4())\n",
+ "\n",
+ "dataset_info = DatasetInfo(sql=sql,\n",
+ " entity_key=entity_key,\n",
+ " feature_names=feature_names,\n",
+ " target_names=target_names,\n",
+ " feature_metadata=feature_metadata,\n",
+ " predictions=predictions)\n",
+ "\n",
+ "ctx = ModelContext(hyperparams=hyperparams,\n",
+ " dataset_info=dataset_info,\n",
+ " artifact_output_path=\"/tmp\",\n",
+ " artifact_input_path=\"/tmp\",\n",
+ " model_version=\"v1\",\n",
+ " model_table=\"aoa_model_v1\",\n",
+ " job_id=job_id)\n",
+ "\n",
+ "score(context=ctx)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "\t
"
+ ],
+ "text/plain": [
+ " job_id PatientId HasDiabetes json_report\n",
+ "0 2d16fcf4-78e3-4801-a052-a9c224814b85 360 1 \n",
+ "1 2d16fcf4-78e3-4801-a052-a9c224814b85 45 1 \n",
+ "2 2d16fcf4-78e3-4801-a052-a9c224814b85 490 0 \n",
+ "3 2d16fcf4-78e3-4801-a052-a9c224814b85 660 1 \n",
+ "4 2d16fcf4-78e3-4801-a052-a9c224814b85 30 1 \n",
+ "5 2d16fcf4-78e3-4801-a052-a9c224814b85 220 1 \n",
+ "6 2d16fcf4-78e3-4801-a052-a9c224814b85 355 1 \n",
+ "7 2d16fcf4-78e3-4801-a052-a9c224814b85 560 0 \n",
+ "8 2d16fcf4-78e3-4801-a052-a9c224814b85 460 0 \n",
+ "9 2d16fcf4-78e3-4801-a052-a9c224814b85 325 0 "
+ ]
+ },
+ "execution_count": 11,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "DataFrame.from_query(f\"SELECT * FROM {database}.pima_patient_predictions_tmp\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Define Model Metadata\n",
+ "\n",
+ "Finally, create the configuration files.\n",
+ "\n",
+ "Requirements file with the dependencies and versions\n",
+ "\n",
+ "```\n",
+ "%%writefile ../model_modules/requirements.txt\n",
+ "xgboost==0.90\n",
+ "scikit-learn==0.24.2\n",
+ "shap==0.36.0\n",
+ "matplotlib==3.3.1\n",
+ "teradataml==17.0.0.4\n",
+ "nyoka==4.3.0\n",
+ "aoa==6.0.0\n",
+ "```\n",
+ "\n",
+ "The hyper parameter configuration (defaults)\n",
+ "```\n",
+ "%%writefile ../config.json\n",
+ "{\n",
+ " \"hyperParameters\": {\n",
+ " \"eta\": 0.2,\n",
+ " \"max_depth\": 6\n",
+ " }\n",
+ "}\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python [conda env:py39]",
+ "language": "python",
+ "name": "conda-env-py39-py"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.12"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/quickstarts/modelops/attachments/ModelOps_Quickstart_BYOM.zip b/quickstarts/modelops/attachments/ModelOps_Quickstart_BYOM.zip
new file mode 100644
index 0000000000..c0fe76c0d0
Binary files /dev/null and b/quickstarts/modelops/attachments/ModelOps_Quickstart_BYOM.zip differ
diff --git a/quickstarts/modelops/attachments/ModelOps_Training_v6.ipynb b/quickstarts/modelops/attachments/ModelOps_Training_v6.ipynb
new file mode 100755
index 0000000000..fb62f9d9cf
--- /dev/null
+++ b/quickstarts/modelops/attachments/ModelOps_Training_v6.ipynb
@@ -0,0 +1,467 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "f6008b6e",
+ "metadata": {},
+ "source": [
+ "## Setup\n",
+ "\n",
+ "\n",
+ "Ensure you have the following packages and python libraries installed \n",
+ "\n",
+ "```code\n",
+ "pip install teradataml==17.0.0.4 aoa==6.1.0 pandas==1.1.5\n",
+ "```\n",
+ "\n",
+ "The remainder of the notebook runs through the following steps\n",
+ "\n",
+ "- Connect to Vantage\n",
+ "- Create DDLs\n",
+ "- Import Data\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "0528bd6a",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdin",
+ "output_type": "stream",
+ "text": [
+ "Host: tdprd.td.teradata.com\n",
+ "Username: wf250003\n",
+ "Password: ···········\n"
+ ]
+ }
+ ],
+ "source": [
+ "from teradataml import create_context\n",
+ "import getpass\n",
+ "import logging\n",
+ "import sys\n",
+ "import urllib\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "\n",
+ "\n",
+ "host = input(\"Host:\")\n",
+ "username = input(\"Username:\")\n",
+ "password = getpass.getpass(\"Password:\")\n",
+ "\n",
+ "\n",
+ "engine = create_context(host=host, username=username, password=urllib.parse.quote(password), logmech=\"TDNEGO\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4eed19e0",
+ "metadata": {},
+ "source": [
+ "### Create DDLs\n",
+ "\n",
+ "Create the following tables \n",
+ "\n",
+ "- aoa_feature_metadata \n",
+ "- aoa_byom_models\n",
+ "- pima_patient_predictions\n",
+ "\n",
+ "`aoa_feature_metadata` is used to store the profiling metadata for the features so that we can consistently compute the data drift and model drift statistics. This table can also be created via the CLI by executing \n",
+ "\n",
+ "```bash\n",
+ "aoa feature create-stats-table -m .\n",
+ "```\n",
+ "\n",
+ "`pima_patient_predictions` is used for storing the predictions of the model scoring for the demo use case"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "9875d156",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ ""
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from aoa import create_features_stats_table\n",
+ "from teradataml import get_context\n",
+ "\n",
+ "# Note: assuming we are using user database for training. If another database (e.g. datalab) is being used, please update.\n",
+ "# Also note, if a shared datalab is being used, only one user should execute the following DDL/DML commands\n",
+ "database = username\n",
+ "\n",
+ "create_features_stats_table(f\"{database}.aoa_feature_metadata\")\n",
+ "\n",
+ "get_context().execute(f\"\"\"\n",
+ "CREATE MULTISET TABLE {database}.aoa_byom_models\n",
+ " (\n",
+ " model_version VARCHAR(255),\n",
+ " model_id VARCHAR(255),\n",
+ " model_type VARCHAR(255),\n",
+ " project_id VARCHAR(255),\n",
+ " deployed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n",
+ " model BLOB(2097088000)\n",
+ " )\n",
+ " UNIQUE PRIMARY INDEX ( model_version );\n",
+ "\"\"\")\n",
+ "\n",
+ "get_context().execute(f\"\"\"\n",
+ "CREATE MULTISET TABLE {database}.pima_patient_predictions\n",
+ " (\n",
+ " job_id VARCHAR(255),\n",
+ " PatientId BIGINT,\n",
+ " HasDiabetes BIGINT,\n",
+ " json_report CLOB(1048544000) CHARACTER SET UNICODE\n",
+ " )\n",
+ " PRIMARY INDEX ( job_id );\n",
+ "\"\"\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b237d537",
+ "metadata": {},
+ "source": [
+ "### Import Data\n",
+ "\n",
+ "Create and import the data for the following two tables\n",
+ "\n",
+ "- pima_patient_features\n",
+ "- pima_patient_diagnoses\n",
+ "- aoa_feature_metadata\n",
+ "\n",
+ "`pima_patient_features` contains the features related to the patients medical history.\n",
+ "\n",
+ "`pima_patient_diagnoses` contains the diabetes diagnostic results for the patients.\n",
+ "\n",
+ "`aoa_feature_metadata` contains the feature statistics data for the `pima_patient_features` and `pima_patient_diagnoses`\n",
+ "\n",
+ "Note the `pima_patient_feature` can be populated via the CLI by executing \n",
+ "\n",
+ "```bash\n",
+ "aoa feature compute-stats -s .PIMA -m . -t continuous -c numtimesprg,plglcconc,bloodp,skinthick,twohourserins,bmi,dipedfunc,age \n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "07461699",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from teradataml import copy_to_sql, DataFrame\n",
+ "from teradatasqlalchemy.types import *\n",
+ "import pandas as pd\n",
+ "\n",
+ "df = pd.read_csv(\"data/pima_patient_features.csv\")\n",
+ "copy_to_sql(df=df, \n",
+ " table_name=\"pima_patient_features\", \n",
+ " schema_name=database,\n",
+ " primary_index=\"PatientId\", \n",
+ " if_exists=\"replace\", \n",
+ " types={\n",
+ " \"PatientId\": INTEGER,\n",
+ " \"NumTimesPrg\": INTEGER, \n",
+ " \"PlGlcConc\": INTEGER,\n",
+ " \"BloodP\": INTEGER,\n",
+ " \"SkinThick\": INTEGER,\n",
+ " \"TwoHourSerIns\": INTEGER,\n",
+ " \"BMI\": FLOAT,\n",
+ " \"DiPedFunc\": FLOAT,\n",
+ " \"Age\": INTEGER\n",
+ " })\n",
+ "\n",
+ "df = pd.read_csv(\"data/pima_patient_diagnoses.csv\")\n",
+ "copy_to_sql(df=df, \n",
+ " table_name=\"pima_patient_diagnoses\", \n",
+ " schema_name=database,\n",
+ " primary_index=\"PatientId\", \n",
+ " if_exists=\"replace\", \n",
+ " types={\n",
+ " \"PatientId\": INTEGER,\n",
+ " \"HasDiabetes\": INTEGER\n",
+ " })\n",
+ "\n",
+ "# we can compute this from the CLI also - but lets import pre-computed for now.\n",
+ "df = pd.read_csv(\"data/aoa_feature_metadata.csv\")\n",
+ "copy_to_sql(df=df, \n",
+ " table_name=\"aoa_feature_metadata\", \n",
+ " schema_name=database,\n",
+ " if_exists=\"append\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2b0cdd53",
+ "metadata": {},
+ "source": [
+ "## ModelOps UI\n",
+ "\n",
+ "#### Add Project\n",
+ "\n",
+ "- create project\n",
+ " - Details\n",
+ " - Name: Demo {your-name}\n",
+ " - Description: ModelOps Demo\n",
+ " - Group: {your-name}\n",
+ " - Path: https://github.com/Teradata/modelops-demo-models \n",
+ " - Credentials: No Credentials\n",
+ " - Branch: master\n",
+ " - Save And Continue\n",
+ " - Service Connection\n",
+ " - Skip for now\n",
+ " - Personal Connection\n",
+ " - Name: Vantage Personal {your-name}\n",
+ " - Description: Vantage Demo Env\n",
+ " - Host: {your-host}\n",
+ " - Database: {your-db}\n",
+ " - VAL Database: {your-val-db}\n",
+ " - BYOM Database: (your-byom-db}\n",
+ " - Login Mech: TDNEGO\n",
+ " - Username/Password\n",
+ " \n",
+ " \n",
+ "#### Add Datasets\n",
+ "\n",
+ "- create dataset template\n",
+ " - Catalog\n",
+ " - Name: PIMA\n",
+ " - Description: PIMA Diabetes\n",
+ " - Feature Catalog: Vantage\n",
+ " - Database: {your-db}\n",
+ " - Table: aoa_feature_metadata\n",
+ " - Features\n",
+ " - Query: `SELECT * FROM {your-db}.pima_patient_features`\n",
+ " - Entity Key: PatientId\n",
+ " - Features: NumTimesPrg, PlGlcConc, BloodP, SkinThick, TwoHourSerIns, BMI, DiPedFunc, Age\n",
+ " - Entity & Target\n",
+ " - Query: `SELECT * FROM {your-db}.pima_patient_diagnoses`\n",
+ " - Entity Key: PatientId\n",
+ " - Target: HasDiabetes\n",
+ " - Predictions\n",
+ " - Database: {your-db}\n",
+ " - Table: pima_patient_predictions\n",
+ " - Entity Selection: `SELECT * FROM pima_patient_features WHERE patientid MOD 5 = 0`\n",
+ " - BYOM Target Column: `CAST(CAST(json_report AS JSON).JSONExtractValue('$.predicted_HasDiabetes') AS INT)`\n",
+ " \n",
+ " \n",
+ "- create training dataset\n",
+ " - Basic\n",
+ " - Name: Train\n",
+ " - Description: Training dataset\n",
+ " - Scope: Training\n",
+ " - Entity & Target\n",
+ " - Query: `SELECT * FROM {your-db}.pima_patient_diagnoses WHERE patientid MOD 5 <> 0`\n",
+ " \n",
+ "- create evaluation dataset\n",
+ " - Basic\n",
+ " - Name: Evaluate\n",
+ " - Description: Evaluation dataset\n",
+ " - Scope: Evaluation\n",
+ " - Entity & Target\n",
+ " - Query: `SELECT * FROM {your-db}.pima_patient_diagnoses WHERE patientid MOD 5 = 0`\n",
+ " \n",
+ "\n",
+ "#### Model Lifecycle\n",
+ "\n",
+ "- Python Diabetes Prediction\n",
+ " - Train\n",
+ " - Evaluate\n",
+ " - Review evaluation report\n",
+ " - Approve \n",
+ " - Deploy \n",
+ " - Deployments/executions\n",
+ " - Retire\n",
+ "- R Diabetes Prediction\n",
+ " - Train\n",
+ " - Evaluate\n",
+ " - Review evaluation report\n",
+ " - Approve \n",
+ " - Deploy \n",
+ " - Deployments/executions\n",
+ " - Retire\n",
+ "- BYOM Diabetes Prediction\n",
+ " - Run BYOM Notebook \n",
+ " - Define BYOM Model \n",
+ " - Import Version\n",
+ " - Evaluate\n",
+ " - Review evaluation report\n",
+ " - Approve \n",
+ " - Deploy \n",
+ " - Deployments/executions\n",
+ " - Retire"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "17a64068",
+ "metadata": {},
+ "source": [
+ "#### View Predictions\n",
+ "\n",
+ "In the next version of ModelOps, you will be able to view the predictions that follow the standard pattern directly via the UI. However, for now, we can view it here. As the same predictions table contains the predictions for all the jobs, we filter by the `airflow_job_id`. You can find this id in the UI under deployment executions."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "904b2fb9",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ ".\n",
+ "```\n",
+ "\n",
+ "`pima_patient_predictions` is used for storing the predictions of the model scoring for the demo use case"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "769f5cfe",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from aoa import create_features_stats_table\n",
+ "from teradataml import get_context\n",
+ "\n",
+ "# Note: assuming we are using user database for training. If another database (e.g. datalab) is being used, please update.\n",
+ "# Also note, if a shared datalab is being used, only one user should execute the following DDL/DML commands\n",
+ "database = username\n",
+ "\n",
+ "create_features_stats_table(f\"{database}.aoa_statistics_metadata\")\n",
+ "\n",
+ "get_context().execute(f\"\"\"\n",
+ "CREATE MULTISET TABLE {database}.aoa_byom_models\n",
+ " (\n",
+ " model_version VARCHAR(255),\n",
+ " model_id VARCHAR(255),\n",
+ " model_type VARCHAR(255),\n",
+ " project_id VARCHAR(255),\n",
+ " deployed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n",
+ " model BLOB(2097088000)\n",
+ " )\n",
+ " UNIQUE PRIMARY INDEX ( model_version );\n",
+ "\"\"\")\n",
+ "\n",
+ "get_context().execute(f\"\"\"\n",
+ "CREATE MULTISET TABLE {database}.pima_patient_predictions\n",
+ " (\n",
+ " job_id VARCHAR(255),\n",
+ " PatientId BIGINT,\n",
+ " HasDiabetes BIGINT,\n",
+ " json_report CLOB(1048544000) CHARACTER SET UNICODE\n",
+ " )\n",
+ " PRIMARY INDEX ( job_id );\n",
+ "\"\"\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "520b92c2",
+ "metadata": {},
+ "source": [
+ "### Import Data\n",
+ "\n",
+ "Create and import the data for the following two tables\n",
+ "\n",
+ "- pima_patient_features\n",
+ "- pima_patient_diagnoses\n",
+ "- aoa_statistics_metadata\n",
+ "\n",
+ "`pima_patient_features` contains the features related to the patients medical history.\n",
+ "\n",
+ "`pima_patient_diagnoses` contains the diabetes diagnostic results for the patients.\n",
+ "\n",
+ "`aoa_statistics_metadata` contains the feature statistics metadata for the `pima_patient_features` and `pima_patient_diagnoses`\n",
+ "\n",
+ "Note the `pima_patient_feature` can be populated via the CLI by executing \n",
+ "\n",
+ "Compute the statistics metadata for the continuous variables\n",
+ "```bash\n",
+ "aoa feature compute-stats \\\n",
+ " -s . \\\n",
+ " -m . \\\n",
+ " -t continuous -c numtimesprg,plglcconc,bloodp,skinthick,twohourserins,bmi,dipedfunc,age\n",
+ "```\n",
+ "\n",
+ "Compute the statistics metadata for the categorical variables\n",
+ "```bash\n",
+ "aoa feature compute-stats \\\n",
+ " -s . \\\n",
+ " -m . \\\n",
+ " -t categorical -c hasdiabetes\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "9dca7bd3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from teradataml import copy_to_sql, DataFrame\n",
+ "from teradatasqlalchemy.types import *\n",
+ "import pandas as pd\n",
+ "\n",
+ "df = pd.read_csv(\"data/pima_patient_features.csv\")\n",
+ "copy_to_sql(df=df, \n",
+ " table_name=\"pima_patient_features\", \n",
+ " schema_name=database,\n",
+ " primary_index=\"PatientId\", \n",
+ " if_exists=\"replace\", \n",
+ " types={\n",
+ " \"PatientId\": INTEGER,\n",
+ " \"NumTimesPrg\": INTEGER, \n",
+ " \"PlGlcConc\": INTEGER,\n",
+ " \"BloodP\": INTEGER,\n",
+ " \"SkinThick\": INTEGER,\n",
+ " \"TwoHourSerIns\": INTEGER,\n",
+ " \"BMI\": FLOAT,\n",
+ " \"DiPedFunc\": FLOAT,\n",
+ " \"Age\": INTEGER\n",
+ " })\n",
+ "\n",
+ "df = pd.read_csv(\"data/pima_patient_diagnoses.csv\")\n",
+ "copy_to_sql(df=df, \n",
+ " table_name=\"pima_patient_diagnoses\", \n",
+ " schema_name=database,\n",
+ " primary_index=\"PatientId\", \n",
+ " if_exists=\"replace\", \n",
+ " types={\n",
+ " \"PatientId\": INTEGER,\n",
+ " \"HasDiabetes\": INTEGER\n",
+ " })\n",
+ "\n",
+ "# we can compute this from the CLI also - but lets import pre-computed for now.\n",
+ "df = pd.read_csv(\"data/aoa_statistics_metadata.csv\")\n",
+ "copy_to_sql(df=df, \n",
+ " table_name=\"aoa_statistics_metadata\", \n",
+ " schema_name=database,\n",
+ " if_exists=\"append\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "97d65765",
+ "metadata": {},
+ "source": [
+ "## ModelOps UI\n",
+ "\n",
+ "#### Add Project\n",
+ "\n",
+ "- create project\n",
+ " - Details\n",
+ " - Name: Demo {your-name}\n",
+ " - Description: ModelOps Demo\n",
+ " - Group: {your-name}\n",
+ " - Path: https://github.com/Teradata/modelops-demo-models \n",
+ " - Credentials: No Credentials\n",
+ " - Branch: master\n",
+ " - Save And Continue\n",
+ " - Service Connection\n",
+ " - Skip for now\n",
+ " - Personal Connection\n",
+ " - Name: Vantage Personal {your-name}\n",
+ " - Description: Vantage Demo Env\n",
+ " - Host: {your-host}\n",
+ " - Database: {your-db}\n",
+ " - VAL Database: {your-val-db}\n",
+ " - BYOM Database: (your-byom-db}\n",
+ " - Login Mech: TDNEGO\n",
+ " - Username/Password\n",
+ " \n",
+ " \n",
+ "#### Add Datasets\n",
+ "\n",
+ "- create dataset template\n",
+ " - Catalog\n",
+ " - Name: PIMA\n",
+ " - Description: PIMA Diabetes\n",
+ " - Feature Catalog: Vantage\n",
+ " - Database: {your-db}\n",
+ " - Table: aoa_statistics_metadata\n",
+ " - Features\n",
+ " - Query: `SELECT * FROM {your-db}.pima_patient_features`\n",
+ " - Entity Key: PatientId\n",
+ " - Features: NumTimesPrg, PlGlcConc, BloodP, SkinThick, TwoHourSerIns, BMI, DiPedFunc, Age\n",
+ " - Entity & Target\n",
+ " - Query: `SELECT * FROM {your-db}.pima_patient_diagnoses`\n",
+ " - Entity Key: PatientId\n",
+ " - Target: HasDiabetes\n",
+ " - Predictions\n",
+ " - Database: {your-db}\n",
+ " - Table: pima_patient_predictions\n",
+ " - Entity Selection: `SELECT * FROM pima_patient_features WHERE patientid MOD 5 = 0`\n",
+ " \n",
+ " \n",
+ "- create training dataset\n",
+ " - Basic\n",
+ " - Name: Train\n",
+ " - Description: Training dataset\n",
+ " - Scope: Training\n",
+ " - Entity & Target\n",
+ " - Query: `SELECT * FROM {your-db}.pima_patient_diagnoses WHERE patientid MOD 5 <> 0`\n",
+ " \n",
+ "- create evaluation dataset\n",
+ " - Basic\n",
+ " - Name: Evaluate\n",
+ " - Description: Evaluation dataset\n",
+ " - Scope: Evaluation\n",
+ " - Entity & Target\n",
+ " - Query: `SELECT * FROM {your-db}.pima_patient_diagnoses WHERE patientid MOD 5 = 0`\n",
+ " \n",
+ "\n",
+ "#### Model Lifecycle\n",
+ "\n",
+ "- Python Diabetes Prediction\n",
+ " - Train\n",
+ " - Evaluate\n",
+ " - Review evaluation report\n",
+ " - Approve \n",
+ " - Deploy \n",
+ " - Deployments/executions\n",
+ " - Retire\n",
+ "- R Diabetes Prediction\n",
+ " - Train\n",
+ " - Evaluate\n",
+ " - Review evaluation report\n",
+ " - Approve \n",
+ " - Deploy \n",
+ " - Deployments/executions\n",
+ " - Retire\n",
+ "- BYOM Diabetes Prediction\n",
+ " - Run BYOM Notebook \n",
+ " - Define BYOM Model \n",
+ " - Import Version\n",
+ " - Evaluate\n",
+ " - Review evaluation report\n",
+ " - Approve \n",
+ " - Deploy \n",
+ " - Deployments/executions\n",
+ " - Retire"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "be1b4671",
+ "metadata": {},
+ "source": [
+ "#### View Predictions\n",
+ "\n",
+ "In the UI, select a deployment from the deployments left hand navigation. Go to the Jobs tab and on the right hand side for each job execution, you can select \"View Predictions\". This will show you a sample of the predictions for that particular job execution.\n",
+ "\n",
+ "Note, your predictions table must have a `job_id` column which matches to the execution job id. If using BYOM, this is done automatically. For you own `scoring.py`, checkout the demo models."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6b812b27",
+ "metadata": {},
+ "source": [
+ "## CLI \n",
+ "\n",
+ "\n",
+ "```bash\n",
+ "pip install aoa>=7.0.0rc3\n",
+ "```\n",
+ "\n",
+ "##### Copy CLI Config\n",
+ "\n",
+ "```\n",
+ "Copy the CLI config from ModelOps UI -> Session Details -> CLI config\n",
+ "```\n",
+ "\n",
+ "##### Add Dataset Connection\n",
+ "\n",
+ "```bash\n",
+ "aoa connection add\n",
+ "```\n",
+ "\n",
+ "##### List Feature Metadata\n",
+ "\n",
+ "```bash\n",
+ "aoa feature list-stats -m {your-db}.aoa_feature_metadata\n",
+ "```\n",
+ "\n",
+ "##### Clone Project\n",
+ "\n",
+ "```bash\n",
+ "aoa clone \n",
+ "```\n",
+ "\n",
+ "```bash\n",
+ "cd modelops-demo-models\n",
+ "```\n",
+ "\n",
+ "##### Install Model Dependencies\n",
+ "\n",
+ "```bash\n",
+ "pip install -r model_definitions/python-diabetes/model_modules/requirements.txt\n",
+ "```\n",
+ "\n",
+ "##### Train Model\n",
+ "\n",
+ "```bash\n",
+ "aoa run\n",
+ "```\n",
+ "\n",
+ "##### Add Model\n",
+ "\n",
+ "```bash\n",
+ "aoa add\n",
+ "```\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "99270257",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/quickstarts/modelops/images/BYOM.png b/quickstarts/modelops/images/BYOM.png
new file mode 100644
index 0000000000..9b1bf00f9b
Binary files /dev/null and b/quickstarts/modelops/images/BYOM.png differ
diff --git a/quickstarts/modelops/images/Define_BYOM_Model.png b/quickstarts/modelops/images/Define_BYOM_Model.png
new file mode 100644
index 0000000000..b9529f93a1
Binary files /dev/null and b/quickstarts/modelops/images/Define_BYOM_Model.png differ
diff --git a/quickstarts/modelops/images/ModelOps_Healthcheck.png b/quickstarts/modelops/images/ModelOps_Healthcheck.png
new file mode 100644
index 0000000000..22d9a47367
Binary files /dev/null and b/quickstarts/modelops/images/ModelOps_Healthcheck.png differ
diff --git a/quickstarts/modelops/images/Personal_Connection.png b/quickstarts/modelops/images/Personal_Connection.png
new file mode 100644
index 0000000000..0bccd123e9
Binary files /dev/null and b/quickstarts/modelops/images/Personal_Connection.png differ
diff --git a/quickstarts/modelops/images/Project_Creating.png b/quickstarts/modelops/images/Project_Creating.png
new file mode 100644
index 0000000000..b9c40bd0a1
Binary files /dev/null and b/quickstarts/modelops/images/Project_Creating.png differ
diff --git a/quickstarts/modelops/images/alert_configuration.png b/quickstarts/modelops/images/alert_configuration.png
new file mode 100644
index 0000000000..15f3a92fd5
Binary files /dev/null and b/quickstarts/modelops/images/alert_configuration.png differ
diff --git a/quickstarts/modelops/images/alert_configuration2.png b/quickstarts/modelops/images/alert_configuration2.png
new file mode 100644
index 0000000000..3a5ce5e259
Binary files /dev/null and b/quickstarts/modelops/images/alert_configuration2.png differ
diff --git a/quickstarts/modelops/images/alert_configuration3.png b/quickstarts/modelops/images/alert_configuration3.png
new file mode 100644
index 0000000000..22e47062c2
Binary files /dev/null and b/quickstarts/modelops/images/alert_configuration3.png differ
diff --git a/quickstarts/modelops/images/alert_configuration4.png b/quickstarts/modelops/images/alert_configuration4.png
new file mode 100644
index 0000000000..17a3592045
Binary files /dev/null and b/quickstarts/modelops/images/alert_configuration4.png differ
diff --git a/quickstarts/modelops/images/alert_new1.png b/quickstarts/modelops/images/alert_new1.png
new file mode 100644
index 0000000000..7015a8061a
Binary files /dev/null and b/quickstarts/modelops/images/alert_new1.png differ
diff --git a/quickstarts/modelops/images/alert_new2.png b/quickstarts/modelops/images/alert_new2.png
new file mode 100644
index 0000000000..3c8f66b1d3
Binary files /dev/null and b/quickstarts/modelops/images/alert_new2.png differ
diff --git a/quickstarts/modelops/images/alert_new3.png b/quickstarts/modelops/images/alert_new3.png
new file mode 100644
index 0000000000..8daa9d7510
Binary files /dev/null and b/quickstarts/modelops/images/alert_new3.png differ
diff --git a/quickstarts/modelops/images/byom_basic.png b/quickstarts/modelops/images/byom_basic.png
new file mode 100644
index 0000000000..84d55bf893
Binary files /dev/null and b/quickstarts/modelops/images/byom_basic.png differ
diff --git a/quickstarts/modelops/images/byom_meth.png b/quickstarts/modelops/images/byom_meth.png
new file mode 100644
index 0000000000..a699ba7ca7
Binary files /dev/null and b/quickstarts/modelops/images/byom_meth.png differ
diff --git a/quickstarts/modelops/images/byom_model.png b/quickstarts/modelops/images/byom_model.png
new file mode 100644
index 0000000000..054f0698c9
Binary files /dev/null and b/quickstarts/modelops/images/byom_model.png differ
diff --git a/quickstarts/modelops/images/byom_monitoring1.png b/quickstarts/modelops/images/byom_monitoring1.png
new file mode 100644
index 0000000000..95cde0a11e
Binary files /dev/null and b/quickstarts/modelops/images/byom_monitoring1.png differ
diff --git a/quickstarts/modelops/images/byom_monitoring2.png b/quickstarts/modelops/images/byom_monitoring2.png
new file mode 100644
index 0000000000..15843f904e
Binary files /dev/null and b/quickstarts/modelops/images/byom_monitoring2.png differ
diff --git a/quickstarts/modelops/images/byom_monitoring_3.png b/quickstarts/modelops/images/byom_monitoring_3.png
new file mode 100644
index 0000000000..26cc16eaa0
Binary files /dev/null and b/quickstarts/modelops/images/byom_monitoring_3.png differ
diff --git a/quickstarts/modelops/images/byom_monitoring_save.png b/quickstarts/modelops/images/byom_monitoring_save.png
new file mode 100644
index 0000000000..f18a6f5569
Binary files /dev/null and b/quickstarts/modelops/images/byom_monitoring_save.png differ
diff --git a/quickstarts/modelops/images/dataset_template.png b/quickstarts/modelops/images/dataset_template.png
new file mode 100644
index 0000000000..990a8455ca
Binary files /dev/null and b/quickstarts/modelops/images/dataset_template.png differ
diff --git a/quickstarts/modelops/images/dataset_template2.png b/quickstarts/modelops/images/dataset_template2.png
new file mode 100644
index 0000000000..122892cb4f
Binary files /dev/null and b/quickstarts/modelops/images/dataset_template2.png differ
diff --git a/quickstarts/modelops/images/dataset_template_features.png b/quickstarts/modelops/images/dataset_template_features.png
new file mode 100644
index 0000000000..273acdc282
Binary files /dev/null and b/quickstarts/modelops/images/dataset_template_features.png differ
diff --git a/quickstarts/modelops/images/dataset_template_prediction.png b/quickstarts/modelops/images/dataset_template_prediction.png
new file mode 100644
index 0000000000..5b60a569cf
Binary files /dev/null and b/quickstarts/modelops/images/dataset_template_prediction.png differ
diff --git a/quickstarts/modelops/images/dataset_template_target.png b/quickstarts/modelops/images/dataset_template_target.png
new file mode 100644
index 0000000000..5f4094d8ec
Binary files /dev/null and b/quickstarts/modelops/images/dataset_template_target.png differ
diff --git a/quickstarts/modelops/images/datasets_created.png b/quickstarts/modelops/images/datasets_created.png
new file mode 100644
index 0000000000..adb5b4ff93
Binary files /dev/null and b/quickstarts/modelops/images/datasets_created.png differ
diff --git a/quickstarts/modelops/images/define_new.png b/quickstarts/modelops/images/define_new.png
new file mode 100644
index 0000000000..a9952d123d
Binary files /dev/null and b/quickstarts/modelops/images/define_new.png differ
diff --git a/quickstarts/modelops/images/deploy.png b/quickstarts/modelops/images/deploy.png
new file mode 100644
index 0000000000..d895dcea09
Binary files /dev/null and b/quickstarts/modelops/images/deploy.png differ
diff --git a/quickstarts/modelops/images/deploy_details1.png b/quickstarts/modelops/images/deploy_details1.png
new file mode 100644
index 0000000000..46d97a93c2
Binary files /dev/null and b/quickstarts/modelops/images/deploy_details1.png differ
diff --git a/quickstarts/modelops/images/deploy_details2.png b/quickstarts/modelops/images/deploy_details2.png
new file mode 100644
index 0000000000..4ea6e9d0a2
Binary files /dev/null and b/quickstarts/modelops/images/deploy_details2.png differ
diff --git a/quickstarts/modelops/images/deploy_details3.png b/quickstarts/modelops/images/deploy_details3.png
new file mode 100644
index 0000000000..4c7961b2b0
Binary files /dev/null and b/quickstarts/modelops/images/deploy_details3.png differ
diff --git a/quickstarts/modelops/images/deploy_job.png b/quickstarts/modelops/images/deploy_job.png
new file mode 100644
index 0000000000..848d3e5bff
Binary files /dev/null and b/quickstarts/modelops/images/deploy_job.png differ
diff --git a/quickstarts/modelops/images/deployment_evaluate.png b/quickstarts/modelops/images/deployment_evaluate.png
new file mode 100644
index 0000000000..7b815a3ae9
Binary files /dev/null and b/quickstarts/modelops/images/deployment_evaluate.png differ
diff --git a/quickstarts/modelops/images/deployment_evaluate2.png b/quickstarts/modelops/images/deployment_evaluate2.png
new file mode 100644
index 0000000000..72e01fc81a
Binary files /dev/null and b/quickstarts/modelops/images/deployment_evaluate2.png differ
diff --git a/quickstarts/modelops/images/deployment_jobs.png b/quickstarts/modelops/images/deployment_jobs.png
new file mode 100644
index 0000000000..7f03703fb2
Binary files /dev/null and b/quickstarts/modelops/images/deployment_jobs.png differ
diff --git a/quickstarts/modelops/images/deployment_jobs2.png b/quickstarts/modelops/images/deployment_jobs2.png
new file mode 100644
index 0000000000..dd87da9b6a
Binary files /dev/null and b/quickstarts/modelops/images/deployment_jobs2.png differ
diff --git a/quickstarts/modelops/images/deployment_predictions.png b/quickstarts/modelops/images/deployment_predictions.png
new file mode 100644
index 0000000000..8a647adf15
Binary files /dev/null and b/quickstarts/modelops/images/deployment_predictions.png differ
diff --git a/quickstarts/modelops/images/deployments.png b/quickstarts/modelops/images/deployments.png
new file mode 100644
index 0000000000..ce5ea1465c
Binary files /dev/null and b/quickstarts/modelops/images/deployments.png differ
diff --git a/quickstarts/modelops/images/enable_alerts.png b/quickstarts/modelops/images/enable_alerts.png
new file mode 100644
index 0000000000..20d9b78b81
Binary files /dev/null and b/quickstarts/modelops/images/enable_alerts.png differ
diff --git a/quickstarts/modelops/images/evaluation2.png b/quickstarts/modelops/images/evaluation2.png
new file mode 100644
index 0000000000..5e0d6dd609
Binary files /dev/null and b/quickstarts/modelops/images/evaluation2.png differ
diff --git a/quickstarts/modelops/images/evaluation2_detail.png b/quickstarts/modelops/images/evaluation2_detail.png
new file mode 100644
index 0000000000..e99fd0c76c
Binary files /dev/null and b/quickstarts/modelops/images/evaluation2_detail.png differ
diff --git a/quickstarts/modelops/images/evaluation_dataset.png b/quickstarts/modelops/images/evaluation_dataset.png
new file mode 100644
index 0000000000..3692382998
Binary files /dev/null and b/quickstarts/modelops/images/evaluation_dataset.png differ
diff --git a/quickstarts/modelops/images/evaluation_dataset_basic.png b/quickstarts/modelops/images/evaluation_dataset_basic.png
new file mode 100644
index 0000000000..0b9127b013
Binary files /dev/null and b/quickstarts/modelops/images/evaluation_dataset_basic.png differ
diff --git a/quickstarts/modelops/images/evaluation_job.png b/quickstarts/modelops/images/evaluation_job.png
new file mode 100644
index 0000000000..a56544d3a0
Binary files /dev/null and b/quickstarts/modelops/images/evaluation_job.png differ
diff --git a/quickstarts/modelops/images/evaluation_report.png b/quickstarts/modelops/images/evaluation_report.png
new file mode 100644
index 0000000000..9c250771e0
Binary files /dev/null and b/quickstarts/modelops/images/evaluation_report.png differ
diff --git a/quickstarts/modelops/images/evaluation_report2.png b/quickstarts/modelops/images/evaluation_report2.png
new file mode 100644
index 0000000000..e2de8f665a
Binary files /dev/null and b/quickstarts/modelops/images/evaluation_report2.png differ
diff --git a/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/DAG_graph.png b/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/DAG_graph.png
new file mode 100644
index 0000000000..69dc4cddfc
Binary files /dev/null and b/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/DAG_graph.png differ
diff --git a/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/DAGs.png b/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/DAGs.png
new file mode 100644
index 0000000000..6f17a8ea20
Binary files /dev/null and b/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/DAGs.png differ
diff --git a/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/LoginPage.png b/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/LoginPage.png
new file mode 100644
index 0000000000..5ba39af71b
Binary files /dev/null and b/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/LoginPage.png differ
diff --git a/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/Workflow.png b/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/Workflow.png
new file mode 100644
index 0000000000..8d68be2d3f
Binary files /dev/null and b/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/Workflow.png differ
diff --git a/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/modelOps1.png b/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/modelOps1.png
new file mode 100644
index 0000000000..7d8fb964ca
Binary files /dev/null and b/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/modelOps1.png differ
diff --git a/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/successTasks.png b/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/successTasks.png
new file mode 100644
index 0000000000..c2e0c8c6f2
Binary files /dev/null and b/quickstarts/modelops/images/execute-airflow-workflows-with-clearscape-analytics-modelops-model-factory-solution/successTasks.png differ
diff --git a/quickstarts/modelops/images/feature_drift.png b/quickstarts/modelops/images/feature_drift.png
new file mode 100644
index 0000000000..957cc0b502
Binary files /dev/null and b/quickstarts/modelops/images/feature_drift.png differ
diff --git a/quickstarts/modelops/images/go.png b/quickstarts/modelops/images/go.png
new file mode 100644
index 0000000000..6d18df6f23
Binary files /dev/null and b/quickstarts/modelops/images/go.png differ
diff --git a/quickstarts/modelops/images/healthcheck.png b/quickstarts/modelops/images/healthcheck.png
new file mode 100644
index 0000000000..5b4aa6cc06
Binary files /dev/null and b/quickstarts/modelops/images/healthcheck.png differ
diff --git a/quickstarts/modelops/images/jobs.png b/quickstarts/modelops/images/jobs.png
new file mode 100644
index 0000000000..eb65981e1c
Binary files /dev/null and b/quickstarts/modelops/images/jobs.png differ
diff --git a/quickstarts/modelops/images/model_evaluate.png b/quickstarts/modelops/images/model_evaluate.png
new file mode 100644
index 0000000000..8895e28286
Binary files /dev/null and b/quickstarts/modelops/images/model_evaluate.png differ
diff --git a/quickstarts/modelops/images/model_evaluate2.png b/quickstarts/modelops/images/model_evaluate2.png
new file mode 100644
index 0000000000..cfa64bdd77
Binary files /dev/null and b/quickstarts/modelops/images/model_evaluate2.png differ
diff --git a/quickstarts/modelops/images/model_version.png b/quickstarts/modelops/images/model_version.png
new file mode 100644
index 0000000000..9bcc58445c
Binary files /dev/null and b/quickstarts/modelops/images/model_version.png differ
diff --git a/quickstarts/modelops/images/modelops-git.png b/quickstarts/modelops/images/modelops-git.png
new file mode 100644
index 0000000000..e4d7ab3431
Binary files /dev/null and b/quickstarts/modelops/images/modelops-git.png differ
diff --git a/quickstarts/modelops/images/performance.png b/quickstarts/modelops/images/performance.png
new file mode 100644
index 0000000000..b45149ba52
Binary files /dev/null and b/quickstarts/modelops/images/performance.png differ
diff --git a/quickstarts/modelops/images/personal1.png b/quickstarts/modelops/images/personal1.png
new file mode 100644
index 0000000000..fdfa547c83
Binary files /dev/null and b/quickstarts/modelops/images/personal1.png differ
diff --git a/quickstarts/modelops/images/prediction_drift.png b/quickstarts/modelops/images/prediction_drift.png
new file mode 100644
index 0000000000..24bab4166b
Binary files /dev/null and b/quickstarts/modelops/images/prediction_drift.png differ
diff --git a/quickstarts/modelops/images/projects.png b/quickstarts/modelops/images/projects.png
new file mode 100644
index 0000000000..4e404e916e
Binary files /dev/null and b/quickstarts/modelops/images/projects.png differ
diff --git a/quickstarts/modelops/images/projects_quickstart.png b/quickstarts/modelops/images/projects_quickstart.png
new file mode 100644
index 0000000000..f8c5e16823
Binary files /dev/null and b/quickstarts/modelops/images/projects_quickstart.png differ
diff --git a/quickstarts/modelops/images/save_continue.png b/quickstarts/modelops/images/save_continue.png
new file mode 100644
index 0000000000..d83105061f
Binary files /dev/null and b/quickstarts/modelops/images/save_continue.png differ
diff --git a/quickstarts/modelops/images/statistics_job.png b/quickstarts/modelops/images/statistics_job.png
new file mode 100644
index 0000000000..b0b927c2d8
Binary files /dev/null and b/quickstarts/modelops/images/statistics_job.png differ
diff --git a/quickstarts/modelops/images/training_dataset.png b/quickstarts/modelops/images/training_dataset.png
new file mode 100644
index 0000000000..f00845b841
Binary files /dev/null and b/quickstarts/modelops/images/training_dataset.png differ
diff --git a/quickstarts/modelops/images/training_dataset_basic.png b/quickstarts/modelops/images/training_dataset_basic.png
new file mode 100644
index 0000000000..e77d86127b
Binary files /dev/null and b/quickstarts/modelops/images/training_dataset_basic.png differ
diff --git a/quickstarts/modelops/images/view_details.png b/quickstarts/modelops/images/view_details.png
new file mode 100644
index 0000000000..195e77c01e
Binary files /dev/null and b/quickstarts/modelops/images/view_details.png differ
diff --git a/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/Dockerfile b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/Dockerfile
new file mode 100644
index 0000000000..dcfb8954c6
--- /dev/null
+++ b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/Dockerfile
@@ -0,0 +1,18 @@
+FROM apache/airflow:2.3.2
+
+USER root
+
+RUN sudo apt update && \
+ sudo apt-get install -y git
+
+
+USER airflow
+
+RUN pip install sqlalchemy && \
+ pip install sqlalchemy-teradata && \
+ pip install teradatasql && \
+ pip install teradatasqlalchemy && \
+ pip install dbt-teradata && \
+ pip install boto3 && \
+ pip install ijson && \
+ pip install --no-cache-dir awscli
diff --git a/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/airflow.cfg b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/airflow.cfg
new file mode 100644
index 0000000000..ec8324bf37
--- /dev/null
+++ b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/airflow.cfg
@@ -0,0 +1,1112 @@
+[core]
+# The folder where your airflow pipelines live, most likely a
+# subfolder in a code repository. This path must be absolute.
+dags_folder = /opt/airflow/dags
+
+# Hostname by providing a path to a callable, which will resolve the hostname.
+# The format is "package.function".
+#
+# For example, default value "socket.getfqdn" means that result from getfqdn() of "socket"
+# package will be used as hostname.
+#
+# No argument should be required in the function specified.
+# If using IP address as hostname is preferred, use value ``airflow.utils.net.get_host_ip_address``
+hostname_callable = socket.getfqdn
+
+# Default timezone in case supplied date times are naive
+# can be utc (default), system, or any IANA timezone string (e.g. Europe/Amsterdam)
+default_timezone = utc
+
+# The executor class that airflow should use. Choices include
+# ``SequentialExecutor``, ``LocalExecutor``, ``CeleryExecutor``, ``DaskExecutor``,
+# ``KubernetesExecutor``, ``CeleryKubernetesExecutor`` or the
+# full import path to the class when using a custom executor.
+executor = SequentialExecutor
+
+# The SqlAlchemy connection string to the metadata database.
+# SqlAlchemy supports many different database engines.
+# More information here:
+# http://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#database-uri
+sql_alchemy_conn = sqlite:////opt/airflow/airflow.db
+
+# The encoding for the databases
+sql_engine_encoding = utf-8
+
+# Collation for ``dag_id``, ``task_id``, ``key`` columns in case they have different encoding.
+# By default this collation is the same as the database collation, however for ``mysql`` and ``mariadb``
+# the default is ``utf8mb3_bin`` so that the index sizes of our index keys will not exceed
+# the maximum size of allowed index when collation is set to ``utf8mb4`` variant
+# (see https://github.com/apache/airflow/pull/17603#issuecomment-901121618).
+# sql_engine_collation_for_ids =
+
+# If SqlAlchemy should pool database connections.
+sql_alchemy_pool_enabled = True
+
+# The SqlAlchemy pool size is the maximum number of database connections
+# in the pool. 0 indicates no limit.
+sql_alchemy_pool_size = 5
+
+# The maximum overflow size of the pool.
+# When the number of checked-out connections reaches the size set in pool_size,
+# additional connections will be returned up to this limit.
+# When those additional connections are returned to the pool, they are disconnected and discarded.
+# It follows then that the total number of simultaneous connections the pool will allow
+# is pool_size + max_overflow,
+# and the total number of "sleeping" connections the pool will allow is pool_size.
+# max_overflow can be set to ``-1`` to indicate no overflow limit;
+# no limit will be placed on the total number of concurrent connections. Defaults to ``10``.
+sql_alchemy_max_overflow = 10
+
+# The SqlAlchemy pool recycle is the number of seconds a connection
+# can be idle in the pool before it is invalidated. This config does
+# not apply to sqlite. If the number of DB connections is ever exceeded,
+# a lower config value will allow the system to recover faster.
+sql_alchemy_pool_recycle = 1800
+
+# Check connection at the start of each connection pool checkout.
+# Typically, this is a simple statement like "SELECT 1".
+# More information here:
+# https://docs.sqlalchemy.org/en/13/core/pooling.html#disconnect-handling-pessimistic
+sql_alchemy_pool_pre_ping = True
+
+# The schema to use for the metadata database.
+# SqlAlchemy supports databases with the concept of multiple schemas.
+sql_alchemy_schema =
+
+# Import path for connect args in SqlAlchemy. Defaults to an empty dict.
+# This is useful when you want to configure db engine args that SqlAlchemy won't parse
+# in connection string.
+# See https://docs.sqlalchemy.org/en/13/core/engines.html#sqlalchemy.create_engine.params.connect_args
+# sql_alchemy_connect_args =
+
+# This defines the maximum number of task instances that can run concurrently in Airflow
+# regardless of scheduler count and worker count. Generally, this value is reflective of
+# the number of task instances with the running state in the metadata database.
+parallelism = 32
+
+# The maximum number of task instances allowed to run concurrently in each DAG. To calculate
+# the number of tasks that is running concurrently for a DAG, add up the number of running
+# tasks for all DAG runs of the DAG. This is configurable at the DAG level with ``max_active_tasks``,
+# which is defaulted as ``max_active_tasks_per_dag``.
+#
+# An example scenario when this would be useful is when you want to stop a new dag with an early
+# start date from stealing all the executor slots in a cluster.
+max_active_tasks_per_dag = 16
+
+# Are DAGs paused by default at creation
+dags_are_paused_at_creation = True
+
+# The maximum number of active DAG runs per DAG. The scheduler will not create more DAG runs
+# if it reaches the limit. This is configurable at the DAG level with ``max_active_runs``,
+# which is defaulted as ``max_active_runs_per_dag``.
+max_active_runs_per_dag = 16
+
+# Whether to load the DAG examples that ship with Airflow. It's good to
+# get started, but you probably want to set this to ``False`` in a production
+# environment
+load_examples = True
+
+# Whether to load the default connections that ship with Airflow. It's good to
+# get started, but you probably want to set this to ``False`` in a production
+# environment
+load_default_connections = True
+
+# Path to the folder containing Airflow plugins
+plugins_folder = /opt/airflow/plugins
+
+# Should tasks be executed via forking of the parent process ("False",
+# the speedier option) or by spawning a new python process ("True" slow,
+# but means plugin changes picked up by tasks straight away)
+execute_tasks_new_python_interpreter = False
+
+# Secret key to save connection passwords in the db
+fernet_key =
+
+# Whether to disable pickling dags
+donot_pickle = True
+
+# How long before timing out a python file import
+dagbag_import_timeout = 30.0
+
+# Should a traceback be shown in the UI for dagbag import errors,
+# instead of just the exception message
+dagbag_import_error_tracebacks = True
+
+# If tracebacks are shown, how many entries from the traceback should be shown
+dagbag_import_error_traceback_depth = 2
+
+# How long before timing out a DagFileProcessor, which processes a dag file
+dag_file_processor_timeout = 50
+
+# The class to use for running task instances in a subprocess.
+# Choices include StandardTaskRunner, CgroupTaskRunner or the full import path to the class
+# when using a custom task runner.
+task_runner = StandardTaskRunner
+
+# If set, tasks without a ``run_as_user`` argument will be run with this user
+# Can be used to de-elevate a sudo user running Airflow when executing tasks
+default_impersonation =
+
+# What security module to use (for example kerberos)
+security =
+
+# Turn unit test mode on (overwrites many configuration options with test
+# values at runtime)
+unit_test_mode = False
+
+# Whether to enable pickling for xcom (note that this is insecure and allows for
+# RCE exploits).
+enable_xcom_pickling = False
+
+# When a task is killed forcefully, this is the amount of time in seconds that
+# it has to cleanup after it is sent a SIGTERM, before it is SIGKILLED
+killed_task_cleanup_time = 60
+
+# Whether to override params with dag_run.conf. If you pass some key-value pairs
+# through ``airflow dags backfill -c`` or
+# ``airflow dags trigger -c``, the key-value pairs will override the existing ones in params.
+dag_run_conf_overrides_params = True
+
+# When discovering DAGs, ignore any files that don't contain the strings ``DAG`` and ``airflow``.
+dag_discovery_safe_mode = True
+
+# The number of retries each task is going to have by default. Can be overridden at dag or task level.
+default_task_retries = 0
+
+# The weighting method used for the effective total priority weight of the task
+default_task_weight_rule = downstream
+
+# Updating serialized DAG can not be faster than a minimum interval to reduce database write rate.
+min_serialized_dag_update_interval = 30
+
+# Fetching serialized DAG can not be faster than a minimum interval to reduce database
+# read rate. This config controls when your DAGs are updated in the Webserver
+min_serialized_dag_fetch_interval = 10
+
+# Maximum number of Rendered Task Instance Fields (Template Fields) per task to store
+# in the Database.
+# All the template_fields for each of Task Instance are stored in the Database.
+# Keeping this number small may cause an error when you try to view ``Rendered`` tab in
+# TaskInstance view for older tasks.
+max_num_rendered_ti_fields_per_task = 30
+
+# On each dagrun check against defined SLAs
+check_slas = True
+
+# Path to custom XCom class that will be used to store and resolve operators results
+# Example: xcom_backend = path.to.CustomXCom
+xcom_backend = airflow.models.xcom.BaseXCom
+
+# By default Airflow plugins are lazily-loaded (only loaded when required). Set it to ``False``,
+# if you want to load plugins whenever 'airflow' is invoked via cli or loaded from module.
+lazy_load_plugins = True
+
+# By default Airflow providers are lazily-discovered (discovery and imports happen only when required).
+# Set it to False, if you want to discover providers whenever 'airflow' is invoked via cli or
+# loaded from module.
+lazy_discover_providers = True
+
+# Number of times the code should be retried in case of DB Operational Errors.
+# Not all transactions will be retried as it can cause undesired state.
+# Currently it is only used in ``DagFileProcessor.process_file`` to retry ``dagbag.sync_to_db``.
+max_db_retries = 3
+
+# Hide sensitive Variables or Connection extra json keys from UI and task logs when set to True
+#
+# (Connection passwords are always hidden in logs)
+hide_sensitive_var_conn_fields = True
+
+# A comma-separated list of extra sensitive keywords to look for in variables names or connection's
+# extra JSON.
+sensitive_var_conn_names =
+
+# Task Slot counts for ``default_pool``. This setting would not have any effect in an existing
+# deployment where the ``default_pool`` is already created. For existing deployments, users can
+# change the number of slots using Webserver, API or the CLI
+default_pool_task_slot_count = 128
+
+[logging]
+# The folder where airflow should store its log files.
+# This path must be absolute.
+# There are a few existing configurations that assume this is set to the default.
+# If you choose to override this you may need to update the dag_processor_manager_log_location and
+# dag_processor_manager_log_location settings as well.
+base_log_folder = /opt/airflow/logs
+
+# Airflow can store logs remotely in AWS S3, Google Cloud Storage or Elastic Search.
+# Set this to True if you want to enable remote logging.
+remote_logging = False
+
+# Users must supply an Airflow connection id that provides access to the storage
+# location.
+remote_log_conn_id =
+
+# Path to Google Credential JSON file. If omitted, authorization based on `the Application Default
+# Credentials
+# `__ will
+# be used.
+google_key_path =
+
+# Storage bucket URL for remote logging
+# S3 buckets should start with "s3://"
+# Cloudwatch log groups should start with "cloudwatch://"
+# GCS buckets should start with "gs://"
+# WASB buckets should start with "wasb" just to help Airflow select correct handler
+# Stackdriver logs should start with "stackdriver://"
+remote_base_log_folder =
+
+# Use server-side encryption for logs stored in S3
+encrypt_s3_logs = False
+
+# Logging level.
+#
+# Supported values: ``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG``.
+logging_level = INFO
+
+# Logging level for Flask-appbuilder UI.
+#
+# Supported values: ``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG``.
+fab_logging_level = WARNING
+
+# Logging class
+# Specify the class that will specify the logging configuration
+# This class has to be on the python classpath
+# Example: logging_config_class = my.path.default_local_settings.LOGGING_CONFIG
+logging_config_class =
+
+# Flag to enable/disable Colored logs in Console
+# Colour the logs when the controlling terminal is a TTY.
+colored_console_log = True
+
+# Log format for when Colored logs is enabled
+colored_log_format = [%%(blue)s%%(asctime)s%%(reset)s] {%%(blue)s%%(filename)s:%%(reset)s%%(lineno)d} %%(log_color)s%%(levelname)s%%(reset)s - %%(log_color)s%%(message)s%%(reset)s
+colored_formatter_class = airflow.utils.log.colored_log.CustomTTYColoredFormatter
+
+# Format of Log line
+log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
+simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
+
+# Specify prefix pattern like mentioned below with stream handler TaskHandlerWithCustomFormatter
+# Example: task_log_prefix_template = {ti.dag_id}-{ti.task_id}-{execution_date}-{try_number}
+task_log_prefix_template =
+
+# Formatting for how airflow generates file names/paths for each task run.
+log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log
+
+# Formatting for how airflow generates file names for log
+log_processor_filename_template = {{ filename }}.log
+
+# Full path of dag_processor_manager logfile.
+dag_processor_manager_log_location = /opt/airflow/logs/dag_processor_manager/dag_processor_manager.log
+
+# Name of handler to read task instance logs.
+# Defaults to use ``task`` handler.
+task_log_reader = task
+
+# A comma\-separated list of third-party logger names that will be configured to print messages to
+# consoles\.
+# Example: extra_logger_names = connexion,sqlalchemy
+extra_logger_names =
+
+# When you start an airflow worker, airflow starts a tiny web server
+# subprocess to serve the workers local log files to the airflow main
+# web server, who then builds pages and sends them to users. This defines
+# the port on which the logs are served. It needs to be unused, and open
+# visible from the main web server to connect into the workers.
+worker_log_server_port = 8793
+
+[metrics]
+
+# StatsD (https://github.com/etsy/statsd) integration settings.
+# Enables sending metrics to StatsD.
+statsd_on = False
+statsd_host = localhost
+statsd_port = 8125
+statsd_prefix = airflow
+
+# If you want to avoid sending all the available metrics to StatsD,
+# you can configure an allow list of prefixes (comma separated) to send only the metrics that
+# start with the elements of the list (e.g: "scheduler,executor,dagrun")
+statsd_allow_list =
+
+# A function that validate the statsd stat name, apply changes to the stat name if necessary and return
+# the transformed stat name.
+#
+# The function should have the following signature:
+# def func_name(stat_name: str) -> str:
+stat_name_handler =
+
+# To enable datadog integration to send airflow metrics.
+statsd_datadog_enabled = False
+
+# List of datadog tags attached to all metrics(e.g: key1:value1,key2:value2)
+statsd_datadog_tags =
+
+# If you want to utilise your own custom Statsd client set the relevant
+# module path below.
+# Note: The module path must exist on your PYTHONPATH for Airflow to pick it up
+# statsd_custom_client_path =
+
+[secrets]
+# Full class name of secrets backend to enable (will precede env vars and metastore in search path)
+# Example: backend = airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
+backend =
+
+# The backend_kwargs param is loaded into a dictionary and passed to __init__ of secrets backend class.
+# See documentation for the secrets backend you are using. JSON is expected.
+# Example for AWS Systems Manager ParameterStore:
+# ``{"connections_prefix": "/airflow/connections", "profile_name": "default"}``
+backend_kwargs =
+
+[cli]
+# In what way should the cli access the API. The LocalClient will use the
+# database directly, while the json_client will use the api running on the
+# webserver
+api_client = airflow.api.client.local_client
+
+# If you set web_server_url_prefix, do NOT forget to append it here, ex:
+# ``endpoint_url = http://localhost:8080/myroot``
+# So api will look like: ``http://localhost:8080/myroot/api/experimental/...``
+endpoint_url = http://localhost:8080
+
+[debug]
+# Used only with ``DebugExecutor``. If set to ``True`` DAG will fail with first
+# failed task. Helpful for debugging purposes.
+fail_fast = False
+
+[api]
+# Enables the deprecated experimental API. Please note that these APIs do not have access control.
+# The authenticated user has full access.
+#
+# .. warning::
+#
+# This `Experimental REST API `__ is
+# deprecated since version 2.0. Please consider using
+# `the Stable REST API `__.
+# For more information on migration, see
+# `UPDATING.md `_
+enable_experimental_api = False
+
+# How to authenticate users of the API. See
+# https://airflow.apache.org/docs/apache-airflow/stable/security.html for possible values.
+# ("airflow.api.auth.backend.default" allows all requests for historic reasons)
+auth_backend = airflow.api.auth.backend.deny_all
+
+# Used to set the maximum page limit for API requests
+maximum_page_limit = 100
+
+# Used to set the default page limit when limit is zero. A default limit
+# of 100 is set on OpenApi spec. However, this particular default limit
+# only work when limit is set equal to zero(0) from API requests.
+# If no limit is supplied, the OpenApi spec default is used.
+fallback_page_limit = 100
+
+# The intended audience for JWT token credentials used for authorization. This value must match on the client and server sides. If empty, audience will not be tested.
+# Example: google_oauth2_audience = project-id-random-value.apps.googleusercontent.com
+google_oauth2_audience =
+
+# Path to Google Cloud Service Account key file (JSON). If omitted, authorization based on
+# `the Application Default Credentials
+# `__ will
+# be used.
+# Example: google_key_path = /files/service-account-json
+google_key_path =
+
+# Used in response to a preflight request to indicate which HTTP
+# headers can be used when making the actual request. This header is
+# the server side response to the browser's
+# Access-Control-Request-Headers header.
+access_control_allow_headers =
+
+# Specifies the method or methods allowed when accessing the resource.
+access_control_allow_methods =
+
+# Indicates whether the response can be shared with requesting code from the given origins.
+# Separate URLs with space.
+access_control_allow_origins =
+
+[lineage]
+# what lineage backend to use
+backend =
+
+[atlas]
+sasl_enabled = False
+host =
+port = 21000
+username =
+password =
+
+[operators]
+# The default owner assigned to each new operator, unless
+# provided explicitly or passed via ``default_args``
+default_owner = airflow
+default_cpus = 1
+default_ram = 512
+default_disk = 512
+default_gpus = 0
+
+# Default queue that tasks get assigned to and that worker listen on.
+default_queue = default
+
+# Is allowed to pass additional/unused arguments (args, kwargs) to the BaseOperator operator.
+# If set to False, an exception will be thrown, otherwise only the console message will be displayed.
+allow_illegal_arguments = False
+
+[hive]
+# Default mapreduce queue for HiveOperator tasks
+default_hive_mapred_queue =
+
+# Template for mapred_job_name in HiveOperator, supports the following named parameters
+# hostname, dag_id, task_id, execution_date
+# mapred_job_name_template =
+
+[webserver]
+# The base url of your website as airflow cannot guess what domain or
+# cname you are using. This is used in automated emails that
+# airflow sends to point links to the right web server
+base_url = http://localhost:8080
+
+# Default timezone to display all dates in the UI, can be UTC, system, or
+# any IANA timezone string (e.g. Europe/Amsterdam). If left empty the
+# default value of core/default_timezone will be used
+# Example: default_ui_timezone = America/New_York
+default_ui_timezone = UTC
+
+# The ip specified when starting the web server
+web_server_host = 0.0.0.0
+
+# The port on which to run the web server
+web_server_port = 8080
+
+# Paths to the SSL certificate and key for the web server. When both are
+# provided SSL will be enabled. This does not change the web server port.
+web_server_ssl_cert =
+
+# Paths to the SSL certificate and key for the web server. When both are
+# provided SSL will be enabled. This does not change the web server port.
+web_server_ssl_key =
+
+# The type of backend used to store web session data, can be 'database' or 'securecookie'
+# Example: session_backend = securecookie
+session_backend = database
+
+# Number of seconds the webserver waits before killing gunicorn master that doesn't respond
+web_server_master_timeout = 120
+
+# Number of seconds the gunicorn webserver waits before timing out on a worker
+web_server_worker_timeout = 120
+
+# Number of workers to refresh at a time. When set to 0, worker refresh is
+# disabled. When nonzero, airflow periodically refreshes webserver workers by
+# bringing up new ones and killing old ones.
+worker_refresh_batch_size = 1
+
+# Number of seconds to wait before refreshing a batch of workers.
+worker_refresh_interval = 6000
+
+# If set to True, Airflow will track files in plugins_folder directory. When it detects changes,
+# then reload the gunicorn.
+reload_on_plugin_change = False
+
+# Secret key used to run your flask app. It should be as random as possible. However, when running
+# more than 1 instances of webserver, make sure all of them use the same ``secret_key`` otherwise
+# one of them will error with "CSRF session token is missing".
+secret_key = g/rHkt7pPrfeHOlAWr5EaQ==
+
+# Number of workers to run the Gunicorn web server
+workers = 4
+
+# The worker class gunicorn should use. Choices include
+# sync (default), eventlet, gevent
+worker_class = sync
+
+# Log files for the gunicorn webserver. '-' means log to stderr.
+access_logfile = -
+
+# Log files for the gunicorn webserver. '-' means log to stderr.
+error_logfile = -
+
+# Access log format for gunicorn webserver.
+# default format is %%(h)s %%(l)s %%(u)s %%(t)s "%%(r)s" %%(s)s %%(b)s "%%(f)s" "%%(a)s"
+# documentation - https://docs.gunicorn.org/en/stable/settings.html#access-log-format
+access_logformat =
+
+# Expose the configuration file in the web server
+expose_config = False
+
+# Expose hostname in the web server
+expose_hostname = True
+
+# Expose stacktrace in the web server
+expose_stacktrace = True
+
+# Default DAG view. Valid values are: ``tree``, ``graph``, ``duration``, ``gantt``, ``landing_times``
+dag_default_view = tree
+
+# Default DAG orientation. Valid values are:
+# ``LR`` (Left->Right), ``TB`` (Top->Bottom), ``RL`` (Right->Left), ``BT`` (Bottom->Top)
+dag_orientation = LR
+
+# The amount of time (in secs) webserver will wait for initial handshake
+# while fetching logs from other worker machine
+log_fetch_timeout_sec = 5
+
+# Time interval (in secs) to wait before next log fetching.
+log_fetch_delay_sec = 2
+
+# Distance away from page bottom to enable auto tailing.
+log_auto_tailing_offset = 30
+
+# Animation speed for auto tailing log display.
+log_animation_speed = 1000
+
+# By default, the webserver shows paused DAGs. Flip this to hide paused
+# DAGs by default
+hide_paused_dags_by_default = False
+
+# Consistent page size across all listing views in the UI
+page_size = 100
+
+# Define the color of navigation bar
+navbar_color = #fff
+
+# Default dagrun to show in UI
+default_dag_run_display_number = 25
+
+# Enable werkzeug ``ProxyFix`` middleware for reverse proxy
+enable_proxy_fix = False
+
+# Number of values to trust for ``X-Forwarded-For``.
+# More info: https://werkzeug.palletsprojects.com/en/0.16.x/middleware/proxy_fix/
+proxy_fix_x_for = 1
+
+# Number of values to trust for ``X-Forwarded-Proto``
+proxy_fix_x_proto = 1
+
+# Number of values to trust for ``X-Forwarded-Host``
+proxy_fix_x_host = 1
+
+# Number of values to trust for ``X-Forwarded-Port``
+proxy_fix_x_port = 1
+
+# Number of values to trust for ``X-Forwarded-Prefix``
+proxy_fix_x_prefix = 1
+
+# Set secure flag on session cookie
+cookie_secure = False
+
+# Set samesite policy on session cookie
+cookie_samesite = Lax
+
+# Default setting for wrap toggle on DAG code and TI log views.
+default_wrap = False
+
+# Allow the UI to be rendered in a frame
+x_frame_enabled = True
+
+# Send anonymous user activity to your analytics tool
+# choose from google_analytics, segment, or metarouter
+# analytics_tool =
+
+# Unique ID of your account in the analytics tool
+# analytics_id =
+
+# 'Recent Tasks' stats will show for old DagRuns if set
+show_recent_stats_for_completed_runs = True
+
+# Update FAB permissions and sync security manager roles
+# on webserver startup
+update_fab_perms = True
+
+# The UI cookie lifetime in minutes. User will be logged out from UI after
+# ``session_lifetime_minutes`` of non-activity
+session_lifetime_minutes = 43200
+
+# Sets a custom page title for the DAGs overview page and site title for all pages
+# instance_name =
+
+# How frequently, in seconds, the DAG data will auto-refresh in graph or tree view
+# when auto-refresh is turned on
+auto_refresh_interval = 3
+
+[email]
+
+# Configuration email backend and whether to
+# send email alerts on retry or failure
+# Email backend to use
+email_backend = airflow.utils.email.send_email_smtp
+
+# Email connection to use
+email_conn_id = smtp_default
+
+# Whether email alerts should be sent when a task is retried
+default_email_on_retry = True
+
+# Whether email alerts should be sent when a task failed
+default_email_on_failure = True
+
+# File that will be used as the template for Email subject (which will be rendered using Jinja2).
+# If not set, Airflow uses a base template.
+# Example: subject_template = /path/to/my_subject_template_file
+# subject_template =
+
+# File that will be used as the template for Email content (which will be rendered using Jinja2).
+# If not set, Airflow uses a base template.
+# Example: html_content_template = /path/to/my_html_content_template_file
+# html_content_template =
+
+# Email address that will be used as sender address.
+# It can either be raw email or the complete address in a format ``Sender Name ``
+# Example: from_email = Airflow
+# from_email =
+
+[smtp]
+
+# If you want airflow to send emails on retries, failure, and you want to use
+# the airflow.utils.email.send_email_smtp function, you have to configure an
+# smtp server here
+smtp_host = localhost
+smtp_starttls = True
+smtp_ssl = False
+# Example: smtp_user = airflow
+# smtp_user =
+# Example: smtp_password = airflow
+# smtp_password =
+smtp_port = 25
+smtp_mail_from = airflow@example.com
+smtp_timeout = 30
+smtp_retry_limit = 5
+
+[sentry]
+
+# Sentry (https://docs.sentry.io) integration. Here you can supply
+# additional configuration options based on the Python platform. See:
+# https://docs.sentry.io/error-reporting/configuration/?platform=python.
+# Unsupported options: ``integrations``, ``in_app_include``, ``in_app_exclude``,
+# ``ignore_errors``, ``before_breadcrumb``, ``transport``.
+# Enable error reporting to Sentry
+sentry_on = false
+sentry_dsn =
+
+# Dotted path to a before_send function that the sentry SDK should be configured to use.
+# before_send =
+
+[celery_kubernetes_executor]
+
+# This section only applies if you are using the ``CeleryKubernetesExecutor`` in
+# ``[core]`` section above
+# Define when to send a task to ``KubernetesExecutor`` when using ``CeleryKubernetesExecutor``.
+# When the queue of a task is the value of ``kubernetes_queue`` (default ``kubernetes``),
+# the task is executed via ``KubernetesExecutor``,
+# otherwise via ``CeleryExecutor``
+kubernetes_queue = kubernetes
+
+[celery]
+
+# This section only applies if you are using the CeleryExecutor in
+# ``[core]`` section above
+# The app name that will be used by celery
+celery_app_name = airflow.executors.celery_executor
+
+# The concurrency that will be used when starting workers with the
+# ``airflow celery worker`` command. This defines the number of task instances that
+# a worker will take, so size up your workers based on the resources on
+# your worker box and the nature of your tasks
+worker_concurrency = 16
+
+# The maximum and minimum concurrency that will be used when starting workers with the
+# ``airflow celery worker`` command (always keep minimum processes, but grow
+# to maximum if necessary). Note the value should be max_concurrency,min_concurrency
+# Pick these numbers based on resources on worker box and the nature of the task.
+# If autoscale option is available, worker_concurrency will be ignored.
+# http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html#cmdoption-celery-worker-autoscale
+# Example: worker_autoscale = 16,12
+# worker_autoscale =
+
+# Used to increase the number of tasks that a worker prefetches which can improve performance.
+# The number of processes multiplied by worker_prefetch_multiplier is the number of tasks
+# that are prefetched by a worker. A value greater than 1 can result in tasks being unnecessarily
+# blocked if there are multiple workers and one worker prefetches tasks that sit behind long
+# running tasks while another worker has unutilized processes that are unable to process the already
+# claimed blocked tasks.
+# https://docs.celeryproject.org/en/stable/userguide/optimizing.html#prefetch-limits
+# Example: worker_prefetch_multiplier = 1
+# worker_prefetch_multiplier =
+
+# Umask that will be used when starting workers with the ``airflow celery worker``
+# in daemon mode. This control the file-creation mode mask which determines the initial
+# value of file permission bits for newly created files.
+worker_umask = 0o077
+
+# The Celery broker URL. Celery supports RabbitMQ, Redis and experimentally
+# a sqlalchemy database. Refer to the Celery documentation for more information.
+broker_url = redis://redis:6379/0
+
+# The Celery result_backend. When a job finishes, it needs to update the
+# metadata of the job. Therefore it will post a message on a message bus,
+# or insert it into a database (depending of the backend)
+# This status is used by the scheduler to update the state of the task
+# The use of a database is highly recommended
+# http://docs.celeryproject.org/en/latest/userguide/configuration.html#task-result-backend-settings
+result_backend = db+postgresql://postgres:airflow@postgres/airflow
+
+# Celery Flower is a sweet UI for Celery. Airflow has a shortcut to start
+# it ``airflow celery flower``. This defines the IP that Celery Flower runs on
+flower_host = 0.0.0.0
+
+# The root URL for Flower
+# Example: flower_url_prefix = /flower
+flower_url_prefix =
+
+# This defines the port that Celery Flower runs on
+flower_port = 5555
+
+# Securing Flower with Basic Authentication
+# Accepts user:password pairs separated by a comma
+# Example: flower_basic_auth = user1:password1,user2:password2
+flower_basic_auth =
+
+# How many processes CeleryExecutor uses to sync task state.
+# 0 means to use max(1, number of cores - 1) processes.
+sync_parallelism = 0
+
+# Import path for celery configuration options
+celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
+ssl_active = False
+ssl_key =
+ssl_cert =
+ssl_cacert =
+
+# Celery Pool implementation.
+# Choices include: ``prefork`` (default), ``eventlet``, ``gevent`` or ``solo``.
+# See:
+# https://docs.celeryproject.org/en/latest/userguide/workers.html#concurrency
+# https://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html
+pool = prefork
+
+# The number of seconds to wait before timing out ``send_task_to_executor`` or
+# ``fetch_celery_task_state`` operations.
+operation_timeout = 1.0
+
+# Celery task will report its status as 'started' when the task is executed by a worker.
+# This is used in Airflow to keep track of the running tasks and if a Scheduler is restarted
+# or run in HA mode, it can adopt the orphan tasks launched by previous SchedulerJob.
+task_track_started = True
+
+# Time in seconds after which Adopted tasks are cleared by CeleryExecutor. This is helpful to clear
+# stalled tasks.
+task_adoption_timeout = 600
+
+# The Maximum number of retries for publishing task messages to the broker when failing
+# due to ``AirflowTaskTimeout`` error before giving up and marking Task as failed.
+task_publish_max_retries = 3
+
+# Worker initialisation check to validate Metadata Database connection
+worker_precheck = False
+
+[celery_broker_transport_options]
+
+# This section is for specifying options which can be passed to the
+# underlying celery broker transport. See:
+# http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-broker_transport_options
+# The visibility timeout defines the number of seconds to wait for the worker
+# to acknowledge the task before the message is redelivered to another worker.
+# Make sure to increase the visibility timeout to match the time of the longest
+# ETA you're planning to use.
+# visibility_timeout is only supported for Redis and SQS celery brokers.
+# See:
+# http://docs.celeryproject.org/en/master/userguide/configuration.html#std:setting-broker_transport_options
+# Example: visibility_timeout = 21600
+# visibility_timeout =
+
+[dask]
+
+# This section only applies if you are using the DaskExecutor in
+# [core] section above
+# The IP address and port of the Dask cluster's scheduler.
+cluster_address = 127.0.0.1:8786
+
+# TLS/ SSL settings to access a secured Dask scheduler.
+tls_ca =
+tls_cert =
+tls_key =
+
+[scheduler]
+# Task instances listen for external kill signal (when you clear tasks
+# from the CLI or the UI), this defines the frequency at which they should
+# listen (in seconds).
+job_heartbeat_sec = 5
+
+# The scheduler constantly tries to trigger new tasks (look at the
+# scheduler section in the docs for more information). This defines
+# how often the scheduler should run (in seconds).
+scheduler_heartbeat_sec = 5
+
+# The number of times to try to schedule each DAG file
+# -1 indicates unlimited number
+num_runs = -1
+
+# Controls how long the scheduler will sleep between loops, but if there was nothing to do
+# in the loop. i.e. if it scheduled something then it will start the next loop
+# iteration straight away.
+scheduler_idle_sleep_time = 1
+
+# Number of seconds after which a DAG file is parsed. The DAG file is parsed every
+# ``min_file_process_interval`` number of seconds. Updates to DAGs are reflected after
+# this interval. Keeping this number low will increase CPU usage.
+min_file_process_interval = 30
+
+# How often (in seconds) to check for stale DAGs (DAGs which are no longer present in
+# the expected files) which should be deactivated.
+deactivate_stale_dags_interval = 60
+
+# How often (in seconds) to scan the DAGs directory for new files. Default to 5 minutes.
+dag_dir_list_interval = 300
+
+# How often should stats be printed to the logs. Setting to 0 will disable printing stats
+print_stats_interval = 30
+
+# How often (in seconds) should pool usage stats be sent to statsd (if statsd_on is enabled)
+pool_metrics_interval = 5.0
+
+# If the last scheduler heartbeat happened more than scheduler_health_check_threshold
+# ago (in seconds), scheduler is considered unhealthy.
+# This is used by the health check in the "/health" endpoint
+scheduler_health_check_threshold = 30
+
+# How often (in seconds) should the scheduler check for orphaned tasks and SchedulerJobs
+orphaned_tasks_check_interval = 300.0
+child_process_log_directory = /opt/airflow/logs/scheduler
+
+# Local task jobs periodically heartbeat to the DB. If the job has
+# not heartbeat in this many seconds, the scheduler will mark the
+# associated task instance as failed and will re-schedule the task.
+scheduler_zombie_task_threshold = 300
+
+# Turn off scheduler catchup by setting this to ``False``.
+# Default behavior is unchanged and
+# Command Line Backfills still work, but the scheduler
+# will not do scheduler catchup if this is ``False``,
+# however it can be set on a per DAG basis in the
+# DAG definition (catchup)
+catchup_by_default = True
+
+# This changes the batch size of queries in the scheduling main loop.
+# If this is too high, SQL query performance may be impacted by
+# complexity of query predicate, and/or excessive locking.
+# Additionally, you may hit the maximum allowable query length for your db.
+# Set this to 0 for no limit (not advised)
+max_tis_per_query = 512
+
+# Should the scheduler issue ``SELECT ... FOR UPDATE`` in relevant queries.
+# If this is set to False then you should not run more than a single
+# scheduler at once
+use_row_level_locking = True
+
+# Max number of DAGs to create DagRuns for per scheduler loop.
+max_dagruns_to_create_per_loop = 10
+
+# How many DagRuns should a scheduler examine (and lock) when scheduling
+# and queuing tasks.
+max_dagruns_per_loop_to_schedule = 20
+
+# Should the Task supervisor process perform a "mini scheduler" to attempt to schedule more tasks of the
+# same DAG. Leaving this on will mean tasks in the same DAG execute quicker, but might starve out other
+# dags in some circumstances
+schedule_after_task_execution = True
+
+# The scheduler can run multiple processes in parallel to parse dags.
+# This defines how many processes will run.
+parsing_processes = 2
+
+# One of ``modified_time``, ``random_seeded_by_host`` and ``alphabetical``.
+# The scheduler will list and sort the dag files to decide the parsing order.
+#
+# * ``modified_time``: Sort by modified time of the files. This is useful on large scale to parse the
+# recently modified DAGs first.
+# * ``random_seeded_by_host``: Sort randomly across multiple Schedulers but with same order on the
+# same host. This is useful when running with Scheduler in HA mode where each scheduler can
+# parse different DAG files.
+# * ``alphabetical``: Sort by filename
+file_parsing_sort_mode = modified_time
+
+# Turn off scheduler use of cron intervals by setting this to False.
+# DAGs submitted manually in the web UI or with trigger_dag will still run.
+use_job_schedule = True
+
+# Allow externally triggered DagRuns for Execution Dates in the future
+# Only has effect if schedule_interval is set to None in DAG
+allow_trigger_in_future = False
+
+# DAG dependency detector class to use
+dependency_detector = airflow.serialization.serialized_objects.DependencyDetector
+
+# How often to check for expired trigger requests that have not run yet.
+trigger_timeout_check_interval = 15
+
+[triggerer]
+# How many triggers a single Triggerer will run at once, by default.
+default_capacity = 1000
+
+[kerberos]
+ccache = /tmp/airflow_krb5_ccache
+
+# gets augmented with fqdn
+principal = airflow
+reinit_frequency = 3600
+kinit_path = kinit
+keytab = airflow.keytab
+
+# Allow to disable ticket forwardability.
+forwardable = True
+
+# Allow to remove source IP from token, useful when using token behind NATted Docker host.
+include_ip = True
+
+[github_enterprise]
+api_rev = v3
+
+[elasticsearch]
+# Elasticsearch host
+host =
+
+# Format of the log_id, which is used to query for a given tasks logs
+log_id_template = {dag_id}-{task_id}-{execution_date}-{try_number}
+
+# Used to mark the end of a log stream for a task
+end_of_log_mark = end_of_log
+
+# Qualified URL for an elasticsearch frontend (like Kibana) with a template argument for log_id
+# Code will construct log_id using the log_id template from the argument above.
+# NOTE: scheme will default to https if one is not provided
+# Example: frontend = http://localhost:5601/app/kibana#/discover?_a=(columns:![message),query:(language:kuery,query:'log_id: "{log_id}"'),sort:![log.offset,asc))
+frontend =
+
+# Write the task logs to the stdout of the worker, rather than the default files
+write_stdout = False
+
+# Instead of the default log formatter, write the log lines as JSON
+json_format = False
+
+# Log fields to also attach to the json output, if enabled
+json_fields = asctime, filename, lineno, levelname, message
+
+# The field where host name is stored (normally either `host` or `host.name`)
+host_field = host
+
+# The field where offset is stored (normally either `offset` or `log.offset`)
+offset_field = offset
+
+[elasticsearch_configs]
+use_ssl = False
+verify_certs = True
+
+[kubernetes]
+# Path to the YAML pod file that forms the basis for KubernetesExecutor workers.
+pod_template_file =
+
+# The repository of the Kubernetes Image for the Worker to Run
+worker_container_repository =
+
+# The tag of the Kubernetes Image for the Worker to Run
+worker_container_tag =
+
+# The Kubernetes namespace where airflow workers should be created. Defaults to ``default``
+namespace = default
+
+# If True, all worker pods will be deleted upon termination
+delete_worker_pods = True
+
+# If False (and delete_worker_pods is True),
+# failed worker pods will not be deleted so users can investigate them.
+# This only prevents removal of worker pods where the worker itself failed,
+# not when the task it ran failed.
+delete_worker_pods_on_failure = False
+
+# Number of Kubernetes Worker Pod creation calls per scheduler loop.
+# Note that the current default of "1" will only launch a single pod
+# per-heartbeat. It is HIGHLY recommended that users increase this
+# number to match the tolerance of their kubernetes cluster for
+# better performance.
+worker_pods_creation_batch_size = 1
+
+# Allows users to launch pods in multiple namespaces.
+# Will require creating a cluster-role for the scheduler
+multi_namespace_mode = False
+
+# Use the service account kubernetes gives to pods to connect to kubernetes cluster.
+# It's intended for clients that expect to be running inside a pod running on kubernetes.
+# It will raise an exception if called from a process not running in a kubernetes environment.
+in_cluster = True
+
+# When running with in_cluster=False change the default cluster_context or config_file
+# options to Kubernetes client. Leave blank these to use default behaviour like ``kubectl`` has.
+# cluster_context =
+
+# Path to the kubernetes configfile to be used when ``in_cluster`` is set to False
+# config_file =
+
+# Keyword parameters to pass while calling a kubernetes client core_v1_api methods
+# from Kubernetes Executor provided as a single line formatted JSON dictionary string.
+# List of supported params are similar for all core_v1_apis, hence a single config
+# variable for all apis. See:
+# https://raw.githubusercontent.com/kubernetes-client/python/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/api/core_v1_api.py
+kube_client_request_args =
+
+# Optional keyword arguments to pass to the ``delete_namespaced_pod`` kubernetes client
+# ``core_v1_api`` method when using the Kubernetes Executor.
+# This should be an object and can contain any of the options listed in the ``v1DeleteOptions``
+# class defined here:
+# https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19
+# Example: delete_option_kwargs = {"grace_period_seconds": 10}
+delete_option_kwargs =
+
+# Enables TCP keepalive mechanism. This prevents Kubernetes API requests to hang indefinitely
+# when idle connection is time-outed on services like cloud load balancers or firewalls.
+enable_tcp_keepalive = True
+
+# When the `enable_tcp_keepalive` option is enabled, TCP probes a connection that has
+# been idle for `tcp_keep_idle` seconds.
+tcp_keep_idle = 120
+
+# When the `enable_tcp_keepalive` option is enabled, if Kubernetes API does not respond
+# to a keepalive probe, TCP retransmits the probe after `tcp_keep_intvl` seconds.
+tcp_keep_intvl = 30
+
+# When the `enable_tcp_keepalive` option is enabled, if Kubernetes API does not respond
+# to a keepalive probe, TCP retransmits the probe `tcp_keep_cnt number` of times before
+# a connection is considered to be broken.
+tcp_keep_cnt = 6
+
+# Set this to false to skip verifying SSL certificate of Kubernetes python client.
+verify_ssl = True
+
+# How long in seconds a worker can be in Pending before it is considered a failure
+worker_pods_pending_timeout = 300
+
+# How often in seconds to check if Pending workers have exceeded their timeouts
+worker_pods_pending_timeout_check_interval = 120
+
+# How often in seconds to check for task instances stuck in "queued" status without a pod
+worker_pods_queued_check_interval = 60
+
+# How many pending pods to check for timeout violations in each check interval.
+# You may want this higher if you have a very large cluster and/or use ``multi_namespace_mode``.
+worker_pods_pending_timeout_batch_size = 100
+
+[smart_sensor]
+# When `use_smart_sensor` is True, Airflow redirects multiple qualified sensor tasks to
+# smart sensor task.
+use_smart_sensor = False
+
+# `shard_code_upper_limit` is the upper limit of `shard_code` value. The `shard_code` is generated
+# by `hashcode % shard_code_upper_limit`.
+shard_code_upper_limit = 10000
+
+# The number of running smart sensor processes for each service.
+shards = 5
+
+# comma separated sensor classes support in smart_sensor.
+sensors_enabled = NamedHivePartitionSensor
+
+# Igor comment
diff --git a/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/airflow_dbt_integration.py b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/airflow_dbt_integration.py
new file mode 100644
index 0000000000..1f6fd5328e
--- /dev/null
+++ b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/airflow_dbt_integration.py
@@ -0,0 +1,47 @@
+from airflow import DAG
+from airflow.operators.python import PythonOperator, BranchPythonOperator
+from airflow.operators.bash import BashOperator
+from airflow.operators.dummy_operator import DummyOperator
+from datetime import datetime
+
+
+default_args = {
+ 'owner': 'airflow',
+ 'depends_on_past': False,
+ 'start_date': datetime(2020,8,1),
+ 'retries': 0
+}
+
+with DAG('airflow_dbt_integration', default_args=default_args, schedule_interval='@once') as dag:
+ task_1 = BashOperator(
+ task_id='dbt_debug',
+ bash_command='cd /opt/airflow && rm -f logs/dbt.log && dbt debug',
+ dag=dag
+ )
+
+ task_2 = BashOperator(
+ task_id='dbt_seed',
+ bash_command='cd /opt/airflow && dbt seed',
+ dag=dag
+ )
+
+ task_3 = BashOperator(
+ task_id='dbt_run',
+ bash_command='cd /opt/airflow && dbt run',
+ dag=dag
+ )
+
+ task_4 = BashOperator(
+ task_id='dbt_test',
+ bash_command='cd /opt/airflow && dbt test',
+ dag=dag
+ )
+
+ task_5 = BashOperator(
+ task_id='dbt_docs_generate',
+ bash_command='cd /opt/airflow && dbt docs generate',
+ dag=dag
+ )
+
+
+ task_1 >> task_2 >> task_3 >> task_4 >> task_5 # Define dependencies
diff --git a/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/db_test_example_dag.py b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/db_test_example_dag.py
new file mode 100644
index 0000000000..7335141e13
--- /dev/null
+++ b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/db_test_example_dag.py
@@ -0,0 +1,152 @@
+from datetime import datetime, timedelta
+from airflow import DAG
+from airflow.models import Variable
+from airflow.operators.python_operator import PythonOperator, BranchPythonOperator
+from airflow.operators.bash_operator import BashOperator
+import pendulum
+import teradatasql
+import logging
+import getpass
+import urllib.parse
+import sqlalchemy
+from sqlalchemy import exc
+from sqlalchemy.dialects import registry
+
+
+db_user = 'airflowtest'
+db_password = 'abcd'
+db_IP_address = '44.236.48.243'
+SQL_string_cleanup = 'drop table employee;drop table organization'
+
+SQL_string_create_employee = 'create table employee (employee_id integer, name varchar(40), emp_position varchar(40), salary integer, organization_id integer);insert into employee (1,\'John Smith\',\'Engineer\',80000,1);insert into employee (2,\'Jennifer Jones\',\'Account Manager\',100000,2);insert into employee (3,\'William Bowman\',\'Product Manager\',90000,3);insert into employee (1,\'Meghan Stein\',\'Project Manager\',75000,1);'
+
+SQL_string_create_organization = 'create table organization (organization_id integer, organization_name varchar(40), organization_status varchar(10)); insert into organization (1,\'Engineering\',\'Active\');insert into organization (2,\'Sales\',\'Active\');insert into organization (3,\'Marketing\',\'Active\');insert into organization (4,\'Engineering-Old\',\'Inactive\')'
+
+SQL_string_select = 'select avg(employee.salary),organization.organization_name from employee,organization where employee.organization_id=organization.organization_id and organization.organization_status=\'Active\' group by organization.organization_name'
+
+
+
+#Execute an SQL statements in a string format; The string can contain one or more SQL commands separated by ";"
+
+def executeSQLString(db_user, db_password, db_IP_address, SQL_string):
+
+
+
+ # all SQL commands (split by ';')
+ sqlCommands = SQL_string.split(';')
+
+ # create database connection
+ try:
+ registry.register("teradatasql", "teradatasqlalchemy.dialect", "TeradataDialect")
+ enginedbc = sqlalchemy.create_engine('teradatasql://'+db_IP_address+'/?user='+db_user+'&password='+db_password, connect_args={'sslmode': "DISABLE"})
+ conn = enginedbc.connect()
+ logging.info ("Database connection with "+db_IP_address+" established successfully.")
+ except Exception as ex:
+ logging.error(str(ex))
+
+
+
+ # Execute every command from the input file
+ for command in sqlCommands:
+ # This will skip and report errors
+ # For example, if the tables do not yet exist, this will skip over
+ # the DROP TABLE commands
+ # Check if sql command empty
+ if not command.strip():
+ continue
+ sqlresp=''
+ try:
+ logging.info("Executing command : "+command.strip('\n'))
+ sqlresp=conn.execute(command)
+ for row in sqlresp:
+ logging.info(row)
+ # for key, value in row.items():
+ # logging.info(str(key) + ' : ' + str(value))
+
+ except exc.SQLAlchemyError as e:
+ logging.warn(type(e))
+ complete_err = str(e.orig.args)
+ # ignore table does not exist, object does not exist, database already exists errors, storage does not exist, view does not exist;
+ # add any errors that you want to be ignored
+ if (("[Error 3802]" in complete_err) or ("[Error 3807]" in complete_err) or ("[Error 6938]" in complete_err) or ("[Error 5612]" in complete_err) or ("[Error 4836]" in complete_err) or ("[Error 3706]" in complete_err)):
+ logging.warn("Ignoring error "+complete_err.partition('\\n'](0])
+ else:
+ logging.error("Terminating execution because of error "+complete_err.partition('\\n'](0])
+ raise
+
+ conn.close
+
+
+def _cleanup():
+ try:
+ logging.info ("Calling execute SQL string.")
+ executeSQLString(db_user, db_password, db_IP_address, SQL_string_cleanup)
+ logging.info ("Completed execute SQL files.")
+ except Exception as ex:
+ logging.error(str(ex))
+
+def _create_employee():
+ try:
+ logging.info ("Calling execute SQL string.")
+ executeSQLString(db_user, db_password, db_IP_address, SQL_string_create_employee)
+ logging.info ("Completed execute SQL files.")
+ except Exception as ex:
+ logging.error(str(ex))
+
+def _create_organization():
+ try:
+ logging.info ("Calling execute SQL string.")
+ executeSQLString(db_user, db_password, db_IP_address, SQL_string_create_organization)
+ logging.info ("Completed execute SQL files.")
+ except Exception as ex:
+ logging.error(str(ex))
+
+def _run_query():
+ try:
+ logging.info ("Calling execute SQL string.")
+ executeSQLString(db_user, db_password, db_IP_address, SQL_string_select)
+ logging.info ("Completed execute SQL files.")
+ except Exception as ex:
+ logging.error(str(ex))
+
+
+
+with DAG("db_test_example", start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ schedule_interval=None, catchup=False) as dag:
+
+
+ cleanup = PythonOperator(
+ task_id="cleanup",
+ python_callable=_cleanup,
+ depends_on_past=False
+ )
+
+ create_employee = PythonOperator(
+ task_id="create_employee",
+ python_callable=_create_employee,
+ depends_on_past=False
+ )
+
+ create_organization = PythonOperator(
+ task_id="create_organization",
+ python_callable=_create_organization,
+ depends_on_past=False
+ )
+
+ run_query = PythonOperator(
+ task_id="run_query",
+ python_callable=_run_query,
+ depends_on_past=False
+ )
+
+
+
+
+
+
+
+
+cleanup >> [create_employee, create_organization] >> run_query
+
+
+
diff --git a/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/discover_dag.py b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/discover_dag.py
new file mode 100644
index 0000000000..17b641f172
--- /dev/null
+++ b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/discover_dag.py
@@ -0,0 +1,489 @@
+# Airflow DAG to load a generic number of parquet, csv and json files into a Teradata 20 database on Amazon Web Services (AWS).
+# The files are assumed to be located on specific S3 buckets (location defined in Airflow variables - csv files go to the csv
+# S3 bucket, json files to the json bucket, parquet into the parquet bucket).
+# The script locates the files, determines the structure of the files (columns, delimiters, etc.) and:
+# 1. Creates the needed databases (scv database for csv files, json database for json files, multiple parquet databases are created based on the
+# parquet directory names). If databases are already created, it skips this step.
+# 2. Creates a teradata foreign table that point to the files
+# 3. Creates a NOS table (uses S3 storage) and uses the previously created foreign tables to load them (doing select from
+# foreign tables / insert into NOS tables).
+#
+# The Airflow environment must be created by using a docker_compose.yaml and Dockerfile to include all the needed packages and libraries.
+
+
+from datetime import datetime, timedelta
+from airflow.decorators import dag,task
+from airflow import AirflowException
+from airflow.models import Variable
+import teradatasql
+import logging
+import getpass
+import urllib.parse
+import sqlalchemy
+from sqlalchemy import exc
+from sqlalchemy.dialects import registry
+from airflow import DAG
+from airflow.operators.python import PythonOperator, BranchPythonOperator
+from airflow.operators.bash import BashOperator
+from airflow.operators.dummy_operator import DummyOperator
+import os
+import sys
+import ijson
+import json
+import subprocess
+import csv
+import boto3
+
+# Airflow variables that must be imported before running this DAG.
+# A sample variables.json file is provided aas an example
+# AWS keys:
+aws_access_key_id =Variable.get("aws_access_key_id")
+aws_secret_access_key =Variable.get("aws_secret_access_key")
+# S3 Locations. Ls locations (to be used for the aws ls command line interface) have a different format than ft locations (where files reside)
+s3_location_parq_ls =Variable.get("s3_location_parq_ls")
+s3_location_parq_ft =Variable.get("s3_location_parq_ft")
+s3_location_csv_ls =Variable.get("s3_location_csv_ls")
+s3_location_csv_create =Variable.get("s3_location_csv_create")
+s3_location_csv_ft =Variable.get("s3_location_csv_ft")
+s3_location_json_ls =Variable.get("s3_location_json_ls")
+s3_location_json_ft =Variable.get("s3_location_json_ft")
+s3_location_json_create =Variable.get("s3_location_json_create")
+# s3 bucket is the top S3 bucket where the data resides (and where the parquet directories start), csv and json are subbuckets where these types of
+# files reside
+s3_bucket =Variable.get("s3_bucket")
+csv_subbucket =Variable.get("csv_subbucket")
+json_subbucket =Variable.get("json_subbucket")
+# Temp file where the list of databases to be created reside
+filenamedb=Variable.get("filenamedb")
+# Location of temporary files where databasename tablename are listed for the program fo create tables
+parqfilenamedbtab=Variable.get("parqfilenamedbtab")
+csvfilenamedbtab=Variable.get("csvfilenamedbtab")
+jsonfilenamedbtab=Variable.get("jsonfilenamedbtab")
+alldbtab =Variable.get("alldbtab")
+# Csv variables, database name and delimiters supported
+csvdb=Variable.get("csvdb")
+supported_csvdelimiters=Variable.get("supported_csvdelimiters")
+supported_csvlineterminator=Variable.get("supported_csvlineterminator")
+# Json db name
+jsondb=Variable.get("jsondb")
+# Target database variables - the DB user must have database create privileges - the csvdb and jsondb are going to be created under the user's
+# datbase.
+DB_username =Variable.get("DB_username")
+DB_password =Variable.get("DB_password")
+DB_ip_address =Variable.get("DB_ip_address")
+# Authorization object name
+auth_name =Variable.get("auth_name")
+region_name=Variable.get("region_name")
+# Temp file used to determine json file format
+output_file =Variable.get("output_file")
+# Sample size (num of lines) to determine csv format
+linenumax=int(Variable.get("linenumax"))
+# Permanent size for csv and json databases. Here one size fits all, to change it has to be slightly modified
+perm_dbsize=Variable.get("perm_dbsize")
+# NOS storage name
+nos_storage=Variable.get("nos_storage")
+# Flags to let the program know which types of files to load. 'Y' to loas the specific file type.
+load_csv=Variable.get("load_csv")
+load_json=Variable.get("load_json")
+load_parquet=Variable.get("load_parquet")
+
+
+
+# Genertes the JSON nos select command for table table_name
+def get_json_nos_select_comm(table_name):
+ try:
+
+ command_getvalues = 'select '
+ json_sample_size = 'top 100'
+
+ logging.info("Generating JSON nos select command")
+
+ command_getfields = 'select * from (SELECT distinct * FROM JSON_KEYS (ON (SELECT ' + json_sample_size + ' payload FROM ' + table_name + ' )) AS j ) as cols;'
+
+ logging.info("Command to get json fields: " + command_getfields)
+
+ registry.register("teradatasql", "teradatasqlalchemy.dialect", "TeradataDialect")
+ enginedbc = sqlalchemy.create_engine('teradatasql://'+DB_ip_address+'/?user='+DB_username+'&password='+DB_password, connect_args={'sslmode': "DISABLE"})
+ conn = enginedbc.connect()
+ logging.info("Database connection with "+DB_ip_address+" established successfully.")
+ sqlrespfields=conn.execute(command_getfields)
+ for row in sqlrespfields:
+ for key, value in row.items():
+ fieldname = '"payload".' + str(value) + ' ' + value.replace('"."','__').replace('"','')
+ print(fieldname)
+ command_getvalues = command_getvalues + fieldname + ', '
+ command_getvalues = command_getvalues[:-3]
+ command_getvalues = command_getvalues + ' from ' + table_name
+ logging.info('JSON nos select command: \n\n' + command_getvalues + '\n')
+ conn.close
+ return(command_getvalues)
+ except Exception as ex:
+ logging.error(str(ex))
+ raise AirflowException
+
+
+
+# Execute a string of SQL commands separated by semicolons (;)
+def execute_sql_commands(commands):
+ try:
+ logging.info ("SQL commands: " + commands)
+ sqlcommands = commands.split(';')
+ registry.register("teradatasql", "teradatasqlalchemy.dialect", "TeradataDialect")
+ enginedbc = sqlalchemy.create_engine('teradatasql://'+DB_ip_address+'/?user='+DB_username+'&password='+DB_password, connect_args={'sslmode': "DISABLE"})
+ conn = enginedbc.connect()
+ logging.info ("Database connection with "+DB_ip_address+" established successfully.")
+
+
+ # files to tbl:
+ for sqlcommand in sqlcommands:
+ try:
+ logging.info ("SQL Command: " + sqlcommand)
+ sqlresp=conn.execute(sqlcommand)
+ for row in sqlresp:
+ logging.info(row)
+
+ except exc.SQLAlchemyError as e:
+ logging.warn(type(e))
+ complete_err = str(e.orig.args)
+ # ignore table does not exist, object does not exist, database already exists errors, storage does not exist, view does not exist
+ if (("[Error 3802]" in complete_err) or ("[Error 3807]" in complete_err) or ("[Error 6938]" in complete_err) or ("[Error 5612]" in complete_err) or ("[Error 4836]" in complete_err) or ("[Error 3706]" in complete_err)):
+ logging.warn("Ignoring error "+complete_err.partition('\\n'](0])
+ continue
+ else:
+ logging.error("Terminating execution because of error "+complete_err.partition('\\n'](0])
+ raise AirflowException
+
+ conn.close
+
+ except Exception as ex:
+ logging.error(str(ex))
+ raise AirflowException
+
+
+
+# Returns bash script string containing the script that creates a file containing the database names to be created
+def create_db_file_bash(filenamedb):
+ empty_bash_str = 'touch ' + filenamedb + ';'
+ csv_bash_str = 'echo \'' + csvdb + '\' >> ' + filenamedb + ';'
+ json_bash_str = 'echo \'' + jsondb + '\' >> ' + filenamedb + ';'
+ parquet_bash_str = 'export AWS_ACCESS_KEY_ID=' + aws_access_key_id + ' ; export AWS_SECRET_ACCESS_KEY=' + aws_secret_access_key + '; aws s3 ls ' + s3_location_parq_ls + ' | awk \'{print $2}\' | sed \'s#/##\' >> ' + filenamedb + ';'
+ create_db_file_bash_command = empty_bash_str
+ if (load_csv == 'Y'):
+ create_db_file_bash_command = create_db_file_bash_command + csv_bash_str
+ if (load_json == 'Y'):
+ create_db_file_bash_command = create_db_file_bash_command + json_bash_str
+ if (load_parquet == 'Y'):
+ create_db_file_bash_command = create_db_file_bash_command + parquet_bash_str
+ logging.info ("Returning db file creation bash command: " + create_db_file_bash_command)
+ return (create_db_file_bash_command)
+
+
+# SQL and Bash scripts
+
+# Bash command to create placeholder empty files
+create_placeholder_files_command = 'touch ' + csvfilenamedbtab + '; touch ' + jsonfilenamedbtab + '; touch ' + parqfilenamedbtab + '; touch ' + alldbtab
+
+# Bash command to create a file containing the names of parquet files to be loaded
+create_parq_db_tab_file_bash_command = 'export AWS_ACCESS_KEY_ID=' + aws_access_key_id + ' ; export AWS_SECRET_ACCESS_KEY=' + aws_secret_access_key + '; for DB in `aws s3 ls ' + s3_location_parq_ls + '| awk \'{print $2}\' | sed \'s#/##\' `; do aws s3 ls ' + s3_location_parq_ls + '$DB/ | awk \'{print db,$2}\' db="${DB}" | sed \'s#/##\'; done > ' + parqfilenamedbtab
+
+# Bash command to create a temporary file containing the names of csv files to be loaded
+create_csv_tab_file_bash_command = 'export AWS_ACCESS_KEY_ID=' + aws_access_key_id + ' ; export AWS_SECRET_ACCESS_KEY=' + aws_secret_access_key + '; aws s3 ls ' + s3_location_csv_ls + ' | awk \'{print "+csv+ " $4}\' | sed \'s#/##\' | tail -n +2 > ' + csvfilenamedbtab
+
+
+# Bash command to create a temporary file containing the names of json files to be loaded
+create_json_tab_file_bash_command = 'export AWS_ACCESS_KEY_ID=' + aws_access_key_id + ' ; export AWS_SECRET_ACCESS_KEY=' + aws_secret_access_key + '; aws s3 ls ' + s3_location_json_ls + ' | awk \'{print "+json+ " $4}\' | sed \'s#/##\' | tail -n +2 > ' + jsonfilenamedbtab
+
+# Bash commands to create a temporary file containing the names of all (json, csv, parquet) files to be loaded
+join_csv_tab_files_bash_command = 'cat ' + csvfilenamedbtab + ' >> ' + alldbtab
+join_json_tab_files_bash_command = 'cat ' + jsonfilenamedbtab + ' >> ' + alldbtab
+join_parquet_tab_files_bash_command = 'cat ' + parqfilenamedbtab + ' >> ' + alldbtab
+
+# Bash command to clean up files containing table and database lists from the previous run if they exist
+cleanup_bash_command = 'rm -f ' + filenamedb + ' ' + parqfilenamedbtab + ' ' + csvfilenamedbtab + ' ' + jsonfilenamedbtab + ' ' + alldbtab
+
+# Returns the delimiter of the csv file. Supported csv delimiters are defined by the supported_csvdelimiters variable.
+# S3 bucket and file name are passed as arguments.
+def csv_delimiter(bucket, file):
+ try:
+ s3 = boto3.resource( 's3', region_name=region_name, aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
+ # bucket = topmost bucket, like tc-001-teracloud-nos-us-west-2-3745abcd, file = filename incl. lower buckets, ex csvdata/inventory.csv
+ # where complete path is tc-001-teracloud-nos-us-west-2-3745abcd/csvdata/inventory.csv
+ obj = s3.Object(bucket,file)
+ line = obj.get(]('Body']._raw_stream.readline().decode('UTF-8')
+ dialect = csv.Sniffer().sniff(line, delimiters=supported_csvdelimiters)
+ delimiter = dialect.__dict__['delimiter']
+ return(delimiter)
+ except Exception as ex:
+ logging.error(str(ex))
+ raise AirflowException
+
+
+
+
+
+# Returns the JSON fields (columns) in the file delimited by the '|' character. In case the Json file is nested the columns are flattened.
+# A sample of the JSON file (numner of lines defined by the linenumax variable) is copied from S3 to the filesystem and examined.
+# then ijson is used to examine it. Linenumax is by default set to 100, but for complex files can be increased.
+def json_fields(bucket, file):
+ try:
+
+ s3 = boto3.resource( 's3', region_name=region_name, aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
+ # bucket = topmost bucket, like tc-perf-001-teracloud-nos-us-west-2-3745a70d0aef, file = filename incl. lower buckets, ex csvdata/inventory.csv
+ # where complete path is tc-perf-001-teracloud-nos-us-west-2-3745a70d0aef/csvdata/inventory.csv
+ logging.info('Json-fields, Bucket: ' + bucket + ', File: ' + file)
+ obj = s3.Object(bucket,file)
+
+ if os.path.exists(output_file):
+ os.remove(output_file)
+
+ f = open(output_file,'w+')
+
+ linenum=1
+ while linenum <= linenumax:
+ # line = print_line(s3_bucket,json_subbucket + '/' + 'pd_review.json')
+ line = obj.get(]('Body']._raw_stream.readline().decode('UTF-8')
+ f.write(line)
+ linenum = linenum + 1
+
+ f.seek(0)
+
+ logging.info('Json-fields, Out Temp File: ' + output_file)
+ objects = ijson.items(f, "", multiple_values=True)
+ logging.info('Json-fields, Json objects: ' + str(objects))
+
+ key_string=""
+
+ for obj in objects:
+ first = True
+ for i in obj.keys():
+ if first:
+ key_string=key_string+i
+ first = False
+ else:
+ key_string=key_string+'|'+i
+ break
+
+ f.close
+ logging.info('Json-fields, Field string: ' + key_string)
+
+ return(key_string)
+
+
+ except Exception as ex:
+ logging.error(str(ex))
+ raise AirflowException
+
+
+
+
+
+default_args = {
+ 'owner': 'airflow',
+ 'depends_on_past': False,
+ 'start_date': datetime(2020,8,1),
+ 'retries': 0
+}
+
+
+@dag(dag_id="discover_dag", schedule_interval=None, start_date=datetime(2022, 4, 2))
+def taskflow():
+
+
+ # Create a temporary file containing the names of all databases.
+ # CSV database name comes from the variable csvdb, JSON database from variable jsondb, Parquet database(s) from the parquet sub-bucket name(s)
+ # The file is created in the directory name defined by the variable filenamedb. By default this is /tmp/db.txt on the host system or
+ # /opt/airflow/tmp/db.txt on the container, but is configurable by changing the variable value and the /tmp mount in the docker_compose.yaml file
+ @task
+ def make_file_db():
+ logging.info ("Cleaning up old files : " + cleanup_bash_command)
+ subprocess.run(cleanup_bash_command, shell=True, check=True, executable='/bin/bash')
+ logging.info ("Executing bash: " + create_db_file_bash(filenamedb))
+ createdb_file_bash_command = create_db_file_bash(filenamedb)
+ subprocess.run(createdb_file_bash_command, shell=True, check=True, executable='/bin/bash')
+ return (filenamedb)
+
+ # Create databases based on the database names found in the file created in the make_file_db task
+ # Notice the password is the same as the database name, manually change the password as needed
+ @task
+ def create_db(filenamedb):
+ try:
+ logging.info ("Opening file " + filenamedb)
+ with open(filenamedb) as file:
+ lines = file.readlines()
+ lines = [line.rstrip() for line in lines]
+ logging.info ("File "+filenamedb+" found, opened, read successfully.")
+
+ # for each line in the db file (i.e. for each database), create user/database and auth object to access S3
+ # databases are created all of the same size because the data will not be loaded into the databasebut in the NOS storage
+ for line in lines:
+ sqlcommandstr = "create user " + line + " as perm=" + perm_dbsize + ",password=" + line + "; grant all on " + line + " to " + line + " with grant option; grant create database on " + line + " to " + line + "; grant EXECUTE FUNCTION on TD_SYSFNLIB to " + line + "; database " + line + "; drop AUTHORIZATION " + line + "." + auth_name + "; CREATE AUTHORIZATION " + line + "." + auth_name + " AS DEFINER TRUSTED USER '" + aws_access_key_id + "' PASSWORD '" + aws_secret_access_key + "';"
+ execute_sql_commands(sqlcommandstr)
+ file.close()
+ os.remove(filenamedb)
+ return(parqfilenamedbtab)
+
+ except Exception as ex:
+ logging.error(str(ex))
+ raise AirflowException
+
+
+
+
+ # Create temporary files containing all the table names. Bash commands use aws command line create a list of files/tables
+ # The argument parquetfilenamedb is a placeholder to support the airflow task flow.
+ @task
+ def make_file_dbtab(parqfilenamedbtab):
+ logging.info ("Create empty files bash: " + create_placeholder_files_command)
+ subprocess.run(create_placeholder_files_command, shell=True, check=True, executable='/bin/bash')
+ if (load_csv == 'Y'):
+ logging.info ("Executing csv bash: " + create_csv_tab_file_bash_command)
+ subprocess.run(create_csv_tab_file_bash_command, shell=True, check=True, executable='/bin/bash')
+ logging.info ("Executing csv join bash: " + join_csv_tab_files_bash_command)
+ subprocess.run(join_csv_tab_files_bash_command, shell=True, check=True, executable='/bin/bash')
+ if (load_json == 'Y'):
+ logging.info ("Executing json bash: " + create_json_tab_file_bash_command)
+ subprocess.run(create_json_tab_file_bash_command, shell=True, check=True, executable='/bin/bash')
+ logging.info ("Executing json join bash: " + join_json_tab_files_bash_command)
+ subprocess.run(join_json_tab_files_bash_command, shell=True, check=True, executable='/bin/bash')
+ if (load_parquet == 'Y'):
+ logging.info ("Executing parq bash: " + create_parq_db_tab_file_bash_command)
+ subprocess.run(create_parq_db_tab_file_bash_command, shell=True, check=True, executable='/bin/bash')
+ logging.info ("Executing parquet join bash: " + join_parquet_tab_files_bash_command)
+ subprocess.run(join_parquet_tab_files_bash_command, shell=True, check=True, executable='/bin/bash')
+ # logging.info ("Executing join_file bash: " + join_tab_files_bash_command)
+ # subprocess.run(join_tab_files_bash_command, shell=True, check=True, executable='/bin/bash')
+ return(alldbtab)
+
+ # Open filename created by the make_file_dbtab task containing the table names and return the content
+ @task
+ def make_tab_list(filename):
+ # Open and read the file as a single buffer, then split sql commnds based on the ";" character, i.e. commands must be separated by ";"
+ logging.info ("Opening file " + filename)
+ try:
+ with open(filename) as file:
+ lines = file.readlines()
+ lines = [line.rstrip() for line in lines]
+ file.close()
+ # os.remove(filename)
+ logging.info ("File "+filename+" found, opened, read successfully.")
+ return (lines)
+ except Exception as ex:
+ logging.error ("File error ", str (ex).split ("\n") [0])
+ raise AirflowException
+
+
+ # Based on the list of tables passed by the previous task and create the tables.
+ # The tables can be csv (prefixed by +csv+), json (prefixed by +json+) or parquet (no +parquet+ prefix, but simply database and table name) .
+ # Each file type has a different creation process and SQL code.
+ @task
+ def create_tables(arg):
+ logging.info ("Creating table for record :"+arg)
+ argstring = arg.split(' ')
+ i = 1
+ db=""
+ tbl=""
+ tbltype=""
+ csvfilename = ""
+ jsonfilename = ""
+ for argstr in argstring:
+ argstr = argstr.strip()
+ logging.info ('Arg passed ' + str(i) + ': ' + argstr + ';')
+ if (i == 1) :
+ if (argstr == '+csv+'):
+ tbltype = 'csv'
+ db = csvdb
+ elif (argstr == '+json+'):
+ tbltype = 'json'
+ db = jsondb
+ else :
+ tbltype = 'parquet'
+ db = argstr
+ i = i + 1
+ elif (i == 2):
+ if (tbltype == 'csv'):
+ csvfilename = argstr
+ tbl = argstr.split('.',1](0]
+ db = csvdb
+ bucketfile = csv_subbucket + '/' + csvfilename
+ csvdelimiter = csv_delimiter(s3_bucket, csv_subbucket + '/' + csvfilename )
+ logging.info ('CSV File path :' + bucketfile)
+ logging.info ('CSV Delimiter :' + csvdelimiter)
+ elif (tbltype == 'json'):
+ jsonfilename = argstr
+ tbl = argstr.split('.',1](0]
+ db = jsondb
+ bucketfile = json_subbucket + '/' + jsonfilename
+ jsonfieldstr = json_fields(s3_bucket, json_subbucket + '/' + jsonfilename )
+ logging.info ('JSON File path :' + bucketfile)
+ logging.info ('JSON Fields String :' + jsonfieldstr)
+ else :
+ tbl = argstr
+
+
+ logging.info ("Table type :" + tbltype + " Table name: " + tbl + " Database: " + db)
+
+
+
+
+
+
+ if (tbltype == 'parquet'):
+
+ sqlstr_parq_ft = "drop FOREIGN TABLE " + db + "." + tbl + "_parq_ft; CREATE FOREIGN TABLE " + db + "." + tbl + "_parq_ft ,EXTERNAL SECURITY DEFINER TRUSTED " + auth_name + " USING ( LOCATION ('" + s3_location_parq_ft + db + "/" + tbl + "/') STOREDAS ('PARQUET') ) NO PRIMARY INDEX PARTITION BY COLUMN; select cast(count(*) as bigint) from " + db + "." + tbl + "_parq_ft;"
+
+ logging.info ("Parquet foreign table string:" + sqlstr_parq_ft)
+
+
+ sqlstr_parq_nosfs = "drop TABLE " + db + "." + tbl + "_parq_nos; CREATE MULTISET TABLE " + db + "." + tbl + "_parq_nos, STORAGE = " + nos_storage + " as ( select * from antiselect ( on " + db + "." + tbl + "_parq_ft using exclude ('location')) as tbl) with data no primary index; select cast(count(*) as bigint) from " + db + "." + tbl + "_parq_nos; select cast(count(*) as bigint) from " + db + "." + tbl + "_parq_ft;"
+
+ logging.info ("Parquet nosfs table string:" + sqlstr_parq_nosfs)
+
+
+ parq_sqlstr_all = sqlstr_parq_ft + sqlstr_parq_nosfs
+
+ execute_sql_commands(parq_sqlstr_all)
+
+
+ elif (tbltype == 'csv'):
+ sqlstr_csv_ft = 'drop FOREIGN TABLE ' + csvdb + '.' + tbl + '_csv_ft; CREATE FOREIGN TABLE ' + csvdb + '.' + tbl + '_csv_ft ,EXTERNAL SECURITY DEFINER TRUSTED ' + auth_name + ' USING ( LOCATION (\'' + s3_location_csv_create + '/' + csvfilename +'\') ROWFORMAT ('+'\'{"field_delimiter":"' + csvdelimiter + '","record_delimiter":"\\n","character_set":"LATIN"}\') HEADER (\'TRUE\')); select cast(count(*) as bigint) from ' + csvdb + '.' + tbl + '_csv_ft;'
+
+ sqlstr_csv_nosfs = "drop TABLE " + csvdb + "." + tbl + "_csv_nos; CREATE MULTISET TABLE " + csvdb + "." + tbl + "_csv_nos, STORAGE = " + nos_storage + " as ( select * from antiselect ( on " + csvdb + "." + tbl + "_csv_ft using exclude ('location')) as tbl) with data no primary index; select cast(count(*) as bigint) from " + csvdb + "." + tbl + "_csv_nos; select cast(count(*) as bigint) from " + csvdb + "." + tbl + "_csv_ft;"
+
+ logging.info ("Csv nosfs table string:" + sqlstr_csv_nosfs)
+
+ csv_sqlstr_all = sqlstr_csv_ft + sqlstr_csv_nosfs
+
+ # sqlcommands = csv_sqlstr_all.split(';')
+ execute_sql_commands(csv_sqlstr_all)
+
+ elif (tbltype == 'json'):
+
+ sqlstr_json_ft = 'drop FOREIGN TABLE ' + jsondb + '.' + tbl + '_json_ft; CREATE FOREIGN TABLE ' + jsondb + '.' + tbl + '_json_ft ,EXTERNAL SECURITY DEFINER TRUSTED ' + auth_name + ' USING ( LOCATION (\'' + s3_location_json_create + '/' + jsonfilename +'\')); select cast(count(*) as bigint) from ' + jsondb + '.' + tbl + '_json_ft;'
+
+
+ execute_sql_commands(sqlstr_json_ft)
+
+ sqlstr_json_select = get_json_nos_select_comm(jsondb + '.' + tbl + '_json_ft')
+
+
+ sqlstr_json_nosfs = "drop TABLE " + jsondb + "." + tbl + "_json_nos; CREATE MULTISET TABLE " + jsondb + "." + tbl + "_json_nos, STORAGE = " + nos_storage + " as ( " + sqlstr_json_select + " ) with data no primary index; select cast(count(*) as bigint) from " + jsondb + "." + tbl + "_json_nos; select cast(count(*) as bigint) from " + jsondb + "." + tbl + "_json_ft;"
+
+ execute_sql_commands(sqlstr_json_nosfs)
+
+
+
+
+
+
+
+
+
+
+
+
+ create_tables.expand(arg=make_tab_list(make_file_dbtab(create_db(make_file_db()))))
+
+
+dag = taskflow()
+
+
diff --git a/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/docker-compose.yaml b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/docker-compose.yaml
new file mode 100644
index 0000000000..82d30f487d
--- /dev/null
+++ b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/docker-compose.yaml
@@ -0,0 +1,351 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
+# 3 workers are created, others can be added.
+# Added nginx web server for the dbt use case.
+#
+# WARNING: This configuration is for local development. Do not use it in a production deployment.
+#
+# This configuration supports basic configuration using environment variables or an .env file
+# The following variables are supported:
+#
+# AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow.
+# Default: apache/airflow:|version|
+# AIRFLOW_UID - User ID in Airflow containers
+# Default: 50000
+# Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
+#
+# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account (if requested).
+# Default: airflow
+# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account (if requested).
+# Default: airflow
+# _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers.
+# Default: ''
+#
+# Feel free to modify this file to suit your needs.
+---
+version: '3'
+x-airflow-common:
+ &airflow-common
+ # In order to add custom dependencies or upgrade provider packages you can use your extended image.
+ # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
+ # and uncomment the "build" line below, Then run `docker-compose build` to build the images.
+ image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.2.4}
+ build: .
+ environment:
+ &airflow-common-env
+ AIRFLOW__CORE__EXECUTOR: CeleryExecutor
+ AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
+ # For backward compatibility, with Airflow <2.3
+ AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
+ AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
+ AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
+ AIRFLOW__CORE__FERNET_KEY: ''
+ AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
+ AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
+ AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth'
+ _PIP_ADDITIONAL_REQUIREMENTS: ''
+ # _PIP_ADDITIONAL_REQUIREMENTS will be implemented in the Dockerfile, that is why it is commented out here
+ # _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:- sqlalchemy sqlalchemy-teradata teradatasql teradatasqlalchemy dbt-teradata}
+ volumes:
+ # Volumes host system directories (in this example host system is an AWS EC2 Linux instance) which will be mounted / readable / writable from all containers.
+ # The first directory is the path on the host system, the second (separated by ":") is the path on the docker container.
+ # These will have to be changed for a different setups / systems
+ # ./dags - Airflow dags directory where the dag python files are places
+ - ./dags:/opt/airflow/dags
+ # ./logs - Airflow logs directory
+ - ./logs:/opt/airflow/logs
+ # plugins - Airflow plugin directory
+ - ./plugins:/opt/airflow/plugins
+ # airflow.cfg - airflow configuration file used when airflow is started on the container
+ - ./config/airflow.cfg:/opt/airflow/airflow.cfg
+ # /tmp - temporary directory used to create / store temporary files
+ - /tmp:/opt/airflow/tmp
+ # The dbt directory (here installed under /home/ec3-user) contains the dbt project
+ - /home/ec2-user/dbt/jaffle_shop/data:/opt/airflow/data
+ - /home/ec2-user/dbt/jaffle_shop/dbt_project.yml:/opt/airflow/dbt_project.yml
+ - /home/ec2-user/dbt/jaffle_shop/etc:/opt/airflow/etc
+ - /home/ec2-user/dbt/jaffle_shop/LICENSE:/opt/airflow/LICENSE
+ - /home/ec2-user/dbt/jaffle_shop/models:/opt/airflow/models
+ # The .dbt directory contain the .dbt configuration files
+ - /home/ec2-user/.dbt:/home/airflow/.dbt
+ - /home/ec2-user/dbt/jaffle_shop/target:/opt/airflow/target
+ user: "${AIRFLOW_UID:-50000}:0"
+ depends_on:
+ &airflow-common-depends-on
+ redis:
+ condition: service_healthy
+ postgres:
+ condition: service_healthy
+
+services:
+ postgres:
+ image: postgres:13
+ environment:
+ POSTGRES_USER: airflow
+ POSTGRES_PASSWORD: airflow
+ POSTGRES_DB: airflow
+ volumes:
+ - postgres-db-volume:/var/lib/postgresql/data
+ healthcheck:
+ test: ["CMD", "pg_isready", "-U", "airflow"]
+ interval: 5s
+ retries: 5
+ restart: always
+
+ redis:
+ image: redis:latest
+ expose:
+ - 6379
+ healthcheck:
+ test: ["CMD", "redis-cli", "ping"]
+ interval: 5s
+ timeout: 30s
+ retries: 50
+ restart: always
+ # nginx added to visualize on a web browser the DBT generated documents. Nginx is here configured on host port 4000
+ nginx:
+ image: nginx
+ ports:
+ - 4000:80
+ volumes:
+ - /home/ec2-user/dbt/jaffle_shop/target:/usr/share/nginx/html
+ healthcheck:
+ test: ["CMD", "curl", "-f", "http://localhost"]
+ interval: 1m30s
+ timeout: 10s
+ retries: 3
+ start_period: 1m #version 3.4 minimum
+
+
+ airflow-webserver:
+ <<: *airflow-common
+ command: webserver
+ ports:
+ - 8080:8080
+ healthcheck:
+ test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
+ interval: 10s
+ timeout: 10s
+ retries: 5
+ restart: always
+ depends_on:
+ <<: *airflow-common-depends-on
+ airflow-init:
+ condition: service_completed_successfully
+
+ airflow-scheduler:
+ <<: *airflow-common
+ command: scheduler
+ healthcheck:
+ test: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"']
+ interval: 10s
+ timeout: 10s
+ retries: 5
+ restart: always
+ depends_on:
+ <<: *airflow-common-depends-on
+ airflow-init:
+ condition: service_completed_successfully
+ # Three workers installed so airflow can in parallel execute up to 3 tasks. If more are needed, just /cut/paste/add/rename additional worker config sessions
+ airflow-worker_1:
+ <<: *airflow-common
+ command: celery worker
+ healthcheck:
+ test:
+ - "CMD-SHELL"
+ - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
+ interval: 10s
+ timeout: 10s
+ retries: 5
+ environment:
+ <<: *airflow-common-env
+ # Required to handle warm shutdown of the celery workers properly
+ # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
+ DUMB_INIT_SETSID: "0"
+ restart: always
+ depends_on:
+ <<: *airflow-common-depends-on
+ airflow-init:
+ condition: service_completed_successfully
+
+ airflow-worker_2:
+ <<: *airflow-common
+ command: celery worker
+ healthcheck:
+ test:
+ - "CMD-SHELL"
+ - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
+ interval: 10s
+ timeout: 10s
+ retries: 5
+ environment:
+ <<: *airflow-common-env
+ # Required to handle warm shutdown of the celery workers properly
+ # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
+ DUMB_INIT_SETSID: "0"
+ restart: always
+ depends_on:
+ <<: *airflow-common-depends-on
+ airflow-init:
+ condition: service_completed_successfully
+
+ airflow-worker_3:
+ <<: *airflow-common
+ command: celery worker
+ healthcheck:
+ test:
+ - "CMD-SHELL"
+ - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
+ interval: 10s
+ timeout: 10s
+ retries: 5
+ environment:
+ <<: *airflow-common-env
+ # Required to handle warm shutdown of the celery workers properly
+ # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
+ DUMB_INIT_SETSID: "0"
+ restart: always
+ depends_on:
+ <<: *airflow-common-depends-on
+ airflow-init:
+ condition: service_completed_successfully
+
+ airflow-triggerer:
+ <<: *airflow-common
+ command: triggerer
+ healthcheck:
+ test: ["CMD-SHELL", 'airflow jobs check --job-type TriggererJob --hostname "$${HOSTNAME}"']
+ interval: 10s
+ timeout: 10s
+ retries: 5
+ restart: always
+ depends_on:
+ <<: *airflow-common-depends-on
+ airflow-init:
+ condition: service_completed_successfully
+
+ airflow-init:
+ <<: *airflow-common
+ entrypoint: /bin/bash
+ # yamllint disable rule:line-length
+ command:
+ - -c
+ - |
+ function ver() {
+ printf "%04d%04d%04d%04d" $${1//./ }
+ }
+ airflow_version=$$(gosu airflow airflow version)
+ airflow_version_comparable=$$(ver $${airflow_version})
+ min_airflow_version=2.2.0
+ min_airflow_version_comparable=$$(ver $${min_airflow_version})
+ if (( airflow_version_comparable < min_airflow_version_comparable )); then
+ echo
+ echo -e "\033[1;31mERROR!!!: Too old Airflow version $${airflow_version}!\e[0m"
+ echo "The minimum Airflow version supported: $${min_airflow_version}. Only use this or higher!"
+ echo
+ exit 1
+ fi
+ if [[ -z "${AIRFLOW_UID}" ]]; then
+ echo
+ echo -e "\033[1;33mWARNING!!!: AIRFLOW_UID not set!\e[0m"
+ echo "If you are on Linux, you SHOULD follow the instructions below to set "
+ echo "AIRFLOW_UID environment variable, otherwise files will be owned by root."
+ echo "For other operating systems you can get rid of the warning with manually created .env file:"
+ echo " See: https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#setting-the-right-airflow-user"
+ echo
+ fi
+ one_meg=1048576
+ mem_available=$$(($$(getconf _PHYS_PAGES) * $$(getconf PAGE_SIZE) / one_meg))
+ cpus_available=$$(grep -cE 'cpu[0-9]+' /proc/stat)
+ disk_available=$$(df / | tail -1 | awk '{print $$4}')
+ warning_resources="false"
+ if (( mem_available < 4000 )) ; then
+ echo
+ echo -e "\033[1;33mWARNING!!!: Not enough memory available for Docker.\e[0m"
+ echo "At least 4GB of memory required. You have $$(numfmt --to iec $$((mem_available * one_meg)))"
+ echo
+ warning_resources="true"
+ fi
+ if (( cpus_available < 2 )); then
+ echo
+ echo -e "\033[1;33mWARNING!!!: Not enough CPUS available for Docker.\e[0m"
+ echo "At least 2 CPUs recommended. You have $${cpus_available}"
+ echo
+ warning_resources="true"
+ fi
+ if (( disk_available < one_meg * 10 )); then
+ echo
+ echo -e "\033[1;33mWARNING!!!: Not enough Disk space available for Docker.\e[0m"
+ echo "At least 10 GBs recommended. You have $$(numfmt --to iec $$((disk_available * 1024 )))"
+ echo
+ warning_resources="true"
+ fi
+ if [[ $${warning_resources} == "true" ]]; then
+ echo
+ echo -e "\033[1;33mWARNING!!!: You have not enough resources to run Airflow (see above)!\e[0m"
+ echo "Please follow the instructions to increase amount of resources available:"
+ echo " https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#before-you-begin"
+ echo
+ fi
+ mkdir -p /sources/logs /sources/dags /sources/plugins
+ chown -R "${AIRFLOW_UID}:0" /sources/{logs,dags,plugins}
+ exec /entrypoint airflow version
+ # yamllint enable rule:line-length
+ environment:
+ <<: *airflow-common-env
+ _AIRFLOW_DB_UPGRADE: 'true'
+ _AIRFLOW_WWW_USER_CREATE: 'true'
+ _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
+ _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
+ user: "0:0"
+ volumes:
+ - .:/sources
+
+ airflow-cli:
+ <<: *airflow-common
+ profiles:
+ - debug
+ environment:
+ <<: *airflow-common-env
+ CONNECTION_CHECK_MAX_COUNT: "0"
+ # Workaround for entrypoint issue. See: https://github.com/apache/airflow/issues/16252
+ command:
+ - bash
+ - -c
+ - airflow
+
+ flower:
+ <<: *airflow-common
+ command: celery flower
+ ports:
+ - 5555:5555
+ healthcheck:
+ test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
+ interval: 10s
+ timeout: 10s
+ retries: 5
+ restart: always
+ depends_on:
+ <<: *airflow-common-depends-on
+ airflow-init:
+ condition: service_completed_successfully
+
+volumes:
+ postgres-db-volume:
diff --git a/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/profiles.yml b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/profiles.yml
new file mode 100644
index 0000000000..691c767cc8
--- /dev/null
+++ b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/profiles.yml
@@ -0,0 +1,15 @@
+jaffle_shop:
+ outputs:
+ dev:
+ type: teradata
+ host: 192.11.25.33
+ user: jaffle_shop
+ password: abcd
+ logmech: TD2
+ schema: jaffle_shop
+ tmode: ANSI
+ threads: 1
+ timeout_seconds: 300
+ priority: interactive
+ retries: 1
+ target: dev
diff --git a/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/variables.json b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/variables.json
new file mode 100644
index 0000000000..0905f82ac7
--- /dev/null
+++ b/quickstarts/other-integrations/attachments/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/variables.json
@@ -0,0 +1,36 @@
+{
+"aws_access_key_id" : "*******",
+"aws_secret_access_key" : "**************",
+"s3_location_parq_ls" : "s3://tc-001-teracloud-nos-us-west-2-374222bfg/soc/nosexports/",
+"s3_location_parq_ft" : "/s3/s3.amazonaws.com/tc-001-teracloud-nos-us-west-2-374222bfg/soc/nosexports/",
+"s3_location_csv_ls" : "s3://tc-001-teracloud-nos-us-west-2-374222bfg/csvdata/",
+"s3_location_csv_create" : "/s3/s3.amazonaws.com/tc-001-teracloud-nos-us-west-2-374222bfg/csvdata",
+"s3_location_csv_ft" : "/s3/s3.amazonaws.com/tc-001-teracloud-nos-us-west-2-374222bfg/nosexports/",
+"s3_location_json_ls" : "s3://tc-001-teracloud-nos-us-west-2-374222bfg/jsondata/",
+"s3_location_json_ft" : "/s3/s3.amazonaws.com/tc-001-teracloud-nos-us-west-2-374222bfg/nosexports/",
+"s3_location_json_create" : "/s3/s3.amazonaws.com/tc-001-teracloud-nos-us-west-2-374222bfg/jsondata",
+"s3_bucket" : "tc-perf-teracloud-nos-us-west-2-374222bfg",
+"csv_subbucket" : "csvdata",
+"json_subbucket" : "jsondata",
+"filenamedb" :"/opt/airflow/tmp/db.txt",
+"parqfilenamedbtab" :"/opt/airflow/tmp/parqdbtbl.txt",
+"csvfilenamedbtab" :"/opt/airflow/tmp/csvdbtbl.txt",
+"jsonfilenamedbtab" :"/opt/airflow/tmp/jsondbtbl.txt",
+"alldbtab" : "/opt/airflow/tmp/alldbtbl.txt",
+"csvdb" :"csvdb",
+"supported_csvdelimiters" : ",:|\t",
+"supported_csvlineterminator" : "\n",
+"jsondb" : "jsondb",
+"DB_username" : "dbc",
+"DB_password" : "dbc",
+"DB_ip_address" : "***.***.***.***",
+"auth_name" : "soc_Auth_NOS",
+"region_name" : "us-west-2",
+"output_file" : "/tmp/outfile.txt",
+"perm_dbsize" : "5e9",
+"nos_storage" : "TD_NOSFS_STORAGE",
+"load_csv" : "Y",
+"load_json" : "Y",
+"load_parquet" : "Y",
+"linenumax" : "100"
+}
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/create-new-source.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/create-new-source.png
new file mode 100644
index 0000000000..3c47435443
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/create-new-source.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/datasets.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/datasets.png
new file mode 100644
index 0000000000..f2665ef5a8
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/datasets.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/entities-list.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/entities-list.png
new file mode 100644
index 0000000000..9fc0de927a
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/entities-list.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/execute.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/execute.png
new file mode 100644
index 0000000000..1dc2d37bfb
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/execute.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/finish-up.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/finish-up.png
new file mode 100644
index 0000000000..b9d0aa83cc
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/finish-up.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/ingestion-icon.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/ingestion-icon.png
new file mode 100644
index 0000000000..ad99432a44
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/ingestion-icon.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/ingestion-result.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/ingestion-result.png
new file mode 100644
index 0000000000..eee487a25a
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/ingestion-result.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/lineage-weather.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/lineage-weather.png
new file mode 100644
index 0000000000..5b09a28376
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/lineage-weather.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/lineage.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/lineage.png
new file mode 100644
index 0000000000..edf5732fc6
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/lineage.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/new-ingestion-source.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/new-ingestion-source.png
new file mode 100644
index 0000000000..24cfd9bfa0
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/new-ingestion-source.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/schema.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/schema.png
new file mode 100644
index 0000000000..3e1b374a66
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/schema.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/select-other-source.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/select-other-source.png
new file mode 100644
index 0000000000..9663a52475
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/select-other-source.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/set-schedule.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/set-schedule.png
new file mode 100644
index 0000000000..dbea9babf4
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-datahub/set-schedule.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/configure-connection.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/configure-connection.png
new file mode 100644
index 0000000000..df14883bda
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/configure-connection.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/configure-driver-string.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/configure-driver-string.png
new file mode 100644
index 0000000000..b073165776
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/configure-driver-string.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/copy-driver.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/copy-driver.png
new file mode 100644
index 0000000000..e73277ed90
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/copy-driver.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/create-connection.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/create-connection.png
new file mode 100644
index 0000000000..f894a5cca2
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/create-connection.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/plug-icon.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/plug-icon.png
new file mode 100644
index 0000000000..fa3901a781
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/plug-icon.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/select-your-database-windows.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/select-your-database-windows.png
new file mode 100644
index 0000000000..d84ca6de29
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/select-your-database-windows.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/select-your-database.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/select-your-database.png
new file mode 100644
index 0000000000..db45329fe9
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/select-your-database.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings-ssh-windows.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings-ssh-windows.png
new file mode 100644
index 0000000000..848d502e0e
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings-ssh-windows.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings-ssh.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings-ssh.png
new file mode 100644
index 0000000000..61d75adc0b
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings-ssh.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings-windows-ldap.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings-windows-ldap.png
new file mode 100644
index 0000000000..e8c6495c42
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings-windows-ldap.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings-windows.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings-windows.png
new file mode 100644
index 0000000000..7283e06f76
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings-windows.png differ
diff --git a/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings.png b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings.png
new file mode 100644
index 0000000000..1b09bf6473
Binary files /dev/null and b/quickstarts/other-integrations/images/configure-a-teradata-connection-in-dbeaver/teradata-connection-settings.png differ
diff --git a/quickstarts/other-integrations/images/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/admin-dropdown.png b/quickstarts/other-integrations/images/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/admin-dropdown.png
new file mode 100644
index 0000000000..09dd180dc3
Binary files /dev/null and b/quickstarts/other-integrations/images/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/admin-dropdown.png differ
diff --git a/quickstarts/other-integrations/images/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/import-variables.png b/quickstarts/other-integrations/images/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/import-variables.png
new file mode 100644
index 0000000000..7cc50d83a4
Binary files /dev/null and b/quickstarts/other-integrations/images/execute-airflow-workflows-that-use-dbt-with-teradata-vantage/import-variables.png differ
diff --git a/quickstarts/other-integrations/images/getting-started-dbt-feast-teradata-pipeline/dbt-feast.png b/quickstarts/other-integrations/images/getting-started-dbt-feast-teradata-pipeline/dbt-feast.png
new file mode 100644
index 0000000000..dd7d8a8d60
Binary files /dev/null and b/quickstarts/other-integrations/images/getting-started-dbt-feast-teradata-pipeline/dbt-feast.png differ
diff --git a/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/add-jar.png b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/add-jar.png
new file mode 100644
index 0000000000..6de7092bce
Binary files /dev/null and b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/add-jar.png differ
diff --git a/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/apply-and-close.png b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/apply-and-close.png
new file mode 100644
index 0000000000..41ddfff79b
Binary files /dev/null and b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/apply-and-close.png differ
diff --git a/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/enter-configuration.png b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/enter-configuration.png
new file mode 100644
index 0000000000..4028ae692e
Binary files /dev/null and b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/enter-configuration.png differ
diff --git a/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/execute-node.png b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/execute-node.png
new file mode 100644
index 0000000000..e1c0725ce2
Binary files /dev/null and b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/execute-node.png differ
diff --git a/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/register-driver.png b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/register-driver.png
new file mode 100644
index 0000000000..d57bb6f83c
Binary files /dev/null and b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/register-driver.png differ
diff --git a/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/start-configuration.png b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/start-configuration.png
new file mode 100644
index 0000000000..e409f70d2a
Binary files /dev/null and b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/start-configuration.png differ
diff --git a/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/test-connection-1.png b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/test-connection-1.png
new file mode 100644
index 0000000000..a87fc46354
Binary files /dev/null and b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/test-connection-1.png differ
diff --git a/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/test-connection-2.png b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/test-connection-2.png
new file mode 100644
index 0000000000..51973412b0
Binary files /dev/null and b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/test-connection-2.png differ
diff --git a/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/test-connection-apply.png b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/test-connection-apply.png
new file mode 100644
index 0000000000..7d11dcfece
Binary files /dev/null and b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/test-connection-apply.png differ
diff --git a/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/view-results-final.png b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/view-results-final.png
new file mode 100644
index 0000000000..add30b8e51
Binary files /dev/null and b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/view-results-final.png differ
diff --git a/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/view-results.png b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/view-results.png
new file mode 100644
index 0000000000..9456d5d2de
Binary files /dev/null and b/quickstarts/other-integrations/images/integrate-teradata-vantage-with-knime/view-results.png differ
diff --git a/quickstarts/vantagecloud-lake/_category_.json b/quickstarts/vantagecloud-lake/_category_.json
new file mode 100644
index 0000000000..58fbcda1af
--- /dev/null
+++ b/quickstarts/vantagecloud-lake/_category_.json
@@ -0,0 +1,4 @@
+{
+ "label": "VantageCloud Lake",
+ "position": 7
+ }
\ No newline at end of file
diff --git a/quickstarts/vantagecloud-lake/_partials/vantagecloud-lake-request.md b/quickstarts/vantagecloud-lake/_partials/vantagecloud-lake-request.md
new file mode 100644
index 0000000000..b46a2ed711
--- /dev/null
+++ b/quickstarts/vantagecloud-lake/_partials/vantagecloud-lake-request.md
@@ -0,0 +1,3 @@
+:::tip
+To request a VantageCloud Lake environment, refer to the form provided in this [link](https://www.teradata.com/about-us/contact). If you already have a VantageCloud Lake environment and seek guidance on configuration, please consult this [guide](https://quickstarts.teradata.com/getting-started-with-vantagecloud-lake.html).
+:::
\ No newline at end of file
diff --git a/quickstarts/vantagecloud-lake/getting-started-with-vantagecloud-lake.md b/quickstarts/vantagecloud-lake/getting-started-with-vantagecloud-lake.md
new file mode 100644
index 0000000000..ba6d19bafc
--- /dev/null
+++ b/quickstarts/vantagecloud-lake/getting-started-with-vantagecloud-lake.md
@@ -0,0 +1,266 @@
+---
+sidebar_position: 1
+author: Vidhan Bhonsle
+email: vidhan.bhonsle@teradata.com
+page_last_update: January 2nd, 2024
+description: Create your own environment in VantageCloud Lake
+keywords: [data warehouses, compute storage separation, teradata, vantage, cloud data platform, business intelligence, enterprise analytics, jupyter, teradatasql, ipython-sql, teradatasqlalchemy, vantagecloud, vantagecloud lake, ip address, public internet, lake]
+---
+# Getting Started with VantageCloud Lake
+
+## Overview
+
+Teradata VantageCloud Lake is Teradata's next-generation, cloud-native analytics and data platform. It provides lakehouse deployment patterns along with the ability to run independent elastic workloads using an object storage-centric design. +
+It empowers organizations to unlock their data, activate analytics, and accelerate value. Customers can optimize their analytics environment using specially configured compute cluster resources that best accommodate their workload requirements. +
+
+
+
+
+VantageCloud Lake provides all the benefits you’d expect in a cloud solution plus Teradata’s differentiated technology stack, including the industry-leading Analytics Database, ClearScape Analytics, and QueryGrid data fabric.
+
+## Sign-on to VantageCloud Lake
+
+IMPORTANT: To get a VantageCloud Lake sign-on link and credentials, fill the [contact form](https://www.teradata.com/about-us/contact) to reach out to Teradata team.
+
+Go to the URL provided by Teradata, for example, **_ourcompany.innovationlabs.teradata.com_** and sign on:
+
+* Existing customers can use their organization admin username(email address) and password to sign on.
+* New customer can sign on with their organization admin username (from welcome letter: email address) and the password you created.
+
+:::note
+Click [here](https://login.customer.teradata.com/ext/pwdreset/Identify?AdapterId=CDSCustomer) to reset the organization admin password.
+:::
+
+
+
+The signing on takes you to VantageCloud Lake welcome page.
+
+
+
+The Welcome page has navigation menu that not only gives you a complete control of your environments but also provides you various necessary tools:
+
+
+
+* Vantage - Home page of VantageCloud Lake portal
+* [Environments](https://docs.teradata.com/r/Teradata-VantageCloud-Lake/Getting-Started-First-Sign-On-by-Organization-Admin/Step-1-Signing-On-and-Creating-Your-First-Environment) - Create your environments and see all the created environments
+* [Organization](https://docs.teradata.com/r/Teradata-VantageCloud-Lake/Introduction-to-VantageCloud-Lake/VantageCloud-Lake-Organizations-and-Environments) - View organizations configuration, manage Organization Admins and view the configuration and status of your account
+* [Consumption](https://docs.teradata.com/r/Teradata-VantageCloud-Lake/Managing-Compute-Resources/Review-Consumption-Usage) - Monitor how your organization consumes compute and storage resources
+* [Cost Calculator](https://docs.teradata.com/r/Teradata-VantageCloud-Lake/Using-VantageCloud-Lake-Console-to-Manage-VantageCloud-Lake/Using-the-Consumption-Estimates) - Calculate the cost and consumption of your environment and whole organization.
+* [Queries](https://docs.teradata.com/r/Teradata-VantageCloud-Lake/Running-and-Monitoring-Queries/Monitoring-and-Managing-Queries) - Inspect an environment's queries to understand their efficiency.
+* [Editor](https://docs.teradata.com/r/Teradata-VantageCloud-Lake/Running-and-Monitoring-Queries) - Create and run queries in an editor.
+* [Data Copy](https://docs.teradata.com/r/Teradata-VantageCloud-Lake/Data-Copy) - Provision, configure and run data copy (also known as Data Mover) jobs from VantageCloud Lake Console.
+
+
+## Create an Environment
+To create a primary cluster environment, click on "Environments" on the navigation menu. In a new opened view, click on "Create" button situated on the top right of the page.
+
+
+
+### Environment configuration
+
+Fill out the Environment configuration fields:
+
+
+| **Item** | **Description** |
+|--------------------|---------------------------------------------------------------------------------|
+| *Environment name* | A contextual name for new environment |
+| *Region* | The available region list was predetermined during the sales process. |
+| *Package* | There are dos service packages available to select from: |
+| | - Lake: Premier 24x7 cloud support |
+| | - Lake: Premier 24x7 Priority cloud support + industry data models |
+
+
+
+
+
+:::important
+The **Consumption estimates**, to your right, provide guidance for configuring the environment. See [Using the Consumption Estimates](https://docs.teradata.com/r/Teradata-VantageCloud-Lake/Using-VantageCloud-Lake-Console-to-Manage-VantageCloud-Lake/Using-the-Consumption-Estimates) for more detail.
+:::
+
+### Primary cluster configuration
+
+Fill out the Primary cluster configuration fields:
+
+
+```mdx-code-block
+
+
+
+
+
+
+```
+
+
+
+
+
+### Database credentials
+
+Fill out the Database credential fields:
+
+| **Item** | **Description** |
+|--------------------|---------------------------------------------------------------------------------|
+| DBC password | The primary administrative account in a Teradata Vantage environment is known as "dbc". Similar to the root user in Linux, the dbc account holds comprehensive administrative privileges. It is recommended to establish additional administrative users for routine tasks, after the environment is created, and to refrain from sharing or utilizing the dbc credentials. |
+
+Set the password for dbc:
+* Between 8 and 64 characters
+* Allows both alphanumeric and special characters
+* No dictionary words
+
+
+
+
+### Advanced options
+
+To quickly get started, you can select **Use Defaults** or define the additional option settings.
+
+
+
+
+
+```mdx-code-block
+| *Item* |Description* |
+|-------------|---------------------------------------------------------------------------------------------------|
+| AMPs per instance | Workload management + Select the number of AMPs per instance for the instance size you selected. |
+| AWS: Storage encryption | Configure encryption for customer data. See [Finding the key ID and key ARN](https://docs.aws.amazon.com/kms/latest/developerguide/find-cmk-id-arn.html)
+
+ ```bash
+ docker run -e "accept_license=Y" -p 127.0.0.1:8888:8888 -v ${PWD}:/home/jovyan/JupyterLabRoot teradata/jupyterlab-extensions
+ ```
+
+
+ ```bash
+ docker run -e "accept_license=Y" -p 127.0.0.1:8888:8888 -v $PWD:/home/jovyan/JupyterLabRoot teradata/jupyterlab-extensions
+ ```
+
+
+ ```bash
+ docker run -e "accept_license=Y" -p 127.0.0.1:8888:8888 -v $PWD:/home/jovyan/JupyterLabRoot teradata/jupyterlab-extensions
+ ```
+
+
+```
+
+Take note of the resulting URL and token; you’ll need them to establish the connection from Visual Studio Code.
+
+
+
+## Visual Studio Code Configuration
+Open `lake-demos` project directory in Visual Studio Code. The repository contains the following project tree:
+
+LAKE_DEMOS
+
+* [UseCases](https://github.com/Teradata/lake-demos/tree/main/UseCases)
+ * [0_Demo_Environment_Setup.ipynb](https://github.com/Teradata/lake-demos/blob/main/0_Demo_Environment_Setup.ipynb)
+ * [1_Load_Base_Demo_Data.ipynb](https://github.com/Teradata/lake-demos/blob/main/1_Load_Base_Demo_Data.ipynb)
+ * [Data_Engineering_Exploration.ipynb](https://github.com/Teradata/lake-demos/blob/main/Data_Engineering_Exploration.ipynb)
+ * [Data_Science_OAF.ipynb](https://github.com/Teradata/lake-demos/blob/main/Data_Science_OAF.ipynb)
+ * [Demo_Admin.ipynb](https://github.com/Teradata/lake-demos/blob/main/Demo_Admin.ipynb)
+* [vars.json file](https://github.com/Teradata/lake-demos/blob/main/vars.json)
+
+### Edit vars.json file
+Edit the *[vars.json](https://github.com/Teradata/lake-demos/blob/main/vars.json)* file to include the required credentials to run the demos +
+
+
+
+| **Variable** | **Value** |
+|--------------|-----------|
+| *"host"* | Public IP value from your VantageCloud Lake environment |
+| *"UES_URI"* | Open Analytics from your VantageCloud Lake environment |
+| *"dbc"* | The master password of your VantageCloud Lake environment. |
+
+
+To retrieve a Public IP address and Open Analytics Endpoint follow these [instructions](https://quickstarts.teradata.com/vantagecloud-lake/vantagecloud-lake-demo-jupyter-docker.html).
+
+:::info
+Change passwords in the vars.json file. You'll see that in the sample vars.json, the passwords of all users are defaulted to "password", this is just for matters of the sample file, you should change all of these password fields to strong passwords, secure them as necessary and follow other password management best practices.
+:::
+
+### Modify path to vars.json in UseCases directory
+
+In the UseCases directory, all .ipynb files use the path ../../vars.json to load the variables from the JSON file when working from Jupyterlab. To work directly from Visual Studio Code, update the code in each .ipynb to point to vars.json.
+
+The quickest way to make these changes is via search feature on the left vertical menu. Search for
+
+```
+'../../vars.json'
+```
+
+and replace with:
+
+```
+'vars.json'
+```
+
+
+
+
+
+### Configuring Jupyter Kernels
+Open *0_Demo_Environment_Setup.ipynb* and click on Select Kernel at the top right corner of Visual Studio Code.
+
+If you have not installed Jupyter and Python extensions, Visual Studio Code will prompt you to install them. These extensions are necessary for Visual Studio Code to detect Kernels. To install them, select 'Install/Enable suggested extensions for Python and Jupyter.'
+
+
+
+Once you've installed the necessary extensions, you'll find options in the drop-down menu. Choose **Existing Jupyter Kernel**.
+
+
+
+Enter the URL of the running Jupyter Server and press enter.
+```
+http://localhost:8888
+```
+
+
+Enter the token found in your terminal when mounting files to the Docker container and press Enter.
+
+
+
+Change Server Display Name (Leave Blank To Use URL)
+
+
+
+You now have access to all the Teradata Vantage extension kernels. Select Python 3 (ipykernel) from the running Jupyter server.
+
+
+
+### Run demos
+Execute all the cells in *0_Demo_Environment_Setup.ipynb* to setup your environment. Followed by *1_Demo_Setup_Base_Data.ipynb* to load the base data required for demo.
+To learn more about the demo notebooks, go to [Teradata Lake demos](https://github.com/Teradata/lake-demos) page on GitHub.
+
+
+
+## Summary
+In this quickstart guide, we configured Visual Studio Code to access VantageCloud Lake demos using Jupyter notebooks.
diff --git a/sidebars.js b/sidebars.js
index 3327580322..1d54a4511c 100644
--- a/sidebars.js
+++ b/sidebars.js
@@ -30,4 +30,4 @@ const sidebars = {
*/
};
-export default sidebars;
+export default sidebars;
\ No newline at end of file
diff --git a/src/components/QuickstartsPageCategories/index.js b/src/components/QuickstartsPageCategories/index.js
new file mode 100644
index 0000000000..f659e9d131
--- /dev/null
+++ b/src/components/QuickstartsPageCategories/index.js
@@ -0,0 +1,75 @@
+import clsx from 'clsx';
+import Heading from '@theme/Heading';
+import styles from './styles.module.css';
+import Link from '@docusaurus/Link';
+
+const FeatureTitle = 'Categories';
+const FeatureList = [
+ {
+ title: `Introduction`,
+ description: 'Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.',
+ href: '/quickstarts/introduction/teradata-vantage-engine-architecture-and-concepts/',
+ },
+ {
+ title: 'Get access to Vantage',
+ description: 'Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.',
+ href: '/quickstarts/get-access-to-vantage/on-your-local/getting-started-vmware/',
+ },
+ {
+ title: 'Connect to Vantage',
+ description: 'Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.',
+ href: '/quickstarts/connect-to-vantage/install-teradata-studio-on-mac-m1-m2/',
+ },
+ {
+ title: 'Manage data',
+ description: 'Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.',
+ href: '/quickstarts/manage-data/nos/',
+ },
+ {
+ title: 'Connect applications',
+ description: 'Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.',
+ href: '/quickstarts/create-applications/jdbc/',
+ },
+ {
+ title: 'Analyze data',
+ description: 'Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.',
+ href: '/quickstarts/analyze-data/jupyter/',
+ },
+ {
+ title: 'Teradata AI Unlimited',
+ description: 'Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.',
+ href: '/ai-unlimited-docs/',
+ },
+ {
+ title: 'VantageCloud Lake',
+ description: 'Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ',
+ href: '/quickstarts/vantagecloud-lake/getting-started-with-vantagecloud-lake/',
+ },
+];
+
+
+function Feature({ title, description, href }) {
+ return (
+
+
+
+ );
+}
diff --git a/src/components/QuickstartsPageCategories/styles.module.css b/src/components/QuickstartsPageCategories/styles.module.css
new file mode 100644
index 0000000000..c972d93c19
--- /dev/null
+++ b/src/components/QuickstartsPageCategories/styles.module.css
@@ -0,0 +1,96 @@
+.features {
+ display: flex;
+ align-items: center;
+ padding: 2rem 0;
+ background: var(--ifm-color-white);
+ gap: 1.5rem;
+}
+
+.features h2 {
+ color: hsl(205, 100%, 12%);
+ margin-bottom: 40px;
+
+ /* Desktop/H2 */
+ font-family: "Inter", sans-serif;
+ font-size: 42px;
+ font-style: normal;
+ font-weight: 300;
+ line-height: 52px; /* 123.81% */
+ letter-spacing: -1.26px;
+}
+
+a.col {
+ text-decoration: none;
+}
+
+.col {
+ padding: 0 0.75rem 1.5rem;
+}
+
+.card {
+ --ifm-spacing-horizontal: 0.25rem;
+
+ border-radius: 12px;
+ border: 1px solid #CED3DA;
+ background: #FFF;
+ margin-bottom: 1rem;
+ padding: 1.5rem;
+}
+
+.card h3 {
+ color: var(--Primary-Navy, #00233c);
+ /* Desktop/H3 */
+ font-family: Inter;
+ font-size: 1.5rem;
+ font-style: normal;
+ font-weight: 600;
+ line-height: 34px; /* 141.667% */
+ letter-spacing: -0.24px;
+ margin-bottom: 0;
+}
+
+.featureSvg {
+ height: 200px;
+ width: 200px;
+}
+
+.container {
+ padding: 0 1.5rem;
+}
+.row {
+ margin: 0 -0.625rem;
+}
+
+
+@media screen and (min-width: 768px) and (max-width: 1024px) {
+ .container {
+ padding: 0 2.5rem 0;
+ }
+
+ .col {
+ --ifm-col-width: 50%;
+ }
+}
+
+@media screen and (max-width: 1024px) {
+ .features h2 {
+ font-size: 26px;
+ line-height: 36px; /* 138.462% */
+ letter-spacing: -0.78px;
+ }
+}
+
+@media screen and (min-width: 1025px) {
+ .container {
+ padding: 0 6.625rem 0;
+ margin: 0 auto;
+ width: 100%;
+ max-width: 1440px;
+ }
+}
+
+@media (min-width: 1440px) {
+ .container {
+ max-width: 1440px;
+ }
+}
diff --git a/src/pages/quickstarts.js b/src/pages/quickstarts.js
new file mode 100644
index 0000000000..f0ac5baeaa
--- /dev/null
+++ b/src/pages/quickstarts.js
@@ -0,0 +1,50 @@
+import clsx from 'clsx';
+import Link from '@docusaurus/Link';
+import useDocusaurusContext from '@docusaurus/useDocusaurusContext';
+import Layout from '@theme/Layout';
+import QuickstartsPageCategories from '@site/src/components/QuickstartsPageCategories';
+import HeroImageUrl from '@site/static/img/quickstarts-hero.png';
+import Translate from '@docusaurus/Translate';
+import Heading from '@theme/Heading';
+import styles from './index.module.css';
+function HomepageHeader() {
+ const { siteConfig } = useDocusaurusContext();
+ return (
+
+
+ );
+}
+
+export default function Home() {
+ const { siteConfig } = useDocusaurusContext();
+ return (
+
+
+
+
+ );
+}
diff --git a/static/img/quickstarts-hero.png b/static/img/quickstarts-hero.png
new file mode 100644
index 0000000000..5ab82c423b
Binary files /dev/null and b/static/img/quickstarts-hero.png differ
| job_id | \n", + "\t\tPatientId | \n", + "\t\tHasDiabetes | \n", + "\t\tjson_report | \n", + "\t
|---|---|---|---|
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t360 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t45 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t490 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t660 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t30 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t220 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t355 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t560 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t460 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t325 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ "Empty DataFrame\n",
+ "Columns: [job_id, PatientId, HasDiabetes, json_report]\n",
+ "Index: []"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import pandas as pd\n",
+ "from teradataml import get_connection\n",
+ "\n",
+ "pd.options.display.max_colwidth = 250\n",
+ "\n",
+ "airflow_job_id = \"5761d5c1-bf57-456b-8076-c3062be0b544-scheduled__2022-07-11T00:00:00+00:00\"\n",
+ "\n",
+ "pd.read_sql(f\"SELECT TOP 5 * FROM pima_patient_predictions WHERE job_id='{airflow_job_id}'\", get_connection())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d479c9cb",
+ "metadata": {},
+ "source": [
+ "## CLI \n",
+ "\n",
+ "\n",
+ "```bash\n",
+ "pip install aoa==6.1.0\n",
+ "```\n",
+ "\n",
+ "##### Copy CLI Config\n",
+ "\n",
+ "```\n",
+ "Copy the CLI config from ModelOps UI -> Session Details -> CLI config\n",
+ "```\n",
+ "\n",
+ "##### Add Dataset Connection\n",
+ "\n",
+ "```bash\n",
+ "aoa connection add\n",
+ "```\n",
+ "\n",
+ "##### List Feature Metadata\n",
+ "\n",
+ "```bash\n",
+ "aoa feature list-stats -m {your-db}.aoa_feature_metadata\n",
+ "```\n",
+ "\n",
+ "##### Clone Project\n",
+ "\n",
+ "```bash\n",
+ "aoa clone \n",
+ "```\n",
+ "\n",
+ "```bash\n",
+ "cd modelops-demo-models\n",
+ "```\n",
+ "\n",
+ "##### Install Model Dependencies\n",
+ "\n",
+ "```bash\n",
+ "pip install -r model_definitions/python-diabetes/model_modules/requirements.txt\n",
+ "```\n",
+ "\n",
+ "##### Train Model\n",
+ "\n",
+ "```bash\n",
+ "aoa run\n",
+ "```\n",
+ "\n",
+ "##### Add Model\n",
+ "\n",
+ "```bash\n",
+ "aoa add\n",
+ "```\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b63bd4d5",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.10"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/quickstarts/modelops/attachments/ModelOps_Training_v7.ipynb b/quickstarts/modelops/attachments/ModelOps_Training_v7.ipynb
new file mode 100644
index 0000000000..dc93b2d16e
--- /dev/null
+++ b/quickstarts/modelops/attachments/ModelOps_Training_v7.ipynb
@@ -0,0 +1,410 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "dcc29d47",
+ "metadata": {},
+ "source": [
+ "## Setup\n",
+ "\n",
+ "The remainder of the notebook runs through the following steps\n",
+ "\n",
+ "- Connect to Vantage\n",
+ "- Create DDLs\n",
+ "- Import Data\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "426c443a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%pip install teradataml==17.20.0.3 aoa==7.0.1 pandas==1.1.5"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8a780585",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from teradataml import create_context\n",
+ "import getpass\n",
+ "import logging\n",
+ "import sys\n",
+ "import urllib\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "\n",
+ "\n",
+ "host = input(\"Host:\")\n",
+ "username = input(\"Username:\")\n",
+ "password = getpass.getpass(\"Password:\")\n",
+ "database = input(\"Database (defaults to user):\")\n",
+ "\n",
+ "if not database:\n",
+ " database = username\n",
+ "\n",
+ "\n",
+ "engine = create_context(host=host, \n",
+ " username=username, \n",
+ " password=urllib.parse.quote(password), \n",
+ " logmech=\"TDNEGO\",\n",
+ " database=database)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88d3dff4",
+ "metadata": {},
+ "source": [
+ "### Create DDLs\n",
+ "\n",
+ "Create the following tables \n",
+ "\n",
+ "- aoa_statistics_metadata \n",
+ "- aoa_byom_models\n",
+ "- pima_patient_predictions\n",
+ "\n",
+ "`aoa_statistics_metadata` is used to store the profiling metadata for the features so that we can consistently compute the data drift and model drift statistics. This table can also be created via the CLI by executing \n",
+ "\n",
+ "```bash\n",
+ "aoa feature create-stats-table -e -m | \n", + " | job_id | \n", + "PatientId | \n", + "HasDiabetes | \n", + "json_report | \n", + "
|---|
| Item | +Description | +||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Instance size | +
+ Select an instance size suitable for your use-case: + Lake +
|
+ ||||||||||||||||||||||||||||||||
| Instance count | +2 to 64 Number of nodes in the primary clusters |
+ ||||||||||||||||||||||||||||||||
| Instance storage | +1 to 72TB per instance | +
- Managed by Teradata
- Customer managed
- Key Alias ARN
+ {title}
+ {description &&
+
+ );
+}
+
+export default function HomepageFeatures() {
+ return (
+ {description}
} +
+
+ {FeatureTitle}
+
+ {FeatureList.map((props, idx) => (
+
+
+
+
+
+ Getting started
+
+
+
+
+ Existing customer or partner? Explore courses at Teradata University. +
+
+ 
+
+
+