diff --git a/quickstarts/create-applications/send-queries-using-rest-api.md b/quickstarts/create-applications/send-queries-using-rest-api.md index 3dc7395b30..142afbd32f 100644 --- a/quickstarts/create-applications/send-queries-using-rest-api.md +++ b/quickstarts/create-applications/send-queries-using-rest-api.md @@ -7,6 +7,8 @@ description: Send queries using REST API. Teradata® Query Service is a middlewa keywords: [query service, teradata, vantage, query, REST API] --- +import ClearscapeDocsNote from '../_partials/vantage_clearscape_analytics.mdx' + # Send queries using REST API ## Overview @@ -18,11 +20,10 @@ This how-to provides examples of common use cases to help you get started with Q ## Prerequisites Before starting, make sure you have: - -import ClearscapeDocsNote from '../_partials/vantage_clearscape_analytics.mdx' - * Access to a VantageCloud system where Query Service is provisioned, or a VantageCore with Query Service enabled connectivity. If you are an admin and need to install Query Service, see [Query Service Installation, Configuration, and Usage Guide](https://docs.teradata.com/r/Teradata-Query-Service-Installation-Configuration-and-Usage-Guide-for-Customers/April-2022). + + * Query Service hostname and system name * Authorization credentials to connect to the database @@ -52,7 +53,7 @@ Provide valid credentials to access the target Analytics Database using HTTP Bas The database username and password are combined into a string (`"username : password"`) which is then encoded using Base64. The API response contains the authorization method and encoded credentials. -Request +**Request** ``` python , id="queryservice_first_query", role="emits-gtm-events" import requests @@ -77,9 +78,8 @@ headers = { print(headers) ``` -Response - -``` +**Response** +```python Basic ZGJjOmRiYw== { 'Content-Type': 'application/json', @@ -92,10 +92,9 @@ Basic ZGJjOmRiYw== Prerequisites: * The user must already exist in the database. - * The database must be JWT enabled. -Request +**Request** ``` python import requests @@ -115,9 +114,9 @@ headers = { print(headers) ``` -Response +**Response** -``` +```python {'Content-Type': 'application/json', 'Authorization': 'Bearer '} ``` @@ -128,14 +127,14 @@ In the following example, the request includes: * `SELECT * FROM DBC.DBCInfo`: The query to the system with the alias ``. * `'format': 'OBJECT'`: The format for response. The formats supported are: JSON object, JSON array, and CSV. -:::note -The JSON object format creates one JSON object per row where the column name is the field name, and the column value is the field value. -::: + :::note + The JSON object format creates one JSON object per row where the column name is the field name, and the column value is the field value. + ::: * `'includeColumns': true`: The request to include column metadata, such as column names and types, in the response. * `'rowLimit': 4`: The number of rows to be returned from a query. -Request +**Request** ``` python url = 'https://:1443/systems//queries' @@ -158,7 +157,7 @@ print('==========================================================') print(response.json()) ``` -Response +**Response** ``` json NUMBER of ROWS 4 @@ -244,7 +243,7 @@ To return an API response in CSV format, set the `*format*` field in the request The CSV format contains only the query results and not response metadata. The response contains a line for each row, where each line contains the row columns separated by a comma. The following example returns the data as comma-separated values. -Request +**Request** ``` python # CSV with all rows included @@ -264,9 +263,9 @@ response = requests.request('POST', url, headers=headers, data=payload_json, ver print(response.text) ``` -Response +**Response** -``` +```python DatabaseName,USEDSPACE_IN_GB,MAXSPACE_IN_GB,Percentage_Used,REMAININGSPACE_IN_GB DBC ,317.7634754180908,1510.521079641879,21.036679308932754,1192.7576042237881 EM ,7.491111755371094E-4,11.546071618795395,0.006488017745513208,11.545322507619858 @@ -298,104 +297,97 @@ Y5WYUUXj ,0.0,0.009313225746154785,0.0,0.009313225746154785 Use explicit sessions when a transaction needs to span multiple requests or when using volatile tables. These sessions are only reused if you reference the sessions in a query request. The request is queued if a request references an explicit session already in use. 1. Create a session + Send a POST request to the `/system//sessions` endpoint. The request creates a new database session and returns the session details as the response. -Send a POST request to the `/system//sessions` endpoint. The request creates a new database session and returns the session details as the response. - -In the following example, the request includes `'auto_commit': True` - the request to commit the query upon completion. - -Request + In the following example, the request includes `'auto_commit': True` - the request to commit the query upon completion. -``` python -# first create a session -url = 'https://:1443/systems//sessions' + **Request** + ``` python + # first create a session + url = 'https://:1443/systems//sessions' -payload = { - 'auto_commit': True -} + payload = { + 'auto_commit': True + } -payload_json = json.dumps(payload) + payload_json = json.dumps(payload) -response = requests.request('POST', url, headers=headers, data=payload_json, verify=False) + response = requests.request('POST', url, headers=headers, data=payload_json, verify=False) -print(response.text) -``` + print(response.text) + ``` -Response - -``` -{ - 'sessionId': 1366010, - 'system': 'testsystem', - 'user': 'dbc', - 'tdSessionNo': 1626922, - 'createMode': 'EXPLICIT', - 'state': 'LOGGINGON', - 'autoCommit': true -} -``` + **Response** + ```python + { + 'sessionId': 1366010, + 'system': 'testsystem', + 'user': 'dbc', + 'tdSessionNo': 1626922, + 'createMode': 'EXPLICIT', + 'state': 'LOGGINGON', + 'autoCommit': true + } + ``` 2. Use the session created in Step 1 to submit queries + + Send a POST request to the `/system//queries` endpoint. -Send a POST request to the `/system//queries` endpoint. + The request submits queries to the target system and returns the release and version number of the target system. -The request submits queries to the target system and returns the release and version number of the target system. - -In the following example, the request includes: - -* `SELECT * FROM DBC.DBCInfo`: The query to the system with the alias ``. -* `'format': 'OBJECT'`: The format for response. -* `'Session' : `: The session ID returned in Step 1 to create an explicit session. + In the following example, the request includes: + * `SELECT * FROM DBC.DBCInfo`: The query to the system with the alias ``. + * `'format': 'OBJECT'`: The format for response. + * `'Session' : `: The session ID returned in Step 1 to create an explicit session. + **Request** + ``` python + # use this session to submit queries afterwards -Request - -``` python -# use this session to submit queries afterwards - -url = 'https://:1443/systems//queries' + url = 'https://:1443/systems//queries' -payload = { - 'query': 'SELECT * FROM DBC.DBCInfo;', - 'format': 'OBJECT', - 'session': 1366010 # <-- sessionId -} -payload_json = json.dumps(payload) - -response = requests.request('POST', url, headers=headers, data=payload_json, verify=False) + payload = { + 'query': 'SELECT * FROM DBC.DBCInfo;', + 'format': 'OBJECT', + 'session': 1366010 # <-- sessionId + } + payload_json = json.dumps(payload) -print(response.text) -``` + response = requests.request('POST', url, headers=headers, data=payload_json, verify=False) -Response + print(response.text) + ``` -``` json -{ - "queueDuration":6, - "queryDuration":41, - "results":[ + **Response** + ``` json { - "resultSet":true, - "data":[ - { - "InfoKey":"LANGUAGE SUPPORT MODE", - "InfoData":"Standard" - }, + "queueDuration":6, + "queryDuration":41, + "results":[ { - "InfoKey":"RELEASE", - "InfoData":"15.10.07.02" - }, - { - "InfoKey":"VERSION", - "InfoData":"15.10.07.02" + "resultSet":true, + "data":[ + { + "InfoKey":"LANGUAGE SUPPORT MODE", + "InfoData":"Standard" + }, + { + "InfoKey":"RELEASE", + "InfoData":"15.10.07.02" + }, + { + "InfoKey":"VERSION", + "InfoData":"15.10.07.02" + } + ], + "rowCount":3, + "rowLimitExceeded":false } - ], - "rowCount":3, - "rowLimitExceeded":false + ] } - ] -} -``` + ``` ## Use asynchronous queries @@ -403,143 +395,134 @@ Response Use asynchronous queries when a system or network performance is affected by querying a large group of data or long running queries. 1. Submit asynchronous queries to the target system and retrieve a Query ID + Send a POST request to the `/system//queries` endpoint. + In the following example, the request includes: + * `SELECT * FROM DBC.DBCInfo`: The query to the system with the alias ``. + * `'format': 'OBJECT'`: The format for response. + * `'spooled_result_set': True`: The indication that the request is asynchronous. + + **Request** + ``` python + ## Run async query . + + url = 'https://:1443/systems//queries' + + payload = { + 'query': 'SELECT * FROM DBC.DBCInfo;', + 'format': 'OBJECT', + 'spooled_result_set': True + } -Send a POST request to the `/system//queries` endpoint. - -In the following example, the request includes: - -* `SELECT * FROM DBC.DBCInfo`: The query to the system with the alias ``. -* `'format': 'OBJECT'`: The format for response. -* `'spooled_result_set': True`: The indication that the request is asynchronous. - - - -Request - -``` python -## Run async query . - -url = 'https://:1443/systems//queries' - -payload = { - 'query': 'SELECT * FROM DBC.DBCInfo;', - 'format': 'OBJECT', - 'spooled_result_set': True -} - -payload_json = json.dumps(payload) -response = requests.request('POST', url, headers=headers, data=payload_json, verify=False) - -print(response.text) -``` + payload_json = json.dumps(payload) + response = requests.request('POST', url, headers=headers, data=payload_json, verify=False) -Response - -``` -{"id":1366025} -``` + print(response.text) + ``` + **Response** + ``` + {"id":1366025} + ``` 2. Get query details using the ID retrieved from Step 1 -+ -Send a GET request to the `/system//queries/` endpoint, replacing `` with the ID retrieved from Step 1. -+ -The request returns the details of the specific query, including `*queryState*`, `*queueOrder*`, `*queueDuration*`, and so on. For a complete list of the response fields and their descriptions, see [Query Service Installation, Configuration, and Usage Guide](https://docs.teradata.com/r/Teradata-Query-Service-Installation-Configuration-and-Usage-Guide-for-Customers/April-2022/Using-the-Query-Service-APIs/Submitting-SQL-Statement/Request-Body). -Request + Send a GET request to the `/system//queries/` endpoint, replacing `` with the ID retrieved from Step 1. -``` python -## response for async query . + The request returns the details of the specific query, including **`queryState`**, **`queueOrder`**, **`queueDuration`**, and so on. For a complete list of the response fields and their descriptions, see [Query Service Installation, Configuration, and Usage Guide](https://docs.teradata.com/r/Teradata-Query-Service-Installation-Configuration-and-Usage-Guide-for-Customers/April-2022/Using-the-Query-Service-APIs/Submitting-SQL-Statement/Request-Body). -url = 'https://:1443/systems//queries/1366025' + **Request** + ``` python + ## response for async query . -payload_json = json.dumps(payload) -response = requests.request('GET', url, headers=headers, verify=False) + url = 'https://:1443/systems//queries/1366025' -print(response.text) -``` + payload_json = json.dumps(payload) + response = requests.request('GET', url, headers=headers, verify=False) -Response + print(response.text) + ``` -``` -{ - "queryId":1366025, - "query":"SELECT * FROM DBC.DBCInfo;", - "batch":false, - "system":"testsystem", - "user":"dbc", - "session":1366015, - "queryState":"RESULT_SET_READY", - "queueOrder":0, - "queueDuration":6, - "queryDuration":9, - "statusCode":200, - "resultSets":{ - - }, - "counts":{ - - }, - "exceptions":{ - - }, - "outParams":{ - - } -} -``` + **Response** + ```python + { + "queryId":1366025, + "query":"SELECT * FROM DBC.DBCInfo;", + "batch":false, + "system":"testsystem", + "user":"dbc", + "session":1366015, + "queryState":"RESULT_SET_READY", + "queueOrder":0, + "queueDuration":6, + "queryDuration":9, + "statusCode":200, + "resultSets":{ + + }, + "counts":{ + + }, + "exceptions":{ + + }, + "outParams":{ + + } + } + ``` 3. View resultset for asynchronous query -Send a GET request to the `/system//queries//results` endpoint, replacing `` with the ID retrieved from Step 1. -The request returns an array of the result sets and update counts produced by the submitted query. + Send a GET request to the `/system//queries//results` endpoint, replacing `` with the ID retrieved from Step 1. -Request + The request returns an array of the result sets and update counts produced by the submitted query. -``` python -url = 'https://:1443/systems//queries/1366025/results' + **Request** -payload_json = json.dumps(payload) -response = requests.request('GET', url, headers=headers, verify=False) + ``` python + url = 'https://:1443/systems//queries/1366025/results' -print(response.text) -``` + payload_json = json.dumps(payload) + response = requests.request('GET', url, headers=headers, verify=False) -Response + print(response.text) + ``` -``` json -{ - "queueDuration":6, - "queryDuration":9, - "results":[ + **Response** + + ``` json { - "resultSet":true, - "data":[ - { - "InfoKey":"LANGUAGE SUPPORT MODE", - "InfoData":"Standard" - }, - { - "InfoKey":"RELEASE", - "InfoData":"15.10.07.02" - }, + "queueDuration":6, + "queryDuration":9, + "results":[ { - "InfoKey":"VERSION", - "InfoData":"15.10.07.02" + "resultSet":true, + "data":[ + { + "InfoKey":"LANGUAGE SUPPORT MODE", + "InfoData":"Standard" + }, + { + "InfoKey":"RELEASE", + "InfoData":"15.10.07.02" + }, + { + "InfoKey":"VERSION", + "InfoData":"15.10.07.02" + } + ], + "rowCount":3, + "rowLimitExceeded":false } - ], - "rowCount":3, - "rowLimitExceeded":false + ] } - ] -} -``` + ``` ## Get a list of active or queued queries Send a GET request to the `/system//queries` endpoint. The request returns the IDs of active queries. -Request +**Request** ``` python url = 'https://:1443/systems//queries' @@ -551,7 +534,7 @@ response = requests.request('GET', url, headers=headers, data=payload, verify=Fa print(response.json()) ``` -Response +**Response** ``` json [ diff --git a/quickstarts/manage-data/run-bulkloads-efficiently-with-teradata-parallel-transporter.md b/quickstarts/manage-data/run-bulkloads-efficiently-with-teradata-parallel-transporter.md index 45888fa99d..5732c0be24 100644 --- a/quickstarts/manage-data/run-bulkloads-efficiently-with-teradata-parallel-transporter.md +++ b/quickstarts/manage-data/run-bulkloads-efficiently-with-teradata-parallel-transporter.md @@ -3,6 +3,7 @@ sidebar_position: 11 author: Adam Tworkiewicz email: adam.tworkiewicz@teradata.com page_last_update: April 6th, 2022 +title: Load data with TPT description: Load data into Vantage efficiently using Teradata Parallel Transporter (TPT). keywords: [data warehouses, compute storage separation, teradata, vantage, cloud data platform, object storage, business intelligence, enterprise analytics, Fastload, Teradata Parallel Transporter, TPT] id: run-bulkloads-efficiently-with-teradata-parallel-transporter diff --git a/quickstarts/manage-data/transforming-external-data-loaded-via-airbyte-in-teradata-vantage-using-dbt.md b/quickstarts/manage-data/transforming-external-data-loaded-via-airbyte-in-teradata-vantage-using-dbt.md index f610a99812..9be2bce23f 100644 --- a/quickstarts/manage-data/transforming-external-data-loaded-via-airbyte-in-teradata-vantage-using-dbt.md +++ b/quickstarts/manage-data/transforming-external-data-loaded-via-airbyte-in-teradata-vantage-using-dbt.md @@ -9,7 +9,7 @@ keywords: [dbt, airbyte, data transformation, data warehouses, compute storage s # Transform data Loaded with Airbyte using dbt -### Overview +## Overview This tutorial demonstrates how to use [dbt (Data Build Tool)](https://docs.getdbt.com/docs/introduction) to transform external data load through [Airbyte](https://github.com/airbytehq/airbyte) (an Open-Source Extract Load tool) in Teradata Vantage. @@ -17,7 +17,7 @@ This tutorial is based on the original [dbt Jaffle Shop tutorial](https://github ![Raw data in Teradata Vantage](../elt/images/getting-started-with-airbyte-dbt/raw_data_vantage_dbeaver.png) -### Prerequisites +## Prerequisites import ClearscapeDocsNote from '../_partials/vantage_clearscape_analytics.mdx' @@ -27,25 +27,25 @@ import ClearscapeDocsNote from '../_partials/vantage_clearscape_analytics.mdx' * Reference dbt project repository: [Jaffle Project with Airbyte.](https://github.com/Teradata/airbyte-dbt-jaffle) * Python 3.7, 3.8, 3.9, 3.10 or 3.11 installed. -### Sample Data Loading -* Follow the steps in the [Airbyte tutorial](https://quickstarts.teradata.com/elt/use-airbyte-to-load-data-from-external-sources-to-teradata-vantage.html). Make sure you load data from the [Jaffle Shop spreadsheet](https://docs.google.com/spreadsheets/d/1-R4F3q8J9KDnFRWpiT3Ysp1RlOoUu3PeQR7xDeLxFts/edit#gid=42273685) and not the default dataset referenced by the Airbyte tutorial. Also, set the `Default Schema` in the Teradata destination to `airbyte_jaffle_shop`. +## Sample Data Loading +* Follow the steps in the [Airbyte tutorial](./use-airbyte-to-load-data-from-external-sources-to-teradata-vantage.md). Make sure you load data from the [Jaffle Shop spreadsheet](https://docs.google.com/spreadsheets/d/1-R4F3q8J9KDnFRWpiT3Ysp1RlOoUu3PeQR7xDeLxFts/edit#gid=42273685) and not the default dataset referenced by the Airbyte tutorial. Also, set the `Default Schema` in the Teradata destination to `airbyte_jaffle_shop`. :::note When you configure a Teradata destination in Airbyte, it will ask for a `Default Schema`. Set the `Default Schema` to `airbyte_jaffle_shop`. ::: -### Clone the project +## Clone the project Clone the tutorial repository and change the directory to the project directory: -``` bash +```bash git clone https://github.com/Teradata/airbyte-dbt-jaffle cd airbyte-dbt-jaffle ``` -### Install dbt +## Install dbt * Create a new python environment to manage dbt and its dependencies. Activate the environment: -``` bash +```bash python3 -m venv env source env/bin/activate ``` @@ -57,14 +57,14 @@ You can activate the virtual environment in Windows by executing the correspondi * Install `dbt-teradata` module and its dependencies. The core dbt module is included as a dependency so you don't have to install it separately: -``` bash +```bash pip install dbt-teradata ``` -### Configure dbt +## Configure dbt * Initialize a dbt project. -``` bash +```bash dbt init ``` @@ -100,7 +100,7 @@ dbt debug If the debug command returned errors, you likely have an issue with the content of `profiles.yml`. If the setup is correct, you will get message `All checks passed!` ![dbt debug output](../elt/images/getting-started-with-airbyte-dbt/dbt_debug.png) -### The Jaffle Shop dbt project +## The Jaffle Shop dbt project `jaffle_shop` is a fictional restaurant that takes orders online. The data of this business consists of tables for `customers`, `orders` and `payments`that follow the entity relations diagram below: @@ -145,7 +145,7 @@ As the method of extracting JSON strings remains consistent across all staging m ::: Below an example of transforming raw JSON data into a view through the `stg_orders.sql` model : -``` sql +```sql WITH source AS ( SELECT * FROM {{ source('airbyte_jaffle_shop', '_airbyte_raw_orders')}} ), @@ -177,7 +177,7 @@ Building a Dimensional Model is a two-step process: ### Executing transformations For executing the transformations defined in the dbt project we run: -``` bash +```bash dbt run ``` @@ -224,12 +224,12 @@ dbt docs serve ![dbt lineage graph](../elt/images/getting-started-with-airbyte-dbt/dbt_docs_serve.png) -### Summary +## Summary This tutorial demonstrated how to use dbt to transform raw JSON data loaded through Airbyte into dimensional model in Teradata Vantage. The sample project takes raw JSON data loaded in Teradata Vantage, creates normalized views and finally produces a dimensional data mart. We used dbt to transform JSON into Normalized views and multiple dbt commands to create models (`dbt run`), test the data (`dbt test`), and generate and serve model documentation (`dbt docs generate`, `dbt docs serve`). -### Further reading +## Further reading * [dbt documentation](https://docs.getdbt.com/docs) * [dbt-teradata plugin documentation](https://github.com/Teradata/dbt-teradata)