Feature/cdm line item

.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md

@@ -4,48 +4,26 @@
 **This PR will result in the following new package version:**
 <!--- Please add details around your decision for breaking vs non-breaking version upgrade. If this is a breaking change, were backwards-compatible options explored? -->
 
-**Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:**
+**Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:**
+<!--- Copy/paste the CHANGELOG for this version below. -->
 
 ## PR Checklist
 ### Basic Validation
 Please acknowledge that you have successfully performed the following commands locally:
-- [ ] dbt compile
-- [ ] dbt run –full-refresh
-- [ ] dbt run
-- [ ] dbt test
-- [ ] dbt run –vars (if applicable)
+- [ ] dbt run –full-refresh && dbt test
+- [ ] dbt run (if incremental models are present) && dbt test
 
 Before marking this PR as "ready for review" the following have been applied:
-- [ ] The appropriate issue has been linked and tagged
-- [ ] You are assigned to the corresponding issue and this PR
+- [ ] The appropriate issue has been linked, tagged, and properly assigned
+- [ ] All necessary documentation and version upgrades have been applied
+- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
 - [ ] BuildKite integration tests are passing
+- [ ] Detailed validation steps have been provided below
 
 ### Detailed Validation
-Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":
-- [ ] You have validated these changes and assure this PR will address the respective Issue/Feature.
-- [ ] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
-- [ ] You have provided details below around the validation steps performed to gain confidence in these changes.
+Please share any and all of your validation steps:
 <!--- Provide the steps you took to validate your changes below. -->
 
-### Standard Updates
-Please acknowledge that your PR contains the following standard updates:
-- Package versioning has been appropriately indexed in the following locations:
-    - [ ] indexed within dbt_project.yml
-    - [ ] indexed within integration_tests/dbt_project.yml
-- [ ] CHANGELOG has individual entries for each respective change in this PR
-    <!--- If there is a parallel upstream change, remember to reference the corresponding CHANGELOG as an individual entry.  -->
-- [ ] README updates have been applied (if applicable)
-    <!--- Remember to check the following README locations for common updates. →
-        <!--- Suggested install range (needed for breaking changes) →
-        <!--- Dependency matrix is appropriately updated (if applicable) →
-        <!--- New variable documentation (if applicable) -->
-- [ ] DECISIONLOG updates have been updated (if applicable)
-- [ ] Appropriate yml documentation has been added (if applicable)
-
-### dbt Docs
-Please acknowledge that after the above were all completed the below were applied to your branch:
-- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
-
 ### If you had to summarize this PR in an emoji, which would it be?
 <!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
-:dancer:
+:dancer:

.github/workflows/auto-release.yml

@@ -0,0 +1,13 @@
+name: 'auto release'
+on:
+  pull_request:
+    types:
+      - closed
+    branches:
+      - main
+
+jobs:
+  call-workflow-passing-data:
+    if: github.event.pull_request.merged
+    uses: fivetran/dbt_package_automations/.github/workflows/auto-release.yml@main
+    secrets: inherit

CHANGELOG.md

@@ -1,3 +1,13 @@
+# dbt_zuora v0.3.0
+[PR #12](https://github.com/fivetran/dbt_zuora/pull/12) includes the following breaking changes:
+
+## Feature Updates
+- Introduced the new `zuora__line_item_enhanced` model. This model includes a line item enriched with invoice, subscription, payment, and refund information. This model has been built with the intention of retaining a common line item schema across all other Fivetran billing data models.
+
+## Under the Hood:
+- Updated the pull request templates.
+- Included auto-releaser GitHub Actions workflow to automate future releases.
+
 # dbt_zuora v0.2.1
 [PR #10](https://github.com/fivetran/dbt_zuora/pull/10) includes the following breaking changes:
 ## 🔧 Fixes

README.md

@@ -60,7 +60,7 @@ Include the following zuora_source package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/zuora
-    version: [">=0.2.0", "<0.3.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.3.0", "<0.4.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 Do NOT include the `zuora_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
 

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'zuora'
-version: '0.2.1'
+version: '0.3.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 

integration_tests/.gitignore

@@ -1,4 +1,5 @@
 target/
 dbt_modules/
 logs/
-env/
+env/
+package-lock.yml

integration_tests/ci/sample.profiles.yml

@@ -49,6 +49,6 @@ integration_tests:
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
       schema: zuora_integration_tests_1
-      threads: 2
+      threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,8 +1,12 @@
 name: 'zuora_integration_tests'
-version: '0.2.1'
+version: '0.3.0'
 profile: 'integration_tests'
 config-version: 2
 
+# For use with validations
+models:
+  +schema: "zuora_{{ var('directed_schema','dev') }}"
+
 vars:
   zuora_schema: zuora_integration_tests_1
   zuora_source:
@@ -114,6 +118,7 @@ seeds:
     zuora_refund_data:
       +column_types: 
         refund_transaction_time: "timestamp"
+        refund_date: "timestamp"
     zuora_subscription_data:
       +column_types: 
         contract_acceptance_date: "timestamp"

integration_tests/packages.yml

@@ -1,2 +1,2 @@
 packages:
-  - local: ../
+  - local: ../

integration_tests/tests/consistency/consistency_line_item_enhanced.sql

@@ -0,0 +1,31 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select *
+    from {{ target.schema }}_zuora_prod.zuora__line_item_enhanced
+),
+
+dev as (
+    select *
+    from {{ target.schema }}_zuora_dev.zuora__line_item_enhanced
+), 
+
+final as (
+    -- test will fail if any rows from prod are not found in dev
+    (select * from prod
+    except distinct
+    select * from dev)
+
+    union all -- union since we only care if rows are produced
+
+    -- test will fail if any rows from dev are not found in prod
+    (select * from dev
+    except distinct
+    select * from prod)
+    )
+
+select *
+from final

integration_tests/tests/consistency/consistency_line_item_enhanced_count.sql

@@ -0,0 +1,21 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test is to make sure the rows counts are the same between versions
+with prod as (
+    select count(*) as prod_rows
+    from {{ target.schema }}_zuora_prod.zuora__line_item_enhanced
+),
+
+dev as (
+    select count(*) as dev_rows
+    from {{ target.schema }}_zuora_dev.zuora__line_item_enhanced
+)
+
+-- test will return values and fail if the row counts don't match
+select *
+from prod
+join dev
+    on prod.prod_rows != dev.dev_rows

models/common_data_models/docs.md

@@ -0,0 +1,135 @@
+{% docs billing_type %}
+description
+{% enddocs %}
+
+{% docs created_at %}
+description
+{% enddocs %}
+
+{% docs currency %}
+description
+{% enddocs %}
+
+{% docs customer_city %}
+description
+{% enddocs %}
+
+{% docs customer_company %}
+description
+{% enddocs %}
+
+{% docs customer_country %}
+description
+{% enddocs %}
+
+{% docs customer_email %}
+description
+{% enddocs %}
+
+{% docs customer_id %}
+description
+{% enddocs %}
+
+{% docs customer_level %}
+description
+{% enddocs %}
+
+{% docs customer_name %}
+description
+{% enddocs %}
+
+{% docs discount_amount %}
+description
+{% enddocs %}
+
+{% docs header_id %}
+description
+{% enddocs %}
+
+{% docs header_status %}
+description
+{% enddocs %}
+
+{% docs line_item_id %}
+description
+{% enddocs %}
+
+{% docs line_item_index %}
+description
+{% enddocs %}
+
+{% docs payment_at %}
+description
+{% enddocs %}
+
+{% docs payment_id %}
+description
+{% enddocs %}
+
+{% docs payment_method %}
+description
+{% enddocs %}
+
+{% docs payment_method_id %}
+description
+{% enddocs %}
+
+{% docs product_category %}
+description
+{% enddocs %}
+
+{% docs product_id %}
+description
+{% enddocs %}
+
+{% docs product_name %}
+description
+{% enddocs %}
+
+{% docs product_type %}
+description
+{% enddocs %}
+
+{% docs quantity %}
+description
+{% enddocs %}
+
+{% docs record_type %}
+description
+{% enddocs %}
+
+{% docs refund_amount %}
+description
+{% enddocs %}
+
+{% docs subscription_id %}
+description
+{% enddocs %}
+
+{% docs subscription_period_ended_at %}
+description
+{% enddocs %}
+
+{% docs subscription_period_started_at %}
+description
+{% enddocs %}
+
+{% docs subscription_status %}
+description
+{% enddocs %}
+
+{% docs tax_amount %}
+description
+{% enddocs %}
+
+{% docs total_amount %}
+description
+{% enddocs %}
+
+{% docs transaction_type %}
+description
+{% enddocs %}
+
+{% docs unit_amount %}
+description
+{% enddocs %}