Code

Quickfix/bumping tenacity version#76

Code

Open

siklosid wants to merge 159 commits intosiklosid:mastersiklosid/dagger:masterfrom chocoapp:quickfix/bumping_tenacity_versionchocoapp/dataeng-dagger:quickfix/bumping_tenacity_versionCopy head branch name to clipboard

+2,639-347

Commits

Commits on Sep 30, 2023

• Turning follow_external_dependency into bool or dict property

  siklosid

  committed
  ea36004

  Copy full SHA for ea36004

• Handling the new dict format of follow_external_dependency

  siklosid

  committed
  5113752

  Copy full SHA for 5113752

• Changing test case to see if it handles sensor parameters properly

  siklosid

  committed
  fc2a2e2

  Copy full SHA for fc2a2e2

Commits on Oct 9, 2023

• Merge pull request #23 from chocoapp/feature/sensor_parameters

  Show description for d0780dd
  siklosid

  authored
  d0780dd

  Copy full SHA for d0780dd

Commits on Oct 23, 2023

• upgrade version of tenacity

  

  claudiazi

  committed
  db6dee2

  Copy full SHA for db6dee2

Commits on Oct 26, 2023

• Merge pull request #24 from chocoapp/feature/DATA-1562-upgrade-tenacity

  Show description for bd5adcf
  claudiazi

  authored
  bd5adcf

  Copy full SHA for bd5adcf

Commits on Nov 15, 2023

• added new dbt config parser module

  kiranvasudev

  committed
  d463438

  Copy full SHA for d463438

• added class with constructor

  kiranvasudev

  committed
  a91a58f

  Copy full SHA for a91a58f

• added method to parse dbt model inputs

  kiranvasudev

  committed
  102620e

  Copy full SHA for 102620e

• added functions to generate dagger input and outputs for dbt models

  kiranvasudev

  committed
  c61e656

  Copy full SHA for c61e656

• added fn to generate io for dbt task

  kiranvasudev

  committed
  29858a9

  Copy full SHA for 29858a9

• black format

  kiranvasudev

  committed
  1a93f05

  Copy full SHA for 1a93f05

• use new module in fn to generate configs

  kiranvasudev

  committed
  847f622

  Copy full SHA for 847f622

• changed output bucket

  kiranvasudev

  committed
  104013a

  Copy full SHA for 104013a

• fixed import

  kiranvasudev

  committed
  de82c14

  Copy full SHA for de82c14

• renamed functions and variables

  kiranvasudev

  committed
  dcb6854

  Copy full SHA for dcb6854

• added type hints and docstrings

  kiranvasudev

  committed
  fb6f8f6

  Copy full SHA for fb6f8f6

• changed how models are selected when config is generated

  kiranvasudev

  committed
  06dc86b

  Copy full SHA for 06dc86b

• black

  kiranvasudev

  committed
  19a13bc

  Copy full SHA for 19a13bc

Commits on Nov 16, 2023

• fix project dir to load the correct profile

  kiranvasudev

  committed
  5f83797

  Copy full SHA for 5f83797

• get external_location from config instead of unrendered config

  kiranvasudev

  committed
  209efd6

  Copy full SHA for 209efd6

• renamed variables for better understanding

  kiranvasudev

  committed
  3c4e4c3

  Copy full SHA for 3c4e4c3

• added doctest for fn

  kiranvasudev

  committed
  1bd525c

  Copy full SHA for 1bd525c

• added files for fixtures and tests

  kiranvasudev

  committed
  4b651ad

  Copy full SHA for 4b651ad

• added fixture for manifest and profiles

  kiranvasudev

  committed
  7123cbc

  Copy full SHA for 7123cbc

• setUp test class

  kiranvasudev

  committed
  77fb2b2

  Copy full SHA for 77fb2b2

• added test for get_dbt_model_parents

  kiranvasudev

  committed
  47c0638

  Copy full SHA for 47c0638

• added tests for generate_dagger_inputs

  kiranvasudev

  committed
  4eced6b

  Copy full SHA for 4eced6b

• added test for generate_dagger_outputs

  kiranvasudev

  committed
  aa767e8

  Copy full SHA for aa767e8

• fixed type hint

  kiranvasudev

  committed
  8cecbe6

  Copy full SHA for 8cecbe6

• fixed type hints

  kiranvasudev

  committed
  04eafe6

  Copy full SHA for 04eafe6

Commits on Nov 17, 2023

• refactored for simplicity

  kiranvasudev

  committed
  745311d

  Copy full SHA for 745311d

• updated tests

  kiranvasudev

  committed
  b52e0b5

  Copy full SHA for b52e0b5

• Merge remote-tracking branch 'origin/feature/dbt-task-split' into feature/dbt-task-split

  kiranvasudev

  committed
  8e8c408

  Copy full SHA for 8e8c408

Commits on Nov 20, 2023

• added dbt profile to default parameters for dbt task

  kiranvasudev

  committed
  6e7ced2

  Copy full SHA for 6e7ced2

• Merge pull request #25 from chocoapp/feature/dbt-task-split

  Show description for 962ef70
  kiranvasudev

  authored
  962ef70

  Copy full SHA for 962ef70

• added follow external dependency as true as default for athena task

  kiranvasudev

  committed
  e8cbc6f

  Copy full SHA for e8cbc6f

• add fn to process seed input

  kiranvasudev

  committed
  e62d35c

  Copy full SHA for e62d35c

• refactor code to incorporate seeds

  kiranvasudev

  committed
  8de6821

  Copy full SHA for 8de6821

• updates fixtures

  kiranvasudev

  committed
  5f26723

  Copy full SHA for 5f26723

• added test for dbt seed

  kiranvasudev

  committed
  9acaff5

  Copy full SHA for 9acaff5

• modified tests for model containing dbt seed as a dependency

  kiranvasudev

  committed
  876010c

  Copy full SHA for 876010c

Commits on Nov 24, 2023

• refactor

  Show description for 9cd0a51
  kiranvasudev

  committed
  9cd0a51

  Copy full SHA for 9cd0a51

• updated tests and fixtures

  kiranvasudev

  committed
  2e6abf6

  Copy full SHA for 2e6abf6

• changed name of seed task generating fn

  kiranvasudev

  committed
  c652d56

  Copy full SHA for c652d56

• removed unused line

  kiranvasudev

  committed
  ac5f48f

  Copy full SHA for ac5f48f

Commits on Nov 27, 2023

• added docstrings

  kiranvasudev

  committed
  46a833d

  Copy full SHA for 46a833d

• removed unused test parameter

  kiranvasudev

  committed
  cdd0fea

  Copy full SHA for cdd0fea

• changed name of function for better understanding

  kiranvasudev

  committed
  829ac94

  Copy full SHA for 829ac94

• added test to check for de-duplication of inputs

  kiranvasudev

  committed
  eaf4286

  Copy full SHA for eaf4286

Commits on Nov 29, 2023

• refactored getting model location function

  Show description for ab9a2c8
  kiranvasudev

  committed
  ab9a2c8

  Copy full SHA for ab9a2c8

• refactor dummy task generation

  kiranvasudev

  committed
  8ef7050

  Copy full SHA for 8ef7050

• generate inputs for intermediate models and updated tests

  kiranvasudev

  committed
  65a07d7

  Copy full SHA for 65a07d7

• uncomment skipping local test

  kiranvasudev

  committed
  e5afcfa

  Copy full SHA for e5afcfa

• refactor generate_dagger_tasks fn to make recursive

  kiranvasudev

  committed
  e28a078

  Copy full SHA for e28a078

• updated fixtures and tests

  kiranvasudev

  committed
  ab48229

  Copy full SHA for ab48229

Commits on Nov 30, 2023

• Merge pull request #26 from chocoapp/feature/dbt-task-split

  Show description for 3c8c92c
  kiranvasudev

  authored
  3c8c92c

  Copy full SHA for 3c8c92c

Commits on Dec 4, 2023

• bugfix

  kiranvasudev

  committed
  d653977

  Copy full SHA for d653977

• only return dummy when stg model

  kiranvasudev

  committed
  48a102f

  Copy full SHA for 48a102f

• adapted tests

  kiranvasudev

  committed
  f6c7226

  Copy full SHA for f6c7226

Commits on Dec 5, 2023

• Merge pull request #27 from chocoapp/fix/external-dag-sensors

  Show description for 32fdd1f
  kiranvasudev

  authored
  32fdd1f

  Copy full SHA for 32fdd1f

• Merge pull request #28 from chocoapp/feature/dbt-task-split

  Show description for 4ceca95
  kiranvasudev

  authored
  4ceca95

  Copy full SHA for 4ceca95

Commits on Dec 6, 2023

• initialize dbt module only when its a dbt pipeline config

  kiranvasudev

  committed
  146846e

  Copy full SHA for 146846e

• format

  kiranvasudev

  committed
  fd2bd0f

  Copy full SHA for fd2bd0f

• made logic to check for dbt task easier

  kiranvasudev

  committed
  fb0c2e9

  Copy full SHA for fb0c2e9

• Merge pull request #29 from chocoapp/feature/dbt-task-split

  Show description for bd8e44c
  kiranvasudev

  authored
  bd8e44c

  Copy full SHA for bd8e44c

Commits on Dec 7, 2023

• fix: follow external dependency for staging models

  kiranvasudev

  committed
  a2b3fdd

  Copy full SHA for a2b3fdd

Commits on Dec 8, 2023

• Merge pull request #30 from chocoapp/feature/dbt-task-split

  Show description for 4f1783b
  kiranvasudev

  authored
  4f1783b

  Copy full SHA for 4f1783b

Commits on Feb 23, 2024

• added logic to get parents of int models

  Show description for 3653646
  kiranvasudev

  committed
  3653646

  Copy full SHA for 3653646

• updated tests and fixtures

  kiranvasudev

  committed
  7dec65a

  Copy full SHA for 7dec65a

Commits on Feb 27, 2024

• Merge pull request #31 from chocoapp/fix/dbt-add-core-sensor-to-int

  Show description for 750f53e
  kiranvasudev

  authored
  750f53e

  Copy full SHA for 750f53e

Commits on Mar 25, 2024

• Turing split_statements on by default

  siklosid

  committed
  bdc3e57

  Copy full SHA for bdc3e57

Commits on Mar 26, 2024

• Merge pull request #32 from chocoapp/feature/psql_split_statements

  Show description for 0e9dd3b
  siklosid

  authored
  0e9dd3b

  Copy full SHA for 0e9dd3b

Commits on Apr 13, 2024

• Moving to python3.9; Upgrading airflow version; removing legacy postgres operator

  siklosid

  committed
  96ad27b

  Copy full SHA for 96ad27b

Commits on Apr 15, 2024

• Making sensor default args more flexible

  siklosid

  committed
  3e62d78

  Copy full SHA for 3e62d78

• Upgrading python in CI

  siklosid

  committed
  41f1554

  Copy full SHA for 41f1554

• Adding graphviz dependency to test

  siklosid

  committed
  d8145ab

  Copy full SHA for d8145ab

• Merge pull request #33 from chocoapp/feature/DATA-1791_defferable_sensors

  Show description for 7b9bd16
  siklosid

  authored
  7b9bd16

  Copy full SHA for 7b9bd16

Commits on Apr 16, 2024

• Upgrading some package versions to remove warnings

  siklosid

  committed
  8cc2ec2

  Copy full SHA for 8cc2ec2

• Merge pull request #34 from chocoapp/fix/update_packages

  Show description for 647a740
  siklosid

  authored
  647a740

  Copy full SHA for 647a740

Commits on Apr 17, 2024

• feat: rename profile_name to target_name

  

  claudiazi

  committed
  f56e6b6

  Copy full SHA for f56e6b6

• Merge pull request #35 from chocoapp/feature/DATA-1493-adjust-dbt-params

  Show description for 52e7572
  claudiazi

  authored
  52e7572

  Copy full SHA for 52e7572

Commits on Apr 26, 2024

• feat: register new databricks_io

  

  claudiazi

  committed
  1b99357

  Copy full SHA for 1b99357

• feat: refactor the DBTParseConfig to parse databricks-dbt manifest

  

  claudiazi

  committed
  e00f555

  Copy full SHA for e00f555

• feat: add unit test for databricks config parser

  

  claudiazi

  committed
  9274fe2

  Copy full SHA for 9274fe2

• chore: black

  

  claudiazi

  committed
  e951c47

  Copy full SHA for e951c47

• Switching to official batch operator

  siklosid

  committed
  4567b3e

  Copy full SHA for 4567b3e

• feat: add another param in dbt task

  

  claudiazi

  committed
  1c2ad82

  Copy full SHA for 1c2ad82

• Complete renaming of classes

  siklosid

  committed
  a8e6471

  Copy full SHA for a8e6471

• Merge pull request #36 from chocoapp/bugfix/DATA-1804_missing_batch_logs

  Show description for 3c5fd12
  siklosid

  authored
  3c5fd12

  Copy full SHA for 3c5fd12

• feat: refactor

  

  claudiazi

  committed
  7e29420

  Copy full SHA for 7e29420

• Merge remote-tracking branch 'origin/master' into feature/DATA-1811-databricks-io

  

  claudiazi

  committed
  432c32d

  Copy full SHA for 432c32d

• feat: adjust the s3 tasks

  

  claudiazi

  committed
  5659c68

  Copy full SHA for 5659c68

Commits on Apr 29, 2024

• feat: adjust the _get_s3_task for different dbt adapters

  

  claudiazi

  committed
  81697d4

  Copy full SHA for 81697d4

• fix: define the correct target_config for databricks adapter

  

  claudiazi

  committed
  7c1aa22

  Copy full SHA for 7c1aa22

• fix: _generate_dagger_output

  

  claudiazi

  committed
  40b28ef

  Copy full SHA for 40b28ef

• extend: command

  

  claudiazi

  committed
  9c0311f

  Copy full SHA for 9c0311f

Commits on Apr 30, 2024

• fix: _generate_command

  

  claudiazi

  committed
  14f9d1f

  Copy full SHA for 14f9d1f

• Merge pull request #37 from chocoapp/feature/DATA-1811-databricks-io

  Show description for 0d3947f
  claudiazi

  authored
  0d3947f

  Copy full SHA for 0d3947f

• fix: _get_model_data_location in DatabricksDBTConfigParser

  

  claudiazi

  committed
  f7cc983

  Copy full SHA for f7cc983

• Merge pull request #38 from chocoapp/feature/DATA-1811-databricks-io

  Show description for 1523648
  claudiazi

  authored
  1523648

  Copy full SHA for 1523648

Commits on May 3, 2024

• fix: generate_task for dbt tasks

  

  claudiazi

  committed
  59cf847

  Copy full SHA for 59cf847

• Merge pull request #39 from chocoapp/feature/DATA-1811-databricks-io

  Show description for a25ccc9
  claudiazi

  authored
  a25ccc9

  Copy full SHA for a25ccc9

Commits on Jun 12, 2024

• Replacing string replacement with jinja in module processor

  siklosid

  committed
  c44a7b5

  Copy full SHA for c44a7b5

• Merge pull request #40 from chocoapp/feature/DATA-1699_using_jinja_in_module_processor

  Show description for e687686
  siklosid

  authored
  e687686

  Copy full SHA for e687686

Commits on Jun 17, 2024

• feat: adjust the dbt config parser so that view/ephemeral staging layer doesnt need a task

  

  claudiazi

  committed
  1bc6bf2

  Copy full SHA for 1bc6bf2

Commits on Jun 18, 2024

• feat: adjust io for the materalised staging model

  

  claudiazi

  committed
  22ca09b

  Copy full SHA for 22ca09b

Commits on Jun 19, 2024

• feat: restructure the dbt dagger task input & output

  

  claudiazi

  committed
  f2f8015

  Copy full SHA for f2f8015

• Merge pull request #41 from chocoapp/feat/DATA-1996-improve-dbt-parser

  Show description for 81e75da
  claudiazi

  authored
  81e75da

  Copy full SHA for 81e75da

Commits on Jul 4, 2024

• Module generation with generalised jinja parameters

  siklosid

  committed
  698358d

  Copy full SHA for 698358d

• Now it's possible to assign task to task groups

  siklosid

  committed
  cd12dda

  Copy full SHA for cd12dda

• Adding default value to the parameter

  siklosid

  committed
  0e6a6a6

  Copy full SHA for 0e6a6a6

• Merge pull request #42 from chocoapp/feature/DATA-2002_generalise_jinja_params

  Show description for f041aa8
  siklosid

  authored
  f041aa8

  Copy full SHA for f041aa8

Commits on Nov 6, 2024

• removed custom dbt task generation logic from Module

  kiranvasudev

  committed
  8d0e23f

  Copy full SHA for 8d0e23f

Commits on Nov 7, 2024

• add plugins path to dagger config

  kiranvasudev

  committed
  77f6026

  Copy full SHA for 77f6026

• added function to load plugins

  kiranvasudev

  committed
  fbb648b

  Copy full SHA for fbb648b

• load plugins and render jinja

  kiranvasudev

  committed
  90bb16b

  Copy full SHA for 90bb16b

• iterate over multiple folders and their subfolders

  kiranvasudev

  committed
  1d09c57

  Copy full SHA for 1d09c57

• exclude all files starting with __

  kiranvasudev

  committed
  da0528a

  Copy full SHA for da0528a

• added plugin to dagger config

  kiranvasudev

  committed
  a8aeff7

  Copy full SHA for a8aeff7

• added logging for plugins

  kiranvasudev

  committed
  6addf98

  Copy full SHA for 6addf98

• fix

  kiranvasudev

  committed
  b8feafb

  Copy full SHA for b8feafb

• added tests for plugins

  kiranvasudev

  committed
  d8c6384

  Copy full SHA for d8c6384

• refactor code

  Show description for a3d7f47
  kiranvasudev

  committed
  a3d7f47

  Copy full SHA for a3d7f47

• refactor tests

  kiranvasudev

  committed
  450b544

  Copy full SHA for 450b544

Commits on Nov 8, 2024

• exclude test folders from directory walk

  kiranvasudev

  committed
  98a6b55

  Copy full SHA for 98a6b55

• remove unused imports

  kiranvasudev

  committed
  ab6ff15

  Copy full SHA for ab6ff15

Commits on Nov 11, 2024

• update readme

  kiranvasudev

  committed
  0fa5389

  Copy full SHA for 0fa5389

Commits on Nov 12, 2024

• fix readme

  kiranvasudev

  committed
  e7f08cd

  Copy full SHA for e7f08cd

Commits on Nov 13, 2024

• Merge pull request #43 from chocoapp/feature/plugin-dbt-parser

  Show description for 7554e62
  kiranvasudev

  authored
  7554e62

  Copy full SHA for 7554e62

Commits on Jan 2, 2025

• Adding new reverse etl operator to dagger inherited from batch operator

  siklosid

  committed
  7d871b6

  Copy full SHA for 7d871b6

• Registering the new operator with dagger

  siklosid

  committed
  bfceb81

  Copy full SHA for bfceb81

• Small type fix to resolve broken cli help command

  siklosid

  committed
  0b2ac50

  Copy full SHA for 0b2ac50

• Adding the possibility that inherited operator can overwrite attribute of base operator

  siklosid

  committed
  e414b84

  Copy full SHA for e414b84

• Smaller fixes; syntax fix; Fixing command creation by extending the existing solution in base class

  siklosid

  committed
  67a8142

  Copy full SHA for 67a8142

Commits on Jan 3, 2025

• Adding dynamo and sns io types

  siklosid

  committed
  0557bab

  Copy full SHA for 0557bab

• Adding dynamo and sns io types

  siklosid

  committed
  4ddfc33

  Copy full SHA for 4ddfc33

• Fixing input/output name for reverse etl so it matches the batch job expected format; Inferring output type (dynamo/sns) based on output type and passing as an argument to the batch job

  siklosid

  committed
  8038d3c

  Copy full SHA for 8038d3c

• Handling hard wired constants as local parameters of the task

  siklosid

  committed
  60c4736

  Copy full SHA for 60c4736

• Removing account_id from io; naming convention; fixing small issues

  siklosid

  committed
  f64910f

  Copy full SHA for f64910f

• Fixing batch job name

  siklosid

  committed
  2ea6f7e

  Copy full SHA for 2ea6f7e

Commits on Jan 7, 2025

• Removing choco specific parameters and moving them to conf file; Making some arguments default value handled by the job itslef; Making some arguments required instead hardwiring default value

  siklosid

  committed
  85c9c2a

  Copy full SHA for 85c9c2a

• Adding unit tests and a small fix

  siklosid

  committed
  3c5a610

  Copy full SHA for 3c5a610

• Adding comments

  siklosid

  committed
  f45c84a

  Copy full SHA for f45c84a

Commits on Jan 8, 2025

• Merge pull request #44 from chocoapp/feature/data-2025_reverse_etl_operator

  Show description for 76c774d
  siklosid

  authored
  76c774d

  Copy full SHA for 76c774d

Commits on Jan 16, 2025

• feat: add application name to the spark job & add kill spark job when timeout

  

  claudiazi

  committed
  e62890b

  Copy full SHA for e62890b

• fix: type of _execution_timeout

  

  claudiazi

  committed
  31085ac

  Copy full SHA for 31085ac

• fix: remove the wrong and uncessary function

  

  claudiazi

  committed
  02283d8

  Copy full SHA for 02283d8

Commits on Jan 17, 2025

• fix: timeout logic

  

  claudiazi

  committed
  c9a1c6f

  Copy full SHA for c9a1c6f

• fix: add missing import

  

  claudiazi

  committed
  bfb935c

  Copy full SHA for bfb935c

• fix: spark_app_name

  

  claudiazi

  committed
  7fba8d5

  Copy full SHA for 7fba8d5

• fix: default spark_app_name

  

  claudiazi

  committed
  37a31f8

  Copy full SHA for 37a31f8

Commits on Jan 20, 2025

• feat: improve the logic to kill the spark job

  

  claudiazi

  committed
  a3422db

  Copy full SHA for a3422db

• chore: black + add log + missing function

  

  claudiazi

  committed
  6046a01

  Copy full SHA for 6046a01

• chore: add info to debug

  

  claudiazi

  committed
  57c394b

  Copy full SHA for 57c394b

• chore: add info to debug

  

  claudiazi

  committed
  6fdb674

  Copy full SHA for 6fdb674

• chore: reformat

  

  claudiazi

  committed
  a5d627c

  Copy full SHA for a5d627c

Commits on Jan 21, 2025

• Merge pull request #45 from chocoapp/feature/DATA-2175-kill-timeout-spark

  Show description for 9ad0c1e
  claudiazi

  authored
  9ad0c1e

  Copy full SHA for 9ad0c1e

Commits on Feb 13, 2025

• Bumping tenacity version

  siklosid

  committed
  3ef83b3

  Copy full SHA for 3ef83b3