Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce TableLastModifiedMetadataBatch and implement BaseAdapter.calculate_freshness_from_metadata_batch #127

Merged
merged 29 commits into from
Apr 12, 2024
Merged
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
2beba0b
extend RelationConfig and MaterializationConfig
MichelleArk Mar 1, 2024
151b131
Merge branch 'main' into batch-metadata-freshness
MichelleArk Mar 11, 2024
727093c
first pass
MichelleArk Mar 11, 2024
ffaf808
accept information schema in calculate_freshness_from_metadata_batch
MichelleArk Mar 11, 2024
b66644a
implement calculate_freshness_from_metadata in terms of calculate_fre…
MichelleArk Mar 11, 2024
54670e0
add TableLastModifiedMetadataBatch capability
MichelleArk Mar 11, 2024
8046a3e
return batched freshness results keyed by (schema, identifier)
MichelleArk Mar 13, 2024
1dbf3e9
Merge branch 'main' into batch-metadata-freshness
MichelleArk Mar 13, 2024
2c28576
Merge branch 'main' into batch-metadata-freshness
MichelleArk Mar 14, 2024
a9cffcd
Merge branch 'main' into batch-metadata-freshness
MichelleArk Mar 22, 2024
9af8c85
handle queries across information_schema in calculate_freshness_from_…
MichelleArk Mar 25, 2024
82d8217
Merge branch 'main' into batch-metadata-freshness
MichelleArk Mar 25, 2024
ed81529
refactor _create_freshness_response
MichelleArk Mar 25, 2024
c44779a
make schema_identifier_to_source type-safe
MichelleArk Mar 25, 2024
62be8df
update TableLastModifiedMetadataBatch description
MichelleArk Mar 25, 2024
9a6c829
changelog entry
MichelleArk Mar 25, 2024
17701ad
refactor to _get_catalog_relations_by_info_schema, _parse_freshness_row
MichelleArk Mar 28, 2024
511cf14
sanitize raw_relation for freshness batch calculation
MichelleArk Apr 2, 2024
01f27a5
ensure a connection is open if possible prior to executing macro
MichelleArk Apr 4, 2024
05c7149
Merge branch 'main' into batch-metadata-freshness
MichelleArk Apr 4, 2024
154e066
fix agate typing
MichelleArk Apr 4, 2024
8c707cc
lazy load agate_helper
MichelleArk Apr 4, 2024
8a1deac
add needs_conn to BaseAdapter.execute_macro
MichelleArk Apr 4, 2024
f3dcac2
docstring
MichelleArk Apr 4, 2024
ebee880
Merge branch 'main' into batch-metadata-freshness
MichelleArk Apr 5, 2024
02988c8
Merge branch 'main' into batch-metadata-freshness
colin-rogers-dbt Apr 11, 2024
0bbf7ed
cleanup adapter_responses parsing in calculate_freshness_from_metadat…
MichelleArk Apr 11, 2024
a43f04c
Merge branch 'main' into batch-metadata-freshness
MichelleArk Apr 12, 2024
7cf501e
Merge branch 'main' into batch-metadata-freshness
MichelleArk Apr 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions dbt/adapters/base/impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -1100,6 +1100,12 @@ def execute_macro(

macro_function = CallableMacroGenerator(macro, macro_context)

# A connection may or may not be required for a given macro execution
connection = self.connections.get_if_exists()
# If a connection exists, ensure it is open prior to executing a macro
if connection:
self.connections.open(connection)

with self.connections.exception_handler(f"macro {macro_name}"):
result = macro_function(**kwargs)
return result
Expand Down