Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] External materialization to S3 does not work if table is empty #510

Merged
merged 2 commits into from
Feb 11, 2025

Conversation

guenp
Copy link
Collaborator

@guenp guenp commented Feb 10, 2025

Workaround for #503 (need to look into fix in duckdb for writing empty tables to S3)

It consists of inserting NULL values (if the table is empty), writing those to S3, and then creating a view that filters those NULL values

add S3 secrets

make separate tests instead of parametrize

use vars instead of hard-coded string

only create dir if not yet exists

make secret persistent if running on md

add random folder name

only yield if dest is s3

fix bug

skip external test on buenavista
@guenp guenp requested a review from jwills February 10, 2025 23:18
Copy link
Collaborator

@jwills jwills left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@guenp guenp merged commit 41ff762 into master Feb 11, 2025
64 checks passed
@guenp guenp deleted the guenp/fix-external-s3 branch February 11, 2025 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants