Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make creation of job status table optional #308

Closed
jeremyprime opened this issue Jan 11, 2022 · 0 comments · Fixed by #311
Closed

Make creation of job status table optional #308

jeremyprime opened this issue Jan 11, 2022 · 0 comments · Fixed by #311
Labels

Comments

@jeremyprime
Copy link
Collaborator

Is your feature request related to a problem? Please describe.

Currently the job status table is created on each write to Vertica, if it doesn't already exist. However this can clutter the database and the option to not create it is desired, or rather don't create it by default and provide an option to create it.

Describe the solution you'd like

Update the connector to not create the job status table by default. But also add a new flag, such as save_metadata_tables, that will create and write to the status table when enabled. This flag can also cover the creation of the rejected rows table in the future (see #293).

Describe alternatives you've considered

We could have the job status table created by default and have the flag to disable that functionality. However, it seems this table is mostly useful for development/debugging purposes so should probably not create the table by default.

Additional context

Note that there is an existing prevent_cleanup flag to persist temporary HDFS data. So it makes sense for the new flag to follow this functionality and not persist metadata tables, such as the job status table, by default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants