You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Currently the job status table is created on each write to Vertica, if it doesn't already exist. However this can clutter the database and the option to not create it is desired, or rather don't create it by default and provide an option to create it.
Describe the solution you'd like
Update the connector to not create the job status table by default. But also add a new flag, such as save_metadata_tables, that will create and write to the status table when enabled. This flag can also cover the creation of the rejected rows table in the future (see #293).
Describe alternatives you've considered
We could have the job status table created by default and have the flag to disable that functionality. However, it seems this table is mostly useful for development/debugging purposes so should probably not create the table by default.
Additional context
Note that there is an existing prevent_cleanup flag to persist temporary HDFS data. So it makes sense for the new flag to follow this functionality and not persist metadata tables, such as the job status table, by default.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Currently the job status table is created on each write to Vertica, if it doesn't already exist. However this can clutter the database and the option to not create it is desired, or rather don't create it by default and provide an option to create it.
Describe the solution you'd like
Update the connector to not create the job status table by default. But also add a new flag, such as
save_metadata_tables
, that will create and write to the status table when enabled. This flag can also cover the creation of the rejected rows table in the future (see #293).Describe alternatives you've considered
We could have the job status table created by default and have the flag to disable that functionality. However, it seems this table is mostly useful for development/debugging purposes so should probably not create the table by default.
Additional context
Note that there is an existing
prevent_cleanup
flag to persist temporary HDFS data. So it makes sense for the new flag to follow this functionality and not persist metadata tables, such as the job status table, by default.The text was updated successfully, but these errors were encountered: