You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
e-belfer opened this issue
Jun 5, 2023
· 0 comments
· Fixed by #2649
Labels
dagsterIssues related to our use of the Dagster orchestratoreia861Anything having to do with EIA Form 861ferc714Anything having to do with FERC Form 714
Right now the EIA 861 and FERC 714 interim ETLs process data for all years for both the full and the fast ETL. As we add more of this data into dagster, and notably as we incorporate the service territory analyses (#2437), the inability to subset this data by year is causing memory issues for the CI.
To make this change, we would need to do the following:
For FERC 714 data: Change the extractor to read in the CSV files (which are not time-subsetted) and subset data by years (closed by Filter FERC714 ETL by year #2649)
zaneselvans
added
ferc714
Anything having to do with FERC Form 714
eia861
Anything having to do with EIA Form 861
dagster
Issues related to our use of the Dagster orchestrator
labels
Jun 5, 2023
dagsterIssues related to our use of the Dagster orchestratoreia861Anything having to do with EIA Form 861ferc714Anything having to do with FERC Form 714
Right now the EIA 861 and FERC 714 interim ETLs process data for all years for both the full and the fast ETL. As we add more of this data into dagster, and notably as we incorporate the service territory analyses (#2437), the inability to subset this data by year is causing memory issues for the CI.
To make this change, we would need to do the following:
The text was updated successfully, but these errors were encountered: