Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Convert EIA-861 + FERC-714 service territory and state electricity demand to Dagster assets #2437

Closed
9 tasks done
Tracked by #1973
zaneselvans opened this issue Mar 22, 2023 · 1 comment · Fixed by #2550
Closed
9 tasks done
Tracked by #1973

Convert EIA-861 + FERC-714 service territory and state electricity demand to Dagster assets #2437

zaneselvans opened this issue Mar 22, 2023 · 1 comment · Fixed by #2550
Assignees
@e-belfer
Labels
analysis Data analysis tasks that involve actually using PUDL to figure things out, like calculating MCOE. dagster Issues related to our use of the Dagster orchestrator eia861 Anything having to do with EIA Form 861 epic Any issue whose primary purpose is to organize other issues into a group. ferc714 Anything having to do with FERC Form 714 geospatial Spatial data and transformations. Anything related to mapping. output Exporting data from PUDL into other platforms or interchange formats. sqlite Issues related to interacting with sqlite databases
Milestone
2023 Spring

Comments

@zaneselvans
Copy link
Member

zaneselvans commented Mar 22, 2023 •
edited by e-belfer
Loading

Notes on how to do this:

  • Interim table transforms in pudl.output.ferc714 should use the standard io manager
  • compile_geoms() should have a factory function, iterating over entity type. The save to geoparquet function has been flaky, if we are able to get it working we could use geoparquet and write a geoparquet io manager. Otherwise, we can strip geometry and save as a table. Extensive additional work on the geometrical component is a low priority in this iteration of the transition.
  • Whatever is implemented in state_demand.py should be used as a default for the ba_ids and util_ids, but these should be parameters passed in by dagster's context to enable manual changes.
  • Best order to implement change in: service_territory -> FERC 714 -> state_demand
  • Rip out the Respondents class in pudl.output.ferc714
  • In pudl.output.ferc714, the summarize_demand and fipsify tables are the best candidates for inclusion into PUDL. The other two tables are just upstream intermediate outputs.

Tasks
Preview Give feedback
@zaneselvans zaneselvans added output Exporting data from PUDL into other platforms or interchange formats. analysis Data analysis tasks that involve actually using PUDL to figure things out, like calculating MCOE. ferc714 Anything having to do with FERC Form 714 eia861 Anything having to do with EIA Form 861 sqlite Issues related to interacting with sqlite databases epic Any issue whose primary purpose is to organize other issues into a group. geospatial Spatial data and transformations. Anything related to mapping. dagster Issues related to our use of the Dagster orchestrator labels Mar 22, 2023
@zaneselvans zaneselvans added this to the 2023Q2 milestone Mar 22, 2023
@zaneselvans zaneselvans changed the title Convert FERC-714 service territory and state demand to Dagster assets Convert EIA-861 + FERC-714 service territory and state electricity demand to Dagster assets Mar 22, 2023
@zaneselvans zaneselvans mentioned this issue Apr 3, 2023
Closed
@zaneselvans zaneselvans moved this from 🆕 New to 🔖 Backlog in Catalyst Megaproject Apr 3, 2023
@e-belfer e-belfer assigned e-belfer and unassigned e-belfer Apr 24, 2023
@e-belfer e-belfer moved this from Backlog to In progress in Catalyst Megaproject Apr 26, 2023
@e-belfer e-belfer self-assigned this Apr 27, 2023
@e-belfer e-belfer mentioned this issue Apr 27, 2023
Merged
12 tasks
@e-belfer e-belfer linked a pull request Apr 27, 2023 that will close this issue
Convert EIA-861 and FERC 714 service territory outputs to Dagster assets #2550
Merged
12 tasks
This was referenced May 31, 2023
Closed
Open
@e-belfer e-belfer moved this from In progress to Backlog in Catalyst Megaproject Jun 5, 2023
@e-belfer e-belfer moved this from Backlog to In review in Catalyst Megaproject Jun 15, 2023
@e-belfer
Copy link
Member

Merged in #2550

@github-project-automation github-project-automation bot moved this from In review to Done in Catalyst Megaproject Jun 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@e-belfer e-belfer

Labels
analysis Data analysis tasks that involve actually using PUDL to figure things out, like calculating MCOE. dagster Issues related to our use of the Dagster orchestrator eia861 Anything having to do with EIA Form 861 epic Any issue whose primary purpose is to organize other issues into a group. ferc714 Anything having to do with FERC Form 714 geospatial Spatial data and transformations. Anything related to mapping. output Exporting data from PUDL into other platforms or interchange formats. sqlite Issues related to interacting with sqlite databases
Projects
Catalyst Megaproject
Archived in project
Milestone
2023 Spring
Development

Successfully merging a pull request may close this issue.

Convert EIA-861 and FERC 714 service territory outputs to Dagster assets catalyst-cooperative/pudl
2 participants
@zaneselvans @e-belfer