Skip to content

2023 Spring

Closed Jan 12, 2024 82% complete

Work that we will include in our Spring, 2023 Release.

Direct big datasette downloads to AWS Open Data Instead cloud Stuff that has to do with adapting PUDL to work in cloud computing context. community datasette Issues related the accessing PUDL data via Datasette. good-first-issue Good issues for first-time contributors. Self-contained, low context, no credentials required. sqlite Issues related to interacting with sqlite databases
#2464 by zaneselvans was closed Jan 7, 2025
Integrate EIA860 EnviroAssoc and EnviroEquip (Schedules 6.1 & 6.2) eia860 Anything having to do with EIA Form 860 epic Any issue whose primary purpose is to organize other issues into a group. inframundo new-data Requests for integration of new data.
#1162 by zaneselvans was closed Oct 1, 2024
Update PUDL to new major versions of key dependencies dependencies Pull requests that update a dependency file epic Any issue whose primary purpose is to organize other issues into a group. inframundo
#2384 by zaneselvans was closed Feb 8, 2024
Remove bandwidth-intensive download options from chonky datasette cloud Stuff that has to do with adapting PUDL to work in cloud computing context. datasette Issues related the accessing PUDL data via Datasette. sqlite Issues related to interacting with sqlite databases
#2465 by zaneselvans was closed Jan 2, 2024 Draft
8 tasks
enable labeling for straightforward rate base calcuations from the exploded FERCtables ferc1 Anything having to do with FERC Form 1 output Exporting data from PUDL into other platforms or interchange formats. rmi
#2600 by cmgosnell was closed Dec 29, 2023
5 of 8 tasks
Remove pudl_settings and pudl_setup in favor of PUDL_INPUT and PUDL_OUTPUT community dagster Issues related to our use of the Dagster orchestrator inframundo
#2293 by jdangerx was closed Dec 6, 2023
Update PUDL to be compatible with SQLAlchemy 2.0 community dependencies Pull requests that update a dependency file good-first-issue Good issues for first-time contributors. Self-contained, low context, no credentials required. inframundo
#2395 by zaneselvans was closed Nov 18, 2023
Update PUDL to SQLAlchemy 2.0 dependencies Pull requests that update a dependency file inframundo
#2267 by zaneselvans was merged Nov 18, 2023 Loading…
4 of 8 tasks
Parallelize extraction of Excel spreadsheets dagster Issues related to our use of the Dagster orchestrator eia860 Anything having to do with EIA Form 860 eia861 Anything having to do with EIA Form 861 eia923 Anything having to do with EIA Form 923 excel Issues involving data in Microsoft Excel spreadsheets good-first-issue Good issues for first-time contributors. Self-contained, low context, no credentials required. inframundo performance Make PUDL run faster! requires-debug Things that have been worked on but hit an issue that requires debugging.
#2385 by zaneselvans was closed Oct 18, 2023
Convert output and analysis tables to assets dagster Issues related to our use of the Dagster orchestrator epic Any issue whose primary purpose is to organize other issues into a group. inframundo
#1973 by bendnorman was closed Oct 16, 2023
Convert FERC1-EIA record linkage outputs to Dagster assets dagster Issues related to our use of the Dagster orchestrator eia860 Anything having to do with EIA Form 860 eia923 Anything having to do with EIA Form 923 ferc1 Anything having to do with FERC Form 1 glue PUDL specific structures & metadata. Stuff that connects datasets together. output Exporting data from PUDL into other platforms or interchange formats. ppe Plant Parts EIA (formerly the EIA plant parts list) sqlite Issues related to interacting with sqlite databases
#2439 by zaneselvans was closed Oct 16, 2023
Dagsterize mega_generators and plant_parts_eia ccai Tasks related to CCAI grant for entity matching
#2714 by katie-lamb was merged Sep 26, 2023 Loading…
7 of 8 tasks
Update PUDL to be compatible with pandas 2.0 dependencies Pull requests that update a dependency file good-first-issue Good issues for first-time contributors. Self-contained, low context, no credentials required. inframundo
#2394 by zaneselvans was closed Aug 30, 2023
Update PUDL to pandas 2.0 dependencies Pull requests that update a dependency file inframundo
#2320 by zaneselvans was merged Aug 30, 2023 Loading…
2 of 8 tasks
Convert MCOE analysis to Dagster assets analysis Data analysis tasks that involve actually using PUDL to figure things out, like calculating MCOE. dagster Issues related to our use of the Dagster orchestrator eia860 Anything having to do with EIA Form 860 eia923 Anything having to do with EIA Form 923 output Exporting data from PUDL into other platforms or interchange formats. sqlite Issues related to interacting with sqlite databases
#2438 by zaneselvans was closed Aug 14, 2023
Dagsterize MCOE output tables analysis Data analysis tasks that involve actually using PUDL to figure things out, like calculating MCOE. dagster Issues related to our use of the Dagster orchestrator
#2553 by zaneselvans was merged Aug 11, 2023 Loading…
5 of 8 tasks
Create design doc for naming convention of PUDL tables dagster Issues related to our use of the Dagster orchestrator inframundo
#2517 by bendnorman was closed Aug 10, 2023
Use tabular calculation fixes in FERC 1 table transformers ferc1 Anything having to do with FERC Form 1 metadata Anything having to do with the content, formatting, or storage of metadata. Mostly datapackages. xbrl Related to the FERC XBRL transition
#2605 by cmgosnell was closed Jul 31, 2023
Generalize Visual FoxPro / DBF extraction code and separate it from FERC Form 1 dbf Data coming from FERC's old Visual FoxPro DBF database file format. epic Any issue whose primary purpose is to organize other issues into a group. ferc1 Anything having to do with FERC Form 1 ferc2 Issues related to the FERC Form 2 dataset ferc6 ferc60 ferceqr Data from FERC's Electric Quarterly Review (EQR) good-first-issue Good issues for first-time contributors. Self-contained, low context, no credentials required. inframundo sqlite Issues related to interacting with sqlite databases
#1984 by zaneselvans was closed Jul 24, 2023
Convert EIA generation and fuel allocations to Dagster analysis Data analysis tasks that involve actually using PUDL to figure things out, like calculating MCOE. dagster Issues related to our use of the Dagster orchestrator eia923 Anything having to do with EIA Form 923 output Exporting data from PUDL into other platforms or interchange formats. sqlite Issues related to interacting with sqlite databases
#2435 by zaneselvans was closed Jul 17, 2023
Run integration & validation tests in parallel in the background. performance Make PUDL run faster! testing Writing tests, creating test data, automating testing, etc.
#2526 by zaneselvans was closed Jul 17, 2023 Loading…
8 tasks
(ferc 💥 metadata) create xbrl_factoid metadata containing tree-like provenance ferc1 Anything having to do with FERC Form 1 metadata Anything having to do with the content, formatting, or storage of metadata. Mostly datapackages. output Exporting data from PUDL into other platforms or interchange formats. xbrl Related to the FERC XBRL transition
#2625 by cmgosnell was closed Jul 17, 2023
Organize FERC 1 XBRL metadata with a calculation forest ferc1 Anything having to do with FERC Form 1 metadata Anything having to do with the content, formatting, or storage of metadata. Mostly datapackages. xbrl Related to the FERC XBRL transition
#2653 by zaneselvans was merged Jul 14, 2023 Loading…
3 of 8 tasks
Convert EIA generation and fuel allocations to Dagster analysis Data analysis tasks that involve actually using PUDL to figure things out, like calculating MCOE. dagster Issues related to our use of the Dagster orchestrator eia923 Anything having to do with EIA Form 923
#2527 by zaneselvans was merged Jul 11, 2023 Loading…
4 of 5 tasks
Integrate minor cleaning fixes from spot-checks of XBRL explosion ferc1 Anything having to do with FERC Form 1 rmi xbrl Related to the FERC XBRL transition
#2599 by e-belfer was closed Jun 22, 2023
Bring Census DP1 database into Dagster censusdp1tract Issues related to the Census DP1 dataset which we distribute as an SQLite DB dagster Issues related to our use of the Dagster orchestrator good-first-issue Good issues for first-time contributors. Self-contained, low context, no credentials required.
#2412 by zaneselvans was closed Jun 21, 2023
Bring the Census DP1 to SQLite ETL into dagster
#2621 by e-belfer was merged Jun 21, 2023 Loading…
7 of 9 tasks
Convert EIA-861 + FERC-714 service territory and state electricity demand to Dagster assets analysis Data analysis tasks that involve actually using PUDL to figure things out, like calculating MCOE. dagster Issues related to our use of the Dagster orchestrator eia861 Anything having to do with EIA Form 861 epic Any issue whose primary purpose is to organize other issues into a group. ferc714 Anything having to do with FERC Form 714 geospatial Spatial data and transformations. Anything related to mapping. output Exporting data from PUDL into other platforms or interchange formats. sqlite Issues related to interacting with sqlite databases
#2437 by zaneselvans was closed Jun 21, 2023
Convert EIA-861 and FERC 714 service territory outputs to Dagster assets dagster Issues related to our use of the Dagster orchestrator service-territories Issues relating to utiltiy service territories and balancing authorities
#2550 by e-belfer was merged Jun 21, 2023 Loading…
6 of 12 tasks
FERC From 2 DBF seems to have gap in the early years bug Things that are just plain broken. ferc2 Issues related to the FERC Form 2 dataset
#2642 by rousik was closed Jun 16, 2023
Integrate FERC Form 6 from dbf dbf Data coming from FERC's old Visual FoxPro DBF database file format. ferc6 sqlite Issues related to interacting with sqlite databases
#2595 by rousik was merged Jun 16, 2023 Loading…
Deduplicate FERC 2 respondent IDs dbf Data coming from FERC's old Visual FoxPro DBF database file format. ferc2 Issues related to the FERC Form 2 dataset
#2661 by zaneselvans was merged Jun 15, 2023 Loading…
8 tasks
Manage duplicate PKs in EIA-861 table transforms data-cleaning Tasks related to cleaning & regularizing data during ETL. eia861 Anything having to do with EIA Form 861
#2648 by zaneselvans was merged Jun 12, 2023 Loading…
8 tasks
Add release notes explaning FERC 1 metadata cleanup data-repair Interpolating or extrapolating data that we don't actually have. docs Documentation for users and contributors. metadata Anything having to do with the content, formatting, or storage of metadata. Mostly datapackages. xbrl Related to the FERC XBRL transition
#2626 by zaneselvans was merged Jun 12, 2023 Loading…
8 tasks
Filter FERC714 ETL by year
#2649 by e-belfer was merged Jun 10, 2023 Loading…
5 of 8 tasks
Fix ferc2 full etl integration test issues ferc2 Issues related to the FERC Form 2 dataset testing Writing tests, creating test data, automating testing, etc.
#2652 by rousik was merged Jun 10, 2023 Loading…
Missing delivery utilities in sales_eia861 bug Things that are just plain broken. eia861 Anything having to do with EIA Form 861
#2636 by christiantfong was closed Jun 8, 2023
Rename and test XBRL metadata calculations ferc1 Anything having to do with FERC Form 1 metadata Anything having to do with the content, formatting, or storage of metadata. Mostly datapackages. rmi xbrl Related to the FERC XBRL transition
#2563 by cmgosnell was merged Jun 8, 2023 Loading…
4 of 8 tasks
Integrate FERC Form 2 dbf formats into ferc_to_sqlite dbf Data coming from FERC's old Visual FoxPro DBF database file format. ferc2 Issues related to the FERC Form 2 dataset
#2564 by rousik was merged Jun 8, 2023 Loading…
Add business_model, service_type to sales_eia861 PK eia861 Anything having to do with EIA Form 861
#2637 by zaneselvans was merged Jun 8, 2023 Loading…
8 tasks
Boiler cooler stack flue table (bb fixes)
#2629 by aesharpe was merged Jun 6, 2023 Loading…
Add correction/balancing account line items for any calculated field that does not compute to the total in the FERC1 calculated fields. data-repair Interpolating or extrapolating data that we don't actually have. ferc1 Anything having to do with FERC Form 1 output Exporting data from PUDL into other platforms or interchange formats. rmi
#2597 by cmgosnell was closed Jun 2, 2023
Add records correcting FERC 1 calculations that are off data-repair Interpolating or extrapolating data that we don't actually have. ferc1 Anything having to do with FERC Form 1 metadata Anything having to do with the content, formatting, or storage of metadata. Mostly datapackages. rmi xbrl Related to the FERC XBRL transition
#2620 by zaneselvans was merged Jun 2, 2023 Loading…
8 tasks
when there is a second dimension (namely utility_type) add a second layer of calcuations to check that utility_type=="total" is calculable ferc1 Anything having to do with FERC Form 1 metadata Anything having to do with the content, formatting, or storage of metadata. Mostly datapackages. rmi xbrl Related to the FERC XBRL transition
#2584 by e-belfer was closed Jun 2, 2023
2 tasks done
Use consistent data column names in balance sheet and income statement tables ferc1 Anything having to do with FERC Form 1 rmi xbrl Related to the FERC XBRL transition
#2581 by zaneselvans was closed May 31, 2023
Integrate greg M's PM code fix
#2446 by cmgosnell was merged May 16, 2023 Loading…
7 of 9 tasks
Warn user when a change to the PUDL DB schema is detected dagster Issues related to our use of the Dagster orchestrator inframundo sqlite Issues related to interacting with sqlite databases
#2377 by jdangerx was closed May 4, 2023
Improve dagster docs dagster Issues related to our use of the Dagster orchestrator docs Documentation for users and contributors.
#2451 by bendnorman was merged May 4, 2023 Loading…
8 tasks
Scope FERC Forms 2 and 6 DBF extraction dbf Data coming from FERC's old Visual FoxPro DBF database file format. ferc2 Issues related to the FERC Form 2 dataset ferc6 inframundo
#2335 by zschira was closed May 2, 2023
4 tasks
Initial refactoring of dbf extraction process dbf Data coming from FERC's old Visual FoxPro DBF database file format. ferc1 Anything having to do with FERC Form 1 ferc2 Issues related to the FERC Form 2 dataset ferc6 ferc60 sqlite Issues related to interacting with sqlite databases
#2536 by rousik was merged Apr 27, 2023 Loading…
11 tasks
Segmentation fault when importing PUDL installed from conda-forge bug Things that are just plain broken. dependencies Pull requests that update a dependency file
#2426 by arengel was closed Apr 24, 2023
Convert FERC1 output tables into Dagster assets dagster Issues related to our use of the Dagster orchestrator ferc1 Anything having to do with FERC Form 1 output Exporting data from PUDL into other platforms or interchange formats. sqlite Issues related to interacting with sqlite databases
#2436 by zaneselvans was closed Apr 24, 2023
Ferc output conversion dagster Issues related to our use of the Dagster orchestrator ferc1 Anything having to do with FERC Form 1 output Exporting data from PUDL into other platforms or interchange formats.
#2521 by e-belfer was merged Apr 20, 2023 Loading…
10 of 13 tasks
Convert original EIA data tables to Dagster assets dagster Issues related to our use of the Dagster orchestrator eia860 Anything having to do with EIA Form 860 eia923 Anything having to do with EIA Form 923 output Exporting data from PUDL into other platforms or interchange formats. sqlite Issues related to interacting with sqlite databases
#2434 by zaneselvans was closed Apr 19, 2023
4 tasks done
Convert EIA entity / annual output tables to Dagster assets dagster Issues related to our use of the Dagster orchestrator eia860 Anything having to do with EIA Form 860 output Exporting data from PUDL into other platforms or interchange formats. sqlite Issues related to interacting with sqlite databases
#2433 by zaneselvans was closed Apr 19, 2023
Dagsterize EIA output tables dagster Issues related to our use of the Dagster orchestrator eia860 Anything having to do with EIA Form 860 eia923 Anything having to do with EIA Form 923 output Exporting data from PUDL into other platforms or interchange formats.
#2519 by zaneselvans was merged Apr 18, 2023 Loading…
5 of 8 tasks
Dagsterize EIA output tables dagster Issues related to our use of the Dagster orchestrator eia860 Anything having to do with EIA Form 860 eia923 Anything having to do with EIA Form 923 inframundo output Exporting data from PUDL into other platforms or interchange formats.
#2490 by e-belfer was closed Apr 11, 2023 Draft
4 of 6 tasks
Add funding and release notes URLs to the project.
#2507 by zaneselvans was merged Apr 11, 2023 Loading…
2 of 8 tasks
Create Dynamic PudlTabl methods dagster Issues related to our use of the Dagster orchestrator output Exporting data from PUDL into other platforms or interchange formats.
#2498 by zschira was merged Apr 8, 2023 Loading…
Convert pu_eia860 to SQL view asset dagster Issues related to our use of the Dagster orchestrator inframundo
#2339 by bendnorman was closed Apr 6, 2023
Create data structure for dagster SQL views dagster Issues related to our use of the Dagster orchestrator inframundo sqlite Issues related to interacting with sqlite databases
#2264 by bendnorman was closed Apr 6, 2023
Create simple SQL view assets dagster Issues related to our use of the Dagster orchestrator metadata Anything having to do with the content, formatting, or storage of metadata. Mostly datapackages. output Exporting data from PUDL into other platforms or interchange formats. sqlite Issues related to interacting with sqlite databases
#2445 by bendnorman was merged Apr 6, 2023 Loading…
8 tasks
Why did our integration tests get so slow? performance Make PUDL run faster! testing Writing tests, creating test data, automating testing, etc.
#2494 by zaneselvans was closed Apr 5, 2023
Merge dev into main after nightly build 2023-04-05
#2495 by zaneselvans was merged Apr 5, 2023 Loading…
2 of 8 tasks
dev -> main 2023-04-05
#2493 by zaneselvans was closed Apr 5, 2023 Draft
8 tasks
Use FERC XBRL Extractor 0.8.2 to allow pandas 2.0 dependencies Pull requests that update a dependency file ferc1 Anything having to do with FERC Form 1 xbrl Related to the FERC XBRL transition
#2492 by zaneselvans was merged Apr 4, 2023 Loading…
2 of 8 tasks
Parallelize tests performance Make PUDL run faster! testing Writing tests, creating test data, automating testing, etc.
#2432 by jdangerx was merged Apr 4, 2023 Loading…
Parallelize Dagster processing of EPA CEMS dagster Issues related to our use of the Dagster orchestrator epacems Integration and analysis of the EPA CEMS dataset. inframundo parquet Issues related to the Apache Parquet file format which we use for long tables. performance Make PUDL run faster!
#2376 by zaneselvans was closed Apr 4, 2023
Parallelize Dagster processing of EPA CEMS dagster Issues related to our use of the Dagster orchestrator epacems Integration and analysis of the EPA CEMS dataset. parquet Issues related to the Apache Parquet file format which we use for long tables. performance Make PUDL run faster!
#2472 by zschira was merged Apr 4, 2023 Loading…
Move project metadata & build specs from setup.py to pyproject.toml packaging Software packaging and distribution of PUDL via pypi, etc.
#2479 by zaneselvans was merged Apr 4, 2023 Loading…