Update FERC rate base tags with RMI guidance #3162

cmgosnell · 2023-12-15T14:25:58Z

Overview

Closes #2600 .

What problem does this address?
This pr integrates updates from the RMI review:

Tasks

Give feedback

updated rate base tags
updated aggregable plant_function tags
updated aggregable plant_status tags
add rate base category types
add income statement revenue requirement tags
update xbrl_factoid-specific utility_types (NOT in transform, as an aggregatable tag instead)
Options

What did you change?

Testing

How did you make sure this worked? How can a reviewer verify this?

To-do list

Give feedback

Make sure full ETL runs & make pytest-integration-full passes locally
Review the PR yourself and call out any questions or issues you have
For major data coverage & analysis changes, run data validation tests
If updating analyses or data processing functions: make sure to update or write data validation tests
Update the release notes: reference the PR and related issues.
Options

this results in a messy calculation table bc rn we generally expect that the parent and child fact have the same dimensions (with the expection of the subtotal calculations). so imma delete this and migrate this change to and aggregatable tag. but wanted to save just in case.

cmgosnell · 2023-12-22T17:18:18Z

src/pudl/output/ferc1.py

-    # Also, these tags may not be applicable to all exploded tables, but
-    # we need to pass in a dataframe with the right structure to all of the exploders,
-    # so we're just re-using this one for the moment.


now all the explosions have tags so bye bye note

cmgosnell · 2023-12-22T17:21:30Z

src/pudl/output/ferc1.py

-    # we need to pass in a dataframe with the right structure to all of the exploders,
-    # so we're just re-using this one for the moment.
-    rate_base_tags = _rate_base_tags(table_dimensions_ferc1=table_dimensions_ferc1)
+    rate_tags = _get_tags("xbrl_factoid_rate_base_tags.csv", table_dimensions_ferc1)


i decided to standardize the tags bc now there are a million so its either _get_tags which just grabs a csv and fills in the dimensions or the per-existing _aggregatable_dimension_tags which is now used for all three of our dimensions.

then instead of a bunch of little merges i used concat to squish em all together.

cmgosnell · 2023-12-22T17:23:31Z

src/pudl/package_data/ferc1/xbrl_factoid_plant_function_tags.csv

its hard to see the changes here because all of the lines changed bc the table_name's changed in the great rename. but the main this to know about all of these changes is that a) they came straight from rmi jon and their utility account expert's expert and b) they only initially caused one error with the explosions that i was very easily able to fix by adding some specified dimensions into the tags (i'll point that out in that csv)

cmgosnell · 2023-12-22T17:43:04Z

src/pudl/package_data/ferc1/xbrl_factoid_rate_base_category_tags.csv

+core_ferc1__yearly_utility_plant_summary_sched200,depreciation_utility_plant_in_service,other,,,net_utility_plant
+core_ferc1__yearly_utility_plant_summary_sched200,depreciation_utility_plant_in_service,gas,,,net_utility_plant
+core_ferc1__yearly_utility_plant_summary_sched200,depreciation_utility_plant_in_service,common,,,net_utility_plant
+core_ferc1__yearly_utility_plant_summary_sched200,depreciation_utility_plant_in_service,other3,,,net_utility_plant
+core_ferc1__yearly_utility_plant_summary_sched200,depreciation_utility_plant_in_service,other2,,,net_utility_plant


this is the only tag from @jrea-rmi that i needed to edit a little bit. the utility_type=="electric" version of this factoid is linked to other more granular facts that have their own rate_base_category that are more specific than net_utility_plant. This was causing the error Calculation forest nodes have conflicting tags: so i removed the generic category and labeled only the non-electric utility types.

cmgosnell · 2023-12-22T17:45:23Z

src/pudl/package_data/ferc1/xbrl_factoid_utility_type_tags.csv

i tried to add these guys into the transform but it broke our convention of the calculations having the same dimensions for the parent and child facts (other than the total-to-subtotal calcs). I could probably figure out how to integrate this into the calc components but this felt much simpler and better overall.

codecov · 2023-12-22T18:58:01Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (539a6f1) 92.6% compared to head (4f85ab8) 92.6%.
Report is 16 commits behind head on dev.

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3162   +/-   ##
=====================================
  Coverage   92.6%   92.6%           
=====================================
  Files        134     134           
  Lines      12611   12615    +4     
=====================================
+ Hits       11682   11686    +4     
  Misses       929     929

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

e-belfer

I have a clarifying question about the table names getting used here. Otherwise this looks straightforward and should be good to go, if it's working as expected on your end!

e-belfer · 2023-12-22T20:44:51Z

src/pudl/package_data/ferc1/xbrl_factoid_plant_function_tags.csv

-electric_operating_expenses_ferc1,customer_service_and_information_expenses,
-electric_operating_expenses_ferc1,sales_expenses,
-electric_operating_expenses_ferc1,administrative_and_general_expenses,general
+core_ferc1__yearly_plant_in_service_sched204,intangible_plant,intangible


Since we're using the table names with the schedules here (which differ from the table names in PUDL), is there a translation step I'm not seeing here?

we ended up landing on table names with the schedules in them!

its a bit of a divergence from the other tables but it ended up feeling like the easiest way to communicate to users which DBF or XBRL table the data originated from.

ah then I'm hopelessly out of date post rename!

e-belfer

Looks good to merge!

cmgosnell added 2 commits December 15, 2023 09:19

edit tags with rmi's updated tagging

3fc3901

add rate base categories

77e1b8c

cmgosnell self-assigned this Dec 15, 2023

cmgosnell added 6 commits December 18, 2023 16:26

update rate base categories with missing ones

bd253cd

Merge branch 'dev' into explode_tag_updates

7816219

rename tags to align with THE GREAT RENAME

bca2571

add in revenue requirement tags and clean up tag making

3bc7f27

transition the utility_type replacements to the output tags

4f85ab8

cmgosnell marked this pull request as ready for review December 22, 2023 17:17

cmgosnell commented Dec 22, 2023

View reviewed changes

cmgosnell requested a review from e-belfer December 22, 2023 17:57

e-belfer reviewed Dec 22, 2023

View reviewed changes

e-belfer self-requested a review December 22, 2023 21:30

e-belfer approved these changes Dec 22, 2023

View reviewed changes

Merge branch 'dev' into explode_tag_updates

9033fe8

cmgosnell enabled auto-merge December 22, 2023 21:31

cmgosnell merged commit 9fcb79a into dev Dec 23, 2023
13 checks passed

cmgosnell deleted the explode_tag_updates branch December 23, 2023 09:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update FERC rate base tags with RMI guidance #3162

Update FERC rate base tags with RMI guidance #3162

cmgosnell commented Dec 15, 2023 •

edited

Loading

Tasks

To-do list

cmgosnell Dec 22, 2023

cmgosnell Dec 22, 2023

cmgosnell Dec 22, 2023 •

edited

Loading

cmgosnell Dec 22, 2023

cmgosnell Dec 22, 2023

codecov bot commented Dec 22, 2023

e-belfer left a comment

e-belfer Dec 22, 2023

cmgosnell Dec 22, 2023

cmgosnell Dec 22, 2023

e-belfer Dec 22, 2023

e-belfer left a comment

Update FERC rate base tags with RMI guidance #3162

Update FERC rate base tags with RMI guidance #3162

Conversation

cmgosnell commented Dec 15, 2023 • edited Loading

Overview

Tasks

Testing

To-do list

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmgosnell Dec 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 22, 2023

Codecov Report

e-belfer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

e-belfer left a comment

Choose a reason for hiding this comment

cmgosnell commented Dec 15, 2023 •

edited

Loading

cmgosnell Dec 22, 2023 •

edited

Loading