Change SEC 10-K table schemas to fix FK errors and use quarterly naming.

[zaneselvans]

Overview

• Nightly builds failed with FK constraint errors in the new SEC 10-K tables.
• These are mostly due to the temporal resolution mismatch between the SEC 10-K tables (quarterly) and the core_eia860__scd_utilities table (annual).
• In addition, the new quarterly tables didn't follow our existing naming conventions, which indicate the temporal frequency in the name.
• However, there are a handful of FK constraint violations even when the FK constraint only includes utility_id_eia.

What did you change?

• Excluded the autogenerated FK constraint between these tables as we have for many of our monthly tables that reference utilities.
• Updated the names of the SEC 10K core tables to reflect their quarterly frequency.
• Dropped the 3 records with the unrecognized utility_id_eia

Documentation

Tasks

Preview Give feedback

1. Update the release notes to highlight the new SEC 10-K ownership data. Reference this PR, Add sec10k metadata directly in PUDL #4035 and Enable sec10k distribution #4026 and Sec distribution fix #4031

Testing

• Materialized the sec10k tables locally and ran pudl_check_fks and got the same failure as the nightly builds.
• Updated the DB schema and FK constraints, rematerialized, and ran pudl_check_fks again and got just 3 records violating the constraint.

To-do list

Preview Give feedback

1. Rename tables to include quarterly prefix.
2. Exclude autogenerated FK constraints with the annual utilities table.
3. Add a new alembic migration to reflect these DB changes.
4. Investigate and fix the remaining 3 FK constraint violations.
5. Make sure that pudl_check_fks succeeds after materializing the SEC 10-K tables before merging.
6. Review the PR yourself and call out any questions or issues you have.
7. Run the build-deploy-pudl GitHub Action manually before merging

[zaneselvans]

@zschira can you provide more extensive table descriptions for these new guys?

[zaneselvans]

It looks like the offending utility_id_eia is 3579 aka Cirro Group, Inc in Texas.

	report_date	filename_sec10k	company_id_sec10k	utility_id_eia	company_name_raw	location_of_incorporation	parent_company_central_index_key	files_sec10k
499327	2010-02-26 00:00:00	edgar/data/715957/0001193125-10-042883.txt	0000715957_cirro group incorporated_texas	3579	cirro group, inc	texas	0000715957	False
523722	2010-02-26 00:00:00	edgar/data/103682/0001193125-10-042883.txt	0000103682_cirro group incorporated_texas	3579	cirro group, inc	texas	0000103682	False
963038	2023-02-23 00:00:00	edgar/data/1013871/0001013871-23-000004.txt	0001013871_cirro group incorporated_texas	3579	cirro group, inc	texas	0001013871	False

[zaneselvans]

Probably this is a rogue utility_id_eia that only exists in the EIA-861 data, and was the only example of such a utility that ended up matching to an SEC company. It should go away if we remove the experimental EIA-861 ID harvesting upstream, as @katie-lamb said she intends to.

[zaneselvans]

Exclude these because they don't have the same (annual) time frequency as the entity table.

[zaneselvans]

Add time frequency prefix.

[zaneselvans]

Rename these to include time frequency prefix.

[zaneselvans]

Branch build seems to have succeeded last night! (with the exception of the Zenodo Sandbox flakeout, which is unrelated)

[zschira]

This looks good to me, thanks for stepping in with the foreign key fixes! I wonder if we should just merge @katie-lamb's metadata into this branch once we finish review, then merge everything into main? I guess it doesn't make too much difference if we get both PR's done today

[zaneselvans]

I think we might have more work to do on the documentation, and it'd be good to get the nightly build ETL passing with the quarterly updates in the works, so I say we merge this one and work on the docs / metadata independently.