520 write an archiver for doe low income energy affordability data lead #536

krivard · 2025-01-22T21:35:07Z

Overview

Closes #520 .

What problem does this address?

Lack of an archiver for DOE LEAD Tool data

What did you change in this PR?

Add optional argument to get_hyperlinks to accept headers to pass to session.get, and improve error messages
Add new archiver for DOE LEAD

See published production archive here: https://zenodo.org/records/14758685

Arguable design decisions

~~The base URL is the DOE page, which does not list past data releases even if they are still available on OIE.~~
- ~~Pros: We'll automatically get the next OIE data release when the DOE page points to a new one~~
- ~~Cons: No seamless way to archive past releases. There's only one past release atm, but still~~
Switched to using hard-coded DOIs and checking the DOE page for new releases (and if we find one, calling for help; this is expected to happen << annually)
The per-state zip files are not unpacked before adding them to the partition zip
- Pros: The partition zip contains 60 files even without unpacking; with unpacking there would be 8 + (52x8) = 424 files, which seems unwieldy
- Cons: Compression will be less efficient; needs an extra unzip step at point of use;
The logging is chatty. We can move most/all to DEBUG depending on group habit, but with a running time of 40m-2h, I needed some reassurance

Testing

How did you make sure this worked? How can a reviewer verify this?

mkdir -p /tmp/doelead
pudl_archiver --datasets doelead \
	--initialize \
	--summary-file doelead-summary.json \
	--depositor fsspec \
	--deposition-path /tmp/doelead/

To-do list

Tasks

Give feedback

Crawl resulting OIE link to fetch state-year links
Download state-year links and zip into year partitions
Return ResourceInfo
Update relevant documentation - like comments, docstrings, README, release notes, etc.
Review the PR yourself and call out any questions or issues you have
Options

… gets passed thru to session.get

Todo: - Crawl resulting oie link to fetch state-year links - Download state-year links and zip into year partitions - Return ResourceInfo

For more information, see https://pre-commit.ci

…bility-data-lead' of github.com:catalyst-cooperative/pudl-archiver into 520-write-an-archiver-for-doe-low-income-energy-affordability-data-lead

src/pudl_archiver/archivers/doelead.py

For more information, see https://pre-commit.ci

…rgy-affordability-data-lead

e-belfer

I didn't run this entirely but it did succeed in starting to download the data, so no problems there so far! I do have a request to reconfigure the way that you're downloading the data though (see below), and some other very minor requests re: docstrings to consider after that. The changes to the get_hyperlinks() method look completely good and reasonable to me!

The base URL is the DOE page, which does not list past data releases even if they are still available on OIE.
    Pros: We'll automatically get the next OIE data release when the DOE page points to a new one
    Cons: No seamless way to archive past releases. There's only one past release atm, but still

See my comment in the code. I'd suggest manually archiving the data at each DOI, then comparing the link you find on the main page to the known last data link and making sure that nothing has changed. That way we get the best of both worlds - we grab all the data from stable locations, and get to check it against the latest release (if there is one). The method you already have for finding the submission link should work just fine for the latter part!

The per-state zip files are not unpacked before adding them to the partition zip
    Pros: The partition zip contains 60 files even without unpacking; with unpacking there would be 8 + (52x8) = 424 files, which seems unwieldy
    Cons: Compression will be less efficient; needs an extra unzip step at point of use;

I think you made the right call here! If we need to reformat the data down the line we certainly can cross that bridge then.

The logging is chatty. We can move most/all to DEBUG depending on group habit, but with a running time of 40m-2h, I needed some reassurance

I err towards this as well, it's no problem for me.

src/pudl_archiver/archivers/doelead.py

…age for new releases, add better docstrings

For more information, see https://pre-commit.ci

…bility-data-lead' of github.com:catalyst-cooperative/pudl-archiver into 520-write-an-archiver-for-doe-low-income-energy-affordability-data-lead

…rgy-affordability-data-lead

e-belfer

This ran as expected for me, I just ran out of time to make a sandbox archive to inspect it with my slow internet connection. All is looking good so far!

…rgy-affordability-data-lead

e-belfer · 2025-01-28T20:31:41Z

src/pudl_archiver/archivers/classes.py

@@ -253,8 +260,8 @@ async def get_hyperlinks(
        # Warn if no links are found
        if not hyperlinks:
            self.logger.warning(
-                f"The archiver couldn't find any hyperlinks that match {filter_pattern}."
-                f"Make sure your filter_pattern is correct or if the structure of the {url} page changed."
+                f"The archiver couldn't find any hyperlinks{('that match' + filter_pattern) if filter_pattern else ''}."


This change may be causing some unexpected problems? https://github.com/catalyst-cooperative/pudl-archiver/actions/runs/13018457111

e-belfer · 2025-01-28T20:34:49Z

src/pudl_archiver/archivers/doelead.py

+)
+from pudl_archiver.frictionless import ZipLayout
+
+TOOL_URL = "https://www.energy.gov/scep/low-income-energy-affordability-data-lead-tool"


This site has gone down, so we should just refer to the DOIs

e-belfer

@krivard Note that the main tool page is no longer online, so we'll just want to archive the DOIs (and quickly, if possible!)

…rgy-affordability-data-lead

cmgosnell

Looks good to me! sad that the actual DOE page went down 😢 ...

…rgy-affordability-data-lead

e-belfer · 2025-01-29T17:02:40Z

.github/workflows/run-archiver.yml

@@ -6,7 +6,7 @@ on:
    inputs:
      datasets:
        description: 'Comma-separated list of datasets to archive (e.g., "ferc2","ferc6").'
-        default: '"doeiraec","eia176","eia191","eia757a","eia860","eia860m","eia861","eia923","eia930","eiaaeo","eiawater","eia_bulk_elec","epacamd_eia","epacems","epapcap","ferc1","ferc2","ferc6","ferc60","ferc714","gridpathratoolkit","mshamines","nrelatb","phmsagas","usgsuspvdb","vcerare"'
+        default: '"doeiraec","doelead","eia176","eia191","eia757a","eia860","eia860m","eia861","eia923","eia930","eiaaeo","eiamecs","eiawater","eia_bulk_elec","epacamd_eia","epacems","epapcap","ferc1","ferc2","ferc6","ferc60","ferc714","gridpathratoolkit","mshamines","nrelatb","phmsagas","usgsuspvdb","vcerare"'


Sneaking in eiamecs which didn't land in #516.

Changes addressed!

Kathryn Mazaitis added 2 commits January 22, 2025 16:31

[wip] feat: permit get_hyperlinks to accept a 'headers' argument that…

9504deb

… gets passed thru to session.get

[wip] feat: add new archiver for DOE LEAD

7fe9ce4

Todo: - Crawl resulting oie link to fetch state-year links - Download state-year links and zip into year partitions - Return ResourceInfo

krivard linked an issue Jan 22, 2025 that may be closed by this pull request

Write an archiver for DOE Low Income Energy Affordability Data (LEAD) #520

Closed

3 tasks

pre-commit-ci bot and others added 6 commits January 22, 2025 21:35

[pre-commit.ci] auto fixes from pre-commit.com hooks

c5ba88e

For more information, see https://pre-commit.ci

the rest of the owl

fe0d44e

resolve conflicts

89937dd

[pre-commit.ci] auto fixes from pre-commit.com hooks

1170476

For more information, see https://pre-commit.ci

[fix] Add missing import

51667b2

Merge branch '520-write-an-archiver-for-doe-low-income-energy-afforda…

c6e1245

…bility-data-lead' of github.com:catalyst-cooperative/pudl-archiver into 520-write-an-archiver-for-doe-low-income-energy-affordability-data-lead

krivard commented Jan 23, 2025

View reviewed changes

src/pudl_archiver/archivers/doelead.py Outdated Show resolved Hide resolved

krivard and others added 3 commits January 23, 2025 16:22

[docs] Add more detail to doelead docstring

020b3cd

[pre-commit.ci] auto fixes from pre-commit.com hooks

3583b4e

For more information, see https://pre-commit.ci

Merge branch 'main' into 520-write-an-archiver-for-doe-low-income-ene…

cb77f65

…rgy-affordability-data-lead

krivard marked this pull request as ready for review January 23, 2025 22:56

krivard requested review from zschira and e-belfer January 23, 2025 22:56

e-belfer requested changes Jan 24, 2025

View reviewed changes

Kathryn Mazaitis and others added 5 commits January 24, 2025 16:58

[fix] switch to hard-coded DOIs for known releases, check LEAD Tool p…

de0ba16

…age for new releases, add better docstrings

[pre-commit.ci] auto fixes from pre-commit.com hooks

96b064d

For more information, see https://pre-commit.ci

[fix] missing refactor in fstring

e46f6d9

Merge branch '520-write-an-archiver-for-doe-low-income-energy-afforda…

7fd864b

…bility-data-lead' of github.com:catalyst-cooperative/pudl-archiver into 520-write-an-archiver-for-doe-low-income-energy-affordability-data-lead

Merge branch 'main' into 520-write-an-archiver-for-doe-low-income-ene…

15cee78

…rgy-affordability-data-lead

krivard requested a review from e-belfer January 24, 2025 22:15

e-belfer assigned krivard Jan 27, 2025

e-belfer added the new-data label Jan 27, 2025

e-belfer reviewed Jan 27, 2025

View reviewed changes

Merge branch 'main' into 520-write-an-archiver-for-doe-low-income-ene…

2a3ef26

…rgy-affordability-data-lead

e-belfer reviewed Jan 28, 2025

View reviewed changes

e-belfer previously requested changes Jan 28, 2025

View reviewed changes

e-belfer and others added 6 commits January 28, 2025 15:52

Drop site that no longer exists, fix class

ac73e65

Download methodology PDFs

9a2a149

Add PDF metadata to archive, add placeholder DOI

99ca4f4

Restore entire archiving workflow

283db91

Update production DOI

1f73afb

Merge branch 'main' into 520-write-an-archiver-for-doe-low-income-ene…

9a98f7d

…rgy-affordability-data-lead

e-belfer requested review from e-belfer and cmgosnell January 28, 2025 23:35

cmgosnell approved these changes Jan 29, 2025

View reviewed changes

Merge branch 'main' into 520-write-an-archiver-for-doe-low-income-ene…

aded9cf

…rgy-affordability-data-lead

e-belfer enabled auto-merge January 29, 2025 16:57

e-belfer disabled auto-merge January 29, 2025 16:58

e-belfer added 3 commits January 29, 2025 12:01

Add to GHA

f87e9f7

Oops also MECS

285479d

Fix bad merge resolution

b9cbbe5

e-belfer reviewed Jan 29, 2025

View reviewed changes

e-belfer enabled auto-merge January 29, 2025 17:03

e-belfer merged commit b2ac313 into main Jan 29, 2025
3 checks passed

e-belfer deleted the 520-write-an-archiver-for-doe-low-income-energy-affordability-data-lead branch January 29, 2025 17:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

520 write an archiver for doe low income energy affordability data lead #536

520 write an archiver for doe low income energy affordability data lead #536

krivard commented Jan 22, 2025 •

edited by e-belfer

Loading

Tasks

e-belfer left a comment •

edited

Loading

e-belfer left a comment

e-belfer Jan 28, 2025

e-belfer Jan 28, 2025

e-belfer left a comment

cmgosnell left a comment

e-belfer Jan 29, 2025

520 write an archiver for doe low income energy affordability data lead #536

520 write an archiver for doe low income energy affordability data lead #536

Conversation

krivard commented Jan 22, 2025 • edited by e-belfer Loading

Overview

Arguable design decisions

Testing

To-do list

Tasks

e-belfer left a comment • edited Loading

Choose a reason for hiding this comment

e-belfer left a comment

Choose a reason for hiding this comment

e-belfer Jan 28, 2025

Choose a reason for hiding this comment

e-belfer Jan 28, 2025

Choose a reason for hiding this comment

e-belfer left a comment

Choose a reason for hiding this comment

cmgosnell left a comment

Choose a reason for hiding this comment

e-belfer Jan 29, 2025

Choose a reason for hiding this comment

krivard commented Jan 22, 2025 •

edited by e-belfer

Loading

e-belfer left a comment •

edited

Loading