Archiver for EIA RECS #534
Conversation
| # housing characteristics | ||
| { | ||
| "base_url": "https://www.eia.gov/consumption/residential/data", | ||
| "php_extension": "index.php?view=characteristics", |
There was a problem hiding this comment.
The RECS webpage is a little bit tricky. It has tabs containing different sequences of datafiles (and PDFs, etc.) that can't be reached directly from the base_url. Instead we need to tack on these strings to the end of the url, which are different for different tabs.
There was a problem hiding this comment.
Might be worth defining the shape of this dictionary as a dataclass, but definitely not a blocking concern.
| "prefix": "state-ce", | ||
| "pattern": re.compile(r"ce(\d)\.(\d{1,2})\.(.*)\.xlsx"), | ||
| }, | ||
| # microdata |
There was a problem hiding this comment.
I'm skipping the microdata for now. Happy to come back and rewrite everything to be more general once I get a better sense from folks who have a sense of how deep we want to go down this rabbit hole.
There was a problem hiding this comment.
Sounds good - we can revisit later. I'm unaware of the context of this rabbit hole - assuming you all had a discussion during the hackathon? @cmgosnell
| zip_path = self.download_directory / f"eia-recs-{year}.zip" | ||
| data_paths_in_archive = set() | ||
| # Loop through different categories of data (all .xlsx) | ||
| for pattern_dict in LINK_PATTERNS: |
There was a problem hiding this comment.
We call get_hyperlinks multiple times: once for each tab of data.
| # We've gotta do a bit of wrangling to get the output filename | ||
| # to match the url somewhat | ||
| n_groups = len(match.groups()) | ||
| output_filename = f"eia-recs-{year}-{pattern_dict['prefix']}" |
There was a problem hiding this comment.
What follows is kinda ugly wrangling to set the output filename appropriately. It would probably be better to specify directly in the dictionary of LINK_PATTERNS above, instead of weird if statements catching the behavior of the regexes in LINK_PATTERNS.
There was a problem hiding this comment.
Hmm, from a cursory reading, this might do something similar-enough so as to be still human-readable? Otherwise defining some helper functions like format_housing_characteristics_filename and referencing them in the LINK_PATTERNS seems nice.
output_filename += "-".join(g for g in match.groups() if g).replace(" ", "_")
| # We've gotta do a bit of wrangling to get the output filename | ||
| # to match the url somewhat | ||
| n_groups = len(match.groups()) | ||
| output_filename = f"eia-recs-{year}-{pattern_dict['prefix']}" |
There was a problem hiding this comment.
Hmm, from a cursory reading, this might do something similar-enough so as to be still human-readable? Otherwise defining some helper functions like format_housing_characteristics_filename and referencing them in the LINK_PATTERNS seems nice.
output_filename += "-".join(g for g in match.groups() if g).replace(" ", "_")| zip_path = self.download_directory / f"eia-recs-{year}.zip" | ||
| data_paths_in_archive = set() | ||
| # Loop through different categories of data (all .xlsx) | ||
| for pattern_dict in LINK_PATTERNS: |
| "prefix": "state-ce", | ||
| "pattern": re.compile(r"ce(\d)\.(\d{1,2})\.(.*)\.xlsx"), | ||
| }, | ||
| # microdata |
There was a problem hiding this comment.
Sounds good - we can revisit later. I'm unaware of the context of this rabbit hole - assuming you all had a discussion during the hackathon? @cmgosnell
| # housing characteristics | ||
| { | ||
| "base_url": "https://www.eia.gov/consumption/residential/data", | ||
| "php_extension": "index.php?view=characteristics", |
There was a problem hiding this comment.
Might be worth defining the shape of this dictionary as a dataclass, but definitely not a blocking concern.
2015 methodology required some changes to allow for downloading html files.
|
Discussed on Slack with Nilay and I'm taking this over the finish line. Draft record (currently including 2020, 2015, 2009) on Zenodo |
Overview
Closes #519.
What problem does this address?
Implements archiver for EIA RECS.
What did you change in this PR?
Added archival of some of the 2020 .xlsx files.
Testing
How did you make sure this worked? How can a reviewer verify this?
pudl_archiver --datasets eiarecs \ --initialize \ --summary-file eiarecs-summary.json \ --depositor fsspec \ --deposition-path /tmp/eia-recs/To-do list
Tasks