-
-
Notifications
You must be signed in to change notification settings - Fork 119
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make it possible to pass another sources
dict to DataSource
#4003
Conversation
@@ -1035,20 +1035,22 @@ def to_rst( | |||
sys.stdout.write(rendered) | |||
|
|||
@classmethod | |||
def from_field_namespace(cls, x: str) -> list["DataSource"]: | |||
def from_field_namespace( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't get called in pudl-archiver
, so I could revert this change. Just figured for consistency it'd make sense to have the same behavior as below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think consistency here is good
src/pudl/metadata/classes.py
Outdated
@@ -1285,6 +1287,7 @@ class Resource(PudlMeta): | |||
"pudl", | |||
"nrelatb", | |||
"vcerare", | |||
"phmsagas", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops! This crossed over from another branch, I'll remove this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems like a reasonable/simple solution to avoid cluttering our main metadata collection
Overview
Addresses a lightweight version of catalyst-cooperative/pudl-archiver#499 in concert with catalyst-cooperative/pudl-archiver#506.
What problem does this address?
In the
pudl-archiver
repository, we want to make it possible to archive datasets that will never end up in the PUDL repository. Eventually, we'll want to fully separate these two repositories, which raises bigger questions about where the metadata should be defined, etc. But for now, we want to make it possible to add a dictionary ofNON_PUDL_SOURCES
to thepudl-archiver
repository, to be called in a method similar tofrom_pudl_metadata
infrictionless.py
.What did you change?
This method imports the
DataSource.from_id()
method from PUDL, which currently hardcodesSOURCES
into the method. In order to make it possible to provide an alternative dictionary of source metadata into this method, I have added asources
parameter to this method which defaults toSOURCES
, but can take a different SOURCE dictionary.Documentation
Make sure to update relevant aspects of the documentation.
Tasks
Testing
How did you make sure this worked? How can a reviewer verify this?
To-do list