Calisphere etl #714
Merged
Calisphere etl #714
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
8 tasks
bibliotechy
approved these changes
Jan 29, 2024
| @@ -56,7 +56,8 @@ def parse_enrichment_url(enrichment_url): | |||
|
|
|||
|
|
|||
| def run_enrichments(records, collection, enrichment_set, page_filename): | |||
| for enrichment_url in collection.get(enrichment_set, []): | |||
| enrichment_urls = collection.get(enrichment_set) or [] | |||
There was a problem hiding this comment.
You've done this in several places, replacing collection.get(some_value, []) with collection.get(some_value) or []. Is there a difference in behavior between those two? It isn't obvious to me what that is. No issue with it, just wondering.
There was a problem hiding this comment.
@amywieliczka @bibliotechy so yeah, it turns out that if the key exists and the value is explicitly None then this is what happens:
Python 3.9.0 (default, May 23 2023, 15:28:21)
[Clang 13.0.0 (clang-1300.0.27.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> mydict = {"foo": None}
>>> for x in mydict.get("foo", []):
... print(x)
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not iterable
Whereas:
>>> mydict.get("foo") or []
[]
amywieliczka
approved these changes
Jan 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The mapper is ready for review.
I see what you mean about the solr API returning a final empty page using nextCursorMark. This causes problems when running the
harvest_collectionDAG where the mapper fans out by page. I can't figure out what the heck is going on with Solr.Since the solr index is static and this is transitional functionality that we'll retire after we cutover to rikolti, I went ahead and added a bit of a hacky workaround to the fetcher's
increment()method, which reports "finished" if we've fetched the number of docs that we're expecting. (See this commit: 2fe62d0).