Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/multi run #23

Merged
merged 7 commits into from
Nov 15, 2021
Merged

Feature/multi run #23

merged 7 commits into from
Nov 15, 2021

Conversation

jbrown-xentity
Copy link
Contributor

@jbrown-xentity jbrown-xentity commented Nov 12, 2021

Related to GSA/data.gov#3527, optimization and cleanup. This adds 2 optimizations:

  • Allows to hit a read-only url for read api calls (defaults to same as api-url, which defaults to read-write catalog URL)
  • Sorts the duplicate identifier list by default, and adds a flag to reverse the sort. This allows to run 2 scripts in parallel, which cuts total run time by about 40% (not quite 50%, running 2 at the same time slows down queries a bit on the catalog).

It also simplifies the API call to get the duplicate identifiers.
Does some tiny cleanup for GSA/data.gov#2744

@robert-bryson robert-bryson merged commit aaab05a into main Nov 15, 2021
@robert-bryson robert-bryson deleted the feature/multi-run branch November 15, 2021 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants