Update to uv and airflow #129
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Updates to use the latest uv (v0.5.28) which changes some things in `pyproject.toml` - Update docs to use `uv run` where appropriate. - Update CI to use uv to run tests. - To simplify things just have one requirements.txt file that is used by Docker to install dependencies with `uv pip` - Changes to tests to accomodate changes in the openalex API
edsu
commented
Feb 5, 2025
| @@ -1,4 +1,4 @@ | |||
| FROM apache/airflow:2.9.3-python3.12 | |||
| FROM apache/airflow:2.10.4-python3.12 | |||
There was a problem hiding this comment.
Updating to the latest version of Airflow. I've run the harvest DAG locally but not in our "production" environment.
edsu
commented
Feb 5, 2025
| @@ -67,28 +67,10 @@ https://sul-rialto-dev.stanford.edu/authors?action=index&commit=Search&controlle | |||
| ### Set-up | |||
|
|
|||
| 1. Install `uv` for dependency management as described in [the uv docs](https://github.com/astral-sh/uv?tab=readme-ov-file#getting-started). | |||
There was a problem hiding this comment.
uv will create the virtual environment as needed when you uv run
edsu
commented
Feb 5, 2025
| @@ -92,7 +91,7 @@ def publications_from_dois(dois: list): | |||
| # TODO: do we need this to stay within 100,000 requests / day API quota? | |||
| time.sleep(1) | |||
|
|
|||
| doi_list = quote("|".join([doi for doi in doi_batch])) | |||
| doi_list = "|".join([doi for doi in doi_batch]) | |||
There was a problem hiding this comment.
The problem with & appearing in the DOI has now been fixed in pyalex:
edsu
commented
Feb 5, 2025
| assert len(pubs[0].keys()) == 53, "first publication has 53 columns" | ||
| assert len(pubs[1].keys()) == 53, "second publication has 53 columns" | ||
| assert len(pubs[0].keys()) == 54, "first publication has 54 columns" | ||
| assert len(pubs[1].keys()) == 54, "second publication has 54 columns" |
There was a problem hiding this comment.
There's a new column coming back from OpenAlex abstract_inverted_index_v3".
edsu
commented
Feb 5, 2025
|
|
||
| - name: Run tests | ||
| run: pytest | ||
| run: uv run pytest |
There was a problem hiding this comment.
uv installs dependencies if needed when you uv run
jmartin-sul
approved these changes
Feb 6, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update uv and Airflow
pyproject.toml.uv runwhere appropriate.uv pip