Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release v0.5.9 -- performance improvements #202

Merged
merged 65 commits into from
Nov 25, 2024
Merged

Release v0.5.9 -- performance improvements #202

merged 65 commits into from
Nov 25, 2024

Conversation

jhkennedy
Copy link
Contributor

No description provided.

jhkennedy and others added 30 commits November 20, 2024 09:17
@jhkennedy
add units
a8eb93c
@jtherrmann
Merge pull request #191 from ASFHyP3/log-message
e99c54d 
add units to log message
@jtherrmann
allow lambda to query hyp3 jobs table
51038d1
@AndrewPlayer3
query dynamo for jobs by status type
90d8e8c
@AndrewPlayer3
ruff
1ae7919
@AndrewPlayer3
updated changelog
ae55f76
@AndrewPlayer3
removed unnecessary comment
9354079
@jtherrmann
Merge pull request #194 from ASFHyP3/query-table-directly
bf876f5 
Query HyP3 jobs table directly
@jtherrmann
Merge branch 'develop' into query-jobs-by-status
880fee0
@jtherrmann
pass jobs table name to lambda
03fe49d
@jtherrmann
finish implementing query_jobs_by_status_code
5d75621
@jhkennedy
handle missing microseconds gracefully
80a2802
@jhkennedy
whitespace
ec45408
@jtherrmann
Update its_live_monitoring/src/main.py
20eb58f
@AndrewPlayer3
add conftest
db6d847
@AndrewPlayer3
Merge branch 'jth-query-jobs-by-status' of https://github.com/asfhyp3…
835e202 
…/its-live-monitoring into jth-query-jobs-by-status
@AndrewPlayer3
added test_query_jobs_by_status_code
9d3a512
@AndrewPlayer3
removed unnecessary params
d9b1497
@jtherrmann
add moto to requirements
8b4d31c
@jtherrmann
remove duplicate moto
3792a52
@AndrewPlayer3
added extra jobs to ensure filtering is working
054168b
@AndrewPlayer3
uncommented
e6342bc
@AndrewPlayer3
Merge branch 'jth-query-jobs-by-status' of https://github.com/asfhyp3…
83b8296 
…/its-live-monitoring into jth-query-jobs-by-status
@jtherrmann
update test_deduplicate_hyp3_pairs
600e6cf
@jtherrmann
Merge branch 'jth-query-jobs-by-status' of github.com:ASFHyP3/its-liv…
795075e 
…e-monitoring into jth-query-jobs-by-status
@jtherrmann
Merge pull request #197 from ASFHyP3/jth-query-jobs-by-status
c1a6c02 
finish implementing `query_jobs_by_status_code`
@jtherrmann
remove todo
13051a6
@AndrewPlayer3
consistant times and extra name case
e834175
@jtherrmann
fix ruff warning
a9d6377
@AndrewPlayer3
fixes
c85f3f6
@jhkennedy
Copy link
Contributor Author

Alright! I've confirmed via too many integration tests that:

  • the deduplication of pending/running jobs via the HyP3 DynamoDB table is working
  • the deduplication of already published (succeeded) jobs via the appropriate S3 bucket is working

So we should be good to release!

@jhkennedy jhkennedy marked this pull request as ready for review November 23, 2024 07:51
@jhkennedy jhkennedy requested a review from a team November 23, 2024 07:51
@jhkennedy jhkennedy linked an issue Nov 23, 2024 that may be closed by this pull request
hyp3_sdk.exceptions.ServerError when de-duplicating new jobs #119
Closed
dependabot bot and others added 4 commits November 25, 2024 19:55
@dependabot
Bump actions/setup-python from 3 to 5
1c8de56 
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 3 to 5.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@v3...v5)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
@dependabot
Bump aws-actions/configure-aws-credentials from 1 to 4
807acd7 
Bumps [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials) from 1 to 4.
- [Release notes](https://github.com/aws-actions/configure-aws-credentials/releases)
- [Changelog](https://github.com/aws-actions/configure-aws-credentials/blob/main/CHANGELOG.md)
- [Commits](aws-actions/configure-aws-credentials@v1...v4)

---
updated-dependencies:
- dependency-name: aws-actions/configure-aws-credentials
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
@dependabot
Bump cfn-lint from 1.19.0 to 1.20.0
2e8e74d 
Bumps [cfn-lint](https://github.com/aws-cloudformation/cfn-lint) from 1.19.0 to 1.20.0.
- [Release notes](https://github.com/aws-cloudformation/cfn-lint/releases)
- [Changelog](https://github.com/aws-cloudformation/cfn-lint/blob/main/CHANGELOG.md)
- [Commits](aws-cloudformation/cfn-lint@v1.19.0...v1.20.0)

---
updated-dependencies:
- dependency-name: cfn-lint
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
@jtherrmann
Merge pull request #207 from ASFHyP3/dependabot/pip/cfn-lint-1.20.0
21f2e82 
Bump cfn-lint from 1.19.0 to 1.20.0
dependabot bot and others added 2 commits November 25, 2024 20:07
@dependabot
Bump ruff from 0.7.4 to 0.8.0
0d8bf09 
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.7.4 to 0.8.0.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.7.4...0.8.0)

---
updated-dependencies:
- dependency-name: ruff
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
@jtherrmann
Merge pull request #205 from ASFHyP3/dependabot/github_actions/action…
2a84603 
…s/setup-python-5

Bump actions/setup-python from 3 to 5
@jtherrmann
Merge pull request #206 from ASFHyP3/dependabot/github_actions/aws-ac…
f05d923 
…tions/configure-aws-credentials-4

Bump aws-actions/configure-aws-credentials from 1 to 4
jtherrmann
jtherrmann reviewed Nov 25, 2024
View reviewed changes
its_live_monitoring/src/main.py
@@ -85,7 +83,7 @@ def get_key(tile_prefixes: Iterable[str], reference: str, secondary: str) -> str
for tile_prefix in tile_prefixes:
prefix = f'{tile_prefix}/{reference}_X_{secondary}'
response = s3.list_objects_v2(
Bucket='its-live-data',
Bucket=os.environ.get('PUBLISH_BUCKET', 'its-live-data'),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a default value here, or could we just require that PUBLISH_BUCKET be defined?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't decide; we provide a default value in other places (e.g., here) so I matched that.

It might be better overall to not provide defaults (or default to test values instead of production...) and maybe add a .env file

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'd err on the side of not providing defaults, but won't hold up the PR for it since we already set the current precedent elsewhere in the code.

@jhkennedy jhkennedy merged commit 1e4d51a into main Nov 25, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jtherrmann jtherrmann jtherrmann approved these changes

Assignees
No one assigned
Labels
patch Bump the patch version number of this project
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

hyp3_sdk.exceptions.ServerError when de-duplicating new jobs
3 participants
@jhkennedy @jtherrmann @AndrewPlayer3