-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Aggregator and launcher for Flexpart #1
Open
ninaburg
wants to merge
37
commits into
main
Choose a base branch
from
trigger_aggregation_flexpartjob
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+3,299
−10
Open
Changes from 22 commits
Commits
Show all changes
37 commits
Select commit
Hold shift + click to select a range
4b2edf1
copied over aviso config and first commit with launch flexpart
53ad009
define S3 config and flexprep time settings in .env
8f3ca40
rm config files transfered to .env
062e010
change env var names with pydantic mechanism
3dfbc4c
script in python instead of bash
95bb6e4
extract launch flexpart from aggregator
7ba581b
adopt mch cookiecutter template
ninaburg 58eee9d
adapt actual code to blueprint
ninaburg 5b0f514
add boto3
ninaburg 991e618
.env file to be included in git
ninaburg 5ff0efd
Fixes
ninaburg d87eb1e
update flexprep image
ninaburg 8b07374
update poetry.lock
ninaburg a9fd246
remove mchpy package and copy files in project
ninaburg 9e8fe83
fix container cleaned up after stopped
ninaburg f767a29
fix env var
ninaburg c1151e2
unittests
ninaburg d4a13d0
[squash commit] adopt mch blueprint and clean for quality tests
ninaburg cc1059b
refactor .env
ninaburg 2fd1206
improve env. var and fix paths
ninaburg cafa117
poetry packages update
ninaburg ec4cfb9
fix exe poetry location and clean up env. var
ninaburg 1c3ab5c
comments: call python script directly, no subprocess
ninaburg a7c080e
comment:rm pre-commit, use mypy from mch template
ninaburg d174d51
comments README.rst
ninaburg 99c402e
comments: rm mchbuild from pyprojecttoml and update README
ninaburg ca4fd31
comments: top comment on mch python common files
ninaburg 2053934
comments: settings description and typo
ninaburg 66c9cdb
comments: add typing in the parsed args
ninaburg 836382d
comments: use logging.getLogger(__name__)
ninaburg 30e663f
comments: extract login_ecr()
ninaburg 2d0dd20
comments: descriptive env. var.
ninaburg 34afdee
comments:renamed module lead_time_aggregator.py
ninaburg b66f191
descriptive function names
ninaburg fc413e8
comments clean up
ninaburg d84223a
comments: improve understability of the aggregator module
ninaburg 98e7dd5
fix tests
ninaburg File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,101 @@ | ||
# Byte-compiled / optimized / DLL files | ||
__pycache__/ | ||
*.py[cod] | ||
*$py.class | ||
|
||
# C extensions | ||
*.so | ||
|
||
# Distribution / packaging | ||
.Python | ||
env/ | ||
build/ | ||
develop-eggs/ | ||
dist/ | ||
downloads/ | ||
eggs/ | ||
.eggs/ | ||
lib/ | ||
lib64/ | ||
parts/ | ||
sdist/ | ||
var/ | ||
wheels/ | ||
*.egg-info/ | ||
.installed.cfg | ||
*.egg | ||
|
||
# PyInstaller | ||
# Usually these files are written by a python script from a template | ||
# before PyInstaller builds the exe, so as to inject date/other infos into it. | ||
*.manifest | ||
*.spec | ||
|
||
# Installer logs | ||
pip-log.txt | ||
pip-delete-this-directory.txt | ||
|
||
# Unit test / coverage reports | ||
htmlcov/ | ||
.tox/ | ||
.coverage | ||
.coverage.* | ||
.cache | ||
nosetests.xml | ||
coverage.xml | ||
*.cover | ||
.hypothesis/ | ||
.pytest_cache/ | ||
junit* | ||
test_reports/ | ||
security_reports/ | ||
|
||
# Translations | ||
*.mo | ||
*.pot | ||
|
||
# Django stuff: | ||
*.log | ||
local_settings.py | ||
|
||
# Flask stuff: | ||
instance/ | ||
.webassets-cache | ||
|
||
# Scrapy stuff: | ||
.scrapy | ||
|
||
# Sphinx documentation | ||
doc/_build/ | ||
|
||
# PyBuilder | ||
target/ | ||
|
||
# Jupyter Notebook | ||
.ipynb_checkpoints | ||
|
||
# celery beat schedule file | ||
celerybeat-schedule | ||
|
||
# SageMath parsed files | ||
*.sage.py | ||
|
||
|
||
# virtualenv | ||
.venv | ||
venv/ | ||
ENV/ | ||
|
||
# IDE project settings | ||
.editorconfig | ||
.spyproject | ||
.idea/ | ||
|
||
# Rope project settings | ||
.ropeproject | ||
|
||
# mkdocs documentation | ||
/site | ||
|
||
# mypy | ||
.mypy_cache/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
# To test with your local Python environment, run 'mchbuild build test'. | ||
# To use a container image, set for example '-s pythonImageName=3.11'. | ||
default: | ||
- build: | ||
- install: | ||
- pythonInstall: | ||
- docs: | ||
- pythonDocs: | ||
pullAlways: false | ||
- test: | ||
- unit: | ||
- pythonTest: | ||
pullAlways: false | ||
- pythonCoverage: | ||
pullAlways: false | ||
- lint: | ||
- pythonLint: | ||
pullAlways: false | ||
- pythonTypeChecker: | ||
pullAlways: false | ||
- verify: | ||
- securityScan: | ||
- securityReport: | ||
format: [ 'html', 'sbom', 'table', 'print' ] | ||
severity: [ 'CRITICAL', 'HIGH' ] | ||
target: file://poetry.lock | ||
qualityGate: | ||
threshold: 5 | ||
criticalFactor: 5 | ||
highFactor: 1 | ||
- publishSbom: | ||
- securityPublishSbom: | ||
- unitWithoutCoverage: | ||
# Set pythonImageName to run with image of non-default Python version | ||
pythonTest: | ||
cacheDeps: false | ||
withCoverage: false | ||
- deploy: | ||
- docs: | ||
openshiftPublishDocs: | ||
docSrc: doc/_build/ | ||
docType: python | ||
- pypi: | ||
pythonPublishPackage: | ||
setPackageVersion: true | ||
pythonImageName: '3.12' | ||
pullAlways: false | ||
|
||
|
||
variables: | ||
project: 'flex-container-orchestrator' | ||
ocpHostNameForEnv: | ||
prod: api.cp.meteoswiss.ch:6443 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,94 @@ | ||
repos: | ||
- repo: https://github.com/pre-commit/pre-commit-hooks | ||
rev: v4.4.0 | ||
hooks: | ||
- id: check-ast | ||
- id: check-case-conflict | ||
- id: check-docstring-first | ||
- id: check-symlinks | ||
- id: check-toml | ||
- id: check-yaml | ||
- id: debug-statements | ||
- id: end-of-file-fixer | ||
- id: trailing-whitespace | ||
- repo: local | ||
hooks: | ||
- id: forbidden-files-git-orig | ||
name: find git merge backup files | ||
language: fail | ||
entry: "Found git merge backup file(s): review and remove them" | ||
files: "\\.orig$" | ||
- repo: local | ||
hooks: | ||
- id: rstcheck | ||
name: rstcheck | ||
description: Check REST files for correctness | ||
language: system | ||
entry: rstcheck | ||
types: [rst] | ||
- repo: local | ||
hooks: | ||
- id: codespell | ||
name: codespell | ||
description: Check for spelling errors | ||
language: system | ||
entry: codespell | ||
- repo: local | ||
hooks: | ||
- id: black | ||
name: black | ||
description: Format Python code | ||
language: system | ||
entry: black | ||
types_or: [python, pyi] | ||
- repo: local | ||
hooks: | ||
- id: isort | ||
name: isort | ||
description: Group and sort Python imports | ||
language: system | ||
entry: isort | ||
types_or: [python, pyi, cython] | ||
- repo: local | ||
hooks: | ||
- id: pydocstyle | ||
name: pydocstyle | ||
description: Check docstrings in Python code for compliance with conventions | ||
language: system | ||
entry: pydocstyle | ||
types: [python] | ||
files: ^src/ | ||
# SR It would be handy to abort if one linter fails b/c the same error often | ||
# SR triggers multiple linters. That way, if flake8 fails, pylint and mypy | ||
# SR (which in large code bases can take some time to run) could be skipped | ||
# SR before fixing the error. Unfortunately, pre-commit only provides the global | ||
# SR option fail_fast, which would abort even after the small fixers and | ||
# SR formatters changed something. A potential solution could be to write a | ||
# SR small bash script run-linters.sh that runs flake8, pylint and run-mypy.sh | ||
# SR in that order and aborts on error. | ||
# TODO: There is significant work involved in getting pylint going. This can be its separate task | ||
# - repo: local | ||
# hooks: | ||
# - id: pylint | ||
# name: pylint | ||
# description: Check Python code for correctness, consistency and adherence to best practices | ||
# language: system | ||
# entry: pylint | ||
# types: [python] | ||
- repo: local | ||
hooks: | ||
- id: flake8 | ||
name: flake8 | ||
description: Check Python code for correctness, consistency and adherence to best practices | ||
language: system | ||
entry: flake8 | ||
types: [python] | ||
- repo: local | ||
hooks: | ||
- id: poetry-check | ||
name: poetry-check | ||
description: run poetry check to validate config | ||
entry: poetry check | ||
language: python | ||
pass_filenames: false | ||
files: ^(.*/)?(poetry\.lock|pyproject\.toml)$ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
3.12 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
======= | ||
History | ||
======= | ||
|
||
0.1.0 (2024-10-25) | ||
------------------ | ||
|
||
* First release on PyPI. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we remove the comment here? Also fine to remove the whole of pre-commit since linting/mypy is checked in CICD ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea to align how the linting is done with how it's done in CI/CD. I removed the dependencies for pre-commit