Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aggregator and launcher for Flexpart #1

Open
wants to merge 37 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
4b2edf1
copied over aviso config and first commit with launch flexpart
Oct 3, 2024
53ad009
define S3 config and flexprep time settings in .env
Oct 3, 2024
8f3ca40
rm config files transfered to .env
Oct 3, 2024
062e010
change env var names with pydantic mechanism
Oct 9, 2024
3dfbc4c
script in python instead of bash
Oct 9, 2024
95bb6e4
extract launch flexpart from aggregator
Oct 10, 2024
7ba581b
adopt mch cookiecutter template
ninaburg Oct 14, 2024
58eee9d
adapt actual code to blueprint
ninaburg Oct 14, 2024
5b0f514
add boto3
ninaburg Oct 15, 2024
991e618
.env file to be included in git
ninaburg Oct 15, 2024
5ff0efd
Fixes
ninaburg Oct 15, 2024
d87eb1e
update flexprep image
ninaburg Oct 15, 2024
8b07374
update poetry.lock
ninaburg Oct 15, 2024
a9fd246
remove mchpy package and copy files in project
ninaburg Oct 17, 2024
9e8fe83
fix container cleaned up after stopped
ninaburg Oct 17, 2024
f767a29
fix env var
ninaburg Oct 17, 2024
c1151e2
unittests
ninaburg Oct 23, 2024
d4a13d0
[squash commit] adopt mch blueprint and clean for quality tests
ninaburg Oct 23, 2024
cc1059b
refactor .env
ninaburg Oct 30, 2024
2fd1206
improve env. var and fix paths
ninaburg Oct 30, 2024
cafa117
poetry packages update
ninaburg Oct 30, 2024
ec4cfb9
fix exe poetry location and clean up env. var
ninaburg Nov 1, 2024
1c3ab5c
comments: call python script directly, no subprocess
ninaburg Nov 5, 2024
a7c080e
comment:rm pre-commit, use mypy from mch template
ninaburg Nov 6, 2024
d174d51
comments README.rst
ninaburg Nov 6, 2024
99c402e
comments: rm mchbuild from pyprojecttoml and update README
ninaburg Nov 6, 2024
ca4fd31
comments: top comment on mch python common files
ninaburg Nov 6, 2024
2053934
comments: settings description and typo
ninaburg Nov 6, 2024
66c9cdb
comments: add typing in the parsed args
ninaburg Nov 6, 2024
836382d
comments: use logging.getLogger(__name__)
ninaburg Nov 6, 2024
30e663f
comments: extract login_ecr()
ninaburg Nov 6, 2024
2d0dd20
comments: descriptive env. var.
ninaburg Nov 6, 2024
34afdee
comments:renamed module lead_time_aggregator.py
ninaburg Nov 6, 2024
b66f191
descriptive function names
ninaburg Nov 6, 2024
fc413e8
comments clean up
ninaburg Nov 6, 2024
d84223a
comments: improve understability of the aggregator module
ninaburg Nov 8, 2024
98e7dd5
fix tests
ninaburg Nov 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 101 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/
junit*
test_reports/
security_reports/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
doc/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# celery beat schedule file
celerybeat-schedule

# SageMath parsed files
*.sage.py


# virtualenv
.venv
venv/
ENV/

# IDE project settings
.editorconfig
.spyproject
.idea/

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
53 changes: 53 additions & 0 deletions .mch-ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# To test with your local Python environment, run 'mchbuild build test'.
# To use a container image, set for example '-s pythonImageName=3.11'.
default:
- build:
- install:
- pythonInstall:
- docs:
- pythonDocs:
pullAlways: false
- test:
- unit:
- pythonTest:
pullAlways: false
- pythonCoverage:
pullAlways: false
- lint:
- pythonLint:
pullAlways: false
- pythonTypeChecker:
pullAlways: false
- verify:
- securityScan:
- securityReport:
format: [ 'html', 'sbom', 'table', 'print' ]
severity: [ 'CRITICAL', 'HIGH' ]
target: file://poetry.lock
qualityGate:
threshold: 5
criticalFactor: 5
highFactor: 1
- publishSbom:
- securityPublishSbom:
- unitWithoutCoverage:
# Set pythonImageName to run with image of non-default Python version
pythonTest:
cacheDeps: false
withCoverage: false
- deploy:
- docs:
openshiftPublishDocs:
docSrc: doc/_build/
docType: python
- pypi:
pythonPublishPackage:
setPackageVersion: true
pythonImageName: '3.12'
pullAlways: false


variables:
project: 'flex-container-orchestrator'
ocpHostNameForEnv:
prod: api.cp.meteoswiss.ch:6443
94 changes: 94 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: check-ast
- id: check-case-conflict
- id: check-docstring-first
- id: check-symlinks
- id: check-toml
- id: check-yaml
- id: debug-statements
- id: end-of-file-fixer
- id: trailing-whitespace
- repo: local
hooks:
- id: forbidden-files-git-orig
name: find git merge backup files
language: fail
entry: "Found git merge backup file(s): review and remove them"
files: "\\.orig$"
- repo: local
hooks:
- id: rstcheck
name: rstcheck
description: Check REST files for correctness
language: system
entry: rstcheck
types: [rst]
- repo: local
hooks:
- id: codespell
name: codespell
description: Check for spelling errors
language: system
entry: codespell
- repo: local
hooks:
- id: black
name: black
description: Format Python code
language: system
entry: black
types_or: [python, pyi]
- repo: local
hooks:
- id: isort
name: isort
description: Group and sort Python imports
language: system
entry: isort
types_or: [python, pyi, cython]
- repo: local
hooks:
- id: pydocstyle
name: pydocstyle
description: Check docstrings in Python code for compliance with conventions
language: system
entry: pydocstyle
types: [python]
files: ^src/
# SR It would be handy to abort if one linter fails b/c the same error often
# SR triggers multiple linters. That way, if flake8 fails, pylint and mypy
# SR (which in large code bases can take some time to run) could be skipped
# SR before fixing the error. Unfortunately, pre-commit only provides the global
# SR option fail_fast, which would abort even after the small fixers and
# SR formatters changed something. A potential solution could be to write a
# SR small bash script run-linters.sh that runs flake8, pylint and run-mypy.sh
# SR in that order and aborts on error.
# TODO: There is significant work involved in getting pylint going. This can be its separate task
# - repo: local
# hooks:
# - id: pylint
# name: pylint
# description: Check Python code for correctness, consistency and adherence to best practices
# language: system
# entry: pylint
# types: [python]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove the comment here? Also fine to remove the whole of pre-commit since linting/mypy is checked in CICD ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea to align how the linting is done with how it's done in CI/CD. I removed the dependencies for pre-commit

- repo: local
hooks:
- id: flake8
name: flake8
description: Check Python code for correctness, consistency and adherence to best practices
language: system
entry: flake8
types: [python]
- repo: local
hooks:
- id: poetry-check
name: poetry-check
description: run poetry check to validate config
entry: poetry check
language: python
pass_filenames: false
files: ^(.*/)?(poetry\.lock|pyproject\.toml)$
1 change: 1 addition & 0 deletions .python-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
3.12
8 changes: 8 additions & 0 deletions HISTORY.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
=======
History
=======

0.1.0 (2024-10-25)
------------------

* First release on PyPI.
Loading