-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] 6054 Integrate azureml for autorunner training #6088
Draft
peterhessey
wants to merge
76
commits into
Project-MONAI:dev
Choose a base branch
from
peterhessey:6054-integrate-azureml-for-autorunner-training
base: dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 74 commits
Commits
Show all changes
76 commits
Select commit
Hold shift + click to select a range
76f3737
➕ Add minimum requirements for triggering AzureML
peterhessey c762d9e
Set AzureML to SDK v1 only
peterhessey 6288cdc
🚧 Set up AzureML environment file
peterhessey 9d0b98f
✨ Enable job submission to run correct command
peterhessey dc8101b
📌 Add hi-ml to AzureML env
peterhessey 2b88637
📌 AzureML environment working
peterhessey 136ec66
🚧 Enable configuring training using task.yaml
peterhessey 49dbb04
🚧 Pass in azureml args as dict
peterhessey c4b8d4e
🚧 Fix AzureML training arg parsing
peterhessey ce16775
🐛 🚧 Fix training to work with AzureML dataroot
peterhessey f5ea9d6
🚧 Attempt to run on Prostate data
peterhessey 218cb8b
🔇 Remove old logs + configs
peterhessey f547930
🙈 Update gitignore for azureml_configs
peterhessey 9a1e7c7
🐛 Fix creating local work_dir when running in Azure
peterhessey 4c024bc
🙈 🐛 Update gitignore and fix work dir creation
peterhessey e6e3456
🚨 Pass linting tests
peterhessey b7cf860
Merge branch 'Project-MONAI:dev' into 6054-integrate-azureml-for-auto…
peterhessey d35d6e7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 2ed7ab0
🐛 Update optional imports
peterhessey 1d40819
Merge branch '6054-integrate-azureml-for-autorunner-training' of http…
peterhessey a3342e4
DCO Remediation Commit for Peter Hessey <[email protected]>
peterhessey 91d7320
Fix mypy
peterhessey 244cf81
📌 Pin packages to remove Azure warnings
peterhessey d7d2af8
✅ 🚧 Add azureml auto3dseg test for heart data
peterhessey dcfee01
🚧 Start building AzureML test using synthetic data
peterhessey 4145a78
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] aeec9f4
Merge branch 'Project-MONAI:dev' into 6054-integrate-azureml-for-auto…
peterhessey e3eacf4
🎨 Restructure auto3dseg synth data functions
peterhessey 9c0cc4d
✅ Add synthetic data test
peterhessey 0eb2cd5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 301dc7d
📌 Update azureml env and remove configs
peterhessey 53db45b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 1103209
Merge branch 'Project-MONAI:dev' into 6054-integrate-azureml-for-auto…
peterhessey 1d9c2f2
Update setup.py
peterhessey 0f39701
Merge branch '6054-integrate-azureml-for-autorunner-training' of http…
peterhessey 8a83655
💄 Fix styling
peterhessey b9e5bc7
💄 Fix styling
peterhessey 18237ae
DCO Remediation Commit for Peter Hessey <[email protected]>
peterhessey b63142c
Fix typing
peterhessey 3a098fa
Fix typing
peterhessey 4a8408b
Merge branch '6054-integrate-azureml-for-autorunner-training' of http…
peterhessey 5e9fa1c
Remove git merge content
peterhessey a53b32b
Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training
peterhessey 26bc937
Merge branch '6054-integrate-azureml-for-autorunner-training' of http…
peterhessey 8f8e63f
Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training
peterhessey 13456a2
Merge branch '6054-integrate-azureml-for-autorunner-training' of http…
peterhessey c96b66a
Add hi-ml-azure to dev reqs
peterhessey ac9ec8c
Test creating separate AML class
peterhessey 354fde1
✅ Pass Auto3dSeg AzureML test with subclass
peterhessey 3a60f6d
📝 Add docstrings
peterhessey a800423
📝 Add more docstrings
peterhessey 662c6b4
Add typing and docstrings
peterhessey 468f893
DCO Remediation Commit for Peter Hessey <[email protected]>
peterhessey 34f51b1
Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training
peterhessey 45e482f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 838abb7
Styling fixes
peterhessey 364f311
Merge branch '6054-integrate-azureml-for-autorunner-training' of http…
peterhessey b7292f4
Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training
peterhessey a073b76
Remove num_fold arg
peterhessey 229a48e
Remove need for --azureml flag
peterhessey a43bd9b
🚧 Reduce AzureML environment complexity
peterhessey fe49bee
Update env again
peterhessey 55d2aaf
DCO Remediation Commit for Peter Hessey <[email protected]>
peterhessey 2b642a5
Add fire to AzureML environment
peterhessey d66c65f
Remove num_fold arg from AMLAutoRunner
peterhessey 0967115
🚧 🐛 Bugfix and env update
peterhessey 345c191
DCO Remediation Commit for Peter Hessey <[email protected]>
peterhessey b8644ff
Merge remote-tracking branch 'upstream/dev' into 6054-integrate-azure…
peterhessey ff5c4f4
mypy fixes
peterhessey 4c275b3
Merge remote-tracking branch 'upstream/dev' into 6054-integrate-azure…
peterhessey 510b141
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 643d713
Black fixes
peterhessey d81bf97
Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training
peterhessey 86adbdf
Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training
peterhessey 2e9c6a1
Merge remote-tracking branch 'upstream/dev' into 6054-integrate-azure…
peterhessey 4833e13
Minor fixes
peterhessey File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| tests/ | ||
| docs/ | ||
| monai/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,241 @@ | ||
| name: monai-azureml-new | ||
| channels: | ||
| - pytorch | ||
| - nvidia | ||
| - defaults | ||
| dependencies: | ||
| - _libgcc_mutex=0.1=main | ||
| - _openmp_mutex=5.1=1_gnu | ||
| - blas=1.0=mkl | ||
| - brotlipy=0.7.0=py38h27cfd23_1003 | ||
| - bzip2=1.0.8=h7b6447c_0 | ||
| - ca-certificates=2023.01.10=h06a4308_0 | ||
| - certifi=2022.12.7=py38h06a4308_0 | ||
| - cffi=1.15.1=py38h5eee18b_3 | ||
| - charset-normalizer=2.0.4=pyhd3eb1b0_0 | ||
| - cryptography=39.0.1=py38h9ce1e76_0 | ||
| - cuda-cudart=11.7.99=0 | ||
| - cuda-cupti=11.7.101=0 | ||
| - cuda-libraries=11.7.1=0 | ||
| - cuda-nvrtc=11.7.99=0 | ||
| - cuda-nvtx=11.7.91=0 | ||
| - cuda-runtime=11.7.1=0 | ||
| - ffmpeg=4.3=hf484d3e_0 | ||
| - filelock=3.9.0=py38h06a4308_0 | ||
| - flit-core=3.8.0=py38h06a4308_0 | ||
| - freetype=2.12.1=h4a9f257_0 | ||
| - giflib=5.2.1=h5eee18b_3 | ||
| - gmp=6.2.1=h295c915_3 | ||
| - gmpy2=2.1.2=py38heeb90bb_0 | ||
| - gnutls=3.6.15=he1e5248_0 | ||
| - idna=3.4=py38h06a4308_0 | ||
| - intel-openmp=2021.4.0=h06a4308_3561 | ||
| - jinja2=3.1.2=py38h06a4308_0 | ||
| - jpeg=9e=h5eee18b_1 | ||
| - lame=3.100=h7b6447c_0 | ||
| - lcms2=2.12=h3be6417_0 | ||
| - ld_impl_linux-64=2.38=h1181459_1 | ||
| - lerc=3.0=h295c915_0 | ||
| - libcublas=11.10.3.66=0 | ||
| - libcufft=10.7.2.124=h4fbf590_0 | ||
| - libcufile=1.6.0.25=0 | ||
| - libcurand=10.3.2.56=0 | ||
| - libcusolver=11.4.0.1=0 | ||
| - libcusparse=11.7.4.91=0 | ||
| - libdeflate=1.17=h5eee18b_0 | ||
| - libffi=3.4.2=h6a678d5_6 | ||
| - libgcc-ng=11.2.0=h1234567_1 | ||
| - libgomp=11.2.0=h1234567_1 | ||
| - libiconv=1.16=h7f8727e_2 | ||
| - libidn2=2.3.2=h7f8727e_0 | ||
| - libnpp=11.7.4.75=0 | ||
| - libnvjpeg=11.8.0.2=0 | ||
| - libpng=1.6.39=h5eee18b_0 | ||
| - libstdcxx-ng=11.2.0=h1234567_1 | ||
| - libtasn1=4.19.0=h5eee18b_0 | ||
| - libtiff=4.5.0=h6a678d5_2 | ||
| - libunistring=0.9.10=h27cfd23_0 | ||
| - libwebp=1.2.4=h11a3e52_1 | ||
| - libwebp-base=1.2.4=h5eee18b_1 | ||
| - lz4-c=1.9.4=h6a678d5_0 | ||
| - markupsafe=2.1.1=py38h7f8727e_0 | ||
| - mkl=2021.4.0=h06a4308_640 | ||
| - mkl-service=2.4.0=py38h7f8727e_0 | ||
| - mkl_fft=1.3.1=py38hd3c417c_0 | ||
| - mkl_random=1.2.2=py38h51133e4_0 | ||
| - mpc=1.1.0=h10f8cd9_1 | ||
| - mpfr=4.0.2=hb69a4c5_1 | ||
| - mpmath=1.2.1=py38h06a4308_0 | ||
| - ncurses=6.4=h6a678d5_0 | ||
| - nettle=3.7.3=hbbd107a_1 | ||
| - networkx=2.8.4=py38h06a4308_1 | ||
| - numpy=1.23.5=py38h14f4228_0 | ||
| - numpy-base=1.23.5=py38h31eccc5_0 | ||
| - openh264=2.1.1=h4ff587b_0 | ||
| - openssl=1.1.1t=h7f8727e_0 | ||
| - pillow=9.4.0=py38h6a678d5_0 | ||
| - pip=23.0.1=py38h06a4308_0 | ||
| - pycparser=2.21=pyhd3eb1b0_0 | ||
| - pyopenssl=23.0.0=py38h06a4308_0 | ||
| - pysocks=1.7.1=py38h06a4308_0 | ||
| - python=3.8.16 | ||
| - pytorch=2.0.0=py3.8_cuda11.7_cudnn8.5.0_0 | ||
| - pytorch-cuda=11.7=h778d358_3 | ||
| - pytorch-mutex=1.0=cuda | ||
| - readline=8.2=h5eee18b_0 | ||
| - requests=2.28.1=py38h06a4308_1 | ||
| - setuptools=65.6.3=py38h06a4308_0 | ||
| - six=1.16.0=pyhd3eb1b0_1 | ||
| - sqlite=3.41.1=h5eee18b_0 | ||
| - sympy=1.11.1=py38h06a4308_0 | ||
| - tk=8.6.12=h1ccaba5_0 | ||
| - torchaudio=2.0.0=py38_cu117 | ||
| - torchtriton=2.0.0=py38 | ||
| - torchvision=0.15.0=py38_cu117 | ||
| - typing_extensions=4.4.0=py38h06a4308_0 | ||
| - urllib3=1.26.15=py38h06a4308_0 | ||
| - wheel=0.38.4=py38h06a4308_0 | ||
| - xz=5.2.10=h5eee18b_1 | ||
| - zlib=1.2.13=h5eee18b_0 | ||
| - zstd=1.5.4=hc292b87_0 | ||
| - pip: | ||
| - absl-py==1.4.0 | ||
| - adal==1.2.7 | ||
| - alembic==1.10.3 | ||
| - applicationinsights==0.11.10 | ||
| - argcomplete==2.1.2 | ||
| - attrs==22.2.0 | ||
| - azure-ai-ml==1.5.0 | ||
| - azure-common==1.1.28 | ||
| - azure-core==1.26.4 | ||
| - azure-graphrbac==0.61.1 | ||
| - azure-identity==1.12.0 | ||
| - azure-mgmt-authorization==3.0.0 | ||
| - azure-mgmt-containerregistry==10.1.0 | ||
| - azure-mgmt-core==1.4.0 | ||
| - azure-mgmt-keyvault==10.2.1 | ||
| - azure-mgmt-resource==21.2.1 | ||
| - azure-mgmt-storage==20.1.0 | ||
| - azure-storage-blob==12.10.0 | ||
| - azure-storage-file-datalake==12.6.0 | ||
| - azure-storage-file-share==12.11.1 | ||
| - azureml-core==1.49.0 | ||
| - azureml-dataprep==4.9.6 | ||
| - azureml-dataprep-native==38.0.0 | ||
| - azureml-dataprep-rslex==2.16.4 | ||
| - azureml-dataset-runtime==1.49.0 | ||
| - azureml-mlflow==1.49.0 | ||
| - azureml-telemetry==1.49.0 | ||
| - azureml-tensorboard==1.49.0 | ||
| - azureml-train-core==1.49.0.post1 | ||
| - azureml-train-restclients-hyperdrive==1.49.0 | ||
| - backports-tempfile==1.0 | ||
| - backports-weakref==1.0.post1 | ||
| - bcrypt==4.0.1 | ||
| - cachetools==5.3.0 | ||
| - click==8.1.3 | ||
| - cloudpickle==2.2.1 | ||
| - colorama==0.4.6 | ||
| - conda-merge==0.2.0 | ||
| - contextlib2==21.6.0 | ||
| - contourpy==1.0.7 | ||
| - cycler==0.11.0 | ||
| - databricks-cli==0.17.6 | ||
| - distro==1.8.0 | ||
| - docker==6.0.1 | ||
| - dotnetcore2==3.1.23 | ||
| - einops==0.6.0 | ||
| - entrypoints==0.4 | ||
| - fire==0.5.0 | ||
| - flask==2.2.3 | ||
| - fonttools==4.39.3 | ||
| - fusepy==3.0.1 | ||
| - gitdb==4.0.10 | ||
| - gitpython==3.1.31 | ||
| - google-api-core==2.11.0 | ||
| - google-auth==2.17.2 | ||
| - google-auth-oauthlib==1.0.0 | ||
| - googleapis-common-protos==1.59.0 | ||
| - greenlet==2.0.2 | ||
| - grpcio==1.53.0 | ||
| - gunicorn==20.1.0 | ||
| - hi-ml-azure==0.2.19 | ||
| - humanfriendly==10.0 | ||
| - importlib-metadata==6.3.0 | ||
| - importlib-resources==5.12.0 | ||
| - isodate==0.6.1 | ||
| - itsdangerous==2.1.2 | ||
| - jeepney==0.8.0 | ||
| - jmespath==1.0.1 | ||
| - joblib==1.2.0 | ||
| - jsonpickle==2.2.0 | ||
| - jsonschema==4.17.3 | ||
| - kiwisolver==1.4.4 | ||
| - knack==0.10.1 | ||
| - llvmlite==0.39.1 | ||
| - mako==1.2.4 | ||
| - markdown==3.4.3 | ||
| - marshmallow==3.19.0 | ||
| - matplotlib==3.7.1 | ||
| - mlflow==2.2.2 | ||
| - mlflow-skinny==2.2.2 | ||
| - git+https://github.com/peterhessey/MONAI.git@6054-integrate-azureml-for-autorunner-training | ||
| - msal==1.21.0 | ||
| - msal-extensions==1.0.0 | ||
| - msrest==0.7.1 | ||
| - msrestazure==0.6.4 | ||
| - ndg-httpsclient==0.5.1 | ||
| - nibabel==5.1.0 | ||
| - numba==0.56.4 | ||
| - oauthlib==3.2.2 | ||
| - opencensus==0.11.2 | ||
| - opencensus-context==0.1.3 | ||
| - opencensus-ext-azure==1.1.9 | ||
| - packaging==21.3 | ||
| - pandas==2.0.0 | ||
| - param==1.13.0 | ||
| - paramiko==2.12.0 | ||
| - pathspec==0.11.1 | ||
| - pkginfo==1.9.6 | ||
| - pkgutil-resolve-name==1.3.10 | ||
| - portalocker==2.7.0 | ||
| - protobuf==3.20.3 | ||
| - psutil==5.9.4 | ||
| - pyarrow==9.0.0 | ||
| - pyasn1==0.4.8 | ||
| - pyasn1-modules==0.2.8 | ||
| - pydash==5.1.2 | ||
| - pygments==2.14.0 | ||
| - pyjwt==2.6.0 | ||
| - pynacl==1.5.0 | ||
| - pyparsing==3.0.9 | ||
| - pyrsistent==0.19.3 | ||
| - python-dateutil==2.8.2 | ||
| - pytz==2022.7.1 | ||
| - pyyaml==6.0 | ||
| - querystring-parser==1.2.4 | ||
| - requests-oauthlib==1.3.1 | ||
| - rsa==4.9 | ||
| - ruamel-yaml==0.17.21 | ||
| - ruamel-yaml-clib==0.2.7 | ||
| - scikit-learn==1.2.2 | ||
| - scipy==1.10.1 | ||
| - secretstorage==3.3.3 | ||
| - shap==0.41.0 | ||
| - slicer==0.0.7 | ||
| - smmap==5.0.0 | ||
| - sqlalchemy==2.0.9 | ||
| - sqlparse==0.4.3 | ||
| - strictyaml==1.7.3 | ||
| - tabulate==0.9.0 | ||
| - tensorboard==2.12.1 | ||
| - tensorboard-data-server==0.7.0 | ||
| - tensorboard-plugin-wit==1.8.1 | ||
| - termcolor==2.2.0 | ||
| - threadpoolctl==3.1.0 | ||
| - tqdm==4.65.0 | ||
| - tzdata==2023.3 | ||
| - websocket-client==1.5.1 | ||
| - werkzeug==2.2.3 | ||
| - zipp==3.15.0 | ||
| prefix: /anaconda/envs/monai-azureml-new | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,5 +10,6 @@ dependencies: | |
| - torchvision | ||
| - pytorch-cuda=11.6 | ||
| - pip | ||
| - python>=3.8 | ||
| - pip: | ||
| - -r requirements-dev.txt | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs to be reviewed and changed back to MONAI repo during PR approval