[WIP] 6054 Integrate azureml for autorunner training #6088

peterhessey · 2023-03-01T17:08:56Z

Finishes first task in #6054 .

Description

Introduces changes necessary to make it possible to train the Auto3dSeg AutoRunner training in AzureML with no/minimal impact to the existing user experience. Requires changes made in this (currently-open) PR for the hi-ml-azure package.

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

Signed-off-by: Peter Hessey <[email protected]>

…runner-training

for more information, see https://pre-commit.ci

…s://github.com/peterhessey/MONAI into 6054-integrate-azureml-for-autorunner-training

I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 76f3737 I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 6288cdc I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 9d0b98f I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: dc8101b I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 2b88637 I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 136ec66 I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 49dbb04 I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: c4b8d4e I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: ce16775 I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: f5ea9d6 I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 218cb8b I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: f547930 I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 9a1e7c7 I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 4c024bc I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: e6e3456 I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 2ed7ab0 Signed-off-by: Peter Hessey <[email protected]>

Signed-off-by: Peter Hessey <[email protected]>

for more information, see https://pre-commit.ci

…runner-training

Signed-off-by: Peter Hessey <[email protected]>

I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 354fde1 I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 3a60f6d I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: a800423 Signed-off-by: Peter Hessey <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Peter Hessey <[email protected]>

…s://github.com/peterhessey/MONAI into 6054-integrate-azureml-for-autorunner-training

Signed-off-by: Peter Hessey <[email protected]>

I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: a43bd9b Signed-off-by: Peter Hessey <[email protected]>

Signed-off-by: Peter Hessey <[email protected]>

I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 0967115 Signed-off-by: Peter Hessey <[email protected]>

…ml-for-autorunner-training Signed-off-by: Peter Hessey <[email protected]>

Signed-off-by: Peter Hessey <[email protected]>

…ml-for-autorunner-training Signed-off-by: Peter Hessey <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Peter Hessey <[email protected]>

lapellaniz · 2023-04-12T12:51:09Z

environment-azureml.yml

+      - matplotlib==3.7.1
+      - mlflow==2.2.2
+      - mlflow-skinny==2.2.2
+      - git+https://github.com/peterhessey/MONAI.git@6054-integrate-azureml-for-autorunner-training


This needs to be reviewed and changed back to MONAI repo during PR approval

…ml-for-autorunner-training Signed-off-by: Peter Hessey <[email protected]>

Signed-off-by: Peter Hessey <[email protected]>

peterhessey and others added 22 commits February 20, 2023 11:04

➕ Add minimum requirements for triggering AzureML

76f3737

Set AzureML to SDK v1 only

c762d9e

Signed-off-by: Peter Hessey <[email protected]>

🚧 Set up AzureML environment file

6288cdc

✨ Enable job submission to run correct command

9d0b98f

📌 Add hi-ml to AzureML env

dc8101b

📌 AzureML environment working

2b88637

🚧 Enable configuring training using task.yaml

136ec66

🚧 Pass in azureml args as dict

49dbb04

🚧 Fix AzureML training arg parsing

c4b8d4e

🐛 🚧 Fix training to work with AzureML dataroot

ce16775

🚧 Attempt to run on Prostate data

f5ea9d6

🔇 Remove old logs + configs

218cb8b

🙈 Update gitignore for azureml_configs

f547930

🐛 Fix creating local work_dir when running in Azure

9a1e7c7

🙈 🐛 Update gitignore and fix work dir creation

4c024bc

🚨 Pass linting tests

e6e3456

Merge branch 'Project-MONAI:dev' into 6054-integrate-azureml-for-auto…

b7cf860

…runner-training

[pre-commit.ci] auto fixes from pre-commit.com hooks

d35d6e7

for more information, see https://pre-commit.ci

🐛 Update optional imports

2ed7ab0

Merge branch '6054-integrate-azureml-for-autorunner-training' of http…

1d40819

…s://github.com/peterhessey/MONAI into 6054-integrate-azureml-for-autorunner-training

Fix mypy

91d7320

Signed-off-by: Peter Hessey <[email protected]>

peterhessey changed the title ~~6054 integrate azureml for autorunner training - WORK IN PROGRESS~~ [WIP] 6054 Integrate azureml for autorunner training Mar 2, 2023

peterhessey and others added 7 commits March 6, 2023 09:29

📌 Pin packages to remove Azure warnings

244cf81

✅ 🚧 Add azureml auto3dseg test for heart data

d7d2af8

🚧 Start building AzureML test using synthetic data

dcfee01

[pre-commit.ci] auto fixes from pre-commit.com hooks

4145a78

for more information, see https://pre-commit.ci

Merge branch 'Project-MONAI:dev' into 6054-integrate-azureml-for-auto…

aeec9f4

…runner-training

🎨 Restructure auto3dseg synth data functions

e3eacf4

Signed-off-by: Peter Hessey <[email protected]>

✅ Add synthetic data test

9c0cc4d

Signed-off-by: Peter Hessey <[email protected]>

peterhessey and others added 26 commits March 23, 2023 12:01

✅ Pass Auto3dSeg AzureML test with subclass

354fde1

📝 Add docstrings

3a60f6d

📝 Add more docstrings

a800423

Add typing and docstrings

662c6b4

Signed-off-by: Peter Hessey <[email protected]>

Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training

34f51b1

[pre-commit.ci] auto fixes from pre-commit.com hooks

45e482f

for more information, see https://pre-commit.ci

Styling fixes

838abb7

Signed-off-by: Peter Hessey <[email protected]>

Merge branch '6054-integrate-azureml-for-autorunner-training' of http…

364f311

…s://github.com/peterhessey/MONAI into 6054-integrate-azureml-for-autorunner-training

Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training

b7292f4

Remove num_fold arg

a073b76

Signed-off-by: Peter Hessey <[email protected]>

Remove need for --azureml flag

229a48e

Signed-off-by: Peter Hessey <[email protected]>

🚧 Reduce AzureML environment complexity

a43bd9b

Update env again

fe49bee

Signed-off-by: Peter Hessey <[email protected]>

DCO Remediation Commit for Peter Hessey <[email protected]>

55d2aaf

I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: a43bd9b Signed-off-by: Peter Hessey <[email protected]>

Add fire to AzureML environment

2b642a5

Signed-off-by: Peter Hessey <[email protected]>

Remove num_fold arg from AMLAutoRunner

d66c65f

Signed-off-by: Peter Hessey <[email protected]>

🚧 🐛 Bugfix and env update

0967115

DCO Remediation Commit for Peter Hessey <[email protected]>

345c191

I, Peter Hessey <[email protected]>, hereby add my Signed-off-by to this commit: 0967115 Signed-off-by: Peter Hessey <[email protected]>

Merge remote-tracking branch 'upstream/dev' into 6054-integrate-azure…

b8644ff

…ml-for-autorunner-training Signed-off-by: Peter Hessey <[email protected]>

mypy fixes

ff5c4f4

Signed-off-by: Peter Hessey <[email protected]>

Merge remote-tracking branch 'upstream/dev' into 6054-integrate-azure…

4c275b3

…ml-for-autorunner-training Signed-off-by: Peter Hessey <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

510b141

for more information, see https://pre-commit.ci

Black fixes

643d713

Signed-off-by: Peter Hessey <[email protected]>

Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training

d81bf97

Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training

86adbdf

lapellaniz reviewed Apr 12, 2023

View reviewed changes

peterhessey added 2 commits April 13, 2023 10:05

Merge remote-tracking branch 'upstream/dev' into 6054-integrate-azure…

2e9c6a1

…ml-for-autorunner-training Signed-off-by: Peter Hessey <[email protected]>

Minor fixes

4833e13

Signed-off-by: Peter Hessey <[email protected]>

wyli mentioned this pull request Jun 28, 2023

Refactor multi-node running command into dedicated functions #6623

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] 6054 Integrate azureml for autorunner training #6088

[WIP] 6054 Integrate azureml for autorunner training #6088

peterhessey commented Mar 1, 2023 •

edited

Loading

lapellaniz Apr 12, 2023

[WIP] 6054 Integrate azureml for autorunner training #6088

Are you sure you want to change the base?

[WIP] 6054 Integrate azureml for autorunner training #6088

Conversation

peterhessey commented Mar 1, 2023 • edited Loading

Description

Types of changes

lapellaniz Apr 12, 2023

Choose a reason for hiding this comment

peterhessey commented Mar 1, 2023 •

edited

Loading