Reworking nnunetv2 predictor

[JupiLogy]

Fixes #7013 .

Description

nnunetv2 version 2.2 introduced breaking changes, so now predict_from_raw_data is no longer a function. Following the changes between their v2.1 and v2.2 I tried to update the nnunetv2 runner.

Haven't documented the changes, not sure if it's required?

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Specific tests passed locally by running python -m tests.test_integration_nnunetv2_runner.

[JupiLogy]

BTW I ran the test that was causing the issue

***All results:***
nnUNetTrainer_1epoch__nnUNetPlans__3d_fullres: 0.6849226238622849

*Best*: nnUNetTrainer_1epoch__nnUNetPlans__3d_fullres: 0.6849226238622849

***Determining postprocessing for best model/ensemble***
Killed

I didn't kill it but I dunno what the outcome was expected to be. Either way the import error is not there any more :D Let me know if you want any more logs to be sure about the performance or whatever. Last time I tried to run the full coverage tests, though, my PC crashed so I'd rather not try that again (sorry)!!

[JupiLogy]

Also I've now signed both the commits. So I'm not sure why it's not passing the DCO.

[wyli]

/build

[wyli]

thanks for lookint into the issue, this currently generates an error AttributeError: 'nnUNetPredictor' object has no attribute 'initialise_from_trained_model_folder'. Did you mean: 'initialize_from_trained_model_folder'?

I think you can test within a docker container (docker run --ipc=host --net=host --gpus all -ti --rm projectmonai/monai):

cd /tmp
git clone --depth 1 --branch 7013nnunetv2-predict --single-branch https://github.com/JupiLogy/MONAI.git
python -m pip install git+https://github.com/julien-blanchon/hiddenlayer.git
python -m pip install nnunetv2
cd MONAI/
python -m tests.test_integration_nnunetv2_runner

[JupiLogy]

I'm not sure how to make this PR a draft again...

But anyway, Note to self sort of, here's the latest error message.

======================================================================
ERROR: test_nnunetv2runner (__main__.TestnnUNetV2Runner.test_nnunetv2runner)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/me/MONAI/tests/test_integration_nnunetv2_runner.py", line 88, in test_nnunetv2runner
    runner.find_best_configuration(configs="3d_fullres")
  File "/home/me/MONAI/monai/apps/nnunet/nnunetv2_runner.py", line 760, in find_best_configuration
    find_best_configuration(
  File "/home/me/.conda/envs/monai/lib/python3.11/site-packages/nnunetv2/evaluation/find_best_configuration.py", line 91, in find_best_configuration
    allowed_trained_models = filter_available_models(deepcopy(allowed_trained_models), dataset_name_or_id)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.conda/envs/monai/lib/python3.11/site-packages/nnunetv2/evaluation/find_best_configuration.py", line 43, in filter_available_models
    raise RuntimeError(f"Trained model {trained_model} does not have an output folder. "
RuntimeError: Trained model {'plans': 'nnUNetPlans', 'configuration': '3d_fullres', 'trainer': 'nnUNetTrainer_1epoch'} does not have an output folder. Expected: ./work_dir/nnUNet_trained_models/Dataset001_dataroot/nnUNetTrainer_1epoch__nnUNetPlans__3d_fullres. Please run the training for this model! (don't forget the --npz flag if you want to ensemble multiple configurations)

----------------------------------------------------------------------

[JupiLogy]

Seems like it's creating a nnunetv2 folder, but looking for an nnunet folder...

[JupiLogy]

I've been delving into the nnunetv2 code all day and it seems to me like it's not actually putting any logs or models or anything into the trained_models folder. Everything in the documentation suggests that it does, however nowhere in the code can I see it doing that during the train loop. Fed up for now :( I wonder if it's because I'm not running it on docker? Please would you be able to test it and see if you get the same result as me @wyli ? 😭 The error is the exact as the last one I posted, but with correct folder names this time...

[JupiLogy]

If it is a problem with nnunet I'll open an issue there

[wyli]

I don't currently have time to look into those API changes... perhaps @dongyang0122 can help

[JupiLogy]

Update, I got access to a machine with Docker, but the GPUs are too old to work with the monai docker. :(

Edit to update more: I got it working on a Docker, and it doesn't have the same folder issue there. Note to self: Now it's just not making the "validation" folder for some reason.