-
|
In MONAI, is there a pre-build function to combine multiple trained models and save predictions for a test set without labels? Since the majority of 3D medical datasets online are challenge datasets without labels for the test set, and to participate, you must submit test predictions. Can I perform ensembles of multiple models and just save the overall predictions, e.g., majority voting, without using test labels and calculating test loss? Moreover, I noticed that the method on the tutorial cannot handle many networks due to memory problems, for example, I can do an ensemble of 5 models but with 10 I have problems. I there a way to overcome this issue? Like some memory-saving tricks or kind of checkpoints in order to don't have limitations with the number of models? |
Beta Was this translation helpful? Give feedback.
Answered by
KumoLiu
Jan 30, 2023
Replies: 1 comment 9 replies
-
|
Hi @cugwu, you could comment And add Since the parameters of the deep learning network are indeed very large, so adding a model for memory will indeed have a relatively large burden, you can try to set the amp to True in |
Beta Was this translation helpful? Give feedback.
9 replies
-
|
Hi @ahxmeds, since the size limit, I can't see the whole error message. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry about that. Here's the full error log: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @ahxmeds, from the error messages, I think the error is due to the mismatched size between your label and prediction. If you are referring to this tutorial. I think you also need to invert your label to make sure that label and prediction have the same shape. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, that worked. Everything ran perfectly and I am able to save the correct ensembled predictions now. I have n=30 images in my test set. But unfortunately, after running for the first 15 images perfectly, I get CUDA out of memory error: I am wrapping my models on 4 GPUs via |
Beta Was this translation helpful? Give feedback.
-
|
@KumoLiu Update: Looks like it is failing only for one of the images (image # 16). When I tried running with only images # 17-30, the code ran perfectly. When I try running with only one image (image # 16), it fails again with the same CUDA out of memory error. The size of the image after applying |
Beta Was this translation helpful? Give feedback.
Answer selected by
cugwu
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi @cugwu, you could comment
key_val_metric, then it will not calculate the metric if you don't have the label.And add
SaveImagedin thepost_transforms, then could save the ensemble results.