Confusing results of custom fine-tuned models

[aloboa]

This is what I have done:

Custom finetuned Models (with the GUI):

• fine_01.pt : with Segmentation Decoder
• fine_02.pt : without Segmentation Decoder
• best.pt: with Segmentation Decoder

micro_sam.automatic_segmentation 
-i imain.tif 
-o imaout1.tif 
-e embeddings_DI17_3_AHE_fine01 
-c fine_01.pt 
-m vit_b 
-d 'cuda' 
--tile_shape 1024 1024 
--halo 256 256 
--pred_iou_thresh 0.7 
--stability_score_thresh 0.9 
--box_nms_thresh 0.1 
--mode amg
--ndim 2 

micro_sam.automatic_segmentation 
-i imain.tif 
-o imaout2.tif 
-e embeddings_DI17_3_AHE_fine02 
-c fine_02.pt 
-m vit_b 
-d 'cuda' 
--tile_shape 1024 1024 
--halo 256 256 
--pred_iou_thresh 0.7 
--stability_score_thresh 0.9 
--box_nms_thresh 0.1 
--mode amg 
--ndim 2 

micro_sam.automatic_segmentation 
-i imain.tif  
-o imaout3.tif 
-e embeddings_DI17_2_AHE_256_fine03 
-c best.pt 
-m vit_b 
-d 'cuda' 
--tile_shape 1024 1024 
--halo 256 256 
--mode ais 
--ndim 2 

According to your explanations:

• with fine_01b.pt: You only have the SAM backbone (i.e. the image encoder, prompt encoder and mask decoder). This part of exporting is done to ensure our checkpoints' cross-tool compatibility for SAM!
• with sam_model/best.pt: You have all finetuned components in here, including the parameters for the additional instance segmentation decoder.

If both *.pt models are only SAM backbones, they should not be affected by the "With segmentation decoder" option. Therefore, fine_01.pt and fine_02.pt should be the same, and thus results imaout1.tif and imaout2.tif should also be the same. But, actually, they are very different.

[anwai98]

Hi @aloboa,

I see the obvious confusion here, and a possible improvement in storing model checkpoints from our side.

To clarify:

• best.pt: This is the filepath, where the best finetuned model (and related components, eg. optimizer state, training time, etc) exists, always stored in your current working directory.

  • This is something I would like to update, where the users can store all files in their desired location, instead of the current working directory.

• <CUSTOM_NAME>.pt: I assume these arise from the part where you export the SAM-backbone to a desired filepath, where the model "only" exports the SAM backbone.

Now coming to your question:

Therefore, fine_01.pt and fine_02.pt should be the same, and thus results imaout1.tif and imaout2.tif should also be the same. But, actually, they are very different.

From what I understand with the context provided, these seem to be two different finetuning runs (one finetuned with the additional segmentation decoder, and one without). Since they are two different models, it is possible that the results could be different (because of two separate training runs).

PS: However, I do see one point which I think I would like to improve in the GUI:

• The possibility of storing the trained checkpoints (eg. best.pt) and related files in a target path, as per the user's choice!

PPS. I've created a follow-up on a running feature request here: #836 (comment).

[aloboa]

I understand that we should expect fine_01.pt and fine_02.pt, and their respective results imaout1.tif and imaout2.tif, to be not identical. However, the problem is that I think that they are too different:

Autosegmentation using a SAM-backbone model created "with segmentation decoder":

Autosegmentation using a SAM-backbone model created "without segmentation decoder":

Maybe you could run a test on your own, with your own data? Simply fine-tune with and without segmentation decoder and autosegment with the respective *.pt models. Just to make sure the problem is not on my side.

[anwai98]

However, the problem is that I think that they are too different:

Interesting. The thing is, we haven't experimented much with different hyperparameters for AMG-based segmentation (I am assuming the screenshots above are from AMG since that's the one we have talked about the most).

Out of curiousity, which of the above segmentation quality matches closest to your expectations? (the one in blue / red / neither)

Maybe you could run a test on your own, with your own data?

The data we work with (biological images), it would be hard to observe these changes as AMG did not really improve the segmentation quality with further finetuning (it worked fine on sparsely distributed objects, but that's less likely to assume this in microscopy images).

Just to make sure the problem is not on my side.

I still don't think there is any issue from your side. The heuristic with which AMG is supported (by putting a grid of points and then estimating valid segmentations using post-processing and NMS) is quite sensitive. In addition, we introduce another stochastic factor here: two different finetuned models.

I can suggest the following: finetune two models with the exact same training setup (for simplicity, let's say without the additional instance segmentation decoder), export the two different paths with two separate names and then validate your observation. If the difference in segmentations still exists, you can make this a part of your study (as this is often encountered in training / finetuning deep learning models)

PS. @constantinpape Let me know if I am missing something obvious here!

[aloboa]

Out of curiousity, which of the above segmentation quality matches closest to your expectations? (the one in blue / red / neither)

The blue one (without) is a bit better, but neither is really good. The one from a best.pt model "with segmentation decoder" is clearly better, although a non-tuned vit_h model is the most useful. The reason is that while the fine-tuned one has higher accuracy, it segments mostly the training objects only. While the vit_h incurs in some error in the training set, it segments a lot of additional objects (ie, it provides a more general result, which is more useful). I am writing all this now and will let you know.

I can suggest the following: finetune two models with the exact same training setup (for simplicity, let's say without the additional instance segmentation decoder), export the two different paths with two separate names and then validate your observation. If the difference in segmentations still exists, you can make this a part of your study (as this is often encountered in training / finetuning deep learning models)

Ok, we'll do it.

[anwai98]

Thanks for the context. I think I understand the details a bit better. Maybe this will all piece in together if you could share with me some ground-truth for this data? (which you use to finetune your custom models?) (if that's okay with you)

Seeing some image and corresponding ground-truth would help us understand the details a bit better!

[aloboa]

Seeing some image and corresponding ground-truth would help us understand the details a bit better!

AI-driven processing of SEM images of thin sections of volcanic rocks: a test with Segment Anything for Microscopy

In section "2.Input information" you can see the 3 original, enhanced and labeled images, just by selecting the tabs.

But I can send you the actual data if you need them.

[anwai98]

In section "2.Input information" you can see the 3 original, enhanced and labeled images, just by selecting the tabs.

I understand it better now. Thanks for sharing the report!