Batch 3D segmentation of multiple point prompts to multiple objects

[psobolewskiPhD]

In 2D Annotator I can place multiple points and segment them at once.
But in 3D this does no appear to be possible? When I place multiple points I get one label.
In prior work I have obtained point markers for every object of interest, so I thought micro-sam would be a good way to get 3D segmentations for those objects!

Likewise, if I correct an a 2D auto-segmentation, I can't use the output of that as the prompt to do a volume segmentation.
Is there anyway to work around that?

[constantinpape]

Hi @psobolewskiPhD ,
the short answer is: both things are in principle possible, but not directly supported in the GUI, because we did not have time to implement it yet.

In a bit more detail:

Segmenting multiple points for multiple objects in the 3D annotator:
We decided against implementing this initially because it adds some complexity when correcting an object in 3D. However, we took this decision before we implemented the batched mode for the 2d annotator. If we follow the same logic it would be possible to do this in 3D as well. This should actually be rather straightfoward to do.

Using the output of 2D auto segmentation to prompt volumetric segmentation:
Currently not implemented in the UI but we have the underlying functionality for it. The function segment_mask_in_volume takes a (2D) mask as input and then uses it as prompt for 3D segmentation, so what you want is quite easy to implement (just outlining the idea):

from micro_sam.multi_dimensional_segmentation import segment_mask_in_volume

# assuming we have the array 'segmentation' and unique ids 'seg_ids' 
for seg_id in seg_ids:
  mask = segmentation == seg_id
  mask_3d = segment_mask_in_volume(
    segmentation=mask,
    predictor=predictor,  # this is the sam model
    image_embeddings=image_embeddings,  # these are the precomputed 3d image embeddings - important to make this efficcient
    segmented_slices=[z],  # the slice id for which you ran auto segmentation, i.e. which contains the mask
   stop_lower=False, stop__upper=False,  # these both have to be set to False for this use-case
   iou_threshold=0.5,  # when to stop segmentation depending on IOU of adjacent slices
   projection="single_point",  # different projection options might affect the result, single_point is the simplest and often works well. 
  )
  ...  # do something to keep track of the 3d masks

I am also not quite sure what would be the best way to integrate this into the GUI to avoid more options and complexity for the user.

In summary:

• batched 3d segmentation is something we will try to add in a future release and should be rather straight-forward to add
• prompting based on 2d auto segmentation is in principle possible but I am not sure how to best integrate it into the UI

Let us know if you would like to explore a contribution here, happy to follow up with more details in that case!

[psobolewskiPhD]

Thanks @constantinpape
I will take a look at the 2d batch approach
Regarding the 2D labels -> volume, I think this would be quite powerful and also intuitive, as manually annotating/correcting 2D labels is quite easy. That said, I understand the challenges of UI complexity.

I will consult with the user I am supporting and see what they think. It may not be till after I2K that I have time to contribute.

[constantinpape]

I will consult with the user I am supporting and see what they think. It may not be till after I2K that I have time to contribute.

I am also not sure if we will have any time for this before I2K. I will create a specific issue on this later and we can coordinate on this there.

Regarding the 2D labels -> volume, I think this would be quite powerful and also intuitive, as manually annotating/correcting 2D labels is quite easy. That said, I understand the challenges of UI complexity.

I agree in principle, but I am also a bit hesitant to add more to the current UI. Long-term I would like to make the UI more modular to enable adding things more easily. In any case, we're very happy with supporting you in creating a small new widget for this as an intermediate step.

[constantinpape]

I created to follow up issues for the two possible feature extensions: #730 #731