Mask R-CNN Instance segmentation extremely slow

[TTkgl]

Hello,
I currently experience performance issues using a Mask R-CNN trained with detectron2 on custom data.

My environment consist of a R50-FPN model, which has been custom trained to a dataset with two category ("a" and "b").
When running the inference, I use sahi to split a high resolution image (6000x4000) into smaller 640x640px slices.

This prediction takes ~120 seconds for one image divided into 120 slices, which results in a inference time of 1 s/img. According to the model zoo, around 0.04 s/img should be possible.

What I found out is that if I run my prediction on the model from the model zoo without applying my custom dataset training to it, the inference time is 11.10 seconds for the 120 slices, which is pretty much what I expected from the model.

cfg = get_cfg()
cfg.merge_from_file("default/mask_rcnn_R_50_FPN_3x.yaml")
cfg.OUTPUT_DIR = "model"

cfg.MODEL.WEIGHTS = "default/model_final_f10217.pkl"
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 2
cfg.MODEL.DEVICE = 'cuda'

cfg.DATASETS.TRAIN = (dataset_name + "_train",)
cfg.DATASETS.TEST = (dataset_name + "_val",)

cfg.DATALOADER.NUM_WORKERS = 2

cfg.SOLVER.MAX_ITER = 1000
cfg.SOLVER.BASE_LR = 0.001
cfg.SOLVER.IMS_PER_BATCH = 2

Above is the configuration I use to train the model, in case that helps, below an excerpt of the training dataset.

I would highly appreciate if anyone can help me solving this performance issue.

[ClemSc]

Hello, I am also having some performance issue with a mask rcnn R50 FPN model trained on with my own data. In my case, the prediction on one slice take ~ 0.5s. However, between slice prediction, up to 10s (but often less) can elapse (between two call of get_prediction() in get_sliced_prediction() of sahi.predict).
Hence, for a large picture (10000px) cut in slices of 1500px with a 0.6 overlap (resulted in 256 slices), it took 5 minutes to predict 80 slices.
I changed for slices of 1600px with a 0.3 overlap, resulting in 81 slices. It took 8 min to process the 81 slices. After the last slice was predicted, another 3 minutes was needed to switch to the next picture.

So for a complete picture cut in 81 slices, I have in total:
~40 sec of inference, 420 sec of process between each slices. And then 180 sec of process between next full picture.

I would be interested in speeding this up :)

[TTkgl]

Hello,
I found out, that this issue is most likely related to SAHI.
When you slice the big image into smaller slices (e.g., 640px x 640px) using slice_image and then perform a batch prediction as follows:
result = predict( model_type='detectron2', model_path=model_path, model_config_path=config_path, image_size=640, model_device="cuda", source="predict_images/slices/", no_standard_prediction=True, slice_height=640, slice_width=640, overlap_height_ratio=0.2, overlap_width_ratio=0.2, verbose=2, novisual=False )

Then, SAHI will perform a prediction on each of the images. For me, as seen in the below image, the whole prediction which beforehand took ~300sec now took place in 10.992 seconds and 8.34 it/s resulting in ~120 ms average image prediction time.
(The prediction result saved contains correct predicitions, so the batch prediction works)

@ClemSc Could you maybe verifiy if you experience the same behavior?
My issue is that now I need to manually merge the results of each slice prediction.

[ClemSc]

@TTkgl Hi. Predicting on manually sliced images without using Sahi is fast, and consistent with what I reported above. I am a bit confused with what you did. I see that you set "no_standard_prediction=True". Checking at predict.py, I would have used "no_slice_prediction=True" to bypass the slice prediction (and thus call get_prediction(), instead of get_sliced_prediction(). But maybe I did not got it right ?
Anyway, the "non-inference" time during the sliced prediction has to be about the image slicing, then the result merging. However, I would expect both tasks to be run one time for each pictures. In my case, I do not explain yet the time "lost" between each slice inference. I will try to have a look at it soon.
(EDIT: now that I check the code carefully, I see that I indeed did not understood what no_standard_prediction was meant for. Actually, I was already activating this option).

Second EDIT: I did not realized that this was the detectron2 repository, and not the SAHI repo. This discussion does not really belong here, whoops.

[TTkgl]

@ClemSc You are right, I also added a discussion in obss/sahi#999 . My expectation is the same, the time lost is most likely an issue. I'll try to investigate further, maybe we continue this in obss/sahi#999