Online stitching camera frames

[horsto]

I have three USB cameras capturing video data of the same frame size and at the same speed. They are just tiled images that have an x,y offset, but no angular offset. I want to stitch them into one big (flat) composite (mosaic) and continue to work only with that. Downstream of that I want to run object detection on the stitched frame.

The idea is similar to this entry.

I am running into out of memory errors quite quickly after starting the workflow. But it does not seem like a lot of memory has been used / is requested. Any ideas?
test_stitching.bonsai.zip

[bruno-f-cruz]

Hey,

I never came across this error before in this pipeline. I wonder if your computer just can't keep up with the online stitching.
Can you describe the setup? What is the FPS of each camera, are they synchronized, Color vs Mono, and resolution?

If you can afford it, I would probably convert everything to grayscale and use WithLatestFrom instead of CombineLatest.

[glopesdev]

It's likely the CombineLatest that is the problem indeed. Assuming a simple 1 MP image at 60Hz from one camera, then CombineLatest > Concat would be allocating 3 MP images at 180Hz.

[horsto]

Thanks! Indeed the images are very large (see screenshot), acquiring at 20fps each. This might be too much data?
The Concat node seems to throw the error- no errors when concat is skipped it seems.
Even streams from two cameras are too much (see screenshot). I wonder why though. Conversion to Gray is not an option since I need the full color information.

[horsto]

Grayscale conversion + WithLatestFrom + grayscale gives me some weirdness on the camera side (so maybe the memory error isn't telling the whole truth ... )

[horsto]

This actually solved itself. Turns out I was starting Bonsai(x86) instead of Bonsai - The latter has zero issues with frame grabbing + stitching.