Possible memory leak while acquiring video

[GalAtlan]

Hi,

I am struggling with an issue in which buffered frames accumulate in the RAM and cause lag and/or crashes of Bonsai. This can happen within one hour. Additionally, the memory stays in use until I close Bonsai.

I have tried using two different cameras (ImagingSource DMK with CameraCapture; and Basler Ace with PylonCapture).
The issue has gotten worse with the latest version of Bonsai, so I tried also v.2.7, manually installing all the compatible packages, but I am still getting this issue.

Acquiring data from a single camera at 60hz with 64gb RAM should be simple right? Something must be off, I just can't locate what it is.

Thank you!

[chiyu1203]

I think that depends on a lots of factor. Eg. CPU speed, image resolution. I have been working with pylon capture for a long time and have no problem large image (1500x1500, 20Hz) or small image (320x240, 100 Hz). You can check out my workflow shown in this discussion.
https://github.com/orgs/bonsai-rx/discussions/1453

[bruno-f-cruz]

Can you provide:

• The workflow you are using to run the experiment
• Camera Resolution
• Camera Frame Format
• Camera Frame Rate
• Video Codec
• Computer specs (mostly CPU model and what hd/sdd are you dumping the data into)
  Thanks

[GalAtlan]

Thank you for your replies.
@chiyu1203, have you been able to image at >30 Hz with larger images?

@bruno-f-cruz regarding your questions:

• I'm attaching the workflow. Although I have encountered the issue even when isolating just the video recording nodes. ex_code.bonsai.zip

• I'm trying to record at 60hz, 1440x1080. I think I can mostly avoid it if I don't go above 30 hz.

• With this workflow the camera is an imaging source DMK 33UX273 at Y800 (1440x1080). I have also tried with a Basler Ace camera.

• I have tried FMP4 and DIVX
  ex_code.bonsai.zip

• The PC is this MinisForum DeskMini with an SSD HD and upgraded to 64GB RAM

[chiyu1203]

My gut feeling is that your CPU is not fast enough. We used Intel-Core-i7-1185G7 to record 1500x1500 @ 20 Hz. I did not try higher sample rate but if I made the image larger (e.g. 2000x2000 @ 20Hz), I would see lags. And your CPU sounds slower than ours. One thing you can try with basler camera is to do test recording (include saving video and repeatedly as you were running a series of real experiments) on pylon viewer. If you also bump into lags there, then this problem is not due to bonsai.
Before Bruno giving a detailed diagnoysis about your problem, things you can work around are

1. reduce image format (either use crop image on bonsai or change image format (Width, Height) on pylon viewer).
2. reduce image resolution (average or sum nearby pixel value with Binning on pylon viewer)
3. reduce frame rate/exposure time
4. get a better CPU to test your setup

[GalAtlan]

Thanks for the input!

I can confirm that recording directly on Pylon viewer with the same settings uses only up to ~35% CPU and is pretty stable around 10-12% RAM usage.
Which makes me feel that is Bonsai related.

Does it make sense that the content of the frame matters (e.g. static frame, vs. recording a moving animal)?

[GalAtlan]

I have also just noticed that a 20 minute video recorded at 1440x1080 using Pylon Viewer weights about 8mb, while the videos I record using Bonsai end up being 10s of GBs large.

Perhaps there lies the issue? Why is Bonsai generating much larger files?

[GalAtlan]

I understand that there may be a more optimal way to approach this. And if there is no other solution, I may be forced to do it.
But I find it frustrating that I can see it working smoothly in the other room, and I can't get it to work on my machine.
I cannot find a way to download the Bonsai - Pylon Library that is compatible with Pylon 5.1, do you know if it is archived somewhere?

[bruno-f-cruz]

You can regenerate a bonsai environment using the bonsai.config as explained in the documentation

[GalAtlan]

Just to follow up on this:
Using the exact settings from the old computer seems to have solved the RAM issue!

Many thanks for your help - I'll definietly look into ffmpeg for future projects.

[glopesdev]

@GalAtlan great to hear you were able to solve this issue.

I noticed from an earlier message that it seems you were using an AMD Ryzen for acquisition? We have consistently noticed performance issues with industrial cameras when using non-Intel processors. The reason I believe is that both camera drivers and OpenCV have performance optimisations that get triggered specifically for Intel processors and not for AMD and those are enough to slow down the pipeline significantly at large resolutions and then creating memory/lag issues.

[chiyu1203]

@GalAtlan I am glad to hear you got that working!! May I know that is your old computer using Intel processor or AMD processor and in the end you solved the problem by switching back to pylon 5.1 runtime and the old bonsai-pylon library?! I am asking this is because I want to increase the fps in my setup and I wonder what would be the best way to avoid the memory, drop frame issue...

[chiyu1203]

We had this discussion before so let me know if you can not download the old pylon library

I understand that there may be a more optimal way to approach this. And if there is no other solution, I may be forced to do it. But I find it frustrating that I can see it working smoothly in the other room, and I can't get it to work on my machine. I cannot find a way to download the Bonsai - Pylon Library that is compatible with Pylon 5.1, do you know if it is archived somewhere?

[GalAtlan]

Thank you. Indeed, I wasn't able to find where to download the old pylon library.
Luckily, it was installed on another computer, and by copying the .config file I was able to get it installed.

My lesson from all of this - make hard copies of your working bonsai .config file in case an update causes issues!

[bruno-f-cruz]

The lesson should be to use version control and environments to manage your code and experiments :). If you version control you can always go back!