Camera Capture Unattributed Latency

[loqoman]

As of the time of writing this, there is currently an unknown latency involved with capturing a frame from the raspberry pi camera when operating in 1080p. The latency

Data (As we have recorded it) (@JeffreyTT Please fill in if I'm missing anything)

• Originally, running a full pipeline (frame capture to target delivery) was operating at approximately 3Hz. It is likely this number derived from a 200ms capture time followed by a 100ms processing time.
• After updates to the camera wrapper, and an improvement on our threading model, @JeffreyTT Was able to upgrade our speed to 5Hz by working in parallel. Currently, the slowest process is camera capture.
• The realPnP pipeline can operate at about 10Hz on a Raspberry Pi 3B+. @dbadb The scripts to test this latency on a Rpi4 should be available 2020/testLatency.py, it would be interesting to see what your latency numbers are.

[dbadb]

I just ran the script with my camera pointing randomly. I had to change one line in algo.py

• if hexagonTarget == None:

• if hexagonTarget is None:

I get a few pages of spewage (which can slow things down).

root DEBUG generatorHexagonVerticies found a half-hexagon
root DEBUG generatorHexagonVerticies found a half-hexagon
root DEBUG Translation Vector:
root DEBUG Rotation Vector:
root DEBUG Transformed Vector:

At the end of it all, i get this:

Total processing time: 3.630576854106039
Average processing time: 0.044275327489098036
Images processed: 82

My raspi4 is 2GB, fwiw.

[JeffreyTT]

I ran the cameraLatencyTest 3 times each on the pi3 and the pi4. Notably, two different cameras were used, but the average time to capture was approximately the same. (Numbers were put into data/). After quite a bit of digging into how the camera captures frames and the picam itself, Dana and I concurred that control of the fps is largely out of our reach; modulating fps and shutterspeed did not affect how many images the pi actually received.

[loqoman]

Here's a data point suggesting our wrapper might be at fault.

On 2020.02.16, Jeffery and I wanted to compare latency between our wrapper (with multithreading) and one of the 'shipped' picam utilities, raspivid. We set up a pi such that we could easily switch between running picamstreamer.py at 1080p and h264player (which is simply raspivid under the hood) at 1080p.

Jeffery has a video showcasing the difference in latency in a more empirical manner, but the two videos suggest that raspivid was able to pull > 5fps off of the camera, while our wrapper was stuck at around 5fps, both at 1080p

@JeffreyTT Upload the videos and post links?

[JeffreyTT]

Here's a gif of the picamstreamer.py fps.

[JeffreyTT]

Here's a gif of the h264player fps