Guidance for setting up low-latency 2-way audio&video

[pnoffke]

Describe The Problem:
Firstly, thanks for this amazing library. I've purchased a G4 Pro that I'm wanting to integrate into Home Assistant. I'm aiming for a custom card in HA the normal audio/video/speak/mic controls, together with an additional button that controls the door release (via a z-wave relay).

I'm hoping you can guide me on 2 issues:

The best framework for achieving low-latency. I first tried using a HLS server, but am getting 2s+ lag. I'm now looking at WebRTC, but please let me know if there's a better way of doing this.

How to link up the audo-stream back to the camera. I'm successfully acessing the camera's MP4 stream, but need pointers for hooking up the clients audio.

Any pointers much appreciated!

Regards
Paul
To Reproduce:
n/a

Logs:
n/a

Screenshots:
n/a

[hjdhjd]

Your welcome! Hope you find it helpful. The Protect API is ever-evolving, but I do my best to keep up with it. I don't have a good answer for you about a low-latency framework for use with HA. One of my other projects, homebridge-unifi-protect achieves most of what you're looking to do within a Homebridge / HomeKit context. You can take a look there for pointers.

In general, I dissuade folks from trying to do one-off projects because the Protect API changes just enough, often enough, that you're taking on a never-ending maintenance burden...it's certainly yours to embrace, but there's so many more projects out there that don't get meaningful time and attention from talented developers, that contributing to an existing project is often a better avenue than a roll-your-own strategy. There's already a few talented developers working on projects utilizing this library, and at least one obsessive one (me!) about maintaining their stuff, that you might want to see if some of those provide a "good enough" experience. I'd encourage you to consider that in your journey.

If all this is a fun hobby to learn the Protect API...enjoy. Best place to see this API in action and some of what you can do with it is in my Homebridge plugin above.

PS: I've also been creating a native UniFi Access API that's a separate project, should your Ubiquiti journey eventually lead you there as well.

[pnoffke]

Thanks for the pointers. I'd certainly like to avoid the maintenance groundhog day and will look to integrate my changes into the existing HA integration.

Some progress from my side. I've given up on WebRTC due to the learning curve and opted for a MediaSource instead (https://developer.mozilla.org/en-US/docs/Web/API/MediaSource). Got that working for the camera video & audio. I'm seeing about 500ms latency which is more than ok. Not sure if the delay is due to buffering on the unifi-protect or MediaSource.

And so now I'm working on the talkback. Your homebridge code seems to have the clues I need ... thanks again for the heads up and pointers.

Cheers
Paul

[hjdhjd]

There's no latency introduced on the API end, so if you're seeing anything, it lives elsewhere...the API sends packets over as soon as it comes across from Protect...in my experience the Protect controller's latency is ~100-200ms at most.

Best of luck!

[github-actions]

This issue is locked to prevent necroposting on closed issues. Please create a new issue for related support requests, bug reports, or feature suggestions.