Add support for local video and caption

[Ovler-Young]

As a student, I often record my classes and transcribe them locally so I can rewatch lectures and study more effectively. I would like to use Hyperaudio to view these recordings as interactive transcripts for better retention and understanding. However, loading local videos and captions currently seems difficult. They will be blocked by the browser and I found that fixing it is beyong my ability as an bioengineering student.

I transcribed recordings using the offline faster-whisper tool whisper-standalone-win as I can tune prompts for better results. It outputs subtitles in .srt format and a JSON with word timestamps. Samples are 2023-11-24 09_50_00_VGA.srt.zip and 2023-11-24 09_50_00_VGA.json

It would greatly improve the student workflow if Hyperaudio could support local video and captions, rather than requiring videos/transcripts to be hosted online. Would it be possible to add support for local video playback and caption loading? Supporting the whisper timestamp JSON format would also allow accurate syncing between video and captions!

[maboa]

Hi @Ovler-Young – I have a number of thoughts on this, I hope it's ok to convert to a discussion for further, er discussion :)

[maboa]

This is a great use case @Ovler-Young (thanks for sharing). I think there's a couple of things you can immediately do to get this working. It sounds like you are trying to reference your media with file:// rather than https:// which browsers will usually have issues with for security reasons.

To use http or https locally you could run your application from a webserver. An example of a straightforward webserver to install is MAMP – free for the non pro version, which should be all you need. You could also run one on the command line using Python, just run this in the same folder as your app. You can access your webserver through localhost:8888/my-app/ (your port number may vary).

You may also be interested in the Hyperaudio Lite Editor (GitHub), which allows files to be saved to local storage/ and we also have a Wordpress plugin that might be useful for a local or private Wordpress installation. Does that help at all?

[Ovler-Young]

Thanks for the suggestions on loading media over HTTP! It works, but just the video half.
I ended up using the software Everything to host my local files - it lets me access all my files through HTTP and starts on boot automatically. Plus I can restrict it to just listen on localhost for some security.
That got my video loading, but the localized subtitles still won't sync properly. The request didn't even sent. I'll paste my trying code tomorrow to show exactly what I tried. Maybe I'm just missing something obvious!

But I do have to say, editing the source code to change the media each time is a bit of a pain. Having some kind of input right on the rendered page to select a local video and captions would make iterating so much faster. Any chance that could get built in?

The Hyperaudio Lite Editor does seem super useful by the way. I took a look but didn't see where to upload a local video in there either. If that's possible, some pointers would be awesome. Or is it on the roadmap?

Anyway, appreciate you taking the time to help! Moving this over to a discussion thread makes sense. I'm definitely eager keep figuring out the best way for students/teachers to easily use Hyperaudio with their own media.

Let me know if any other specifics from my end would be helpful!

[maboa]

Great that you got it (mostly) working :) I think for security reasons you need to load everything (including captions) over http. Although you could inline them somehow.

I general Hyperaudio Lite is meant as a library that you can build web apps upon (see the Hyperaudio Lite Editor or Wordpress Plugin as an example). I think the uploading video functionality you mention would be better suited to an app not a library.

[Ovler-Young]

Understood and thanks for help!
Today, by the help of AI, I added the upload ability to watch the video from the user end, which also support srt and whisper json.
Code might be a bit mess, I've adjusted something more, removed the functions I might won't use, but it works for my use case. First upload the video, then upload the subtitle, srt or whisper json, then everything works!
If you have time, maybe check this! https://github.com/Ovler-Young/hyperaudio-lite/blob/test/test.html

I might add the ability to 'upload' the video and subtitle together, but that might be after this semester‘s exam.

[Ovler-Young]

GitHub pages： https://ovler-young.github.io/hyperaudio-lite/test.html