Frame extraction rule

[road2empire]

Hi @byjlw

The frame extraction works fine on the multiple scene(different background & setup) videos, but in single scene(no change on the background), it is not detecting the new text overlays.

Video: https://streamable.com/gqeyzj

It only extracts single frame(attached), but there are other overlay text appearance too.

I tried changing the frame.py with Claude sonnet 3.5, and it offered me new implementation to detect the text changes, but the change made the frame.py so sensitive that every other frame is being captured.

What do you think about this?

[byjlw]

So given the smaller sized video, you should be getting all keyframes by default, and you should have a keyframe for each different text. You can run the command with --keep-frames you'll get the frames in the output directory and you can inspect.
I suspect that the real issue is that the LLM isn't taking looking at the text in the frame every time you can use --prompt to explicitly say that you need to record any text found in the frame

[road2empire]

I had the same thought of LLM not being able to read the frames, but then I started inspecting the extracted frames by looking at the saved frames folders, and the problem is that it doesn't extract the all frames.

I will do more testing with Claude to change the frame.py, let's see how it goes.

[byjlw]

Interesting. That's interesting. I'm surprised the text changes weren't enough to trigger a frame difference. I'll look into it, later this week.

[byjlw]

I have time to look into this now but the video isn't available right now.
My suspicion is that the delta between two frames isn't great enough once creating to grayscale to hit the threshold of 10

[byjlw]

I've landed some changes to reduce the complexity to frame selection and added details on how it works to docs/DESIGN.md
You can change the variables frames_per_minute and FRAME_DIFFERENCE_THRESHOLD to see if you can get the desired behavior. If you're running at 60 frames a second, you'll be missing a lot of frames at 10 frames a minute (which is the default).

[road2empire]

hey man, sorry for delayed response, I was trying different solution to get the full picture on the frame extraction. I'm in process of testing the "pysceneDetect", any thought on this?

I'll test your solution tomorrow.

[road2empire]

Hi @byjlw , I did test with new frame.py, but I don't seem to get better results.

And for the above video, I get 0 frames(I was getting 1 previously).

If you'd like to test on your end: https://we.tl/t-Za7vRVQ4k6

[byjlw]

looks like i need a login to download that video.

[road2empire]

@byjlw is it okay if I reach you out in DMs? Not sure how to do it on github, it would be way faster for this, I changed the frame extraction to the pyscenedetect, and getting betters results, i'd like to show you firsthand