Feature suggestions

[sethcohn]

Playing with this... and having fun imagining uses for it.

Some suggestions:

• Make Text detection optional
• Add some visual beat option, or maybe a beat visualization layer?
• allow for slight offset for beat/video sync (so that you can adjust to correct for visual/auditory lag) (ie appear to change just on the beat)
• allow setting specific videos to only certain time windows (so you could ensure Video X is only at end, or Video Y won't be used after halfway point, etc...)
• order video clips so that they are chronological (ie pick from beginning of videos at start, then pick later as song progresses...)
• overlapping clips - similar to the way the Cup Song works, adding new non fullscreen clips to the beat...
• stay nearby - pick the next clip nearby from the last one, for at least X clips, before switching video sources
• ken burns the clips (zoom and pan)
• leave clip audio intact (or mix with song?)

[scherroman]

Hey thanks for trying this out, these are some great ideas!

I'm in the process of rewriting mugen from the ground up for an initial release, and I'm definitely including things like:

• high level of control over which filters to use, including optional text detection

• keeping the original audio instead of mixing in a song

• custom cut locations (locations where a change between clips should occur).

visual beat option/visualization layer: Do you mean adding visual effects to some beats, instead of always cutting? Like the visual pulses in this amv after the 50 sec mark? There they do pulses on bass and cuts on snare. Though later on, they start cutting to the melody more often, and use pulses for most of the beats.

slight offset for beat/video sync: This would surely be useful. I'm curious what kind of sync/lag issues you've seen? Do the cuts between clips sometimes appear to be slightly off-beat? Are there some songs where the beat detection is completely off?

ken burns (zoom and pan): I've noticed many amv creators do use slow-zoom and pan effects, especially on clips without much motion. I didn't realize thats called the ken burns effect! Ideally, if there were an option to add a mix of ken burns effects to the clips, we'd only want to add them to clips which don't already have zooms or pans (i.e. we'd want a method to detect if a clip has a zoom or pan).

I've added some of your other ideas to my notes. "stay nearby" is an interesting one. Some of them i've definitely been thinking about, like control over chronology and time windows.

[sethcohn]

visual beat option/visualization layer

Yes, that's one way to do it (pulsing)... and I've seen that used to good effect.
Another is something like this: https://www.youtube.com/watch?v=6ksA1BaOpOU (adding the audio in a visual way...)

I was actually thinking of yet a 3rd way, which is to show the beats on a moving timeline. Sorta of like guitar hero or karaok-ish. I recently found perl code to do this: https://github.com/jd284/mkbeat
(Turning your beat analysis into this text format would be perfect, that's a really flexible format, and hand editable...) If I do a sample video, I'll let you know.
This is a flexible idea: Here's how someone does it with a video editor: https://www.youtube.com/watch?v=c6FOuC0g7_w

slight offset

I haven't played a lot (yet), but I think I noticed places where it felt like it wasn't 'quite right', and unsure why... being able to go into the json, it's certainly possible to manually fix this, but it might be nice to have a command line option to tweak widely. (like "hmm, if I cut .005 earlier, it feels better")

ken burns

True, a method to detect might be good, but applying it to a moving clip wouldn't hurt it, it would only change the speed/direction of the movement. Make it something you can turn off or adjust in json per clip, so if you watch your video and notice a clip that doesn't need, or needs tweaking to focus it differently, etc. I am pretty sure there is an basic algorithm to figure out the focal point of at least a still image, to direct zoom/pan movement toward it. Examples I found quickly: https://github.com/jonom/jquery-focuspoint
https://github.com/thumbor/thumbor/wiki/Extract-Focal-Points
(and bam, there's the algo:)
https://github.com/thumbor/thumbor/wiki/Detection-algorithms#using-focal-points-for-cropping