Silero-VAD V5

[dgoryeo]

Silero VAD V5 was just released. The published benchmarks look promising.
Would stable-ts need any updates to use the new version of Silero VAD, or the exisiting code pickes up the latest?

Thanks

[jianfch]

By default, model is downloaded to the local cache for the first time then it loads it from the cache each time it loads Silero VAD.
So you'll just need to update the files in the cache.

import torch
torch.hub.load(repo_or_dir='snakers4/silero-vad:master', model='silero_vad', force_reload=True)

Another way after b309530 is to usevad=dict(force_reload=True, cache=False) once instead of vad=True to update it.

[dgoryeo]

Hi @jianfch ,

I was not sure if I should open a new post for this, but though of mentioning it here: I did some tests using the latest update with Silero V5, and I notice that in several occasions the dialogs do not get captured. The previous versions of stable-ts were able to capture those dialogs. Assuming that V5 is not yet ready for the primetime, is it possible to specify an older version? I personally believe Silero V3 was the best option for working in tandem with Whisper. Just a thought.

Thanks

[jianfch]

By default, the VAD outputs shouldn't affect the contents of the transcription because only timestamps are adjusted after the fact and contents are not removed even if they completely overlap a section that is determined to be silent by the VAD. There might something else causing the missing dialogs. It would be helpful if you could share the settings and the audio that you use for tests you did.

After b309530, a local path to the directory of Silero VAD can be specified with vad=dict(repo_or_dir="./silero-vad-3.0", cache=False). You can download the source code of various versions from the Silero VAD releases.

[dgoryeo]

Thanks @jianfch for the workaround options. I'll send you a sample audio and corresponding subs and settings later this week. The one that I have in hand is copyright'd. I need to sanitise it before sharing.

[dgoryeo]

Hi @jianfch ,

Here are a sample test audio, and a comparison table of results after I ran that through Stable-ts Silero 5.0 and Standalone-Faster-Whisper-XXL Silero 3.1, and Pyannote 3.

Test-Audio-3-Minutes

Excel-sheet-comp-table

The common settings were:

--model large-v2 (faster-whisper on both) 
-- VAD-threashold 2.0 (for Stable-ts)
--beam_size 5 
--temperature 0   
--patience 3 
--repetition_penalty 2  
--no_repeat_ngram_size 3 

I hope this helps

Thanks

[dgoryeo]

Thanks. I'm going to review the colab during the weekend. I just took a look and noticed that I can learn quite a bit already from the code. Thank you!

[dgoryeo]

Hi @jianfch,

Thank you for the helpful notebook. I now have a better understanding of how VAD is applied to the transcription results.

I have a suggestion: would it be possible for stable_ts to include a function that allows users to apply any given timestamps to the results of a transcription? This way, users could obtain segment timestamps from various VAD tools (such as Pyannote, Nemo, or Silero) and supply these timestamps to the function, which would then apply them to generate the final SRT file.

[jianfch]

stable-ts/stable_whisper/result.py

Lines 653 to 659 in 5f1461f

	def suppress_silence(self,
	silent_starts: np.ndarray,
	silent_ends: np.ndarray,
	min_word_dur: Optional[float] = None,
	word_level: bool = True,
	nonspeech_error: float = 0.3,
	use_word_position: bool = True):

When vad=True the timestamps produce the VAD are passed into this method. The Silence Suppression section goes into detail what the parameters do.

import numpy as np
result = model.transcribe(...)
silent_starts = np.array([0.0, 3.1])
silent_ends = np.array([2.3, 5.5])
result.suppress_silence(silent_starts, silent_ends)

If the model hallucinates produces transcription for parts of the audio that contains no speech (i.e. overlap in the provided silent timestamps), this method will remove those parts even if min_word_dur=0. You will have to separately remove the words/segments with zero duration.

[dgoryeo]

Hi @jianfch

I tried to use your guidleines above to test supress_silence, but I'm not sure where I go wrong. Here is my code

    model = stable_whisper.load_model('small')
    result = model.transcribe(audio_file)

    # Applying test timestamps to suppress silence
    silent_starts = np.array([40.0, 48.0])
    silent_ends = np.array([95.0, 99.0])
    result.suppress_silence(silent_starts, silent_ends)
    result.update_nonspeech_sections(silent_starts, silent_ends)
    result.set_current_as_orig()
    result.to_srt_vtt("random.srt", segment_level=True, word_level=False)

The result however remains unchanged regardless of any timestamps given. It is like supress_silence has no impact. Also note that I assume vad is by default set as False. I have also tried without :

    result.update_nonspeech_sections(silent_starts, silent_ends)
    result.set_current_as_orig()

But I get the same outcome / behaviour.

Where do I go wrong?

[jianfch]

Under certain conditions, some silent sections will be ignored and the timestamps will remain unchanged. See https://github.com/jianfch/stable-ts?tab=readme-ov-file#silence-suppression