[WIP][Model] Whisper + vLLM

[kylesayrs]

Purpose

• Support whisper model with huggingface and vllm backends

TODO

• Better arguments support
• Better vllm batching
• Evaluation replication
• Explore expanding model definition to be shared across architectures such as qwen_audio, ultravox

Testing

lmms-eval --model=whisper --model_args="pretrained=openai/whisper-tiny" --tasks librispeech
lmms-eval --model=whisper_vllm --model_args="pretrained=openai/whisper-tiny" --tasks librispeech

https://huggingface.co/openai/whisper-tiny
7.547098647858638

whisper (pretrained=openai/whisper-tiny), gen_kwargs: (), limit: None, num_fewshot: None, batch_size: 1
|          Tasks          |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
|-------------------------|-------|------|-----:|------|---|------:|---|------|
|librispeech              |    N/A|      |      |      |   |       |   |      |
| - librispeech_dev_clean |Yaml   |none  |     0|wer   |↓  | 8.2813|±  |   N/A|
| - librispeech_dev_other |Yaml   |none  |     0|wer   |↓  |16.3908|±  |   N/A|
| - librispeech_test_clean|Yaml   |none  |     0|wer   |↓  | 7.5773|±  |   N/A|
| - librispeech_test_other|Yaml   |none  |     0|wer   |↓  |17.5674|±  |   N/A|

whisper_vllm (pretrained=openai/whisper-tiny), gen_kwargs: (), limit: None, num_fewshot: None, batch_size: 1
|          Tasks          |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
|-------------------------|-------|------|-----:|------|---|------:|---|------|
|librispeech              |    N/A|      |      |      |   |       |   |      |
| - librispeech_dev_clean |Yaml   |none  |     0|wer   |↓  | 7.8324|±  |   N/A|
| - librispeech_dev_other |Yaml   |none  |     0|wer   |↓  |16.4548|±  |   N/A|
| - librispeech_test_clean|Yaml   |none  |     0|wer   |↓  | 7.6640|±  |   N/A|
| - librispeech_test_other|Yaml   |none  |     0|wer   |↓  |17.1136|±  |   N/A|

[Luodian]

Hi do you still want to contribute whisper? Looking forward to it!