[Misc] add mm_processor_kwargs to extra_body for Qwen2.5-VL

[wulipc]

In Qwen2.5-VL online inference, the fps parameter in mm_processor_kwargs is essential for accurately calculating the second_pre_grid_t value. However, the OpenAI interface currently does not support passing mm_processor_kwargs via extra_body. This PR fixes this problem (the previous issue #11652).

You can now interact with the vLLM server using the following example, which has been self-tested and verified to work correctly. If additional test cases are required, please tell me where they should be added. @DarkLight1337

FIX #11652

import base64
import numpy as np
from PIL import Image
from io import BytesIO
from openai import OpenAI
from qwen_vl_utils import process_vision_info


# Set OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8899/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)


video_messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": [
        {"type": "text", "text": "请用表格总结一下视频中的商品特点"},
        {
            "type": "video",
            "video": "https://duguang-labelling.oss-cn-shanghai.aliyuncs.com/qiansun/video_ocr/videos/50221078283.mp4",
            "total_pixels": 20480 * 28 * 28, "min_pixels": 16 * 28 * 2, 
            'fps': 3.0  # The default value is 2.0, but for demonstration purposes, we set it to 3.0.
        }]
    },
]


def prepare_message_for_vllm(content_messages):
    """
    The frame extraction logic for videos in `vLLM` differs from that of `qwen_vl_utils`.
    Here, we utilize `qwen_vl_utils` to extract video frames, with the `media_typ`e of the video explicitly set to `video/jpeg`.
    By doing so, vLLM will no longer attempt to extract frames from the input base64-encoded images.
    """
    vllm_messages, fps_list = [], []
    for message in content_messages:
        message_content_list = message["content"]
        if not isinstance(message_content_list, list):
            vllm_messages.append(message)
            continue

        new_content_list = []
        for part_message in message_content_list:
            if 'video' in part_message:
                video_message = [{'content': [part_message]}]
                image_inputs, video_inputs, video_kwargs = process_vision_info(video_message, return_video_kwargs=True)
                assert video_inputs is not None, "video_inputs should not be None"
                video_input = (video_inputs.pop()).permute(0, 2, 3, 1).numpy().astype(np.uint8)
                print("video_kwargs", video_kwargs, video_input.shape)
                fps_list.extend(video_kwargs.get('fps', []))

                # encode image with base64
                base64_frames = []
                for frame in video_input:
                    img = Image.fromarray(frame)
                    output_buffer = BytesIO()
                    img.save(output_buffer, format="jpeg")
                    byte_data = output_buffer.getvalue()
                    base64_str = base64.b64encode(byte_data).decode("utf-8")
                    base64_frames.append(base64_str)

                part_message = {
                    "type": "video_url",
                    "video_url": {"url": f"data:video/jpeg;base64,{','.join(base64_frames)}"}
                }
            new_content_list.append(part_message)
        message["content"] = new_content_list
        vllm_messages.append(message)
    return vllm_messages, {'fps': fps_list}


video_messages, video_kwargs = prepare_message_for_vllm(video_messages)
chat_response = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    messages=video_messages,
    extra_body={
        "mm_processor_kwargs": video_kwargs
    }
)
print("Chat response:", chat_response)

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

Thanks, can you update the docs with this example? We don't have to test this as the code is straightforward enough.

[wulipc]

Thanks, can you update the docs with this example? We don't have to test this as the code is straightforward enough.

ok, which docs in vLLM should be updated? This case has been updated in our Qwen repo.

[DarkLight1337]

You can add it under Online Serving of this page

[DarkLight1337]

Also, it would be best to verify that your PR works with #13516 once it's merged.

[wulipc]

You can add it under Online Serving of this page

Currently, only the Qwen2.5-VL model requires passing the mm_processor_kwargs parameter, and the above case is a bit cumbersome. Given the specificity of this use case and to maintain the simplicity of the vLLM documentation, I prefer not to include this case in the vLLM documentation. If users have related needs, they can refer to this issue or the official Qwen documentation for more details.

[ywang96]

You can add it under Online Serving of this page

Currently, only the Qwen2.5-VL model requires passing the mm_processor_kwargs parameter, and the above case is a bit cumbersome. Given the specificity of this use case and to maintain the simplicity of the vLLM documentation, I prefer not to include this case in the vLLM documentation. If users have related needs, they can refer to this issue or the official Qwen documentation for more details.

I think it's okay if you want to include this example in https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_client_for_multimodal.py (maybe video_with_kwargs as another chat-type), but yea as you mentioned, it's probably also a better idea to update this in the README of https://github.com/QwenLM/Qwen2.5-VL since this is only relavant to Qwen2.5VL

[wulipc]

update this in the README of https://github.com/QwenLM/Qwen2.5-VL since this is only relavant to Qwen2.5VL

Let's keep it simple and only update this in the README of the Qwen2.5-VL: https://github.com/QwenLM/Qwen2.5-VL.

[DarkLight1337]

I just realized that fps can be a list of float, can you update the model file with the correct type annotation? Otherwise LGTM

[wulipc]

Also, it would be best to verify that your PR works with #13516 once it's merged.

@DarkLight1337 I found that after merging the fix into the main branch, passing the fps parameter through mm_processor_kwargs (mm_processor_kwargs: {'fps': []}) results in the following error. I checked and found that it was caused by the fps parameter being of list type.

[DarkLight1337]

Can you try converting it into a tuple inside the processor?

[DarkLight1337]

Another way would be to construct HashableList classes that use the tuple of its elements as the hash (similar to HashableDict)

[wulipc]

HashableList

@DarkLight1337 I change the list to a tuple in the merge_mm_kwargs function. Using a tuple instead of HashableList should also be fine, right? Everything seems to be working fine so far, and after merging, the PR works with #13516.

[DarkLight1337]

To follow HF's type hints, let's use HashableList

[wulipc]

To follow HF's type hints, let's use HashableList

done

[ywang96]

LGTM - Thanks for the continuous contribution to vLLM!