[Bug]: base64 string leads to gibberish with latest vLLM server and pixtral-12b

[michael-brunzel]

Your current environment

The output of `python collect_env.py`

PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.12.8 (main, Dec  4 2024, 08:54:12) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.8.0-1021-aws-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A10G
Nvidia driver version: 550.127.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        48 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               8
On-line CPU(s) list:                  0-7
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 7R32
CPU family:                           23
Model:                                49
Thread(s) per core:                   2
Core(s) per socket:                   4
Socket(s):                            1
Stepping:                             0
BogoMIPS:                             5599.99
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            128 KiB (4 instances)
L1i cache:                            128 KiB (4 instances)
L2 cache:                             2 MiB (4 instances)
L3 cache:                             16 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-7
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow:   Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] flashinfer==0.1.6+cu121torch2.4
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] torch==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.47.1
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.6.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-7     0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.1.0
LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/cv2/../../lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
CUDA_MODULE_LOADING=LAZY

Model Input Dumps

No response

🐛 Describe the bug

I use the following snippet https://huggingface.co/mistralai/Pixtral-12B-2409/discussions/6 to create a base64 string which is sent as a payload to a docker container which is spin up from the latest vllm-image. I run the following model: https://huggingface.co/SeanScripts/pixtral-12b-nf4 and only get gibberish as model output. Everything is run within an EC2 server with a g5.2xlarge VM (A10 GPU).
If I don't pass the base64 string but a regular image url, everything works as intended. The prompt seems to be correctly formatted.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

Can you show the base64 string you sent?

[michael-brunzel]

Sure please find the string attached. I was also able to decode it again for recreation of the image. Btw I updated the output of python collect_env.py
base64_str.txt

[DarkLight1337]

When you use OpenAI API, image_url should be a valid base64 data URL (with a prefix like data:image/jpg;base64,, not just the base64 data).

[michael-brunzel]

I did that already. The file is just the base64 data though.

[DarkLight1337]

Are you sure the file format (e.g. jpg) in the data URL is correct?

[michael-brunzel]

Yes I also tried JPG and jpeg file and corresponding data URLs, but it still yields (sometimes human readable, but gibberish results).

[DarkLight1337]

Is it possible for you to share a HTTP link to the image so I can test it?

[michael-brunzel]

Sure please find the demo image attached. The base_str file from above is the corresponding base64 string.

[DarkLight1337]

Quick question: Do you get similar issues using the original HF model, or only on the quantized model? Can you show the command you used to serve vLLM?

[michael-brunzel]

And a link to it: https://drive.google.com/file/d/1UkhiFtWTDYNvbVg-berCkPu7GKhQDQ1k/view?usp=sharing

[michael-brunzel]

I did not try the original HF model, but I thought it should not be a quantization issue, since a image url works

[michael-brunzel]

My docker run command is: docker run -d --runtime nvidia --gpus all -v ~/.cache/huggingface:/root/.cache/huggingface -p 8000:8000 --ipc=host pixtral-vllm-4bit

and in the CMD of my dockerfile I have: ["--model", "SeanScripts/pixtral-12b-nf4", "--quantization", "bitsandbytes", "--trust-remote-code", "--load-format", "bitsandbytes", "--max-model-len", "8192", "--served-model-name", "pixtral", "--chat-template", ".."]

[DarkLight1337]

What is this chat template that you're using? From my understanding, the model should already define one so there is no need to override it.

[michael-brunzel]

It is a custom one, since the chat template can't be inferred, because the model is not auto recognized because it is not main repo but a quantized one on HF. So I also tested the url for the picture and the base64 output is consistent with output of the image url. So the image url and base64 probably also seems to work for an example image like https://picsum.photos/id/237/400/300 but not for my image. Are there specific requirements for the image?

[michael-brunzel]

Can confirm that https://picsum.photos/id/237/400/300 also works as a base64 string

[DarkLight1337]

It is possible that the performance of the model may not be consistent for very detailed images, especially since you're using a quantized model.

[DarkLight1337]

But, there should be no difference between base64 and HTTP URL. Can you set the temperature to zero and see if the outputs are the same for your image?

[michael-brunzel]

the outputs are the same for the other example images and also for my image. For the example image it works well (reasonable answer), but not for my image.

[michael-brunzel]

"It is possible that the performance of the model may not be consistent for very detailed images, especially since you're using a quantized model." No i tested the quantized model also in a notebook (no vllm) and it worked fine on that image

[michael-brunzel]

So for smaller, simpler images it seems to be work. Maybe it is tied to the way vLLM dequantizes. I am not entirely sure, that it works perfectly for SeanScripts/pixtral-12b-nf4. The performance is definitely not consistent with running the quantized model with HF though.

[DarkLight1337]

@mgoin could you offer some insights on this?