6 ms·

text is mostly beaten to death, so you can expect good defaults to work for vllm. VLMs specifically are quite sensitive to quantization, especially if you want

text is mostly beaten to death, so you can expect good defaults to work for vllm. VLMs specifically are quite sensitive to quantization, especially if you want it to do fine-grained localization (time or spatial), and the vllm default params can be way off for your use-case. For example, most vision models don't need 256K context length for modes like Qwen3.8-27B when all you care about is single-image captioning, so you can technically save on KV cache. Video reasoning does require that context length, so it's a different set of deployment parameters that need to be enabled. All of this to say that the providers that offer these models, are simply using vLLM / SGLang, and mostly cater to the text inference use-case (coding, etc). Vision always seems to be a bit of an afterthought.

msp26 5d ago
Got it, I'll test more deployment params when I benchmark models for vision tasks. Do you have any tips for Gemma 4 31B in particular? I quite like the model but I feel like I'm underutilizing my rented GPU hard due to skill issues. Throughput should be way higher than this, right? --- Gemma 4 31B NVFP4, vLLM 0.29, single B200 (modal), FlashInfer, fp8 KV, prefix caching, 32k ctx Workload: ~6k-token shared prefix (~98% cache hit) + short input, ~250 tokens out, ~250 seqs running. Get ~5-6k output tok/s, flat from 128 to 384 concurrency. Spec decode (DFlash, n-gram) didn't give a good boost which is annoying because I feel like this task should be easy for a draft model to predict. Individual request latency doesn't matter I just need as much throughput as possible. Its only active for a few hours when I need it.