4 ms·

What configuration are you using? On both vllm and llama-cpp, I get significantly higher speeds from gemma4 than qwen3.6 (with their respective speculative deco

What configuration are you using? On both vllm and llama-cpp, I get significantly higher speeds from gemma4 than qwen3.6 (with their respective speculative decoding methods). Output TPS in vllm for instance: - Gemma4 26B-A4B: 200-300TPS - Qwen3.6 35B-A3B: 120-180TPS - Gemma4 31B: 80-120TPS - Qwen3.6 27B: 60-80TPS This is for a first request on a dual 5090 setup, with their respective speculative decoding methods.

petu 1mo ago
Single 3090 under llama.cpp: | model | size | test | t/s | | ------------------- | ------- | ------ | ---- | | gemma4 31B Q4_0 | 16.1 GB | pp2048 | 1248 | | gemma4 31B Q4_0 | 16.1 GB | tg512 | 40 | | qwen35 27B Q4_K | 15.9 GB | pp2048 | 1248 | | qwen35 27B Q4_K | 15.9 GB | tg512 | 39 | | gemma4 26B.A4B Q4_0 | 13.3 GB | pp2048 | 4304 | | gemma4 26B.A4B Q4_0 | 13.3 GB | tg512 | 160 | | qwen35 35B.A3B Q3_K | 15.7 GB | pp2048 | 3329 | | qwen35 35B.A3B Q3_K | 15.7 GB | tg512 | 144 | > with their respective speculative decoding methods You're benchmarking drafter acceptance rate, then. Which is real life values, yes, but attributing worse drafter performance to the other 95% of the model being inherently slower.
- trouve_search 29d ago
  I think it's a vllm vs llama_cpp performance thing, will pay more into it. One note I had between the two is that gemma has a much higher prefix cache hit rate in general.
xfalcox 1mo ago
Have you tried running it on a single 5090? Dual 5090 require https://github.com/aikitoria/open-gpu-kernel-modules https://github.com/aikitoria/open-gpu-kernel-modules for higher perf. Are you using TP?
- trouve_search 29d ago
  Yes, I mentioned the setup, but on vllm you can only use TP with speculative decoding or pipeline parallelism without, so there's tradeoff to both. I gave general numbers of what I'm getting above, the performance ratios seemed similar regardless of setup (eg. getting a AWQ-in4 quant on a single GPU vs PP without speculative decoding vs TP with speculative decoding). Overall single GPU is fastest, and TP+speculative decoding is still faster than PP, but for fp8 models you need dual GPUs whether you want it or not.
mirekrusin 1mo ago
Dual 4090, getting 85-113 t/s depending on task (draft seems to speed up quite a lot, disproportionately more for content like svg etc): ./llama.cpp/llama-server \ -hf unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL \ --webui-mcp-proxy \ --no-mmproj \ --parallel 1 \ --kv-unified \ --flash-attn on \ --fit off \ --split-mode tensor \ -ngl 999 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ -ub 256 \ --no-context-shift \ --host 0.0.0.0 \ --tools all \ --jinja \ --ctx-size 262144 \ --spec-type draft-mtp \ --spec-draft-n-max 3 \ --reasoning on \ --chat-template-kwargs '{"reasoning_effort":"medium"}' \ --reasoning-preserve \ --temp 1.0 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 0.0 \ --repeat-penalty 1.0 Use claude/codex/whatever with /goal to optimize params for you. IMHO draft model support on dense models is great alternative to MoE on GPUs (high bandwidth, less memory) – more intelligence, speed somewhere mid way there which is usually sufficient.
- trouve_search 29d ago
  thanks for posting your setup! I think it's smart to set the reasoning effort default to something saner in the base config. Here's a VLLM command for 3.6 (I'll update to 3.8 today) to test out: ``` PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ vllm serve Qwen/Qwen3.6-27B-FP8 \ --dtype auto \ --kv-cache-dtype fp8 \ --enable-chunked-prefill \ --enable-prefix-caching \ --trust-remote-code \ --enable-auto-tool-choice \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}' \ --default-chat-template-kwargs '{ "enable_thinking": true, "reasoning_effort":"medium" }' \ --tensor-parallel-size 2 \ --max-model-len 250000 \ --gpu-memory-utilization 0.9 \ --max-num-batched 12000 \ --max-num-seqs 24 ``` I took the liberty of adding your reasoning effort chat template to my setup. You can play around with the last few parameters. In generall VLLM will be better in higher concurrency scenarios, so if you only use it for a personal vibe coding assistant and less as a general home model for task execution llama.cpp may be better.
  - mirekrusin 29d ago

petu 1mo ago

Single 3090 under llama.cpp: | model | size | test | t/s | | ------------------- | ------- | ------ | ---- | | gemma4 31B Q4_0 | 16.1 GB | pp2048 | 1248 | | gemma4 31B Q4_0 | 16.1 GB | tg512 | 40 | | qwen35 27B Q4_K | 15.9 GB | pp2048 | 1248 | | qwen35 27B Q4_K | 15.9 GB | tg512 | 39 | | gemma4 26B.A4B Q4_0 | 13.3 GB | pp2048 | 4304 | | gemma4 26B.A4B Q4_0 | 13.3 GB | tg512 | 160 | | qwen35 35B.A3B Q3_K | 15.7 GB | pp2048 | 3329 | | qwen35 35B.A3B Q3_K | 15.7 GB | tg512 | 144 | > with their respective speculative decoding methods You're benchmarking drafter acceptance rate, then. Which is real life values, yes, but attributing worse drafter performance to the other 95% of the model being inherently slower.

trouve_search 29d ago
I think it's a vllm vs llama_cpp performance thing, will pay more into it. One note I had between the two is that gemma has a much higher prefix cache hit rate in general.

xfalcox 1mo ago

Have you tried running it on a single 5090? Dual 5090 require https://github.com/aikitoria/open-gpu-kernel-modules https://github.com/aikitoria/open-gpu-kernel-modules for higher perf. Are you using TP?

trouve_search 29d ago
Yes, I mentioned the setup, but on vllm you can only use TP with speculative decoding or pipeline parallelism without, so there's tradeoff to both. I gave general numbers of what I'm getting above, the performance ratios seemed similar regardless of setup (eg. getting a AWQ-in4 quant on a single GPU vs PP without speculative decoding vs TP with speculative decoding). Overall single GPU is fastest, and TP+speculative decoding is still faster than PP, but for fp8 models you need dual GPUs whether you want it or not.

mirekrusin 1mo ago

Dual 4090, getting 85-113 t/s depending on task (draft seems to speed up quite a lot, disproportionately more for content like svg etc): ./llama.cpp/llama-server \ -hf unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL \ --webui-mcp-proxy \ --no-mmproj \ --parallel 1 \ --kv-unified \ --flash-attn on \ --fit off \ --split-mode tensor \ -ngl 999 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ -ub 256 \ --no-context-shift \ --host 0.0.0.0 \ --tools all \ --jinja \ --ctx-size 262144 \ --spec-type draft-mtp \ --spec-draft-n-max 3 \ --reasoning on \ --chat-template-kwargs '{"reasoning_effort":"medium"}' \ --reasoning-preserve \ --temp 1.0 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 0.0 \ --repeat-penalty 1.0 Use claude/codex/whatever with /goal to optimize params for you. IMHO draft model support on dense models is great alternative to MoE on GPUs (high bandwidth, less memory) – more intelligence, speed somewhere mid way there which is usually sufficient.

trouve_search 29d ago
thanks for posting your setup! I think it's smart to set the reasoning effort default to something saner in the base config. Here's a VLLM command for 3.6 (I'll update to 3.8 today) to test out: ``` PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ vllm serve Qwen/Qwen3.6-27B-FP8 \ --dtype auto \ --kv-cache-dtype fp8 \ --enable-chunked-prefill \ --enable-prefix-caching \ --trust-remote-code \ --enable-auto-tool-choice \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}' \ --default-chat-template-kwargs '{ "enable_thinking": true, "reasoning_effort":"medium" }' \ --tensor-parallel-size 2 \ --max-model-len 250000 \ --gpu-memory-utilization 0.9 \ --max-num-batched 12000 \ --max-num-seqs 24 ``` I took the liberty of adding your reasoning effort chat template to my setup. You can play around with the last few parameters. In generall VLLM will be better in higher concurrency scenarios, so if you only use it for a personal vibe coding assistant and less as a general home model for task execution llama.cpp may be better.
- mirekrusin 29d ago