5 ms·
"Multi-token prediction gives a free speedup of up to 2x on many models" - at the expense of halving prompt processing speed
by skolos 2mo ago
"Multi-token prediction gives a free speedup of up to 2x on many models" - at the expense of halving prompt processing speed
- hedgehog 2mo agoMTP has no effect on prompt processing.
- skolos 2mo agoInteresting - in my setup (llama.cpp rtx5090 qwen-3.6 27b) prompt processing with mtp is almost half vs non mtp. Sounds like I need to investigate what is wrong.
- hedgehog 2mo agoMaybe enabling MTP causes some weights to be displaced to host memory? MTP itself doesn't do anything during prefill so that should be exactly unchanged, decode will vary depending on settings but with 2-4 proposals depending on workload I've never seen an overall slowdown. edit: I recommend building recent llama.cpp from source, I've been updating about once a week, as there has been a fair amount of work related to MTP recently. If you're running a lot of tool calling on Qwen you might also benefit from one of the bugfixed chat templates like the Froggeric version.
- tingletech 2mo agoit works quite will for me in llama.cpp, but I get more like 20% to 40% speedup on tokens per second. I generally use spec-type = draft-mtp,ngram-mod spec-draft-n-max = 4 I have not observed any effect on prompt processing, which is usually an order of magnitude faster than generation on my spark.