11 ms·
For what it's worth, I've spent some time with Claude to develop a local runner for `llama.cpp`. I run Qwen3.6-35B-A3B-MTP (fast!) and Qwen3.8-27B (20 tokens/s)
by jwr 15d ago
For what it's worth, I've spent some time with Claude to develop a local runner for `llama.cpp`. I run Qwen3.6-35B-A3B-MTP (fast!) and Qwen3.8-27B (20 tokens/s).
This was definitely worth the effort. Benchmarking and testing various approaches and various options really paid off. For example, one thing that surprised me was that MTP made things slower, not faster for Qwen3.8-27B.
I use a 64GB MacBook Pro (M4 Max).
- nixon_why69 15d agoI find mtp=3 does well with that model, only at 4 it becomes unprofitable. Check your quants, its worth having the mtp layer be a bigger quant if it leads to 2x throughput from more accepted tokens.
- ch_sm 15d agoI’m not an expert, but my understanding is that MTPs are smaller LLMs fine-tuned to "mimic" / predict a specific model’s response. It’s possible that the MTP you’re using isn’t trained well enough on Qwen 3.8. What accept rate are you getting?
- nixon_why69 15d agoFor qwen, it's an additional transformer layer at the very back, it ships as part of the model.
- jwr 11d agoAcceptance rate is good, but MTP doesn't help in my case because of my machine's memory bandwidth constraints (M4 Max). Turn out it's better to turn MTP off.