6 ms·
You and I have the same machine. Do you mind sharing the model ID you're using? I'm on oMLX also but I haven't seen anything above ~20 tok/s out of 3.8 27B, eve
by anon373839 19d ago
You and I have the same machine. Do you mind sharing the model ID you're using? I'm on oMLX also but I haven't seen anything above ~20 tok/s out of 3.8 27B, even with MTP and generating code.
- seanmcdirmid 18d agohttps://huggingface.co/Jundot/Qwen3.6-35B-A3B-oQ6-fp16-mtp https://huggingface.co/Jundot/Qwen3.6-35B-A3B-oQ6-fp16-mtp is my favorite right now (and nothing else is even close), you should be able to get 80-90 tok/sec enabling MTP and disabling turboquant. Supposedly the 6-bit quant is essential for decent MTP performance, although I didn’t test the 4-bit quant.