6 ms·
> Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope ot
by spider-mario 8d ago
> Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it.
Have you tried it with MTPLX? I get around 30 tok/s with it, also on an M1 Max with 64GB.
- Xeoncross 8d agoNice, which model quantization is this? Is it on huggingface?
- spider-mario 8d agoMTPLX is this software: https://github.com/youssofal/MTPLX https://github.com/youssofal/MTPLX I tried it with the author’s 4-bit quant of Qwen 3.8 27B: https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized... (but no need to download it manually; MTPLX will ask which one you want).
- SwellJoe 8d agoEven at 30 t/s, 3.8 thinks so long, even on medium, it still takes 3x or more longer than any cloud model, in my testing.
- lowbloodsugar 8d agoI've got an M1 Max 64GB too. It's just not an LLM-class workstation. Give it a year and buy an M7 and you'll be laughing. Right now is a really bad time to invest in anything - using the cloud is the cheapest option, especially for open weight models.
- bellowsgulch 8d agoThanks, man! I’ll go use that now that I know. llama-server the last time I used it for inference with this model wasn’t able to produce work fast enough to reach those numbers.