6 ms·

Have you tried a mixture of experts model? Dense models have been quite slow for me, as I have only 6 GB VRAM. But with llama.cpp and --cpu-moe I get 200 t/s in

Have you tried a mixture of experts model? Dense models have been quite slow for me, as I have only 6 GB VRAM. But with llama.cpp and --cpu-moe I get 200 t/s input and almost 30 t/s output with Gemma 4 26B A3B, which feels ok to use. Would be interested about your mileage there.

b112 25d ago
I wish qwen3.8 had a MoE variant, but the skinny is it won't be coming.
tharkun__ 25d ago
If I use the 12B Unified (dense) model I mentioned without MTP, then I get 37t/s, input ~700t/s. It's all still quite frustrating in the end, like a Claude from a very long time ago by now but usable. If I want 64k context, I can't use MTP. I still haven't decided whether I'd rather have 37t/s but it's "less dumb" or I want MTP speed but it's going off the rails more. All of this is also with `-ctv q4_0 -ctk q4_0)`, which is not ideal. I'm actually right now contending with 35k context but using q8_0 KV quantization. More like 35t/s coz with those settings I can't use MTP. But I'm not ready to go back to 4t/s. It's not interactive enough for me. That said, I had tried to use the Gemma E4B for example to have it build itself that websearch/fetch skill. It utterly failed, as did previous qwens. I don't see a Gemma 4 26B A3B GGUF for download, but there is a gemma-4-26B-A4B-it-MXFP4_MOE.gguf that should fit into my overall RAM and then use lots of CPU like the Qwen 3.8. I guess I'll give it a try just to see the difference in speed though I don't expect anything "usable" out of that tbh.
- tharkun__ 25d ago
  Haha, OK, so it processes input tokens at 55t/s if started like the 12B. That's so slow that I don't even have the patience to wait for the context of the session I aborted mid-processing to test this, to load. Heck loading the system prompt alone is not fun. That said, after that it seems to run at the same ~35t/s speed but, I was able to start it up with a 128k context (and no KV quantization), humm humm. I am now playing more with all the parameters like I did to get the 12B going e.g. I'm already up to ~125t/s for inputs now and with an 8bit MTP it's running about 45-85t/s output when its coding. I run with `--spec-type draft-mtp --spec-draft-p-min 0.8 --spec-draft-n-max 8` and when it was that fast (during actual code generation) got some `draft acceptance = 0.91801 ( 3482 accepted / 3793 generated), mean len = 7.92`. I also tried 16 with no real improvement for those phases but it seemed slower, wasting too much with low 50% acceptances during other phases like thinking. EDIT: Oh look, I can now use larger batch sizes again and that gets prompt_eval reads back to 700+ Regarding how good it is: Probably, maybe, better than the 12B? It immediately noticed an error the 12B made, where it hadn't followed a skill's format description. But it's also annoyingly not following instructions the other one was and instead just gives me its opinion lol! And it couldn't figure out how to read a simple pi SKILL.md and run an included script. It complained that it wasn't registered as a tool. But like all other model changes, will get used to it and adjust I suppose :)
  - aktenlage 20d ago
    Thank you, very interesting to read. I haven't considered batch sizes at all up to now. Seems I should look into it. How did you get the prediction that good? The default spec-draft-max-n is 3 and if I go beyond, the output rate stays the same or gets lower any magic spell you could share?
    - aktenlage 18d ago
      Wow, setting the "ubatch" size to 2048 (default is 512), like the "batch" size roughly doubled prompt processing speed.
    - tharkun__ 16d ago
      Regarding the MTP, if you haven't found out why yet, the `--spec-draft-p-min 0.8`. Something to play with and of course it's non-deterministic and depends on both the model itself and whatever you're actually throwing at it. This is what I read before I went to try it: https://github.com/ggml-org/llama.cpp/discussions/25198 https://github.com/ggml-org/llama.cpp/discussions/25198

b112 25d ago

I wish qwen3.8 had a MoE variant, but the skinny is it won't be coming.

tharkun__ 25d ago

If I use the 12B Unified (dense) model I mentioned without MTP, then I get 37t/s, input ~700t/s. It's all still quite frustrating in the end, like a Claude from a very long time ago by now but usable. If I want 64k context, I can't use MTP. I still haven't decided whether I'd rather have 37t/s but it's "less dumb" or I want MTP speed but it's going off the rails more. All of this is also with `-ctv q4_0 -ctk q4_0)`, which is not ideal. I'm actually right now contending with 35k context but using q8_0 KV quantization. More like 35t/s coz with those settings I can't use MTP. But I'm not ready to go back to 4t/s. It's not interactive enough for me. That said, I had tried to use the Gemma E4B for example to have it build itself that websearch/fetch skill. It utterly failed, as did previous qwens. I don't see a Gemma 4 26B A3B GGUF for download, but there is a gemma-4-26B-A4B-it-MXFP4_MOE.gguf that should fit into my overall RAM and then use lots of CPU like the Qwen 3.8. I guess I'll give it a try just to see the difference in speed though I don't expect anything "usable" out of that tbh.

tharkun__ 25d ago
Haha, OK, so it processes input tokens at 55t/s if started like the 12B. That's so slow that I don't even have the patience to wait for the context of the session I aborted mid-processing to test this, to load. Heck loading the system prompt alone is not fun. That said, after that it seems to run at the same ~35t/s speed but, I was able to start it up with a 128k context (and no KV quantization), humm humm. I am now playing more with all the parameters like I did to get the 12B going e.g. I'm already up to ~125t/s for inputs now and with an 8bit MTP it's running about 45-85t/s output when its coding. I run with `--spec-type draft-mtp --spec-draft-p-min 0.8 --spec-draft-n-max 8` and when it was that fast (during actual code generation) got some `draft acceptance = 0.91801 ( 3482 accepted / 3793 generated), mean len = 7.92`. I also tried 16 with no real improvement for those phases but it seemed slower, wasting too much with low 50% acceptances during other phases like thinking. EDIT: Oh look, I can now use larger batch sizes again and that gets prompt_eval reads back to 700+ Regarding how good it is: Probably, maybe, better than the 12B? It immediately noticed an error the 12B made, where it hadn't followed a skill's format description. But it's also annoyingly not following instructions the other one was and instead just gives me its opinion lol! And it couldn't figure out how to read a simple pi SKILL.md and run an included script. It complained that it wasn't registered as a tool. But like all other model changes, will get used to it and adjust I suppose :)
- aktenlage 20d ago
  Thank you, very interesting to read. I haven't considered batch sizes at all up to now. Seems I should look into it. How did you get the prediction that good? The default spec-draft-max-n is 3 and if I go beyond, the output rate stays the same or gets lower any magic spell you could share?
  - aktenlage 18d ago
    Wow, setting the "ubatch" size to 2048 (default is 512), like the "batch" size roughly doubled prompt processing speed.
  - tharkun__ 16d ago
    Regarding the MTP, if you haven't found out why yet, the `--spec-draft-p-min 0.8`. Something to play with and of course it's non-deterministic and depends on both the model itself and whatever you're actually throwing at it. This is what I read before I went to try it: https://github.com/ggml-org/llama.cpp/discussions/25198 https://github.com/ggml-org/llama.cpp/discussions/25198