7 ms·

Really looking forward to this, 27B is a struggle with a strix halo and Laguna 2.1 can do stupid things for tooling calls.

cpburns2009 24d ago
Yeah 27B is way too slow for the Strix Halo. Laguna was better but still slow when I tried it. Qwen3.6 35B is still the best today.
- deleted 24d ago
  [deleted]
- cyanydeez 24d ago
  it'll hopefully improve with more MoE and half the prefill/generation. I think it's the sweet spot for the strix halo for smarter or vibe tasks.
- puzzlingcaptcha 24d ago
  What sort of pp/tg speed do you get on a Strix Halo?
  - cpburns2009 24d ago
    This is the best I got, all with Unsloth's quantizations. Laguna-S-2.1:UD-Q4_K_XL (no MTP) pp=186.4 t/s tg=27.8 t/s Qwen3.6-35B:UD-Q4_K_XL (with MTP) pp=404.4 t/s tg=83.2 t/s Qwen3.6-27B:UD-Q4_K_XL (recorded pre-MTP) pp=343 t/s tg=12.1 t/s Laguna actually performed better than I remembered. I thought it was slower.
    - deleted 24d ago
      [deleted]
    - htrp 24d ago
      Have you benchmarked against full precision models for accuracy/ performance?
      - cpburns2009 24d ago
        Not full precision. I've only benchmarked 27B across Q3-6 quants using lm-eval. I lack the hardware to bench 27B at BF16 but I might be able to do Q8_0. I haven't gotten around to doing 35B. I really should upload my collection of results to Github or somewhere. Here's a summary of what I have for 27B. I used unsloth's UD-Q{3-6}_K_XL quants across 11 evals. The values are pretty linear between Q3 and Q6. Qwen3.6-27B Q3 Q6 ARC-Challenge 97.0 97.0 BIG-Bench Hard 57.9 59.3 GPQA Diamond 77.8 83.3 GSM8K 92.4 92.6 Hendrycks Math 35.5 38.9 HumanEval 80.5 85.4 HumanEval+ 75.0 79.3 IFEval 87.3 88.0 MBPP 75.2 77.2 MBPP+ 88.4 88.9 MMLU-Pro 83.1 83.5
  - ascii0eks84 24d ago
    What are pp/tg? I get 30t/s on 27B qwen.
    - throwawayffffas 24d ago
      pp is prompt processing how fast it processes the prompt. Tg is token generation how fast, it generates tokens.
      - ascii0eks84 19d ago
        okay, so prefill and decode would be the terms I was already familiar with.
- SparkyMcUnicorn 24d ago
  Have you given Ornith-1.5-35B a shot? It's been a pretty decent step up for me compared to Qwen3.6 https://news.ycombinator.com/item?id=49362401 https://news.ycombinator.com/item?id=49362401
  - cpburns2009 24d ago
    I know it benchmarks very well. I haven't tried it yet though.
corysama 24d ago
So, I know https://cactuscompute.com/needle https://cactuscompute.com/needle is designed only to enable tool calling on tiny devices. But, I wonder if anyone has used it as a CPU-side mediator between a tool and a GPU-side local LLM making semi-natural-language tool requests...

cpburns2009 24d ago

Yeah 27B is way too slow for the Strix Halo. Laguna was better but still slow when I tried it. Qwen3.6 35B is still the best today.

deleted 24d ago
[deleted]
cyanydeez 24d ago
it'll hopefully improve with more MoE and half the prefill/generation. I think it's the sweet spot for the strix halo for smarter or vibe tasks.
puzzlingcaptcha 24d ago
What sort of pp/tg speed do you get on a Strix Halo?
- cpburns2009 24d ago
  This is the best I got, all with Unsloth's quantizations. Laguna-S-2.1:UD-Q4_K_XL (no MTP) pp=186.4 t/s tg=27.8 t/s Qwen3.6-35B:UD-Q4_K_XL (with MTP) pp=404.4 t/s tg=83.2 t/s Qwen3.6-27B:UD-Q4_K_XL (recorded pre-MTP) pp=343 t/s tg=12.1 t/s Laguna actually performed better than I remembered. I thought it was slower.
  - deleted 24d ago
    [deleted]
  - htrp 24d ago
    Have you benchmarked against full precision models for accuracy/ performance?
    - cpburns2009 24d ago
      Not full precision. I've only benchmarked 27B across Q3-6 quants using lm-eval. I lack the hardware to bench 27B at BF16 but I might be able to do Q8_0. I haven't gotten around to doing 35B. I really should upload my collection of results to Github or somewhere. Here's a summary of what I have for 27B. I used unsloth's UD-Q{3-6}_K_XL quants across 11 evals. The values are pretty linear between Q3 and Q6. Qwen3.6-27B Q3 Q6 ARC-Challenge 97.0 97.0 BIG-Bench Hard 57.9 59.3 GPQA Diamond 77.8 83.3 GSM8K 92.4 92.6 Hendrycks Math 35.5 38.9 HumanEval 80.5 85.4 HumanEval+ 75.0 79.3 IFEval 87.3 88.0 MBPP 75.2 77.2 MBPP+ 88.4 88.9 MMLU-Pro 83.1 83.5
- ascii0eks84 24d ago
  What are pp/tg? I get 30t/s on 27B qwen.
  - throwawayffffas 24d ago
    pp is prompt processing how fast it processes the prompt. Tg is token generation how fast, it generates tokens.
    - ascii0eks84 19d ago
      okay, so prefill and decode would be the terms I was already familiar with.
SparkyMcUnicorn 24d ago
Have you given Ornith-1.5-35B a shot? It's been a pretty decent step up for me compared to Qwen3.6 https://news.ycombinator.com/item?id=49362401 https://news.ycombinator.com/item?id=49362401
- cpburns2009 24d ago
  I know it benchmarks very well. I haven't tried it yet though.

corysama 24d ago

So, I know https://cactuscompute.com/needle https://cactuscompute.com/needle is designed only to enable tool calling on tiny devices. But, I wonder if anyone has used it as a CPU-side mediator between a tool and a GPU-side local LLM making semi-natural-language tool requests...