6 ms·

how much performance (tok/s) can you expect from 128GB Strix Halo? assuming this model will be released with FP8 also can you use it for fine tuning?

SwellJoe 23d ago
The Strix Halo and DGX Spark are pretty danged slow, relatively speaking. I don't recall exact numbers, but with MoE models in this size ballpark (Laguna S 2.1), I seem to recall I was seeing about 20-25 t/s with a big context, which is close to usable. Qwen 3.8 27B crawls on this hardware, though, at 10-16 t/s, definitely not comfortable for interactive use. (Though this makes it seem like you can cook pretty good with a 4-bit ROCmFP4 quantization: https://github.com/julianmb/q38rocm https://github.com/julianmb/q38rocm the model does get notably dumber below six bits.) A model similar in size to Laguna S 2.1, but with only 6B active parameters, should be a notable amount faster, so I would imagine 25-30 t/s would be a reasonable guess for where Qwen 3.8 Flash Next will land. DFlash2 might improve all these numbers. It wasn't available last I was testing new models on the Strix Halo; I've only used MTP (which doesn't generally improve MoE models, but I believe DFlash2 can). Given software improvements, I'm hopeful an MoE in this size range will be the sweet spot that pushes past 40 t/s and is also smart enough for real work. Qwen 3.8 27B is finally a self-hostable model that's smart enough, but it thinks so hard it still isn't really useful for agentic interactive use. Note also prefill with large models is pretty slow on the Strix Halo (300 t/s, maybe). Time to first token is a painful wait, when using it interactively with large models.
- decide1000 22d ago
  On the DGX I get 44.5 tokens per second (NVFP4). With 8 concurrent it's 241 t/s total. I am using the PrismaAQUA standard 9.7 t/s + Dflash2 30 t/s + torch-compile 37 t/s c8 = 177 t/s
  - SwellJoe 22d ago
    What model? Also, I don't know what "the PrismaAQUA" means, ddg thinks it's a CPAP machine, which seems unlikely to help with inference performance. Also, 4-bit has measurable intelligence loss. Sometimes worth it, but, at this size models are barely smart enough at 8 or 6.
    - decide1000 22d ago
      Qwen3.8-27B-PrismaAQUA-5.5bit-vllm The output quality is higher. It's held at full precision (not quantized).
downrightmike 22d ago
You can only use up to 90gb for the GPU, so it doesn't fit
- SwellJoe 22d ago
  No. In Linux, you can ignore the BIOS (well, set it to 512MB) and set GTT to allow almost the entirety of memory for GPU. If you don't run a desktop GUI (and why would you), you don't need memory for much of anything else. Just a few GB for the OS and llama.cpp. These are roughly the settings I use: https://github.com/kyuz0/amd-strix-halo-toolboxes#kernel-parameters-tested-on-fedora-42 https://github.com/kyuz0/amd-strix-halo-toolboxes#kernel-par...

SwellJoe 23d ago

The Strix Halo and DGX Spark are pretty danged slow, relatively speaking. I don't recall exact numbers, but with MoE models in this size ballpark (Laguna S 2.1), I seem to recall I was seeing about 20-25 t/s with a big context, which is close to usable. Qwen 3.8 27B crawls on this hardware, though, at 10-16 t/s, definitely not comfortable for interactive use. (Though this makes it seem like you can cook pretty good with a 4-bit ROCmFP4 quantization: https://github.com/julianmb/q38rocm https://github.com/julianmb/q38rocm the model does get notably dumber below six bits.) A model similar in size to Laguna S 2.1, but with only 6B active parameters, should be a notable amount faster, so I would imagine 25-30 t/s would be a reasonable guess for where Qwen 3.8 Flash Next will land. DFlash2 might improve all these numbers. It wasn't available last I was testing new models on the Strix Halo; I've only used MTP (which doesn't generally improve MoE models, but I believe DFlash2 can). Given software improvements, I'm hopeful an MoE in this size range will be the sweet spot that pushes past 40 t/s and is also smart enough for real work. Qwen 3.8 27B is finally a self-hostable model that's smart enough, but it thinks so hard it still isn't really useful for agentic interactive use. Note also prefill with large models is pretty slow on the Strix Halo (300 t/s, maybe). Time to first token is a painful wait, when using it interactively with large models.

decide1000 22d ago
On the DGX I get 44.5 tokens per second (NVFP4). With 8 concurrent it's 241 t/s total. I am using the PrismaAQUA standard 9.7 t/s + Dflash2 30 t/s + torch-compile 37 t/s c8 = 177 t/s
- SwellJoe 22d ago
  What model? Also, I don't know what "the PrismaAQUA" means, ddg thinks it's a CPAP machine, which seems unlikely to help with inference performance. Also, 4-bit has measurable intelligence loss. Sometimes worth it, but, at this size models are barely smart enough at 8 or 6.
  - decide1000 22d ago
    Qwen3.8-27B-PrismaAQUA-5.5bit-vllm The output quality is higher. It's held at full precision (not quantized).

downrightmike 22d ago

You can only use up to 90gb for the GPU, so it doesn't fit

SwellJoe 22d ago
No. In Linux, you can ignore the BIOS (well, set it to 512MB) and set GTT to allow almost the entirety of memory for GPU. If you don't run a desktop GUI (and why would you), you don't need memory for much of anything else. Just a few GB for the OS and llama.cpp. These are roughly the settings I use: https://github.com/kyuz0/amd-strix-halo-toolboxes#kernel-parameters-tested-on-fedora-42 https://github.com/kyuz0/amd-strix-halo-toolboxes#kernel-par...