4 ms·

Can anyone recommend the perfect sweet spot for someone who wants to run their own inference?

by Axsuul 23d ago

AbsurdCensor 23d ago
For me it's be Strix Halo, 128gb machine, especially running Qwen models. Except when I bought it, it was $1,900, now it's $4,600 for the same box. (Wow that's insane) For tinkering and learning, it's been great. Tie it into something like Hermes and you have a pretty powerful AI assistant in a box. And when you need to step up your model, you just do something like OpenRouter and it makes it pretty easy.
- bsagdiyev 23d ago
  Seconded. I just got a Strix Halo box a few months back and it is great. Does everything I need it to.
rkangel 23d ago
Thinkstation PGX maybe? Got the recommendation from these articles: https://www.xda-developers.com/qwen-3-8-27b-reverse-engineering-job-frontier-model/ https://www.xda-developers.com/qwen-3-8-27b-reverse-engineer... https://www.xda-developers.com/lenovo-thinkstation-pgx-review/ https://www.xda-developers.com/lenovo-thinkstation-pgx-revie... But haven't had a chance to try it myself.
netsroht 22d ago
I have been looking for a good local setup for a while now. Qwen 3.8 27b is really good for a dense model of this size IMO. I already had an RTX 4090 and I forked ninfer [0] with the obsession to squeeze everything out of this card for this model. Results: 149 tok/s decode speed (aggregate with concurrency about ~270 tok/s) with prefill speeds faster than 2500 tok/s. And all of this with full 262k albeit quantized context. Fast prefill speed is really important when launching multiple clients such as opencode or pi at the same time and especially if they launch subagents. This is why I also implemented a caching tier so computed contexts can be faster loaded from RAM (or disk). Speeds feel almost like with official SOTA openai or anthropic models. Im currently measuring a pareto front in J/tok in order to set power limits of this card without sacrificing too much performance. Since we are talking about full power draw of ~480W which is fine during the day (with solar panels) but during night when the sun doesn't shine (even with a battery) I'd like to limit this a little bit. [0] https://github.com/tensorninja/ninfer-4090 https://github.com/tensorninja/ninfer-4090
- ilteris 18d ago
  Could you please tell me what kind of machine you are using to operate it?
  - netsroht 17d ago
    Sure, just consumer grade products: an ~8yr old 16 core AMD Threadripper CPU + 64 GB RAM, Samsung NVMe drive(s) and obviously an nvidia RTX 4090 (watercooled). Only the GPU is important though.

Can anyone recommend the perfect sweet spot for someone who wants to run their own inference?

by Axsuul 23d ago

Can anyone recommend the perfect sweet spot for someone who wants to run their own inference?

For me it's be Strix Halo, 128gb machine, especially running Qwen models. Except when I bought it, it was $1,900, now it's $4,600 for the same box. (Wow that's insane) For tinkering and learning, it's been great. Tie it into something like Hermes and you have a pretty powerful AI assistant in a box. And when you need to step up your model, you just do something like OpenRouter and it makes it pretty easy.

bsagdiyev 23d ago
Seconded. I just got a Strix Halo box a few months back and it is great. Does everything I need it to.

rkangel 23d ago

Thinkstation PGX maybe? Got the recommendation from these articles: https://www.xda-developers.com/qwen-3-8-27b-reverse-engineering-job-frontier-model/ https://www.xda-developers.com/qwen-3-8-27b-reverse-engineer... https://www.xda-developers.com/lenovo-thinkstation-pgx-review/ https://www.xda-developers.com/lenovo-thinkstation-pgx-revie... But haven't had a chance to try it myself.

netsroht 22d ago

I have been looking for a good local setup for a while now. Qwen 3.8 27b is really good for a dense model of this size IMO. I already had an RTX 4090 and I forked ninfer [0] with the obsession to squeeze everything out of this card for this model. Results: 149 tok/s decode speed (aggregate with concurrency about ~270 tok/s) with prefill speeds faster than 2500 tok/s. And all of this with full 262k albeit quantized context. Fast prefill speed is really important when launching multiple clients such as opencode or pi at the same time and especially if they launch subagents. This is why I also implemented a caching tier so computed contexts can be faster loaded from RAM (or disk). Speeds feel almost like with official SOTA openai or anthropic models. Im currently measuring a pareto front in J/tok in order to set power limits of this card without sacrificing too much performance. Since we are talking about full power draw of ~480W which is fine during the day (with solar panels) but during night when the sun doesn't shine (even with a battery) I'd like to limit this a little bit. [0] https://github.com/tensorninja/ninfer-4090 https://github.com/tensorninja/ninfer-4090

ilteris 18d ago
Could you please tell me what kind of machine you are using to operate it?
- netsroht 17d ago
  Sure, just consumer grade products: an ~8yr old 16 core AMD Threadripper CPU + 64 GB RAM, Samsung NVMe drive(s) and obviously an nvidia RTX 4090 (watercooled). Only the GPU is important though.