7 ms·
I built a similar system, meanwhile I've sold one of the RTX 3090's. Local inference is fun and feels liberating, but it's also slow, and once I was used to the
by lifeinthevoid 1y ago
I built a similar system, meanwhile I've sold one of the RTX 3090's. Local inference is fun and feels liberating, but it's also slow, and once I was used to the immense power of the giant hosted models, the fun quickly disappeared.
I've kept a single GPU to still be able to play a bit with light local models, but not anymore for serious use.
- nenenejej 1y agoGraphics cards are so expensive (list price) they are cheap (no depreciation liquid market)
- Our_Benefactors 1y agoDid you really claim GPUs have zero depreciation? That’s obviously false.
- deleted 1y ago[deleted]
- imiric 1y agoI have a similar setup as the author with 2x 3090s. The issue is not that it's slow. 20-30 tk/s is perfectly acceptable to me. The issue is that the quality of the models that I'm able to self-host pales in comparison to that of SOTA hosted models. They hallucinate more, don't follow prompts as well, and simply generate overall worse quality content. These are issues that plague all "AI" models, but they are particularly evident on open weights ones. Maybe this is less noticeable on behemoth 100B+ parameter models, but to run those I would need to invest much more into this hobby than I'm willing to do. I still run inference locally for simple one-off tasks. But for anything more sophisticated, hosted models are unfortunately required.
- mycall 1y ago> 20-30 tk/s or ~2.2M tk/day. This is how we should be thinking about it imho.
- elsombrero 1y agoOn my 2x 3090s I am running glm4.5 air q1 and it runs at ~300pp and 20/30 tk/s works pretty well with roo code on vscode, rarely misses tool calls and produces decent quality code. I also tried to use it with claude code with claude code router and it's pretty fast. Roo code uses bigger contexts, so it's quite slower than claude code in general, but I like the workflow better. this is my snippet for llama-swap ``` models: "glm45-air": healthCheckTimeout: 300 cmd: | llama.cpp/build/bin/llama-server -hf unsloth/GLM-4.5-Air-GGUF:IQ1_M --split-mode layer --tensor-split 0.48,0.52 --flash-attn on -c 82000 --ubatch-size 512 --cache-type-k q4_1 --cache-type-v q4_1 -ngl 99 --threads -1 --port ${PORT} --host 0.0.0.0 --no-mmap -hfd mradermacher/GLM-4.5-DRAFT-0.6B-v3.0-i1-GGUF:Q6_K -ngld 99 --kv-unified ```
- imiric 1y agoThanks, but I find it hard to believe that a Q1 model would produce decent results. I see that the Q2 version is around 42GB, which might be doable on 2x 3090s, even if some of it spills over to CPU/RAM. Have you tried Q2?
- elsombrero 1y agowell, I tried it and it works for me. llm output is hard to properly evaluate without actually using it. I read a lot of good comments on r/localllama, with most people suggesting qwen3 coder 30ba3b, but I never got it to work as well as GLM 4.5 air Q1. As for using Q2, it will fit in vram, but with very small context or spill over to RAM, but with quite an impact on speed depending on your setup. I have slow ddr4 ram and going for Q1 has been a good compromise for me, but YMMV.
- ericdotlee 1y agoWhat is llama-swap? Been looking for more details about software configs on https://llamabuilds.ai https://llamabuilds.ai
- elsombrero 1y ago
- ThatPlayer 1y ago> behemoth 100B+ parameter models, but to run those I would need to invest much more into this hobby than I'm willing to do. Have you tried newer MoE models with llama.cpp's recent '--n-cpu-moe' option to offload MoE layers to the CPU? I can run gpt-oss-120b (5.1B active) on my 4080 and get a usable ~20 tk/s. Had to upgrade my system RAM, but that's easier. https://github.com/ggml-org/llama.cpp/discussions/15396 https://github.com/ggml-org/llama.cpp/discussions/15396 has a bit on getting that running
- imiric 1y agoI use Ollama which offloads to the CPU automatically IIRC. IME the performance drops dramatically when that happens, and it hogs the CPU making the system unresponsive for other tasks, so I try to avoid it.
- ThatPlayer 1y agoI don't believe that's the same thing. That should be the generic offloading that ollama will do to any too big model, while this feature requires MoE models. https://github.com/ollama/ollama/issues/11772 https://github.com/ollama/ollama/issues/11772 is the feature request for similar on ollama. One comment in that thread mentions getting almost 30tk/s from gpt-oss-120b on a 3090 with llama.cpp compared to 8tk/s with ollama. This feature is limited to MoE models, but those seem to be gaining traction with gpt-oss, glm-4.5, and qwen3
- imiric 1y agoAh, I was not aware of that, thanks. I'll give it a try.
- NicoJuicy 1y agoIf you have a 24 gb 3090. Try out qwen:30b-a3b-instruct-2507-q4_K_M ( ollama ) It's pretty good.
- naabb 1y agotbf I also run that on a 16GB 5070TI at 25T/S, it's amazing how fast it runs on consumer grade hardware. I think you could push up to a bigger model but I don't know enough about local llama.
- jszymborski 1y agoDon't need a 3090, it runs really fast on an RTX 2080 too.