6 ms·

I'm using qwen 3.6 35B unsloth 4 bit with my 5950x (128 gb memory) and a 3060 12 gb gpu with a self made harness. At 10k context I get about 40 tps generation

by tyfon 2mo ago

I'm using qwen 3.6 35B unsloth 4 bit with my 5950x (128 gb memory) and a 3060 12 gb gpu with a self made harness. At 10k context I get about 40 tps generation and 500 tps prefill. At 100k context I get about 25 tps generation and 400 tps prefill. It works, but I often use gpt or claude to make a detailed enumerated plan of what I want to do first, then have qwen follow it. I'm not sure if it is economical or not, but I have solar on the roof so the power use is not really an issue and I already have the hardware. The biggest benefit for me is that it's all done locally, and I know the harness is not uploading anything or sending telemetry to someone else.

johnvanommen 2mo ago
> The biggest benefit for me is that it's all done locally, and I know the harness is not uploading anything or sending telemetry to someone else. Are there any articles you’d recommend for this? I have Qwen running on an HP Z8. Very nice platform. I have mine in a sandbox, due to privacy fears. Your solution sounds more elegant.
- tyfon 2mo ago
  Articles regarding my own harness or how I set up llama.cpp etc? I really just iterated over the harness over and over for about two weeks with opencode until I was sort of satisfied (still lots to do there :). For the llama.cpp I asked claude fable to optimize it for my hardware and iterated a few times. In the end I landed on the following: https://pastebin.com/2PpJFUC0 https://pastebin.com/2PpJFUC0