5 ms·
I'm getting like 25 tok/s on 2x RTX Pro 6000. This is with llama.cpp, but I had GPT tune it for me. I was under the impression vLLM was at most ~2x faster, and
by apitman 1mo ago
I'm getting like 25 tok/s on 2x RTX Pro 6000. This is with llama.cpp, but I had GPT tune it for me. I was under the impression vLLM was at most ~2x faster, and usually for highly parallel loads. Any tips on where I should look first for an obvious blunder?
I'm guessing tensor parallelism or similar?
- lcampbell 1mo agoYou're gonna want a custom vLLM build. Here's a runbook: https://github.com/local-inference-lab/rtx6kpro/blob/master/models/deepseek-v4-flash.md https://github.com/local-inference-lab/rtx6kpro/blob/master/... If the newer builds aren't working, you might try running the old v6 build (based on the eldritch-enlightenment image). gilded-gnosis gave me some problems that I haven't bothered to track down, the old builds are still gonna blow away llama-server performance. And that's before you get hooked on vLLM's PagedAttention and can run multiple sequences without a ton of extra overhead.