5 ms·

Which model/quant/command line did you use? I can barely get 100 token/secs and for sure clearly not a full context. With vllm, I am limited to 130k tokens with

Which model/quant/command line did you use? I can barely get 100 token/secs and for sure clearly not a full context. With vllm, I am limited to 130k tokens with vllm + nvfp4.

iv42 1mo ago
If you've got RTX 5090, maybe try ninfer (https://github.com/Neroued/ninfer https://github.com/Neroued/ninfer). Folks over on /r/localllama have been reporting wild prefill/token gen speeds with ninfer (NVFP4; 256k ctx).
- Foobar8568 1mo ago
  Looks like slope benchmarks and results, and as usually, people are mixing MTP numbers with non MTP numbers. Or just 100 token input benchmarks. Or just failed ones as actual measures. https://github.com/Neroued/ninfer/blob/master/docs/performance.md https://github.com/Neroued/ninfer/blob/master/docs/performan... Category MTP3 stochastic sampler DFlash stochastic sampler DFlash greedy Code 1/15 natural stops; 0/15 prompt-complete 2/15 natural stops; 0/15 prompt-complete 0/15 natural stops Story 9/15 natural stops; the nine Chinese outputs pass requested division and minimum length 8/15 natural stops; the eight Chinese outputs pass requested division and minimum length 10/15 natural stops; five Chinese dialogue outputs are under length Translation 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural checks Structured 0/15 satisfy the requested complete record/script contract 0/15 satisfy the requested complete record/script contract 0/15 satisfy the requested complete record/script contract And on my "own" "quick" benchmark, it's slower than vllm.
  - iv42 1mo ago
    I don't have a 5090, so I can't really comment, but here's the relevant reddit thread from today where they report the numbers (including ninfer ones), and where you can make your case: https://www.reddit.com/r/LocalLLaMA/comments/1vqjeub/how_many_tokenssecond_output_are_you_getting_with/ https://www.reddit.com/r/LocalLLaMA/comments/1vqjeub/how_man...
pich 1mo ago
[dead]

Which model/quant/command line did you use? I can barely get 100 token/secs and for sure clearly not a full context. With vllm, I am limited to 130k tokens with vllm + nvfp4.

iv42 1mo ago

If you've got RTX 5090, maybe try ninfer (https://github.com/Neroued/ninfer https://github.com/Neroued/ninfer). Folks over on /r/localllama have been reporting wild prefill/token gen speeds with ninfer (NVFP4; 256k ctx).

Foobar8568 1mo ago
Looks like slope benchmarks and results, and as usually, people are mixing MTP numbers with non MTP numbers. Or just 100 token input benchmarks. Or just failed ones as actual measures. https://github.com/Neroued/ninfer/blob/master/docs/performance.md https://github.com/Neroued/ninfer/blob/master/docs/performan... Category MTP3 stochastic sampler DFlash stochastic sampler DFlash greedy Code 1/15 natural stops; 0/15 prompt-complete 2/15 natural stops; 0/15 prompt-complete 0/15 natural stops Story 9/15 natural stops; the nine Chinese outputs pass requested division and minimum length 8/15 natural stops; the eight Chinese outputs pass requested division and minimum length 10/15 natural stops; five Chinese dialogue outputs are under length Translation 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural checks Structured 0/15 satisfy the requested complete record/script contract 0/15 satisfy the requested complete record/script contract 0/15 satisfy the requested complete record/script contract And on my "own" "quick" benchmark, it's slower than vllm.
- iv42 1mo ago
  I don't have a 5090, so I can't really comment, but here's the relevant reddit thread from today where they report the numbers (including ninfer ones), and where you can make your case: https://www.reddit.com/r/LocalLLaMA/comments/1vqjeub/how_many_tokenssecond_output_are_you_getting_with/ https://www.reddit.com/r/LocalLLaMA/comments/1vqjeub/how_man...

pich 1mo ago

[dead]