4 ms·
Note this is the 07/31 release of DSv4 flash and not the "preview" that they put out a couple months or so ago. I've been running this model locally for a week
by ak_t 1mo ago
Note this is the 07/31 release of DSv4 flash and not the "preview" that they put out a couple months or so ago.
I've been running this model locally for a week, and the preview version before that. This updated one feels like a whole tier up. It's very capable for debugging and analyzing documents/data I upload.
The killer feature, IMO, is the speed. On 2x RTX Pro 6000 Blackwell, its ~8k tok/s prefill and ~250 tok/s on a single stream. I saw 1000 tok/s with ~64 concurrent streams on vLLM.
That's fast enough that you can interactively chat with it without switching tabs while you wait, and its a ~300B (13B active, hence the speed) model so the responses are also very good. It's actually more convenient now for me to direct 95%+ of my day to day usage to my local model, and only use Claude Fable for really big coding tasks.
Until this model was released, I was contemplating spending even more money on hardware to run GLM5.2 (~750B) at reasonable speeds, but I no longer feel that need. This is smart enough, and I think it only gets much better for local models from here.
- MrDrMcCoy 1mo agoI wish I could say that it performs reasonably on my hardware. 8x Radeon AI Pro 9700XTs, and I can't get it to hit double-digit tokens per second. Neither vllm nor llama.cpp, with various combinations of quants, draft models, and parallelism methods can get it to run at a tolerable speed. I'll be sticking with StepFun 3.7-Flash for the foreseeable future :(
- ComputerGuru 1mo agoWhat quantization level is that? Because official endpoints are slow.
- olejorgenb 1mo agoI get around 60-110 TPS on tensorx.ai hosted models.
- bel8 1mo agoFrom opencode go $10/mo plan I get between 60 t/s and 100 token/s even with large contexts of 150k+ tokens. I wouldn't call 80 t/s slow.
- ponyous 1mo agoYou are right, relatively to other llm providers this is not slow. But if you think what is possible when you have 1000t/s a sec you might find it slow.
- hatefulmoron 1mo agoThat's across 64 concurrent streams; you could make more concurrent requests to DeepSeek API no?
- ak_t 1mo agoIt doesn't need extra quantization. The official weights are natively mixed precision FP4/FP8, so it fits in ~160GB. The API slowness is probably from being batched with other concurrent user requests. The provider's aggregate throughput gets higher but per-stream speed slows down.
- zargon 1mo agoV4 Flash fits entirely in two RTX Pro 6000s without any quantization at all.
- apitman 1mo agoI'm getting like 25 tok/s on 2x RTX Pro 6000. This is with llama.cpp, but I had GPT tune it for me. I was under the impression vLLM was at most ~2x faster, and usually for highly parallel loads. Any tips on where I should look first for an obvious blunder? I'm guessing tensor parallelism or similar?
- lcampbell 1mo agoYou're gonna want a custom vLLM build. Here's a runbook: https://github.com/local-inference-lab/rtx6kpro/blob/master/models/deepseek-v4-flash.md https://github.com/local-inference-lab/rtx6kpro/blob/master/... If the newer builds aren't working, you might try running the old v6 build (based on the eldritch-enlightenment image). gilded-gnosis gave me some problems that I haven't bothered to track down, the old builds are still gonna blow away llama-server performance. And that's before you get hooked on vLLM's PagedAttention and can run multiple sequences without a ton of extra overhead.
- namr2000 1mo agoWhat runtime are you using with the 2x RTX Pro 6000 Blackwell machine? I have the same setup and tried DSv4 Flash on vLLM and ran into a ton of kernel bugs that don't seem to have been fixed yet.
- deleted 1mo ago[deleted]
- mshachkov 1mo agoThis one works great. https://github.com/ormandj/sglang-deepseek-v4-flash-sm120 https://github.com/ormandj/sglang-deepseek-v4-flash-sm120
- natsucks 1mo agoso..you guys have $20k+ rigs?
- complex1314 1mo agoThis also works great: https://github.com/antirez/ds4 https://github.com/antirez/ds4
- kouteiheika 1mo ago> On 2x RTX Pro 6000 Blackwell, its ~8k tok/s prefill and ~250 tok/s on a single stream. For reference, on a 1x B300 it's over 400 tok/s decode on a single stream.