6 ms·
DeepSeek V4 Flash is natively FP4 MoE with very compact KV cache. Say 8 GB/s. Qwen 27B is about 60 GB/s at full FP16 precision.
by mordae 1mo ago
DeepSeek V4 Flash is natively FP4 MoE with very compact KV cache. Say 8 GB/s. Qwen 27B is about 60 GB/s at full FP16 precision.
- SparkyMcUnicorn 1mo agoIf active weight bandwidth is everything, then wouldn't Qwen 35b a3b (FP8) be roughly half the cost of dsv4f (FP4)? dsv4f appears bigger and more memory/computationally expensive to run any way I look at it.
- mordae 1mo agoYeah, it should be basically free. No idea why it is not. I guess KV cache taking up RAM and possibly bad business sense or amortized engineering costs, I honestly do not know.
- petu 1mo ago> and more memory/computationally expensive to run any way I look at it. Just speculating, but look at attention. 35B-A3B context is heavier -- about 20GB per 1M tokens vs ~7GB. So if you have ≥10M tokens of context per inference node, then DS4 Flash uses less memory than Qwen 35B. Qwen attention is probably more computationaly expensive as well. Also I guess not much interest/competition in serving Qwen edge model.