5 ms·
If active weight bandwidth is everything, then wouldn't Qwen 35b a3b (FP8) be roughly half the cost of dsv4f (FP4)? dsv4f appears bigger and more memory/comput
by SparkyMcUnicorn 1mo ago
If active weight bandwidth is everything, then wouldn't Qwen 35b a3b (FP8) be roughly half the cost of dsv4f (FP4)?
dsv4f appears bigger and more memory/computationally expensive to run any way I look at it.
- mordae 1mo agoYeah, it should be basically free. No idea why it is not. I guess KV cache taking up RAM and possibly bad business sense or amortized engineering costs, I honestly do not know.
- petu 1mo ago> and more memory/computationally expensive to run any way I look at it. Just speculating, but look at attention. 35B-A3B context is heavier -- about 20GB per 1M tokens vs ~7GB. So if you have ≥10M tokens of context per inference node, then DS4 Flash uses less memory than Qwen 35B. Qwen attention is probably more computationaly expensive as well. Also I guess not much interest/competition in serving Qwen edge model.