5 ms·
You'd need hundreds of GB alone for the KV cache of each user. For something like LLama 3 405B you need ~67GB at ~130k tokens. A single CS-3 has 44GB on-chip sr
by philipportner 1mo ago
You'd need hundreds of GB alone for the KV cache of each user. For something like LLama 3 405B you need ~67GB at ~130k tokens.
A single CS-3 has 44GB on-chip sram.
So, afaik, Cerebras are optimizing for ultra-low latency batch=1 inference.
https://newsletter.semianalysis.com/p/cerebras-faster-tokens-please https://newsletter.semianalysis.com/p/cerebras-faster-tokens... goes quite in-depth.
- YetAnotherNick 1mo agoLLama 3 405B had the most unoptimized kv cache usage by far. Deepseek v4 pro uses 2.4GB for the same context length[1]. [1]: https://vllm.ai/blog/2026-04-24-deepseek-v4 https://vllm.ai/blog/2026-04-24-deepseek-v4
- philipportner 1mo agoGood point, thanks! I haven't been keeping up with most of the new model internals.