4 ms·
Cerebras is targeting a distinctly different point on the cost/latency curve. They are betting that there will be some high value applications where latency and
by Eridrus 22d ago
Cerebras is targeting a distinctly different point on the cost/latency curve. They are betting that there will be some high value applications where latency and not just throughput is super important.
- porridgeraisin 22d agoIt is being used as part of a combined system. For example AWS is pushing for Trainium + WSE 3. The WSE 3 does the decode and the Trainium does the prefill. Even in nvidia land rubin + LPU does a similar thing. It has its downsides of course - if your traffic swings prefill heavy to decode heavy, you can't suddenly use your lpu for prefill. With GPUs they're totally interchangeable. Tradeoffs.
- Eridrus 22d agoAFAIK You can use WSE/LPU for prefill, it's just less efficient to do so.
- porridgeraisin 22d agoWell ya, that efficiency is why it's split. There is also the other idea where you run your attention layer on the GPU/TPU/Trainium and the FFN on the SRAM accelerator. Because KV cache is more difficult on cerebras etc, while MOE latency is easier to deal with