5 ms·
AFAIK You can use WSE/LPU for prefill, it's just less efficient to do so.
by Eridrus 22d ago
AFAIK You can use WSE/LPU for prefill, it's just less efficient to do so.
- porridgeraisin 22d agoWell ya, that efficiency is why it's split. There is also the other idea where you run your attention layer on the GPU/TPU/Trainium and the FFN on the SRAM accelerator. Because KV cache is more difficult on cerebras etc, while MOE latency is easier to deal with