6 ms·
Why was this your first thought? Is a limiting factor to transformers the MLP layer? I thought the bottleneck was in the renormalization part.
by mp187 2y ago
Why was this your first thought? Is a limiting factor to transformers the MLP layer? I thought the bottleneck was in the renormalization part.
- brrrrrm 2y agoAt small input size, yes the MLP dominates compute. At large input attention matters more