7 ms·
RNN with extra steps?
by ottaborra 2y ago
RNN with extra steps?
- biofox 2y agoIsn't that all of modern AI?
- immibis 2y agoTransformers are completely unlike RNNs.
- tripplyons 2y agoThere are some interesting connections between them. If you remove the softmax from the attention formula, you end up with linear attention, which has a recurrent form. I haven't read it, but the Mamba 2 paper claims to establish a stronger connection.
- kadushka 2y ago* If you remove the softmax from the attention formula, you end up with linear attention* Sorry, what?
- tripplyons 2y agoHere is a paper explaining it: https://arxiv.org/abs/2006.16236 https://arxiv.org/abs/2006.16236
- tripplyons 2y agoThere are many papers that use a recurrence across sub-sequences and attention within sub-sequences. Google did this with Infini-Attention and one of the variants from the Titans paper. However, I think the earliest example of this is Transformer-XL.