7 ms·
Attention is basically routing, these other routing schemes put a less fine-grained choice for the model, which potentially makes it easier to train
by imranq 2y ago
Attention is basically routing, these other routing schemes put a less fine-grained choice for the model, which potentially makes it easier to train
- whimsicalism 2y agoHow is attention basically routing?
- visarga 2y agoIt routes values based on linear combinations taken from the attention map.
- whimsicalism 2y agoBut all of those values are created using an MLP with the same parameters, so there is no routing to different parameters.
- visarga 2y agoYou have to look at it as a sequence of time steps which can interact. You can implement this interaction in many ways, such as transformer, mamba, rwkv or mlp-mixer. But the purpose is always to allow communication across time. You use three distinct linear projections, one for queries, one for keys and one for values. From Q and K you compute the attention matrix A, and using A you construct linear combinations from V. But depending on A, for example for a token V_i there might be input from two other tokens, V_j or V_k, so information is moved between the tokens.
- pizza 2y agoThink of it like an edge flow matrix
- whimsicalism 2y agoThat doesn't clarify it for me. The same parameters are being used for every layer for every token. Yes, there is this differentiable lookup in attention like in MoE - but routing is about more than just differentiable lookup, it is about selecting on parameters not state.