6 ms·
Can you explain like I'm 5 why this matters distinctly from how transformers are normally trained with autodiff and what its possible applications are?
by LukeB42 4y ago
Can you explain like I'm 5 why this matters distinctly from how transformers are normally trained with autodiff and what its possible applications are?
- adamnemecek 4y agoI’m talking about attention only transformers. Those don’t have an autodiff but still learn. The math is actually really cool.
- lostmsu 4y ago> attention only transformers Can you share any good link on the subject?
- adamnemecek 4y agohttps://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2021/framework/index.html
- lostmsu 4y agoMaybe I am missing something, but I don't see any learning without autodiff.
- adamnemecek 4y agoI thought you were asking about attention only transformers. This paper touches on some of it https://arxiv.org/abs/2212.10559v2 https://arxiv.org/abs/2212.10559v2.
- lostmsu 4y agoThe paper speculates that it is analogous to gradient descent and empirically confirms it is similar in behavior, but it is not a rigorous proof of any kind. The momentum experiment they made also does not seem related. E.g. it just adds past values to V, which extends the effective context length.
- adamnemecek 4y ago> but it is not a rigorous proof of any kind. Such is the nature of early theories.