4 ms·
Transformers Are Multi-State RNNs
- Icko 3y agoI've seen at least 6 such papers, all being like "<popular architecture> are actually <a bit older concept>". Neural networks are generic enough that you can make them equivalent to almost everything.
- civilized 3y agoHas anybody proved that transformers are just kernel SVM yet?
- blackbear_ 3y agoI hope you're satisfied with Gaussian Processes: https://arxiv.org/abs/1806.07572 https://arxiv.org/abs/1806.07572
- deleted 3y ago[deleted]
- haswell 3y agoLinking the latest generation tech to the previous generations is actually really helpful from my perspective. All of the terminology for this tech is still emerging, and it can be quite difficult to formulate a reliable mental model for any of it due to how quickly it’s changing. If hammers were more difficult to understand, I could imagine someone writing about the fact that hammers are in fact, just a piece of steel mounted on a handle made of wood. > Neural networks are generic enough that you can make them equivalent to almost everything. Which to me is why papers like this are useful. They help newcomers conceptualize what the latest <popular architecture> is actually made of in terms of <a bit older concept> that the reader may already understand. It will take some time for this information space to stabilize.
- howrar 3y agoIt's also helpful in that you can now take insights from <older concept> and apply it to <popular architecture>. Many things are easier to reason about when framed a little differently.
- dboreham 3y agoSee: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_representation_theorem https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr...
- nh23423fefe 3y agooblique
- joewferrara 3y agoThey show that a decoder only transformer (which gpts are) are rnns with infinite hidden state size. Infinite hidden state size is a pretty strong thing! Sounds interesting to me.
- visarga 3y agonot infinite, just scaling linearly with length