6 ms·
How does N relate to the number of parameters that is frequently mentioned?
by zorr 3y ago
How does N relate to the number of parameters that is frequently mentioned?
- throwawaymaths 3y agoIn my screed, N is the attention width. (How many token it looks at at a time) number of parameters is O(KxNxNxL) where k is the vector size of your tokens, and l is the # of layers. There are other parameters floating around, like in the encoder and decoder matrices, but the NXN matrix dominates.
- 8thcross 3y agoThis is an awesome explanation. You guys are the real heroes