6 ms·
Keep in mind that the model is thinking in a token space, itself a compressive representation of language. (Note: there's still a huge grammar penalty, so, ugh
by chaboud 1mo ago
Keep in mind that the model is thinking in a token space, itself a compressive representation of language.
(Note: there's still a huge grammar penalty, so, ugh do think small.)
- kzrdude 1mo agoIt selects tokens but they expand to embedding vectors which are huge, also in memory and attention requirements, I think?
- qeternity 1mo agoThe real breakthrough is going to be thinking in latent space.
- rcxdude 1mo agoArguably this is already happening: the whole state of the model gets fed through from token to token, and even just shoving a bunch of dashes in between the input tokens and the model's output can improve performance (thinking tokens from the model help a little bit more, but the difference is not as large as you mught expect).