6 ms·
He forgot the tokens! It's not simple weights and numbers all the way down. The available output is pre-set by the tokens we allow it to predict. There was a
by bwest87 4mo ago
He forgot the tokens!
It's not simple weights and numbers all the way down. The available output is pre-set by the tokens we allow it to predict.
There was a whole bit in there about not having a language module or using words. But it does. We tell it.
Humans do not come pre programmed with a set of possible "tokens". We just figure it out and I believe that fact captures something very essential. Maybe the missing piece of AGI. The fact that humans can just be awash in pure sense data, and somehow just figure out what is important and what to do. Never ceases to amaze me.
- beering 4mo agoThe set of tokens is learned, more or less. So I don’t get what point you’re trying to make here. There’s not a human manually deciding what tokens make up the token dictionary.
- brainwad 3mo agoLLMs can work without tokens: https://arxiv.org/abs/2412.09871 https://arxiv.org/abs/2412.09871. Tokens are more or less a performance optimisation.