5 ms·

Reading stuff like this (as a layman), diminishing these things as 'Next token predictors' seems absurdly reductive. At some point we'll need to concede that 's

Reading stuff like this (as a layman), diminishing these things as 'Next token predictors' seems absurdly reductive. At some point we'll need to concede that 'selection' is a better term for this than prediction.

chrisjj 14d ago
> diminishing these things as 'Next token predictors' seems absurdly reductive. This shows a deep misunderstanding of the paper's claims, which in no way challenge the established view that these bots are next-token predictors. Regardless, if all you want is a next-token selector, save your money and roll a die.
- squidbeak 14d ago
  > This shows a deep misunderstanding of the paper's claims, which in no way challenge the established view that these bots are next-token predictors. No, this shows an appreciation of the symbolic richness behind that token 'prediction' which the paper leads on. > Regardless, if all you want is a next-token selector, save your money and roll a die. Tell me, where is the emergent symbology guiding that dice?
  - chrisjj 14d ago
    > No, this shows an appreciation of the symbolic richness behind that token 'prediction' which the paper leads on. The paper claims no symbolic richness beyond that evident from the undisputed next-token prediction. > Tell me, where is the emergent symbology guiding that dice? There's none. That's my point.
- xg15 14d ago
  > which in no way challenge the established view that these bots are next-token predictors. I mean, of course they are, that's literally what the inference loop does. You can look at the source of your favorite model runner and you'll see exactly that. What I find misleading about this term is that it focuses attention on the "next token" part and glosses over the "prediction" part as some sort of unspecified "statistical algorithm" - even though this is where most of the work happens and where the interesting questions are.
  - chrisjj 13d ago
    I've seen nothing to suggest it misleads anyone else.
    - xg15 13d ago
      There are other next token prediction algorithms such as markov chains or HMMs that also "fit the same interface", but are vastly simpler than what LLMs use. I've seen various posters talk about "stochastic parrots" or about how LLMs were just "using very simple word statistics" to get their results, which sounded to me very much as if they thought of LLMs simply as glorified markov chains. I would consider that a big misunderstanding.
      - chrisjj 13d ago
        No misunderstanding at all. "In machine learning, the term stochastic parrot is a metaphor that frames large language models as systems that statistically mimic text without real understanding." https://en.wikipedia.org/wiki/Stochastic_parrot https://en.wikipedia.org/wiki/Stochastic_parrot "Large Language Models as Markov Chains" https://arxiv.org/abs/2410.02724 https://arxiv.org/abs/2410.02724

chrisjj 14d ago

> diminishing these things as 'Next token predictors' seems absurdly reductive. This shows a deep misunderstanding of the paper's claims, which in no way challenge the established view that these bots are next-token predictors. Regardless, if all you want is a next-token selector, save your money and roll a die.

squidbeak 14d ago
> This shows a deep misunderstanding of the paper's claims, which in no way challenge the established view that these bots are next-token predictors. No, this shows an appreciation of the symbolic richness behind that token 'prediction' which the paper leads on. > Regardless, if all you want is a next-token selector, save your money and roll a die. Tell me, where is the emergent symbology guiding that dice?
- chrisjj 14d ago
  > No, this shows an appreciation of the symbolic richness behind that token 'prediction' which the paper leads on. The paper claims no symbolic richness beyond that evident from the undisputed next-token prediction. > Tell me, where is the emergent symbology guiding that dice? There's none. That's my point.
xg15 14d ago
> which in no way challenge the established view that these bots are next-token predictors. I mean, of course they are, that's literally what the inference loop does. You can look at the source of your favorite model runner and you'll see exactly that. What I find misleading about this term is that it focuses attention on the "next token" part and glosses over the "prediction" part as some sort of unspecified "statistical algorithm" - even though this is where most of the work happens and where the interesting questions are.
- chrisjj 13d ago
  I've seen nothing to suggest it misleads anyone else.
  - xg15 13d ago
    There are other next token prediction algorithms such as markov chains or HMMs that also "fit the same interface", but are vastly simpler than what LLMs use. I've seen various posters talk about "stochastic parrots" or about how LLMs were just "using very simple word statistics" to get their results, which sounded to me very much as if they thought of LLMs simply as glorified markov chains. I would consider that a big misunderstanding.
    - chrisjj 13d ago
      No misunderstanding at all. "In machine learning, the term stochastic parrot is a metaphor that frames large language models as systems that statistically mimic text without real understanding." https://en.wikipedia.org/wiki/Stochastic_parrot https://en.wikipedia.org/wiki/Stochastic_parrot "Large Language Models as Markov Chains" https://arxiv.org/abs/2410.02724 https://arxiv.org/abs/2410.02724