9 ms·

It seems like we forget that LLMs are next token prediction systems. Using raw models without instruction following and chat completion bells and whistles will

by mohsen1 3mo ago

It seems like we forget that LLMs are next token prediction systems. Using raw models without instruction following and chat completion bells and whistles will give you a better feeling of what LLMs are. The current interface to LLMs are heavily biased towards "predict the next token in the context of a user with a helpful assistant" but LLMs are capable of other modes of next token prediction too. Before the ChatGPT release people often measured LLM performance by how well they could produce a coherent story or a poem. that's where Anthropic model names are originating from I am guessing.

post-it 3mo ago
> It seems like we forget that LLMs are next token prediction systems. It's pretty clear to me that above a certain size threshold, LLMs are more than a sum of their parts. The sheer amount of training data seems to embed a higher level of reasoning.
- gmueckl 3mo ago
  There cannot be any reasoning embedded in the model. The algorithm is literally "predict the most likely next token". Anything beyond that is just patterns in the predictions fooling us humans into ascribing more to the system than it is actually producing.
  - thepasch 3mo ago
    > The algorithm is literally "predict the most likely next token". That's confusing the training objective with the learned behavior. It's like saying "Stockfish's algorithm is literally 'minimize this number', and therefore, it can't actually play Chess."
    - gmueckl 3mo ago
      Not a valid comparison. Chess algorithms are built around the rules of chess, most notably the turn taking nature of the game (min/max with alpha/beta pruning based on lists of valid moves in any position).
      - etskinner 3mo ago
        Who's to say the "rules of reasoning" aren't just predicting the next thing that an intelligent person (you) would do? Emergent behavior isn't magic, it's just emergent.
  - wizzwizz4 3mo ago
    LLMs are great big finite state machines. Finite state machines can perform mechanical reasoning. A priori, there can be reasoning embedded in the model. I agree that (these) LLMs don't generally reason (even when they're writing words like "I reason that, since X, we have Y, therefore Z"), but that's not because a model inherently cannot do that.
  - aspenmartin 3mo ago
    This is just a misconception of how LLMs work and also what reasoning is. “There cannot be any reasoning embedded in the model” a strong statement, what do you mean by reasoning because by any reasonable definition I’m aware of, they clearly are able to exhibit reasoning. The fact that the pre training objective is next token loss has nothing to do with capabilities or their ability to reason. To be highly successful at next token prediction you NEED to reason. I’m quite confused here.
    - applicative 3mo ago
      LLM output is a kind of dreaming but with the whole of past human text output as dream material. It turns out to be useful if you can direct the hallucination
    - gmueckl 3mo ago
      LLM output produces the illusion of reasoning. The underlying computation, however, is not reasoning.
      - aspenmartin 3mo ago
        If you don’t mind actually taking a few more words to be more specific that would be helpful because what you’re saying doesn’t really make sense at all. You don’t need to trust that the reasoning traces are all faithful representation of an internal reasoning trace. Plenty of other ways to probe models (see anthropics work using circuit tracing).
        gmueckl 3mo ago
        What else is there to say? LLMs can at most regurgitate approximations of human reasoning steps in the limited forms in which they may be expressed in the training data or interpolations thereof. That's the core essence of what they are. There is no proper reasoning to be found.
        aspenmartin 3mo ago
  - linzhangrun 3mo ago
    Quantity change leads to quality change. You can check out this Kurzgesagt video on emergence: https://www.youtube.com/watch?v=16W7c0mb-rE https://www.youtube.com/watch?v=16W7c0mb-rE
- geon 3mo ago
  No. The reasoning is just an illusion. As soon as you look at it closer, it falls apart completely. https://yosefk.com/blog/llms-arent-world-models.html https://yosefk.com/blog/llms-arent-world-models.html https://www.reddit.com/r/programming/comments/1mnc9qf/llms_arent_world_models/ https://www.reddit.com/r/programming/comments/1mnc9qf/llms_a... https://news.ycombinator.com/item?id=44854518 https://news.ycombinator.com/item?id=44854518
  - post-it 3mo ago
    This is an excellent comment from your HN link: > The core issue there isn't that the LLM isn't building internal models to represent its world, it's that its world is limited to tokens. Anything not represented in tokens, or token relationships, can't be modeled by the LLM, by definition. I've noticed that LLMs struggle with tic tac toe, even when I get them to draw it out. And it makes sense! Tic tac toe is an extremely visual game, and difficult to reason about in a single dimension.
- solid_fuel 3mo ago
  > It's pretty clear to me that above a certain size threshold, LLMs are more than a sum of their parts. The sheer amount of training data seems to embed a higher level of reasoning. You're staring at a mirror. I have never seen any compelling evidence that LLMs actually reason. They are good at predicting the next token, and that is all they do. This action may be useful, but it is not reasoning.
jhbadger 3mo ago
The problem with that argument is it is trivial to write a Markov chain program that takes in text and then can generate the most probable series of words given a starting word. I myself wrote such a program in BASIC on a 64K 8-bit computer in the 1980s after reading one of A.K. Dewdney's columns. That wasn't at all an LLM though. There's a connection, sure, but one that is equating a paper airplane to a jet airliner.
- Charon77 3mo ago
  The issue with Markov Chain is you can't get good next token prediction on long enough context because once you see the last 1000 words instead of just 2, it's quite unlikely that your 'frequency' is populated for that exact combination, and markov chain don't work on token embedding that allows some encoding of meaning.
  - AlecSchueler 3mo ago
    > and markov chain don't work on token embedding that allows some encoding of meaning. Working on an "encoding of meaning" sure sounds a lot like reasoning.
    - tanseydavid 3mo ago
      Consider the classic ambiguity example from Natural Language Processing: "Time flies like an arrow; fruit flies like a banana". Markov chains are mostly about memorized local co-occurrence, while transformers are much better at context-sensitive disambiguation and compositional structure. I do not have deep knowledge about these subjects but this is what I think was meant by "encoding of meaning" (LLMs). I also do not think this capability has to be thought of as "reasoning".

by mohsen1 3mo ago

post-it 3mo ago

> It seems like we forget that LLMs are next token prediction systems. It's pretty clear to me that above a certain size threshold, LLMs are more than a sum of their parts. The sheer amount of training data seems to embed a higher level of reasoning.

gmueckl 3mo ago
There cannot be any reasoning embedded in the model. The algorithm is literally "predict the most likely next token". Anything beyond that is just patterns in the predictions fooling us humans into ascribing more to the system than it is actually producing.
- thepasch 3mo ago
  > The algorithm is literally "predict the most likely next token". That's confusing the training objective with the learned behavior. It's like saying "Stockfish's algorithm is literally 'minimize this number', and therefore, it can't actually play Chess."
  - gmueckl 3mo ago
    Not a valid comparison. Chess algorithms are built around the rules of chess, most notably the turn taking nature of the game (min/max with alpha/beta pruning based on lists of valid moves in any position).
    - etskinner 3mo ago
      Who's to say the "rules of reasoning" aren't just predicting the next thing that an intelligent person (you) would do? Emergent behavior isn't magic, it's just emergent.
- wizzwizz4 3mo ago
  LLMs are great big finite state machines. Finite state machines can perform mechanical reasoning. A priori, there can be reasoning embedded in the model. I agree that (these) LLMs don't generally reason (even when they're writing words like "I reason that, since X, we have Y, therefore Z"), but that's not because a model inherently cannot do that.
- aspenmartin 3mo ago
  This is just a misconception of how LLMs work and also what reasoning is. “There cannot be any reasoning embedded in the model” a strong statement, what do you mean by reasoning because by any reasonable definition I’m aware of, they clearly are able to exhibit reasoning. The fact that the pre training objective is next token loss has nothing to do with capabilities or their ability to reason. To be highly successful at next token prediction you NEED to reason. I’m quite confused here.
  - applicative 3mo ago
    LLM output is a kind of dreaming but with the whole of past human text output as dream material. It turns out to be useful if you can direct the hallucination
  - gmueckl 3mo ago
    LLM output produces the illusion of reasoning. The underlying computation, however, is not reasoning.
    - aspenmartin 3mo ago
      If you don’t mind actually taking a few more words to be more specific that would be helpful because what you’re saying doesn’t really make sense at all. You don’t need to trust that the reasoning traces are all faithful representation of an internal reasoning trace. Plenty of other ways to probe models (see anthropics work using circuit tracing).
      - gmueckl 3mo ago
        What else is there to say? LLMs can at most regurgitate approximations of human reasoning steps in the limited forms in which they may be expressed in the training data or interpolations thereof. That's the core essence of what they are. There is no proper reasoning to be found.
        aspenmartin 3mo ago
- linzhangrun 3mo ago
  Quantity change leads to quality change. You can check out this Kurzgesagt video on emergence: https://www.youtube.com/watch?v=16W7c0mb-rE https://www.youtube.com/watch?v=16W7c0mb-rE
geon 3mo ago
No. The reasoning is just an illusion. As soon as you look at it closer, it falls apart completely. https://yosefk.com/blog/llms-arent-world-models.html https://yosefk.com/blog/llms-arent-world-models.html https://www.reddit.com/r/programming/comments/1mnc9qf/llms_arent_world_models/ https://www.reddit.com/r/programming/comments/1mnc9qf/llms_a... https://news.ycombinator.com/item?id=44854518 https://news.ycombinator.com/item?id=44854518
- post-it 3mo ago
  This is an excellent comment from your HN link: > The core issue there isn't that the LLM isn't building internal models to represent its world, it's that its world is limited to tokens. Anything not represented in tokens, or token relationships, can't be modeled by the LLM, by definition. I've noticed that LLMs struggle with tic tac toe, even when I get them to draw it out. And it makes sense! Tic tac toe is an extremely visual game, and difficult to reason about in a single dimension.
solid_fuel 3mo ago
> It's pretty clear to me that above a certain size threshold, LLMs are more than a sum of their parts. The sheer amount of training data seems to embed a higher level of reasoning. You're staring at a mirror. I have never seen any compelling evidence that LLMs actually reason. They are good at predicting the next token, and that is all they do. This action may be useful, but it is not reasoning.

jhbadger 3mo ago

The problem with that argument is it is trivial to write a Markov chain program that takes in text and then can generate the most probable series of words given a starting word. I myself wrote such a program in BASIC on a 64K 8-bit computer in the 1980s after reading one of A.K. Dewdney's columns. That wasn't at all an LLM though. There's a connection, sure, but one that is equating a paper airplane to a jet airliner.

Charon77 3mo ago
The issue with Markov Chain is you can't get good next token prediction on long enough context because once you see the last 1000 words instead of just 2, it's quite unlikely that your 'frequency' is populated for that exact combination, and markov chain don't work on token embedding that allows some encoding of meaning.
- AlecSchueler 3mo ago
  > and markov chain don't work on token embedding that allows some encoding of meaning. Working on an "encoding of meaning" sure sounds a lot like reasoning.
  - tanseydavid 3mo ago
    Consider the classic ambiguity example from Natural Language Processing: "Time flies like an arrow; fruit flies like a banana". Markov chains are mostly about memorized local co-occurrence, while transformers are much better at context-sensitive disambiguation and compositional structure. I do not have deep knowledge about these subjects but this is what I think was meant by "encoding of meaning" (LLMs). I also do not think this capability has to be thought of as "reasoning".