12 ms·

> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a

> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins. i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?

Veedrac 12d ago
The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.
- mannykannot 12d ago
  Yes, but I think the same construction could also be used to characterize the first system; it determines the next move based on a prediction of its reward signal, where its reward signal is a measure of how likely it is that a grand master would make that move. Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.
  - Veedrac 12d ago
    A classically pretrained LLM does not have a concept of having determined its previous tokens — it has only ever observed inputs that it had no causal influence over. This is why it's valid to say its actions are predictive and not determinative.
- jayd16 12d ago
  > _determining_ the next token based on a prediction Also known as predicting.
  - Borealid 12d ago
    I think the most useful word in both cases is "extrapolating". An LLM extrapolates from its context window to the immediate next token. This word applies whether you view what's happening as "reasoning", "prediction", or as a math function.
  - Veedrac 12d ago
    No, those aren't synonyms at all. If I steer a car to avoid a predicted collision with a wall, this is not me 'predicting' the car. I am steering the car based on a prediction.
    - Dylan16807 12d ago
      If you're assigning steering 70 of your 100 output points because it's what you think we should go with most of the time in this situation, I'm going to call that a prediction of how to steer.
      - MrScruff 12d ago
        The point was, if your internal model of the world makes a prediction of a negative outcome at some point in the future, and you optimise your individual actions to avoid that negative outcome, then wouldn’t it make sense to focus on the fact you’re building and optimizing towards an internal world model rather than the fact you’re executing your actions one at a time in series?
        Dylan16807 11d ago
        If you purged and rebuilt that works model every action, the consequences of doing so would be worth a good amount of focus. And if you didn't make the final decision on the action yourself that would be worth mentioning.
danielmarkbruce 12d ago
The word "predict" has a meaning. I don't "predict" my next move in chess. I might predict what someone elses first move is.
- stanleykm 12d ago
  In any case this is all very pedantic. In the process of selecting a move to make there is a prediction. Whether that prediction is the opponent’s next move or what your next move should be based on the game’s existing state, there is a prediction that the next move you make will improve your chance to win. Maybe the probability in that selection is 100%. You have no other possible move. It doesn’t matter. All we are doing here as far as I can tell is arguing over where the prediction happens and whether that counts as predicting something.
  - danielmarkbruce 12d ago
    There is no truth for RLHF or RLVR. You can't predict against something if you can't check against the truth. It's not pedantry. The objective function changes. The optimization changes. THese are real things when training a model, not hand wavy philosophical ideas.
- deleted 12d ago
  [deleted]
- ordersofmag 12d ago
  The LLM does not determine the next token. It generate odds for all of the tokens it knows as to their likelihood of being 'next'. It's up to the harness running the LLM (and in most cases the a temperature setting) to actually decide on a particular next token. I think it's more accurate to call the thing the LLM actually generates (an ensemble of probabilities) a 'prediction'. It might be accurate to say the harness decides on the next token based on the prediction from the LLM. The role of the LLM is much more akin to predicting your opponents move than deciding your own.
  - danielmarkbruce 12d ago
garrinm 12d ago
In the article I made 3 claims, and I agree it was a bit clumsy. 1st I say that "working forwards" in the sense of outputting one token at a time could be some form of prediction, I don't argue against that. This is what LLMs do at inference time. 2nd I say that to me what really constitutes a prediction is the pre-training. Here it's the classic setting for the word prediction in ML. The model outputs a prediction of the ground truth label: the next token. 3rd I argue that in RL there is no ground truth next token, so prediction doesn't apply here anymore. Back to your question then: you're asking points 3 and 1 are different. Working backwards from a set of win states is basically what RL does in training. Working forward from the current state is what inference does. To me there is a distinction worth thinking about. First between the mechanism at inference time and at train time. Then between what happens in pre-training vs. RL post training.
- vrighter 10d ago
  Neither is there a ground truth in rock paper scissors (the pattern varies by the player). But one would still be trying to detect patterns and predict the next "token" the opponent is going to output.

Veedrac 12d ago

The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.

mannykannot 12d ago
Yes, but I think the same construction could also be used to characterize the first system; it determines the next move based on a prediction of its reward signal, where its reward signal is a measure of how likely it is that a grand master would make that move. Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.
- Veedrac 12d ago
  A classically pretrained LLM does not have a concept of having determined its previous tokens — it has only ever observed inputs that it had no causal influence over. This is why it's valid to say its actions are predictive and not determinative.
jayd16 12d ago
> _determining_ the next token based on a prediction Also known as predicting.
- Borealid 12d ago
  I think the most useful word in both cases is "extrapolating". An LLM extrapolates from its context window to the immediate next token. This word applies whether you view what's happening as "reasoning", "prediction", or as a math function.
- Veedrac 12d ago
  No, those aren't synonyms at all. If I steer a car to avoid a predicted collision with a wall, this is not me 'predicting' the car. I am steering the car based on a prediction.
  - Dylan16807 12d ago
    If you're assigning steering 70 of your 100 output points because it's what you think we should go with most of the time in this situation, I'm going to call that a prediction of how to steer.
    - MrScruff 12d ago
      The point was, if your internal model of the world makes a prediction of a negative outcome at some point in the future, and you optimise your individual actions to avoid that negative outcome, then wouldn’t it make sense to focus on the fact you’re building and optimizing towards an internal world model rather than the fact you’re executing your actions one at a time in series?
      - Dylan16807 11d ago
        If you purged and rebuilt that works model every action, the consequences of doing so would be worth a good amount of focus. And if you didn't make the final decision on the action yourself that would be worth mentioning.

danielmarkbruce 12d ago

The word "predict" has a meaning. I don't "predict" my next move in chess. I might predict what someone elses first move is.

stanleykm 12d ago
In any case this is all very pedantic. In the process of selecting a move to make there is a prediction. Whether that prediction is the opponent’s next move or what your next move should be based on the game’s existing state, there is a prediction that the next move you make will improve your chance to win. Maybe the probability in that selection is 100%. You have no other possible move. It doesn’t matter. All we are doing here as far as I can tell is arguing over where the prediction happens and whether that counts as predicting something.
- danielmarkbruce 12d ago
  There is no truth for RLHF or RLVR. You can't predict against something if you can't check against the truth. It's not pedantry. The objective function changes. The optimization changes. THese are real things when training a model, not hand wavy philosophical ideas.
deleted 12d ago
[deleted]
ordersofmag 12d ago
The LLM does not determine the next token. It generate odds for all of the tokens it knows as to their likelihood of being 'next'. It's up to the harness running the LLM (and in most cases the a temperature setting) to actually decide on a particular next token. I think it's more accurate to call the thing the LLM actually generates (an ensemble of probabilities) a 'prediction'. It might be accurate to say the harness decides on the next token based on the prediction from the LLM. The role of the LLM is much more akin to predicting your opponents move than deciding your own.
- danielmarkbruce 12d ago

garrinm 12d ago

In the article I made 3 claims, and I agree it was a bit clumsy. 1st I say that "working forwards" in the sense of outputting one token at a time could be some form of prediction, I don't argue against that. This is what LLMs do at inference time. 2nd I say that to me what really constitutes a prediction is the pre-training. Here it's the classic setting for the word prediction in ML. The model outputs a prediction of the ground truth label: the next token. 3rd I argue that in RL there is no ground truth next token, so prediction doesn't apply here anymore. Back to your question then: you're asking points 3 and 1 are different. Working backwards from a set of win states is basically what RL does in training. Working forward from the current state is what inference does. To me there is a distinction worth thinking about. First between the mechanism at inference time and at train time. Then between what happens in pre-training vs. RL post training.

vrighter 10d ago
Neither is there a ground truth in rock paper scissors (the pattern varies by the player). But one would still be trying to detect patterns and predict the next "token" the opponent is going to output.