10 ms·

The fight is about the predictor language in some cases. Because it's only a trivial difference to those who don't understand the details of how these things ar

The fight is about the predictor language in some cases. Because it's only a trivial difference to those who don't understand the details of how these things are made. In pre-training the model really is trained to predict the next token. What is being emitted by the model is, by structure, by training and by optimization, a prediction of the very next token. What is emitted by a model during RLHF and RLVR is not, by structure, training or optimization, a prediction of the next token.

wat10000 12d ago
Right. But that's never what the discussion is about. This is the first time I've seen someone say, "it's not just a next-token predictor because a bunch of the training isn't about predicting the next token." I have seen many, many people say "it's not just a next-token predictor because there's all this smart stuff going on inside the model." Even here, in the comments for an article that's all about the former, most of the objections to "next-token predictor" are of the latter form.
- danielmarkbruce 12d ago
  Yup, you are mostly right. I guess the people in my camp find the "it's just a next token predictor" stupid in that it's like saying "it's just a bunch of carbon and hydrogen", but it's also one of those things where people like to think they are clever because they think they are theoretically correct. But they aren't even that. So it's like double stupid. But the "next token predictor" part is at least technically correct (like, carbon and hydrogen right) for pretraining, so the debate can't really be won there.
  - dijksterhuis 12d ago
    they are cleverly arranged / configured next-most-likely-token predictors, possibly with some clever procedures / attachments on top.
    - danielmarkbruce 12d ago
      Nope. This isn't right.
      - dijksterhuis 12d ago
        > "it's not just a next-token predictor because a bunch of the training isn't about predicting the next token." clever procedures on top of the base transformer architecture. i used simplified words/phrases to summarise the same thing you two were saying (the intent being: here's a version that may be digestible when discussing with others). apparently that means i'm wrong though, no idea why because it seems you've decided to be dismissive rather than constructively elaborate on why this simplified and digestible version might be wrong :shrug:
        danielmarkbruce 12d ago
        They aren't predicting the next token. It's quite literally not a prediction.
        deleted 12d ago
        [deleted]
        gwerbin 12d ago
        They're estimating a probability distribution over the next token, from which a sample is taken. Close enough.
        danielmarkbruce 12d ago
        It's not an estimation of something. It's a policy.
        gwerbin 11d ago
        Sure, you're right. But it's a policy learned from a next-token prediction task. You could also call it an inferrer or generator or whatever. The point is that it takes as input a sequence of preceding tokens and emits one more token to continue the sequence.
  - Dylan16807 12d ago
    It really is an important distinction, though. Being a next token predictor doesn't stop it from writing good sentences, but it does mean an LLM by itself can't play the number guessing game with you.
    - danielmarkbruce 12d ago
      This is pedantic, but, actually RL has improved the quality of sentence construction in LLMs quite dramatically... And once you do some RL on that model, it aint a next token prediction machine any longer.
      - Dylan16807 12d ago
        That's just training, isn't it? (Even if the term is "post-training") It doesn't change how inference works. My understanding is it usually strengthens the "thinking ahead" part of inference, but that part was already there, and it's still at the end of the day picking one token and then purging internal state in a way that can only partially be recovered from.
        wat10000 12d ago
        How inference works is you do a bunch of math and token weights come out, and then you pick one based on some sampling policy. “Prediction” is really about how they’re trained than how they run afterwards.
        deleted 12d ago
        [deleted]
        danielmarkbruce 12d ago
        The discussion is basically: what is a model trying to do? One may reasonably assert it isn't trying to do anything. But, in practice, if you give it an objective function and optimize it, the model is basically trained to "do" something. So what is it trained to "do"? During pre training it is trained to produce a distribution which is a prediction of the next token in it's training data samples. During RLVR and RLHF, it is trained to produce a distribution of tokens that will maximize a scoring function over many steps - not just the next step. The fact that it produces a distribution of potential choices for the next step doesn't mean the next step is a prediction. It's more of a "strategy" or "probabilistic path choice". The word used in RL is a "policy". It's a decent word to describe what the model is. So, modern LLMs are trying to produce a good sequence of tokens. They are "good token sequence producer machines". Not "next token prediction machines". Pre RLHF (in practice, go back to pre chatgpt) they really were "next token prediction machines".

wat10000 12d ago

Right. But that's never what the discussion is about. This is the first time I've seen someone say, "it's not just a next-token predictor because a bunch of the training isn't about predicting the next token." I have seen many, many people say "it's not just a next-token predictor because there's all this smart stuff going on inside the model." Even here, in the comments for an article that's all about the former, most of the objections to "next-token predictor" are of the latter form.

danielmarkbruce 12d ago
Yup, you are mostly right. I guess the people in my camp find the "it's just a next token predictor" stupid in that it's like saying "it's just a bunch of carbon and hydrogen", but it's also one of those things where people like to think they are clever because they think they are theoretically correct. But they aren't even that. So it's like double stupid. But the "next token predictor" part is at least technically correct (like, carbon and hydrogen right) for pretraining, so the debate can't really be won there.
- dijksterhuis 12d ago
  they are cleverly arranged / configured next-most-likely-token predictors, possibly with some clever procedures / attachments on top.
  - danielmarkbruce 12d ago
    Nope. This isn't right.
    - dijksterhuis 12d ago
      > "it's not just a next-token predictor because a bunch of the training isn't about predicting the next token." clever procedures on top of the base transformer architecture. i used simplified words/phrases to summarise the same thing you two were saying (the intent being: here's a version that may be digestible when discussing with others). apparently that means i'm wrong though, no idea why because it seems you've decided to be dismissive rather than constructively elaborate on why this simplified and digestible version might be wrong :shrug:
      - danielmarkbruce 12d ago
        They aren't predicting the next token. It's quite literally not a prediction.
        deleted 12d ago
        [deleted]
        gwerbin 12d ago
        They're estimating a probability distribution over the next token, from which a sample is taken. Close enough.
        danielmarkbruce 12d ago
        It's not an estimation of something. It's a policy.
        gwerbin 11d ago
        Sure, you're right. But it's a policy learned from a next-token prediction task. You could also call it an inferrer or generator or whatever. The point is that it takes as input a sequence of preceding tokens and emits one more token to continue the sequence.
- Dylan16807 12d ago
  It really is an important distinction, though. Being a next token predictor doesn't stop it from writing good sentences, but it does mean an LLM by itself can't play the number guessing game with you.
  - danielmarkbruce 12d ago
    This is pedantic, but, actually RL has improved the quality of sentence construction in LLMs quite dramatically... And once you do some RL on that model, it aint a next token prediction machine any longer.
    - Dylan16807 12d ago
      That's just training, isn't it? (Even if the term is "post-training") It doesn't change how inference works. My understanding is it usually strengthens the "thinking ahead" part of inference, but that part was already there, and it's still at the end of the day picking one token and then purging internal state in a way that can only partially be recovered from.
      - wat10000 12d ago
        How inference works is you do a bunch of math and token weights come out, and then you pick one based on some sampling policy. “Prediction” is really about how they’re trained than how they run afterwards.
        deleted 12d ago
        [deleted]
      - danielmarkbruce 12d ago
        The discussion is basically: what is a model trying to do? One may reasonably assert it isn't trying to do anything. But, in practice, if you give it an objective function and optimize it, the model is basically trained to "do" something. So what is it trained to "do"? During pre training it is trained to produce a distribution which is a prediction of the next token in it's training data samples. During RLVR and RLHF, it is trained to produce a distribution of tokens that will maximize a scoring function over many steps - not just the next step. The fact that it produces a distribution of potential choices for the next step doesn't mean the next step is a prediction. It's more of a "strategy" or "probabilistic path choice". The word used in RL is a "policy". It's a decent word to describe what the model is. So, modern LLMs are trying to produce a good sequence of tokens. They are "good token sequence producer machines". Not "next token prediction machines". Pre RLHF (in practice, go back to pre chatgpt) they really were "next token prediction machines".