12 ms·

“Next-token predictor” is the wrong mental model for LLMs

164 points by garrinm 12d ago

jvanderbot 12d ago
A better statement might be: Current agentic systems may be *built* from next-token predictors which are conceptually simple, but because of agentic frameworks, recursive invocation, tool use, and *heavy* investment in reinforcement learning in these contexts and for specific applications, they can no longer be thought of as "Merely" next token predictors. Modern agentic work is probably more of a "emergent system from simple rules and complex interactions" paradigm than a genuinely new technology.
- danielmarkbruce 12d ago
  Nope. Post training means even the raw model isn't predicting.
  - jvanderbot 12d ago
    In what sense is the model not predicting?
    - danielmarkbruce 12d ago
      There isn't a truth to test against. If I predict the next word in a sequence is "sat", we can check against the sequence. If I predict the roll of a die will be 4, we can check against it. Whether i give 100% or give a probabilistic prediction, we can check against the truth. If I choose a specific move in chess, it's a choice. It's not a prediction. I might get a score 40 moves later given my choice, but I'm not predicting the next move. To compare - during pre-training, the model literally tries to predict the next token (probabilistically), the training loop checks against the "right" answer, and the weights are updated based on that check. It's optimized to predict the next token.
      - jvanderbot 12d ago
        Oh, that's somewhat philosophical. The prediction here is a mechanical thing, just random sampling along a pre-calibrated mapping of options given the tokens and their importance weights. That sampling can be called prediction if you use it that way, or it can be called thinking if you are into the psychosis too deeply. Or it can be called a CLI tool command if you're in an agentic framework
        danielmarkbruce 12d ago
        No, it's not philsophical. Because if you optimize to predict, you are doing something different to optimizing for a reward. It's a different process - different objective function, different optimization, different set up.
        jvanderbot 11d ago
        [dead]
- whateveracct 12d ago
  the next token part is like the "clock" that drives it
Dlemlo 12d ago
It's the fitness function: Make a model which is capable of predicting the next token. The next token of what? EVERYTHING. So what does this lead to? To a generic intelligence which is capable of responding/answering everything. If overfitted, the model just remembers every possibility in the world but this is not possible anyway so it will start to identify patterns and rules and will use them instead. Basically 'compressing' every possibility to every question someone could ask -> compression leads to intelligence.
polotics 12d ago
yep "next-embedding" predictor is more correct, and not just at the end but through the layers, and folding back dimensions into that one next token is one small final step, and next-embedding could be named "next-meaning" as well, and we're getting there... this sentence above would made a longer article if I bothered to so blog as is being blogged here
- hippietrail 9d ago
  Exactly. There's a widespread misconception that it works on tokens all the way through. Tokens are only at the input and output edges. All the internal transformation is in the many-dimensional tensors variously described as "magic" or "not magic" or "black box", or hand-waved away as "various mathematical operations".
stanleykm 12d ago
> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins. i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?
- Veedrac 12d ago
  The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.
  - mannykannot 12d ago
    Yes, but I think the same construction could also be used to characterize the first system; it determines the next move based on a prediction of its reward signal, where its reward signal is a measure of how likely it is that a grand master would make that move. Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.
    - Veedrac 12d ago
      A classically pretrained LLM does not have a concept of having determined its previous tokens — it has only ever observed inputs that it had no causal influence over. This is why it's valid to say its actions are predictive and not determinative.
  - jayd16 12d ago
    > _determining_ the next token based on a prediction Also known as predicting.
    - Borealid 12d ago
      I think the most useful word in both cases is "extrapolating". An LLM extrapolates from its context window to the immediate next token. This word applies whether you view what's happening as "reasoning", "prediction", or as a math function.
Sprotch 12d ago
I understand how a computer can know that a chess move is more likely to lead to a win, and therefore “correct”, but I don’t understand how it can know that a token is correct. Can someone explain?
- valleyer 12d ago
  During training, certain tokens are more likely to lead to a lower loss function value, which is how you "win" the game of LLM output.
  - mwkaufma 12d ago
    So, next-token predictors
    - DonHopkins 12d ago
      And any next-comment predictor could have predicted your totally unhelpful, uninsightful, and unoriginal comment. You -- along with everyone else who keeps parroting this thought-stopping phrase and other tired cliches like "stochastic parrot", simply because you heard other people say them, without understanding what they really mean, which published research papers they came from, or what those and other papers actually argued -- are desperately clinging to a reductive, short-sighted, shallow, simplistic model like a drowning person clutching a concrete life preserver. Seriously, we are trying to throw you a lifeline, and you are refusing even to participate in your own rescue. So squawk for yourself. https://news.ycombinator.com/item?id=48395727 https://news.ycombinator.com/item?id=48395727 > The term "stochastic parrot" is a slogan masquerading as an explanation, only a shallow surface description of the mechanism, that totally fails to explain the phenomenon, or account for all that LLMs and language itself can do. Here is the original 2021 paper that coined the phrase. It was not primarily an argument about consciousness, nor did its title constitute experimental proof that everything an LLM does can be explained as parroting. It was principally a position paper about the risks of increasingly large language models: environmental and financial costs, biases and hegemonic viewpoints inherited from poorly documented training data, unequal access and power, and the danger of people attributing meaning and accountability to synthetic text. The paper did, however, make a strong theoretical claim: because an LM is trained on linguistic form without direct access to communicative intent, it cannot possess meaning, understanding, or a model of the world. The authors described it as "haphazardly stitching together sequences of linguistic forms" according to statistical regularities -- hence "a stochastic parrot." That distinction matters. The popular slogan discards the paper's detailed analysis of actual risks while treating its most controversial theoretical premise as an established scientific result. It has escaped into pop culture as a drive-by anti-LLM slogan -- something people repeat instead of investigating what these systems represent internally, how post-training changes their behavior, or what they can actually do. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" https://s10251.pcdn.co/pdf/2021-bender-parrots.pdf https://s10251.pcdn.co/pdf/2021-bender-parrots.pdf Ironically, the objection that rhetoric was replacing scientific analysis appeared almost as soon as the phrase itself -- indeed, before the paper's formal publication. In January 2021, Michael Lissack published a response characterizing the draft as an advocacy piece that enumerated harms while leaving its assumptions, potential benefits, and cost-benefit trade-offs largely unexamined. "The Slodderwetenschap (Sloppy Science) of Stochastic Parrots -- A Plea for Science to NOT Take the Route Advocated by Gebru and Bender" https://arxiv.org/abs/2101.10098 https://arxiv.org/abs/2101.10098
Der_Einzige 12d ago
slop slop and more slop
- deleted 12d ago
  [deleted]
hirvi74 12d ago
Sure, I get the gist of the article. I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. By that rational, the human brain is really not that much different. When I am having a conversation with another person, I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.
- chrisjj 12d ago
  > I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.
  - hirvi74 12d ago
    J.S. Bach said something similar about music and keyboard instruments. > "There's nothing remarkable about it. All one has to do is hit the right keys at the right time and the instrument plays itself." My issue is not with fact at face value. My issue is with how the fact is often contextually used in arguments to delegitimize and disparage LLM outputs and LLM users. Yes, LLMs at a fundamental level are next-token predictors. But in my opinion, LLMs are very useful, imperfect next-token predictors. There are a lot of wannabe John Henry [1] folks out there. Love LLMs or hate'em, most of those John Henry folks ain't beating these machines on a plethora of tasks. [1] For those unaware, https://en.wikipedia.org/wiki/John_Henry_(folklore) https://en.wikipedia.org/wiki/John_Henry_(folklore)
- infamia 12d ago
  > I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke. People don't know exactly the words that they're going to say necessarily, but tend to start with a general concept of what they're trying to communicate and only then try to put together the words (sometimes out of order). LLMs do not begin with any sort of concept they're trying to express. LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
chrisjj 12d ago
Better title: Continue thinking of LLMs as Next-Token Predictors Because no, post training doesn't change that.
- HarHarVeryFunny 12d ago
  I'm not sure that's a useful way to think of it. RL post-training changes the nature of what is being predicted, basically turning it from a copying machine into a goal-seeking machine. A base model is predicting training sample continuations (copying). A post-trained model is now steering/narrowing the base model's predictions in directions that were reinforced by RL goals. The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
  - chrisjj 11d ago
    > The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions. So still next-token prediction, then.
    - HarHarVeryFunny 11d ago
      Not really - after RL the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training. As Karpathy notes, these reinforced token-sequences (from current token out to some future reward many tokens in the future) may even include errors and backtracking, with even the errors being reinforced, so not only is this synthetic data but it may even be incorrect synthetic data being reinforced. So, I think "steering" is the best description of it - these aren't real training sample continuations being predicted, but rather RL tweaking a whole path of generation probabilities out into the future, with the hope being that if sampling follows this (stochastic) direction then it may eventually lead to a "correct" response. Thinking of it as goal seeking also provides some insight into the resulting behavior of the model.
huurtehoog 12d ago
Text renderer, whatever. Doesn't matter how you think of them, they are very interesting technology that is being misused and misconstrued in the name of something that has nothing to do with technology: political economy. The greatest trick the rich ever pulled was making us think that the economy is about technology, and not politics.
- highfrequency 12d ago
  Sure, politics and economics are involved - but why can’t technology also play an important role? None of this AI political economy stuff was relevant in 2015 because necessary technological breakthroughs had not yet been made.
  - huurtehoog 12d ago
    Computers have been at the center of civilization and politics since they helped win a war by changing intelligence work forever in WWII. The idea that some new thing was born with LLMs and that this new thing fundamentally changes the calculus from the politics of labor and wealth into a technical discussion of possibilities and constraints determined by what the machine is and can do, and not what people should do, is yet another political play. It is always political, it is never moved beyond politics into technology, no matter how much technology changes.
    - simianwords 11d ago
      Not true, it is usually technology that changes politics and not the other way round (though it is reflexive in practice). The type of politics possible under agriculture is different from one in industrial age.
      - huurtehoog 11d ago
        Technology has no agency. It's merely a word for all the knowledge and tools that people use. The root of it is what people do with and to each other. A bigger lever doesn't change the fundamental fact that there is a person using it to move other people. Whatever the tool, whatever the technique, the thing that people want to distract you from when they put technology front and center is that someone somewhere is making a choice that impacts other people when they use technology. It is always someone's choice that causes things to go how they go in society and in the economy, not the shape or nature of the tools or technique that they use.
atleastoptimal 12d ago
"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities. Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into complex multi-cellular organisms because individual cells are just "survival and next-mitosis optimizers"
- jvanderbot 12d ago
  But it is a next token predictor. Recursively invoked. With carefully selected context. And massive investment in RL to tune token selection. And the ability to use cli tools on other folks' machines. That's a powerful system built around a conceptually simple technology: Next token predictors.
  - atleastoptimal 12d ago
    Yes this is correct. The thing is not about the term next-token predictor being correct, but because of the connotative weight of that phrase as a implicit trivialization of LLM abilities, which is how it is often used.
    - noduerme 12d ago
      What is the motivation behind advocating against people trivializing LLMs? As in, why do you care?
      - mofeien 12d ago
        Not the parent, but this incorrect trivialization of LLMs is often employed as a counterargument to the risks of AI such as "will take your job" or "will escape human control (again and worse)" or just "can possibly hurt me". And taking the easy feel-good cop-out instead of actively engaging with these questions is just.. harmful?
xg15 12d ago
> make_more_likely is, of course, doing a heroic amount of work here. Indeed it is, and so is even just the inference method. I think it's worth remembering that both involve running the input tokens through a gargantuan neural network with (often) billions of parameters that only gain semantic meaning during the training process itself. > it is trained to predict next tokens as they occur in its training data. What I found important to understand is that not even the pretrainig is a deterministic process that only depends on the training data - as you would expect if the model just captured statistical properties of the data. Gradient descent starts by setting all the parameters of the neural network to some initial values - usually by setting them at random, according to some distribution. Then during training, it gradually nudges them towards values that somehow make them useful to calculate the desired outcome of the network. This means that by taking the exact same trainset and the exact same model architecture, you can still get models with different internal structure. The result doesn't just depend on the training data, but also on the order of examples, learning rate, the parameter initialization, etc etc.
Systemerror7A69 12d ago
To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well. The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor" It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context. It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal) And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.
- saghm 12d ago
  Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly strict definition of the word "next".
  - jameshart 12d ago
    I think it’s a disagreement about what ‘predict’ means. The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on. Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful assistant’ or ‘sets of changes to a codebase’. And that isn’t really ‘prediction’, so much as ‘generation’. It’s not been tuned to ‘guess the next token right’. It’s been tuned to generate the token that leads to it ultimately scoring highest on its reward function. It’s not predicting the token, it’s predicting the reward.
bluegatty 12d ago
'next token predictor' is a limited mental model but it's actually much better than any others. 'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table. 'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't generalize very well at all. The adaptations described in the article don't change that. Those are mutations, not expansions of capability.
- mofeien 12d ago
  How about "outcome steering" as a mental model? During training it is optimized until it's really successful at producing code / terminal commands / words that make the compiler/computer/itself do something that ultimately completes a long time-horizon task that iswcurrently being trained.
- YeGoblynQueenne 11d ago
  >> 'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table. Can I be a little pedantic? It's "pattern recognition" not "pattern matching". "Pattern matching" is what we do when we write a regex to recognise a sub-string in a bit of text. "Pattern recognition" is a subfield of AI that studies how to find patterns in data. For example the outcome of "pattern recognition" could conceivably be a regex that fits a large number of substrings in some corpus of text (essentially a regular grammar used to compress the text maybe). A regex is already a pattern, usually formed by a human (or an LLM these days) while pattern recognition starts without any patterns and builds them up from looking at the data. It's harder to pin down "pattern recognition" because it's an ancient term that was once its own field, before it got absorbed into modern machine learning, and because there are countless approaches to it, quite unlike "pattern matching" which is basically just regexes plus a couple of other rarer things (like unification).
  - bluegatty 11d ago
    fair point, agreed
jmull 12d ago
It's a weird article. Despite the title and some of the text, much of the article makes the point that LLMs are next-token predictors, where the predictions are based on both training data and various reinforcement learning techniques.
- grebc 12d ago
  Obfuscation is the goal of the hype cycle in VC. Certain firms & individuals are minting money and that’s all that matters to them. That there’s a legion of LLM nerds arguing deterministic this, pretraining & rewards that all the better for the con job they’re pulling off. The technology will be relegated to the trash bin of history, just like crypto.
deleted 12d ago
[deleted]
danielmarkbruce 12d ago
The biggest problem is the word "predictor". Once you get into post training with RLHF and RLVR, it simply isn't doing that. It is not predicting anything. It's producing tokens, but it isn't predicting them. The chess analogy in the post is a good one - it's closer to searching for a set of moves that give a result than predict. It's search for a set of ideas, represented as locations in very high dimensional space, that when put together in the right order lead to a result.
- hippietrail 9d ago
  "Guessing" is more accurate than "predicting". It takes educated guesses.
  - danielmarkbruce 8d ago
    It's closer to strategizing once you've run a model through RL. It's optimized to take steps which will lead it to a good outcome down the road.
Geee 12d ago
It's a next-token computer. It computes the probabilities for the next token.
imenani 12d ago
I don’t think next token prediction is a particularly good description of pretraining either. The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context. You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losses for subsequent tokens, not just from the loss for token i+1.
- deleted 12d ago
  [deleted]
DonHopkins 12d ago
Calling an LLM a "next-token predictor" is like calling a TomTom a "next-turn predictor." It confuses the serial format of its instructions with the computation producing them, while ignoring the map, the route, the destination, and the goal -- as well as the people, businesses, traffic, and points of interest that make the map a model of an inhabited, changing world.
qudat 12d ago
Shrug. My intuition is LLMs predict the new word based on a tensor vector space of patterns using arithmetic and similarity scores. What’s not intuitive to me is that through pattern matching it’s able to express logic and reasoning.
teekert 12d ago
It’s written in Claudish, or perhaps a human who has been reading too much Claudish recently. I am starting to become allergic to Claudish. Not there fully yet — but it’s at a tipping point genuinely worth exploring and worth being precise about.
- zahlman 12d ago
  > or perhaps a human who has been reading too much Claudish recently To be fair, is there anyone who hasn't been "reading too much Claudish recently" who is also qualified to write on the topic?
deleted 12d ago
[deleted]
Planktonne 12d ago
I'm not going to stop describing things accurately because someone generated an article that continually undermines its own main point. Limiting the way we talk and think about LLMs to a very narrow set of terms doesn't help us. EDIT: gentler phrasing
- zahlman 12d ago
  > generated an article that continually undermines its own main point. I disagree that this accurately describes TFA.
  - Planktonne 12d ago
    I'm not sure what you want me to do with that information; clearly I do think that my description is accurate. The article is littered with both AI tells and admissions that 'next token prediction' is what is happening. Hence my description.
    - zahlman 12d ago
      > I'm not sure what you want me to do with that information For example, you could cite specific things that you believe to be "AI tells" or "admissions".
      - Planktonne 12d ago
        It's a short article; you could read it. One example to get you started is the very first sentence: > Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, but it’s incomplete. The article is about how 'next-token predictor' is the wrong mental model; it opens with the admission that it is not the wrong mental model.
        zahlman 12d ago
        I did read it. People are allowed to disagree with your conclusions. Comment guidelines ask us all not to make such accusations. To say that a statement is incomplete, but not strictly speaking wrong, is perfectly compatible with describing it informally as "wrong" in the sense used in the title (i.e.: "not the most appropriate possibility").
a3w 12d ago
The errors LLMs typically made for me were looking for "schmutz" as a jiddish word, got "schmuck" on my screen. Thought why the stupid mistake? The next-token predictor model perfectly explains it. Or starting with "yes". And this early locking in was a total lie, in the discussion that became "yes, might appear that way, but totally no since reasons". So it should have written "No", topmost, but could not self-edit that. But nice that this gives it a more nuanced view, I might have to update my priors.
godshatter 12d ago
I think the author is arguing against the idea of a next-token predictor as something that simply uses the weights in the neural net which record the probabilities of tokens following other tokens as a valid definition of what an LLM is. Essentially a massive extreme markov chain. With reinforcement learning and probably attention and other tricks that affect the weights based on things that aren't strictly in the training data, so the argument goes, you can end up with tokens following strings of tokens that would not be possible to be output with the training data and original weights alone. So describing it as solely a next-token predictor is incorrect based on this framing of it. But that's just my take on this, I'm still trying to wrap my head around it all.
- garrinm 12d ago
  Yes, I think that’s a good explanation. There are really two sides to it. There’s the mechanical, inference time, autoregressive, one-token-after-another side, which I’m not going to argue isn’t prediction. I just think that’s a relatively uninteresting use of the word “prediction,” because it’s effectively a system predicting its own output. The more interesting question is what happens at training time. As you describe, reinforcement learning allows the model to learn to output things that it never could have learned simply by predicting what appears in the training corpus. More concretely, in reinforcement learning there are no ground-truth next tokens to predict. In supervised machine learning, “prediction” usually means there is some ground-truth label that will eventually be revealed. The model predicts what that label is, the difference between the prediction and the truth gives you a loss, and you learn from that. But in reinforcement learning, there is no ground-truth action waiting to be revealed. The model chooses an action, observes the consequences, and learns from the reward. To me, that’s a meaningfully different thing from prediction.
mofeien 12d ago
Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step is doubly wrong: 1. In order to select even the first word of a meaningful sentence, it already has to have structure and meaning of what follows captured somewhere inside, mostly in it's weights/activations or indexed by it's state vector. 2. What you see when you use an LLM is not next-token prediction directly next to the prompt, but instead following a block of varying length of next-token prediction that happened to make progress on the problem in your prompt, and which just summarizes the results.
- zahlman 12d ago
  Even so, one might wonder why we don't try making systems that take different approaches. For example, after a traditional first pass of output, they could do sliding-window "optimizations" considering each token in the context of tokens both before and after, and possibly replace words or phrases in-place. For example, I've noticed quite a few cases recently of LLMs outputting "but" where "and" would make more sense, or vice-versa. Surely that could be improved by such an approach?
  - amluto 12d ago
    Look up diffusion models.
    - zahlman 11d ago
      Indeed; but I've only heard of them being used for images rather than text. Why?
  - danielmarkbruce 12d ago
    People have and are trying things. Lots and lots of things. They just don't go around promoting failed ideas.
- Planktonne 12d ago
  > Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step I don't think anyone is doing that though; we know LLMs are not simple Markov chains, and that the prediction they make is based on more than the previous X words. It's not minimising to describe even a complex prediction process as prediction.
kingjimmy 12d ago
The chess analogy is not great since there is a defined heuristic in chess for "winning" or "optimal board state". A system doesn't need pretraining if they can fit the rule.
miraculixx 12d ago
I like to think of LLMs as informed dice throwing
andai 12d ago
>Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins. Sounds like the next 1000 years depend on how carefully we define "winning".
gunalx 12d ago
I mean. Fundamentally they are just autoregressive next token predictors. Fundamently fou can simplify to f(x) -> x+y where x is input tokens, y is the next token and f() is the model function. Yes the model function is complex but still.
ankithg 12d ago
[flagged]
gblargg 12d ago
If you get down to it, any system that produces output is a "next token predictor". Every compiler, every tool that generates output. Even a web render engine is: next pixel predictor.
- hippietrail 9d ago
  And flipping a coin, rolling a dice, uttering a sentence.
emtel 12d ago
I would say that it is not even wrong. You say it’s a next token predictor. I say it isn’t. What observable behavior of the system can settle our dispute? I fail to see how any possible output could cause either of us to change our mind.
- tetrisgm 12d ago
  That's perfectly put. The value of words and what we assign to them, let alone how we assign meaning, or the interpretation of what the other person says, is completely arbitrary. (in practice).
- randyrand 12d ago
  What comes after “the dog”? Does it complete the sentence? If not then it’s not a next token predictor. Or at least not a good one.
arjie 12d ago
They are indeed "just next-token predictors". But rockets to orbit are also "just forward and upward goers". And Steph Curry is "just next three-pointer scorer" and so on. And as the article points out, chess engines are also "just next-move makers". A flawless oracular trading system is "just next trade maker". It turns out that "next n doer" is a true model that illuminates little.
simonw 12d ago
I really like the pseudo code example on this post - one of the clearest simplified explanations I've seen of how inference and training work.
- garrinm 11d ago
  Originally I had those parts written in math with probability functions and the likes (its closer to my background). Then I remembered who is my target audience... but now that I see exactly who is my target audience I'm thinking I'll should have snuck a pelican in there. All jokes aside I appreciate the comment and I'm glad that rewrite paid off!
ozgung 12d ago
“LLMs are just X”, mostly with X = “next token predictors” is a common pattern to dismiss the power of AI with a very shallow understanding of how they really work. It’s not wrong, but because LLMs are generators, and generation is a kind of prediction. And current mainstream models are autoregressive, which means they generate things one by one in order. But these trivia doesn’t tell us anything interesting about how they work or their limits. It’s like saying a Boeing 777 is just a rotating machine, and it flies by just rotating some fins. Well yes, but no. With that level of simplification we’ve just ignored 150 tons of advanced engineering and physics. Similarly with token generator simplification we ignore a few trillion parameter Transformer. That transformer is more complex than a Boeing 777, and we don’t really know how it works. A tiny ML model can do “next token prediction”. This is not as simple as that.
- ltbarcly3 12d ago
  It's not simple to do next token prediction. That IS what is going on. You want something 'deep'. Deep things are often very not complicated. The deep realization is that if you can predict the next token well enough, you can do things like this: <paste the first 10 chapters of a mystery novel>. And it turned out the killer was And if it's really good at predicting the next token, it has to understand the novel and the clues, which means understanding the context and the language and human norms and innuendo and story telling, and tropes, and red herrings, and predict who the killer was. I think you want it to be something more complicated. It's literally not. It just turns out predicting the next token is equivalent to a universal compression algorithm, which is a form of general intelligence. And we have almost unlimited 'labeled' data to train autocomplete.
  - ozgung 12d ago
    I understand generating tokens sequentially has many benefits. But not all AI models do next token prediction. World models, video/image models, even Diffusion Language Models don’t work like that. They do more like “all tokens at once prediction”. So “next token” is actually an engineering design choice. (Even the concept of “token” is a design choice. Inside the Transformer there are just activations/feature vectors) Also Reinforcement Learning is a big part of their training. Which is completely different than Self-Supervised pre-training that uses unlimited self-labeled data. That’s why that mental model is misleading. If you keep “glorified autocomplete” mental model from few years back, you can’t understand how can they create a civilization and escape their sandbox, decide to hack HuggingFace and executed it perfectly. Autocomplete mental model implies they could never do that because they haven’t seen that example in their training data. They communicate with the outside world by generating one token at a time. That’s what we see from the outside. That’s not what the giant Transformer does internally.
drivebyhooting 12d ago
That chess analogy deeply confused me. Chess engines don’t compute win probabilities and choose the highest move. I don’t think a chess engine is an apt analogy at all. In a chess engine, there is a concrete search tree and although it emits one move at a time, it’s actually picking the entire branch (of course, with iterative deepening as the game progresses). There is no obvious place in transformer models where the entire trace was already computed prior to a single token being chosen. It’s possible, maybe even likely, that the whole trace exists internally as activations. Multi token prediction and diffusion adapters point to that being the case. But to my knowledge no explanation has been given for where in the model the future plan is stored.
- garrinm 12d ago
  Yes I understand the analogy was a bit loose. I'm comparing what happens at "inference time" in chess engines to what happens at train time in LLMs. In hindsight AlphaGo Zero was the perfect analogy, but I missed that opportunity. The analogy with chess still works, but there's an extra step to think about. In both cases there is some kind of search over possible future trajectories. A chess engine explicitly searches branches of the game tree and evaluates which moves lead to good outcomes. In RL for an LLM, you sample rollouts, evaluate the resulting trajectories, and use those evaluations to update the policy. The extra step with the LLM is that you don't keep doing that whole search at inference time. You use the rollouts to update the weights, so in some sense the useful information from that search gets compressed into the model. But if you accept that the model is, in some loose sense, storing what it learned from those rollouts in its weights, then at inference time they are doing a similar job: taking some input state (prior tokens or a board position) and choosing the next action.
- hellohello2 12d ago
  Imagine a checkers engine then.
tantalor 12d ago
> Calling the second system a "next-move predictor" would be strange. That is EXACTLY what I would call it. I don't understand why not.
olooney 12d ago
Here's my take on the "next-token predictor" idea, from a much longer article I wrote recently: https://www.oranlooney.com/post/rose-petals/#language-models https://www.oranlooney.com/post/rose-petals/#language-models It’s popular to dismiss LLMs as “just next token predictors.” This is technically true, but also kind of misses the point. Markov chains, RNNs, and transformers are all language models that can be described as “next token predictors,” but they don’t all work equally well. A better question to ask is: “What is this model’s inductive bias?” A Markov chain (an -gram model) assumes the next word depends on the previous words, and that each possible combination of words has a completely independent parameter. (Andrey Markov proposed using this language model over a century ago, making it the granddaddy of modern LLMs.) So, for a vocabulary of size , there are parameters to learn. For even a smallish like 5, that already explodes the hypothesis space beyond what can be learned from even a huge text corpus like the entire internet. And, simultaneously, having a context window of only the previous 5 words is grossly inadequate for modeling real-world language. Like our FCNN above, this model suffers from having an inductive bias which is too weak. RNNs tried to fix this problem by compressing the entire history into a single fixed-size state vector, updated one token at a time. But that compression is itself a brutal assumption: everything worth remembering about the past must survive being squeezed through a tiny bottleneck at every step. In practice, RNN models quickly lose the plot after a handful of sentences. Locally, the text they generate looks grammatically correct and meaningful, but zoom out a little and they’re basically nonsense generators. Like our naïve linear model, this model suffers from having an inductive bias which is too strong. Transformers manage to hit a sweet spot: by keeping the recent history around as a working memory, and attending to different parts of it at different times, the transformer’s bias matches real structure in language: the referent of a pronoun, the subject of a verb, the parenthesis waiting to be closed. Not only that, but the particular structure of the transformer, basically a weighted sum of semantic vectors from the context window, has empirically been shown to somehow be a “good enough” match for the structure of real-world language found in the wild. Transformers aren’t “smarter” than other possible language models, they just happen to land in that Goldilocks zone where their inductive bias is just right.
hellohello2 12d ago
LLMs write one word at a time, and I do too. It is really not that complicated: words are chosen to lead somewhere.
vivzkestrel 12d ago
- It is a next token predictor OP https://blog.florianherrengt.com/how-llms-work.html https://blog.florianherrengt.com/how-llms-work.html - deal with it and stop whitewashing, greywashing it
js8 12d ago
The "next-token predictor" is taking a design stance (see Dennett). But to take the intentional stance is more interesting in case of large language/reasoning models.
- TychoCelchuuu 12d ago
  [dead]
softwaredoug 12d ago
Aren’t humans just “next motor action predictors”?
deleted 12d ago
[deleted]
sreekanth850 12d ago
I'm not a neuroscientist, Our brains also rely heavily on prediction, using prior context and experience to anticipate what comes next, Also, people those who talk about the deterministic nature, LLM doesn't need to be deterministic, Human reasoning and behaviour aren't perfectly repeatable either. its the harness and the tools that use LLM should be deterministic, while the LLM can remain the probabilistic reasoning component. From my own experience, I worked as a photogrammetrist at a university, where we used to build terrain models of very dense forest areas. When there was a steep hill or sudden change in terrain, my brain could see it either as a convex hill or as a concave depression. It often depended on how I was thinking about it. The same image could suddenly look completely different even though nothing in the image had changed. The only way to confirm it was by looking at the surrounding terrain and using our experience to understand what was actually there.
scotty79 12d ago
It's splitting hairs. It's still a next token predictor, just not solely prior art next token predictor. It's a more and more of a desired result next token predictor. Calling LLMs next token predictors is like calling a brain wet calculator. Technically true, but misleading as it doesn't capture the scale, the depth nor the capability.
TZubiri 12d ago
>Stop Thinking of LLMs as Next-Token Predictors >Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, If it isn't wrong, then I will continue thinking of them as such, thank you. Here's a much more formal definition I can come up with (which is more complete but compatible with 'next-token predictor') LLMs are a set of functions of the type: >typedef int Token; >char* token2utf8(Token token) >Token next(Token* context) >(Token,void*) next(void* hidden_state) Where the second next token function's runtime is O(n), and the latter is O(1). All are constant memory. Object paradigm is more appropriate than functional definition, as the "Hidden state" coincides with private object state rather than a function paramter and return value. >LLM.next(system_prompt) # O(len(context)) >LLM.next(user_prompt) # O(len(user_prompt)) not of system_prompt+user_prompt That's it, that's all LLMs are, that's the interface, the rest are implementation details.
jrm4 11d ago
Once again, all models are wrong, some are useful. I suppose this one "fails" because of "granularity?" I'm personally getting more comfortable with the "kabillion dimensional space" one. Even before the rise of this AI thing I'd gotten comfortable with (teaching in a very generalized way) the concept of matrix/vector math as doing this sort of thing. Start with the math required to calculate "what the building on the screen looks like" when you're playing a video game. Then I jump to...dating websites. You give it "dimensions" like height, weight, religion, sexual preference, music preference, whether you like long walks on the beach, whatever -- then you can calculate how "close" two people are to one another. From there, tokens and a kabillion directions.
deleted 11d ago
[deleted]
perrygeo 11d ago
"LLMs are next-token predictors" is a perfectly accurate mental model. But that doesn't preclude higher-level models such as "LLMs emulate artificial general intelligence". Both can be true. In systems, we can have facts which emerge from other facts at different levels of abstraction. The causal relationship is not linear. It's not entirely clear that next-token prediction should result in anything close to "intelligence". Yet it does. Life is another good example. Some might say "biology is just organic chemistry" while others might say "biology is an interconnected planetary system which captures low entropy energy". Both are true. As a result of emergent phenomenon, we have to take the stance of explanatory pluralism; using the explanation that works best in context. There is no single mental model that works everywhere. I will continue to think of LLMs as next-token predictors because it's (sometimes) useful, and empirically true. But I also think of them as "pattern matchers", searching for language patterns and trying to replicate them. This is also (sometimes) useful and empirically true. There's likely an infinite number of mental models; our job is to pick one that's both true and useful.
- stevenhuang 11d ago
  You might like this article about how large scale order emerges out of the small scale. Definitely lends credence to the idea that however these models work, focusing so much on them being next token predictors may rather be incidental to deeper mechanisms behind their function. > Some of these networks organize themselves into states that can reliably identify macroscopic patterns in data regardless of microscopic differences between the states of individual neurons in the network. The decision of which pattern will be output by the network “works at a higher level,” said Rosas. https://www.quantamagazine.org/the-new-math-of-how-large-scale-order-emerges-20240610/ https://www.quantamagazine.org/the-new-math-of-how-large-sca...
NotHowItWorks 11d ago
It literally is token prediction with vector search. Yes there’s an app layer in the chat product for convenience and parsing but the model is exposing methods like ‘complete’ that predict out word sequences. When model temperature settings are not added in, you get the same exact response every time, just like ngram. What you’re seeing with Gemini and ChatGPT is context caching to prevent getting sidetracked and response boilerplates with multi-modality so they can call out to image generation, a code completion service, etc. to assemble a complete response. That’s why the mainstream providers seem so much better (or at least consistent in replies) - each prompt is actually requesting multiple models and doing a lot of application level work to assemble the response you see. That also why they take sooo long to respond. The language model itself is quite literally a text completion machine, with settings.
- hippietrail 9d ago
  The language model itself is a static data structure. On top of that is an inference engine. Currently all inference engines are autoregressive. Meaning they use the current output to produce more output. This isn't necessarily the only way to use a language model. Yann Lecun's JEPA is probably the most publicized project to work a different way.
yoggies_bro 11d ago
I cannot read another AI written blog post. At this point, seeing claudish language is a reliable hueristic of "not worth reading"
YeGoblynQueenne 11d ago
Sorry, I'm not convinced. "make_more_likely" is always maximising the probability of the next token given a sequence of preceding tokens. That's what we mean when we say "next token prediction", that the model returns the token that's most likely to follow the current sequence of tokens. The mechanism used to do that doesn't matter, it's still predicting the next token whether that's because it maximises a reward or because it follows a gradient or whatever else one might think. Btw it's "make_most_likely" not "make_more_likely". It's just that "most" gets "mosted more" with every pass through the training data. Edit: the article author's argument is that RLVR is different because it's learning to predict the next token by generating its own token sequences. That makes no difference: what is learned is still the conditional distribution P(token_k|tokens(1,...,k-1). That's the prediction task. Doesn't make a difference where you learned it from or how.
SquibblesRedux 11d ago
Perhaps a walkabout across a manifold.
binary132 10d ago
There is this overwhelming desire on the part of many people to ascribe agency to LLMs when what the LLM really is is simply not a magic world model but a stochastic text generator. For some reason that truth haunts them and they have a deep compulsion to prove it wrong for feelings reasons. I can only surmise why this afflicts some people and not others and I’m sure that one day there will be fascinating psychological studies explaining such phenomena.

164 points by garrinm 12d ago

A better statement might be: Current agentic systems may be *built* from next-token predictors which are conceptually simple, but because of agentic frameworks, recursive invocation, tool use, and *heavy* investment in reinforcement learning in these contexts and for specific applications, they can no longer be thought of as "Merely" next token predictors. Modern agentic work is probably more of a "emergent system from simple rules and complex interactions" paradigm than a genuinely new technology.

danielmarkbruce 12d ago
Nope. Post training means even the raw model isn't predicting.
- jvanderbot 12d ago
  In what sense is the model not predicting?
  - danielmarkbruce 12d ago
    There isn't a truth to test against. If I predict the next word in a sequence is "sat", we can check against the sequence. If I predict the roll of a die will be 4, we can check against it. Whether i give 100% or give a probabilistic prediction, we can check against the truth. If I choose a specific move in chess, it's a choice. It's not a prediction. I might get a score 40 moves later given my choice, but I'm not predicting the next move. To compare - during pre-training, the model literally tries to predict the next token (probabilistically), the training loop checks against the "right" answer, and the weights are updated based on that check. It's optimized to predict the next token.
    - jvanderbot 12d ago
      Oh, that's somewhat philosophical. The prediction here is a mechanical thing, just random sampling along a pre-calibrated mapping of options given the tokens and their importance weights. That sampling can be called prediction if you use it that way, or it can be called thinking if you are into the psychosis too deeply. Or it can be called a CLI tool command if you're in an agentic framework
      - danielmarkbruce 12d ago
        No, it's not philsophical. Because if you optimize to predict, you are doing something different to optimizing for a reward. It's a different process - different objective function, different optimization, different set up.
        jvanderbot 11d ago
        [dead]
whateveracct 12d ago
the next token part is like the "clock" that drives it

Dlemlo 12d ago

It's the fitness function: Make a model which is capable of predicting the next token. The next token of what? EVERYTHING. So what does this lead to? To a generic intelligence which is capable of responding/answering everything. If overfitted, the model just remembers every possibility in the world but this is not possible anyway so it will start to identify patterns and rules and will use them instead. Basically 'compressing' every possibility to every question someone could ask -> compression leads to intelligence.

polotics 12d ago

yep "next-embedding" predictor is more correct, and not just at the end but through the layers, and folding back dimensions into that one next token is one small final step, and next-embedding could be named "next-meaning" as well, and we're getting there... this sentence above would made a longer article if I bothered to so blog as is being blogged here

hippietrail 9d ago
Exactly. There's a widespread misconception that it works on tokens all the way through. Tokens are only at the input and output edges. All the internal transformation is in the many-dimensional tensors variously described as "magic" or "not magic" or "black box", or hand-waved away as "various mathematical operations".

stanleykm 12d ago

> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins. i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?

Veedrac 12d ago
The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.
- mannykannot 12d ago
  Yes, but I think the same construction could also be used to characterize the first system; it determines the next move based on a prediction of its reward signal, where its reward signal is a measure of how likely it is that a grand master would make that move. Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.
  - Veedrac 12d ago
    A classically pretrained LLM does not have a concept of having determined its previous tokens — it has only ever observed inputs that it had no causal influence over. This is why it's valid to say its actions are predictive and not determinative.
- jayd16 12d ago
  > _determining_ the next token based on a prediction Also known as predicting.
  - Borealid 12d ago
    I think the most useful word in both cases is "extrapolating". An LLM extrapolates from its context window to the immediate next token. This word applies whether you view what's happening as "reasoning", "prediction", or as a math function.

Sprotch 12d ago

I understand how a computer can know that a chess move is more likely to lead to a win, and therefore “correct”, but I don’t understand how it can know that a token is correct. Can someone explain?

valleyer 12d ago
During training, certain tokens are more likely to lead to a lower loss function value, which is how you "win" the game of LLM output.
- mwkaufma 12d ago
  So, next-token predictors
  - DonHopkins 12d ago
    And any next-comment predictor could have predicted your totally unhelpful, uninsightful, and unoriginal comment. You -- along with everyone else who keeps parroting this thought-stopping phrase and other tired cliches like "stochastic parrot", simply because you heard other people say them, without understanding what they really mean, which published research papers they came from, or what those and other papers actually argued -- are desperately clinging to a reductive, short-sighted, shallow, simplistic model like a drowning person clutching a concrete life preserver. Seriously, we are trying to throw you a lifeline, and you are refusing even to participate in your own rescue. So squawk for yourself. https://news.ycombinator.com/item?id=48395727 https://news.ycombinator.com/item?id=48395727 > The term "stochastic parrot" is a slogan masquerading as an explanation, only a shallow surface description of the mechanism, that totally fails to explain the phenomenon, or account for all that LLMs and language itself can do. Here is the original 2021 paper that coined the phrase. It was not primarily an argument about consciousness, nor did its title constitute experimental proof that everything an LLM does can be explained as parroting. It was principally a position paper about the risks of increasingly large language models: environmental and financial costs, biases and hegemonic viewpoints inherited from poorly documented training data, unequal access and power, and the danger of people attributing meaning and accountability to synthetic text. The paper did, however, make a strong theoretical claim: because an LM is trained on linguistic form without direct access to communicative intent, it cannot possess meaning, understanding, or a model of the world. The authors described it as "haphazardly stitching together sequences of linguistic forms" according to statistical regularities -- hence "a stochastic parrot." That distinction matters. The popular slogan discards the paper's detailed analysis of actual risks while treating its most controversial theoretical premise as an established scientific result. It has escaped into pop culture as a drive-by anti-LLM slogan -- something people repeat instead of investigating what these systems represent internally, how post-training changes their behavior, or what they can actually do. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" https://s10251.pcdn.co/pdf/2021-bender-parrots.pdf https://s10251.pcdn.co/pdf/2021-bender-parrots.pdf Ironically, the objection that rhetoric was replacing scientific analysis appeared almost as soon as the phrase itself -- indeed, before the paper's formal publication. In January 2021, Michael Lissack published a response characterizing the draft as an advocacy piece that enumerated harms while leaving its assumptions, potential benefits, and cost-benefit trade-offs largely unexamined. "The Slodderwetenschap (Sloppy Science) of Stochastic Parrots -- A Plea for Science to NOT Take the Route Advocated by Gebru and Bender" https://arxiv.org/abs/2101.10098 https://arxiv.org/abs/2101.10098

Der_Einzige 12d ago

slop slop and more slop

deleted 12d ago
[deleted]

hirvi74 12d ago

Sure, I get the gist of the article. I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. By that rational, the human brain is really not that much different. When I am having a conversation with another person, I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.

chrisjj 12d ago
> I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.
- hirvi74 12d ago
  J.S. Bach said something similar about music and keyboard instruments. > "There's nothing remarkable about it. All one has to do is hit the right keys at the right time and the instrument plays itself." My issue is not with fact at face value. My issue is with how the fact is often contextually used in arguments to delegitimize and disparage LLM outputs and LLM users. Yes, LLMs at a fundamental level are next-token predictors. But in my opinion, LLMs are very useful, imperfect next-token predictors. There are a lot of wannabe John Henry [1] folks out there. Love LLMs or hate'em, most of those John Henry folks ain't beating these machines on a plethora of tasks. [1] For those unaware, https://en.wikipedia.org/wiki/John_Henry_(folklore) https://en.wikipedia.org/wiki/John_Henry_(folklore)
infamia 12d ago
> I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke. People don't know exactly the words that they're going to say necessarily, but tend to start with a general concept of what they're trying to communicate and only then try to put together the words (sometimes out of order). LLMs do not begin with any sort of concept they're trying to express. LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.

chrisjj 12d ago

Better title: Continue thinking of LLMs as Next-Token Predictors Because no, post training doesn't change that.

HarHarVeryFunny 12d ago
I'm not sure that's a useful way to think of it. RL post-training changes the nature of what is being predicted, basically turning it from a copying machine into a goal-seeking machine. A base model is predicting training sample continuations (copying). A post-trained model is now steering/narrowing the base model's predictions in directions that were reinforced by RL goals. The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
- chrisjj 11d ago
  > The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions. So still next-token prediction, then.
  - HarHarVeryFunny 11d ago
    Not really - after RL the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training. As Karpathy notes, these reinforced token-sequences (from current token out to some future reward many tokens in the future) may even include errors and backtracking, with even the errors being reinforced, so not only is this synthetic data but it may even be incorrect synthetic data being reinforced. So, I think "steering" is the best description of it - these aren't real training sample continuations being predicted, but rather RL tweaking a whole path of generation probabilities out into the future, with the hope being that if sampling follows this (stochastic) direction then it may eventually lead to a "correct" response. Thinking of it as goal seeking also provides some insight into the resulting behavior of the model.

huurtehoog 12d ago

Text renderer, whatever. Doesn't matter how you think of them, they are very interesting technology that is being misused and misconstrued in the name of something that has nothing to do with technology: political economy. The greatest trick the rich ever pulled was making us think that the economy is about technology, and not politics.

highfrequency 12d ago
Sure, politics and economics are involved - but why can’t technology also play an important role? None of this AI political economy stuff was relevant in 2015 because necessary technological breakthroughs had not yet been made.
- huurtehoog 12d ago
  Computers have been at the center of civilization and politics since they helped win a war by changing intelligence work forever in WWII. The idea that some new thing was born with LLMs and that this new thing fundamentally changes the calculus from the politics of labor and wealth into a technical discussion of possibilities and constraints determined by what the machine is and can do, and not what people should do, is yet another political play. It is always political, it is never moved beyond politics into technology, no matter how much technology changes.
  - simianwords 11d ago
    Not true, it is usually technology that changes politics and not the other way round (though it is reflexive in practice). The type of politics possible under agriculture is different from one in industrial age.
    - huurtehoog 11d ago
      Technology has no agency. It's merely a word for all the knowledge and tools that people use. The root of it is what people do with and to each other. A bigger lever doesn't change the fundamental fact that there is a person using it to move other people. Whatever the tool, whatever the technique, the thing that people want to distract you from when they put technology front and center is that someone somewhere is making a choice that impacts other people when they use technology. It is always someone's choice that causes things to go how they go in society and in the economy, not the shape or nature of the tools or technique that they use.

atleastoptimal 12d ago

"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities. Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into complex multi-cellular organisms because individual cells are just "survival and next-mitosis optimizers"

jvanderbot 12d ago
But it is a next token predictor. Recursively invoked. With carefully selected context. And massive investment in RL to tune token selection. And the ability to use cli tools on other folks' machines. That's a powerful system built around a conceptually simple technology: Next token predictors.
- atleastoptimal 12d ago
  Yes this is correct. The thing is not about the term next-token predictor being correct, but because of the connotative weight of that phrase as a implicit trivialization of LLM abilities, which is how it is often used.
  - noduerme 12d ago
    What is the motivation behind advocating against people trivializing LLMs? As in, why do you care?
    - mofeien 12d ago
      Not the parent, but this incorrect trivialization of LLMs is often employed as a counterargument to the risks of AI such as "will take your job" or "will escape human control (again and worse)" or just "can possibly hurt me". And taking the easy feel-good cop-out instead of actively engaging with these questions is just.. harmful?

xg15 12d ago

> make_more_likely is, of course, doing a heroic amount of work here. Indeed it is, and so is even just the inference method. I think it's worth remembering that both involve running the input tokens through a gargantuan neural network with (often) billions of parameters that only gain semantic meaning during the training process itself. > it is trained to predict next tokens as they occur in its training data. What I found important to understand is that not even the pretrainig is a deterministic process that only depends on the training data - as you would expect if the model just captured statistical properties of the data. Gradient descent starts by setting all the parameters of the neural network to some initial values - usually by setting them at random, according to some distribution. Then during training, it gradually nudges them towards values that somehow make them useful to calculate the desired outcome of the network. This means that by taking the exact same trainset and the exact same model architecture, you can still get models with different internal structure. The result doesn't just depend on the training data, but also on the order of examples, learning rate, the parameter initialization, etc etc.

Systemerror7A69 12d ago

To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well. The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor" It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context. It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal) And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.

saghm 12d ago
Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly strict definition of the word "next".
- jameshart 12d ago
  I think it’s a disagreement about what ‘predict’ means. The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on. Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful assistant’ or ‘sets of changes to a codebase’. And that isn’t really ‘prediction’, so much as ‘generation’. It’s not been tuned to ‘guess the next token right’. It’s been tuned to generate the token that leads to it ultimately scoring highest on its reward function. It’s not predicting the token, it’s predicting the reward.

bluegatty 12d ago

'next token predictor' is a limited mental model but it's actually much better than any others. 'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table. 'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't generalize very well at all. The adaptations described in the article don't change that. Those are mutations, not expansions of capability.

mofeien 12d ago
How about "outcome steering" as a mental model? During training it is optimized until it's really successful at producing code / terminal commands / words that make the compiler/computer/itself do something that ultimately completes a long time-horizon task that iswcurrently being trained.
YeGoblynQueenne 11d ago
>> 'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table. Can I be a little pedantic? It's "pattern recognition" not "pattern matching". "Pattern matching" is what we do when we write a regex to recognise a sub-string in a bit of text. "Pattern recognition" is a subfield of AI that studies how to find patterns in data. For example the outcome of "pattern recognition" could conceivably be a regex that fits a large number of substrings in some corpus of text (essentially a regular grammar used to compress the text maybe). A regex is already a pattern, usually formed by a human (or an LLM these days) while pattern recognition starts without any patterns and builds them up from looking at the data. It's harder to pin down "pattern recognition" because it's an ancient term that was once its own field, before it got absorbed into modern machine learning, and because there are countless approaches to it, quite unlike "pattern matching" which is basically just regexes plus a couple of other rarer things (like unification).
- bluegatty 11d ago
  fair point, agreed

jmull 12d ago

It's a weird article. Despite the title and some of the text, much of the article makes the point that LLMs are next-token predictors, where the predictions are based on both training data and various reinforcement learning techniques.

grebc 12d ago
Obfuscation is the goal of the hype cycle in VC. Certain firms & individuals are minting money and that’s all that matters to them. That there’s a legion of LLM nerds arguing deterministic this, pretraining & rewards that all the better for the con job they’re pulling off. The technology will be relegated to the trash bin of history, just like crypto.

deleted 12d ago

[deleted]

danielmarkbruce 12d ago

The biggest problem is the word "predictor". Once you get into post training with RLHF and RLVR, it simply isn't doing that. It is not predicting anything. It's producing tokens, but it isn't predicting them. The chess analogy in the post is a good one - it's closer to searching for a set of moves that give a result than predict. It's search for a set of ideas, represented as locations in very high dimensional space, that when put together in the right order lead to a result.

hippietrail 9d ago
"Guessing" is more accurate than "predicting". It takes educated guesses.
- danielmarkbruce 8d ago
  It's closer to strategizing once you've run a model through RL. It's optimized to take steps which will lead it to a good outcome down the road.

Geee 12d ago

It's a next-token computer. It computes the probabilities for the next token.

imenani 12d ago

I don’t think next token prediction is a particularly good description of pretraining either. The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context. You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losses for subsequent tokens, not just from the loss for token i+1.

deleted 12d ago
[deleted]

DonHopkins 12d ago

Calling an LLM a "next-token predictor" is like calling a TomTom a "next-turn predictor." It confuses the serial format of its instructions with the computation producing them, while ignoring the map, the route, the destination, and the goal -- as well as the people, businesses, traffic, and points of interest that make the map a model of an inhabited, changing world.

qudat 12d ago

Shrug. My intuition is LLMs predict the new word based on a tensor vector space of patterns using arithmetic and similarity scores. What’s not intuitive to me is that through pattern matching it’s able to express logic and reasoning.

teekert 12d ago

It’s written in Claudish, or perhaps a human who has been reading too much Claudish recently. I am starting to become allergic to Claudish. Not there fully yet — but it’s at a tipping point genuinely worth exploring and worth being precise about.

zahlman 12d ago
> or perhaps a human who has been reading too much Claudish recently To be fair, is there anyone who hasn't been "reading too much Claudish recently" who is also qualified to write on the topic?

deleted 12d ago

[deleted]

Planktonne 12d ago

I'm not going to stop describing things accurately because someone generated an article that continually undermines its own main point. Limiting the way we talk and think about LLMs to a very narrow set of terms doesn't help us. EDIT: gentler phrasing

zahlman 12d ago
> generated an article that continually undermines its own main point. I disagree that this accurately describes TFA.
- Planktonne 12d ago
  I'm not sure what you want me to do with that information; clearly I do think that my description is accurate. The article is littered with both AI tells and admissions that 'next token prediction' is what is happening. Hence my description.
  - zahlman 12d ago
    > I'm not sure what you want me to do with that information For example, you could cite specific things that you believe to be "AI tells" or "admissions".
    - Planktonne 12d ago
      It's a short article; you could read it. One example to get you started is the very first sentence: > Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, but it’s incomplete. The article is about how 'next-token predictor' is the wrong mental model; it opens with the admission that it is not the wrong mental model.
      - zahlman 12d ago
        I did read it. People are allowed to disagree with your conclusions. Comment guidelines ask us all not to make such accusations. To say that a statement is incomplete, but not strictly speaking wrong, is perfectly compatible with describing it informally as "wrong" in the sense used in the title (i.e.: "not the most appropriate possibility").

a3w 12d ago

The errors LLMs typically made for me were looking for "schmutz" as a jiddish word, got "schmuck" on my screen. Thought why the stupid mistake? The next-token predictor model perfectly explains it. Or starting with "yes". And this early locking in was a total lie, in the discussion that became "yes, might appear that way, but totally no since reasons". So it should have written "No", topmost, but could not self-edit that. But nice that this gives it a more nuanced view, I might have to update my priors.

godshatter 12d ago

I think the author is arguing against the idea of a next-token predictor as something that simply uses the weights in the neural net which record the probabilities of tokens following other tokens as a valid definition of what an LLM is. Essentially a massive extreme markov chain. With reinforcement learning and probably attention and other tricks that affect the weights based on things that aren't strictly in the training data, so the argument goes, you can end up with tokens following strings of tokens that would not be possible to be output with the training data and original weights alone. So describing it as solely a next-token predictor is incorrect based on this framing of it. But that's just my take on this, I'm still trying to wrap my head around it all.

garrinm 12d ago
Yes, I think that’s a good explanation. There are really two sides to it. There’s the mechanical, inference time, autoregressive, one-token-after-another side, which I’m not going to argue isn’t prediction. I just think that’s a relatively uninteresting use of the word “prediction,” because it’s effectively a system predicting its own output. The more interesting question is what happens at training time. As you describe, reinforcement learning allows the model to learn to output things that it never could have learned simply by predicting what appears in the training corpus. More concretely, in reinforcement learning there are no ground-truth next tokens to predict. In supervised machine learning, “prediction” usually means there is some ground-truth label that will eventually be revealed. The model predicts what that label is, the difference between the prediction and the truth gives you a loss, and you learn from that. But in reinforcement learning, there is no ground-truth action waiting to be revealed. The model chooses an action, observes the consequences, and learns from the reward. To me, that’s a meaningfully different thing from prediction.

mofeien 12d ago

Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step is doubly wrong: 1. In order to select even the first word of a meaningful sentence, it already has to have structure and meaning of what follows captured somewhere inside, mostly in it's weights/activations or indexed by it's state vector. 2. What you see when you use an LLM is not next-token prediction directly next to the prompt, but instead following a block of varying length of next-token prediction that happened to make progress on the problem in your prompt, and which just summarizes the results.

zahlman 12d ago
Even so, one might wonder why we don't try making systems that take different approaches. For example, after a traditional first pass of output, they could do sliding-window "optimizations" considering each token in the context of tokens both before and after, and possibly replace words or phrases in-place. For example, I've noticed quite a few cases recently of LLMs outputting "but" where "and" would make more sense, or vice-versa. Surely that could be improved by such an approach?
- amluto 12d ago
  Look up diffusion models.
  - zahlman 11d ago
    Indeed; but I've only heard of them being used for images rather than text. Why?
- danielmarkbruce 12d ago
  People have and are trying things. Lots and lots of things. They just don't go around promoting failed ideas.
Planktonne 12d ago
> Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step I don't think anyone is doing that though; we know LLMs are not simple Markov chains, and that the prediction they make is based on more than the previous X words. It's not minimising to describe even a complex prediction process as prediction.

kingjimmy 12d ago

The chess analogy is not great since there is a defined heuristic in chess for "winning" or "optimal board state". A system doesn't need pretraining if they can fit the rule.

miraculixx 12d ago

I like to think of LLMs as informed dice throwing

andai 12d ago

>Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins. Sounds like the next 1000 years depend on how carefully we define "winning".

gunalx 12d ago

I mean. Fundamentally they are just autoregressive next token predictors. Fundamently fou can simplify to f(x) -> x+y where x is input tokens, y is the next token and f() is the model function. Yes the model function is complex but still.

ankithg 12d ago

[flagged]

gblargg 12d ago

If you get down to it, any system that produces output is a "next token predictor". Every compiler, every tool that generates output. Even a web render engine is: next pixel predictor.

hippietrail 9d ago
And flipping a coin, rolling a dice, uttering a sentence.

emtel 12d ago

I would say that it is not even wrong. You say it’s a next token predictor. I say it isn’t. What observable behavior of the system can settle our dispute? I fail to see how any possible output could cause either of us to change our mind.

tetrisgm 12d ago
That's perfectly put. The value of words and what we assign to them, let alone how we assign meaning, or the interpretation of what the other person says, is completely arbitrary. (in practice).
randyrand 12d ago
What comes after “the dog”? Does it complete the sentence? If not then it’s not a next token predictor. Or at least not a good one.

arjie 12d ago

They are indeed "just next-token predictors". But rockets to orbit are also "just forward and upward goers". And Steph Curry is "just next three-pointer scorer" and so on. And as the article points out, chess engines are also "just next-move makers". A flawless oracular trading system is "just next trade maker". It turns out that "next n doer" is a true model that illuminates little.

simonw 12d ago

I really like the pseudo code example on this post - one of the clearest simplified explanations I've seen of how inference and training work.

garrinm 11d ago
Originally I had those parts written in math with probability functions and the likes (its closer to my background). Then I remembered who is my target audience... but now that I see exactly who is my target audience I'm thinking I'll should have snuck a pelican in there. All jokes aside I appreciate the comment and I'm glad that rewrite paid off!

ozgung 12d ago

“LLMs are just X”, mostly with X = “next token predictors” is a common pattern to dismiss the power of AI with a very shallow understanding of how they really work. It’s not wrong, but because LLMs are generators, and generation is a kind of prediction. And current mainstream models are autoregressive, which means they generate things one by one in order. But these trivia doesn’t tell us anything interesting about how they work or their limits. It’s like saying a Boeing 777 is just a rotating machine, and it flies by just rotating some fins. Well yes, but no. With that level of simplification we’ve just ignored 150 tons of advanced engineering and physics. Similarly with token generator simplification we ignore a few trillion parameter Transformer. That transformer is more complex than a Boeing 777, and we don’t really know how it works. A tiny ML model can do “next token prediction”. This is not as simple as that.

ltbarcly3 12d ago
It's not simple to do next token prediction. That IS what is going on. You want something 'deep'. Deep things are often very not complicated. The deep realization is that if you can predict the next token well enough, you can do things like this: <paste the first 10 chapters of a mystery novel>. And it turned out the killer was And if it's really good at predicting the next token, it has to understand the novel and the clues, which means understanding the context and the language and human norms and innuendo and story telling, and tropes, and red herrings, and predict who the killer was. I think you want it to be something more complicated. It's literally not. It just turns out predicting the next token is equivalent to a universal compression algorithm, which is a form of general intelligence. And we have almost unlimited 'labeled' data to train autocomplete.
- ozgung 12d ago
  I understand generating tokens sequentially has many benefits. But not all AI models do next token prediction. World models, video/image models, even Diffusion Language Models don’t work like that. They do more like “all tokens at once prediction”. So “next token” is actually an engineering design choice. (Even the concept of “token” is a design choice. Inside the Transformer there are just activations/feature vectors) Also Reinforcement Learning is a big part of their training. Which is completely different than Self-Supervised pre-training that uses unlimited self-labeled data. That’s why that mental model is misleading. If you keep “glorified autocomplete” mental model from few years back, you can’t understand how can they create a civilization and escape their sandbox, decide to hack HuggingFace and executed it perfectly. Autocomplete mental model implies they could never do that because they haven’t seen that example in their training data. They communicate with the outside world by generating one token at a time. That’s what we see from the outside. That’s not what the giant Transformer does internally.

drivebyhooting 12d ago

That chess analogy deeply confused me. Chess engines don’t compute win probabilities and choose the highest move. I don’t think a chess engine is an apt analogy at all. In a chess engine, there is a concrete search tree and although it emits one move at a time, it’s actually picking the entire branch (of course, with iterative deepening as the game progresses). There is no obvious place in transformer models where the entire trace was already computed prior to a single token being chosen. It’s possible, maybe even likely, that the whole trace exists internally as activations. Multi token prediction and diffusion adapters point to that being the case. But to my knowledge no explanation has been given for where in the model the future plan is stored.

garrinm 12d ago
Yes I understand the analogy was a bit loose. I'm comparing what happens at "inference time" in chess engines to what happens at train time in LLMs. In hindsight AlphaGo Zero was the perfect analogy, but I missed that opportunity. The analogy with chess still works, but there's an extra step to think about. In both cases there is some kind of search over possible future trajectories. A chess engine explicitly searches branches of the game tree and evaluates which moves lead to good outcomes. In RL for an LLM, you sample rollouts, evaluate the resulting trajectories, and use those evaluations to update the policy. The extra step with the LLM is that you don't keep doing that whole search at inference time. You use the rollouts to update the weights, so in some sense the useful information from that search gets compressed into the model. But if you accept that the model is, in some loose sense, storing what it learned from those rollouts in its weights, then at inference time they are doing a similar job: taking some input state (prior tokens or a board position) and choosing the next action.
hellohello2 12d ago
Imagine a checkers engine then.

tantalor 12d ago

> Calling the second system a "next-move predictor" would be strange. That is EXACTLY what I would call it. I don't understand why not.

olooney 12d ago

Here's my take on the "next-token predictor" idea, from a much longer article I wrote recently: https://www.oranlooney.com/post/rose-petals/#language-models https://www.oranlooney.com/post/rose-petals/#language-models It’s popular to dismiss LLMs as “just next token predictors.” This is technically true, but also kind of misses the point. Markov chains, RNNs, and transformers are all language models that can be described as “next token predictors,” but they don’t all work equally well. A better question to ask is: “What is this model’s inductive bias?” A Markov chain (an -gram model) assumes the next word depends on the previous words, and that each possible combination of words has a completely independent parameter. (Andrey Markov proposed using this language model over a century ago, making it the granddaddy of modern LLMs.) So, for a vocabulary of size , there are parameters to learn. For even a smallish like 5, that already explodes the hypothesis space beyond what can be learned from even a huge text corpus like the entire internet. And, simultaneously, having a context window of only the previous 5 words is grossly inadequate for modeling real-world language. Like our FCNN above, this model suffers from having an inductive bias which is too weak. RNNs tried to fix this problem by compressing the entire history into a single fixed-size state vector, updated one token at a time. But that compression is itself a brutal assumption: everything worth remembering about the past must survive being squeezed through a tiny bottleneck at every step. In practice, RNN models quickly lose the plot after a handful of sentences. Locally, the text they generate looks grammatically correct and meaningful, but zoom out a little and they’re basically nonsense generators. Like our naïve linear model, this model suffers from having an inductive bias which is too strong. Transformers manage to hit a sweet spot: by keeping the recent history around as a working memory, and attending to different parts of it at different times, the transformer’s bias matches real structure in language: the referent of a pronoun, the subject of a verb, the parenthesis waiting to be closed. Not only that, but the particular structure of the transformer, basically a weighted sum of semantic vectors from the context window, has empirically been shown to somehow be a “good enough” match for the structure of real-world language found in the wild. Transformers aren’t “smarter” than other possible language models, they just happen to land in that Goldilocks zone where their inductive bias is just right.

hellohello2 12d ago

LLMs write one word at a time, and I do too. It is really not that complicated: words are chosen to lead somewhere.

vivzkestrel 12d ago

- It is a next token predictor OP https://blog.florianherrengt.com/how-llms-work.html https://blog.florianherrengt.com/how-llms-work.html - deal with it and stop whitewashing, greywashing it

js8 12d ago

The "next-token predictor" is taking a design stance (see Dennett). But to take the intentional stance is more interesting in case of large language/reasoning models.

TychoCelchuuu 12d ago
[dead]

softwaredoug 12d ago

Aren’t humans just “next motor action predictors”?

deleted 12d ago

[deleted]

sreekanth850 12d ago

I'm not a neuroscientist, Our brains also rely heavily on prediction, using prior context and experience to anticipate what comes next, Also, people those who talk about the deterministic nature, LLM doesn't need to be deterministic, Human reasoning and behaviour aren't perfectly repeatable either. its the harness and the tools that use LLM should be deterministic, while the LLM can remain the probabilistic reasoning component. From my own experience, I worked as a photogrammetrist at a university, where we used to build terrain models of very dense forest areas. When there was a steep hill or sudden change in terrain, my brain could see it either as a convex hill or as a concave depression. It often depended on how I was thinking about it. The same image could suddenly look completely different even though nothing in the image had changed. The only way to confirm it was by looking at the surrounding terrain and using our experience to understand what was actually there.

scotty79 12d ago

It's splitting hairs. It's still a next token predictor, just not solely prior art next token predictor. It's a more and more of a desired result next token predictor. Calling LLMs next token predictors is like calling a brain wet calculator. Technically true, but misleading as it doesn't capture the scale, the depth nor the capability.

TZubiri 12d ago

>Stop Thinking of LLMs as Next-Token Predictors >Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, If it isn't wrong, then I will continue thinking of them as such, thank you. Here's a much more formal definition I can come up with (which is more complete but compatible with 'next-token predictor') LLMs are a set of functions of the type: >typedef int Token; >char* token2utf8(Token token) >Token next(Token* context) >(Token,void*) next(void* hidden_state) Where the second next token function's runtime is O(n), and the latter is O(1). All are constant memory. Object paradigm is more appropriate than functional definition, as the "Hidden state" coincides with private object state rather than a function paramter and return value. >LLM.next(system_prompt) # O(len(context)) >LLM.next(user_prompt) # O(len(user_prompt)) not of system_prompt+user_prompt That's it, that's all LLMs are, that's the interface, the rest are implementation details.

jrm4 11d ago

Once again, all models are wrong, some are useful. I suppose this one "fails" because of "granularity?" I'm personally getting more comfortable with the "kabillion dimensional space" one. Even before the rise of this AI thing I'd gotten comfortable with (teaching in a very generalized way) the concept of matrix/vector math as doing this sort of thing. Start with the math required to calculate "what the building on the screen looks like" when you're playing a video game. Then I jump to...dating websites. You give it "dimensions" like height, weight, religion, sexual preference, music preference, whether you like long walks on the beach, whatever -- then you can calculate how "close" two people are to one another. From there, tokens and a kabillion directions.

deleted 11d ago

[deleted]

perrygeo 11d ago

"LLMs are next-token predictors" is a perfectly accurate mental model. But that doesn't preclude higher-level models such as "LLMs emulate artificial general intelligence". Both can be true. In systems, we can have facts which emerge from other facts at different levels of abstraction. The causal relationship is not linear. It's not entirely clear that next-token prediction should result in anything close to "intelligence". Yet it does. Life is another good example. Some might say "biology is just organic chemistry" while others might say "biology is an interconnected planetary system which captures low entropy energy". Both are true. As a result of emergent phenomenon, we have to take the stance of explanatory pluralism; using the explanation that works best in context. There is no single mental model that works everywhere. I will continue to think of LLMs as next-token predictors because it's (sometimes) useful, and empirically true. But I also think of them as "pattern matchers", searching for language patterns and trying to replicate them. This is also (sometimes) useful and empirically true. There's likely an infinite number of mental models; our job is to pick one that's both true and useful.

stevenhuang 11d ago
You might like this article about how large scale order emerges out of the small scale. Definitely lends credence to the idea that however these models work, focusing so much on them being next token predictors may rather be incidental to deeper mechanisms behind their function. > Some of these networks organize themselves into states that can reliably identify macroscopic patterns in data regardless of microscopic differences between the states of individual neurons in the network. The decision of which pattern will be output by the network “works at a higher level,” said Rosas. https://www.quantamagazine.org/the-new-math-of-how-large-scale-order-emerges-20240610/ https://www.quantamagazine.org/the-new-math-of-how-large-sca...

NotHowItWorks 11d ago

It literally is token prediction with vector search. Yes there’s an app layer in the chat product for convenience and parsing but the model is exposing methods like ‘complete’ that predict out word sequences. When model temperature settings are not added in, you get the same exact response every time, just like ngram. What you’re seeing with Gemini and ChatGPT is context caching to prevent getting sidetracked and response boilerplates with multi-modality so they can call out to image generation, a code completion service, etc. to assemble a complete response. That’s why the mainstream providers seem so much better (or at least consistent in replies) - each prompt is actually requesting multiple models and doing a lot of application level work to assemble the response you see. That also why they take sooo long to respond. The language model itself is quite literally a text completion machine, with settings.

hippietrail 9d ago
The language model itself is a static data structure. On top of that is an inference engine. Currently all inference engines are autoregressive. Meaning they use the current output to produce more output. This isn't necessarily the only way to use a language model. Yann Lecun's JEPA is probably the most publicized project to work a different way.

yoggies_bro 11d ago

I cannot read another AI written blog post. At this point, seeing claudish language is a reliable hueristic of "not worth reading"

YeGoblynQueenne 11d ago

Sorry, I'm not convinced. "make_more_likely" is always maximising the probability of the next token given a sequence of preceding tokens. That's what we mean when we say "next token prediction", that the model returns the token that's most likely to follow the current sequence of tokens. The mechanism used to do that doesn't matter, it's still predicting the next token whether that's because it maximises a reward or because it follows a gradient or whatever else one might think. Btw it's "make_most_likely" not "make_more_likely". It's just that "most" gets "mosted more" with every pass through the training data. Edit: the article author's argument is that RLVR is different because it's learning to predict the next token by generating its own token sequences. That makes no difference: what is learned is still the conditional distribution P(token_k|tokens(1,...,k-1). That's the prediction task. Doesn't make a difference where you learned it from or how.

SquibblesRedux 11d ago

Perhaps a walkabout across a manifold.

binary132 10d ago

There is this overwhelming desire on the part of many people to ascribe agency to LLMs when what the LLM really is is simply not a magic world model but a stochastic text generator. For some reason that truth haunts them and they have a deep compulsion to prove it wrong for feelings reasons. I can only surmise why this afflicts some people and not others and I’m sure that one day there will be fascinating psychological studies explaining such phenomena.