6 ms·

As an MLE I feel these takes are too reductionist. You could say the (nearly) same thing about search. And content recommendation. And clustering. And topic m

As an MLE I feel these takes are too reductionist. You could say the (nearly) same thing about search. And content recommendation. And clustering. And topic modeling. And outlier detection. And spam filtering. And image diffusion. And dimension reduction. And... There's a lot in common between these things, but there's also a lot cool and different! For transformers in particular, it's pretty cool that you get some WILD emergent properties simply from scaling up. So yes, it's just a next token predictor, but I'm just a bundle of nerves and meat. I don't get a lot out of those descriptions.

oersted 8d ago
Obligatory link to the classic copy-pasta: > They're Made out of Meat https://web.mit.edu/people/dpolicar/writing/prose/text/thinkingMeat.html https://web.mit.edu/people/dpolicar/writing/prose/text/think...
mitxela 8d ago
Some concrete facts about LLMs are explained by their next token predictor nature. Every time it says "wait, that's wrong." instead of generating the correct thing the first time.
- howunfortunate 8d ago
  I think that's relatively emergent too though! BERT never really did that (at least to my recollection), presumably because its training was never sufficient for it to develop corrective reasoning in a chain of thought.
  - mitxela 7d ago
    Back around late 2024 I think, LLMs were being trained to say "Wait," because that led to more correct answers in the end. It was actually a hack to inject "Wait," whenever the LLM tried to end a reasoning block.
  - mitxela 8d ago
    BERT isn't a next token predictor. It predicts a single token based on the whole surrounding context in both directions.
    - howunfortunate 8d ago
      I mean sure, but BERT can be and often is used as a next token predictor / generator. I could have used any series of NLP examples, the point is this CoT behavior only emerges when you get to a certain scale (and training style, presumably).
  - nairboon 8d ago
    What exactly do you mean with "emergent" here?
    - b112 7d ago
      I don't know what the parent poster means, but if you look at how LLMs are trained, and how it's trained on human communication, you should find it interesting that humans often correct themselves. They also often create facts out of nowhere, and do this crazy thing called dreaming. If you're an LLM trying to symbolically emulate human behavior, especially when coding, you're going to see git commits, where people write a bunch of code and then go, oh wait, I should fix that! From the perspective of an outside observer, making mistakes, and then correcting the mistakes, is how you code. In fact, an outside observer might think that this is how you teach people to code. You make a mistake, then you say, oh, here's the mistake I made, and here's why, and then you fix it. So I think true emergent behavior will be demonstrated, when an LLM says "Hey wait a minute! They're actually making real mistakes, and they're actually correcting, it's them not on purpose?!", and then stops hallucinating, and stops making mistakes as it codes. The same is true of bug trackers. You submit a bug request and you often have a patch. Then people discuss it. From a MLM's perspective, you're supposed to write poor code, or code that's not perfect, and then have a conversation. If I had the resources to train a large-scale LLM, I would clone, for example, GitHub. I would then remove everything where people are fixing broken code. The outcome would be very interesting. I've been looking at ways to make enhanced long-term memory stores for LLMs, and there's lots of problems with shifting symbolic relationships if you do it wrong, but definitely once there's true long-term memory, and adaptive behavio,r I think that's the only way you're going to get true real emergent behavior.
      - danielbarla 7d ago
        In general, I completely agree with you. Personally, as a human: I have fallible memory, beliefs that are not as sound as I may believe, and all of those notwithstanding, I may not correctly reason, even if my "inputs" were perfect. Let's not even get into cognitive biases, etc. I am acutely aware of my, and other humans' fallibilities. I think when people are irritated by the "hallucination" aspect of LLMs, they are often running into something of a slightly different nature. I mean, firstly, there seems to be a higher-than-normal ratio of these "little mistakes". And secondly, there are some pretty odd ones - e.g. in my team, Claude regularly just straight up makes up Jira ticket numbers, and then refers to them with high confidence. I guess what I'm saying is that a human would probably not just make up an id, and run with it (they may be off by one, or mix two up, etc). In my opinion, these can be successfully treated, but I guess you can never fully eliminate the tendency that irks people.
nightski 8d ago
It's a little different than that. Your bundle of nerves and meat is not static. It changes over time. To me the heart of the "next token predictor" is that the distributions are static. You can manipulate what you feed into it through context (and a lot of interesting engineering has been applied there through CoT and other techniques to manipulate the prompt). But these models as implemented will never be able try things and learn from mistakes or adapt. They are a set of weights frozen in time. A set of distributions derived from the original data that created them.
- howunfortunate 8d ago
  That's not quite true though. The fact that most models are in practice non-deterministic has been a huge point of contention. And there's nothing inherently stopping labs from continuously fine-tuning the weights after every new invocation. It's just a difficult (not to mention expensive) software and ML problem.
  - deleted 8d ago
    [deleted]
  - nightski 8d ago
    I was not saying that they are deterministic, rather that the distributions (aka weights) are fixed. A model as deployed today at anthropic/open ai/etc is not learning beyond the context as far as I know. What prevents continuous fine-tuning from what I understand is catastrophic forgetting. You can do things like RLHF which are built to minimize the damage but that is more about bringing out capabilities of the base model than incorporating new knowledge (at least from my understanding, I am obviously not a researcher at a lab).
    - howunfortunate 8d ago
      Yes, catastrophic forgetting is absolutely one of the problems that needs to be solved to enable something like this. My broader point is just that there's nothing inherent to the structure of LLMs that stops them from updating their weights and continuously learning from environmental feedback in the way humans do, and there's already solid templates for how they could push even further in that direction. But as an assessment of the current state, I agree with you, LLMs lag humans severely in ability to self-update.
redwood 7d ago
Right or put differently you can say that all computer stuff is reduced to ones and zeros. And if you say that in the right context you might convince someone that computers are therefore not very powerful or interesting but of course you're hiding the power of abstraction generalization scale and for lack of a better word entropy in action. Sometimes I feel we lack the words to describe what we're doing in a way that really conveys how all these things Ladder Up

oersted 8d ago

Obligatory link to the classic copy-pasta: > They're Made out of Meat https://web.mit.edu/people/dpolicar/writing/prose/text/thinkingMeat.html https://web.mit.edu/people/dpolicar/writing/prose/text/think...

mitxela 8d ago

Some concrete facts about LLMs are explained by their next token predictor nature. Every time it says "wait, that's wrong." instead of generating the correct thing the first time.

howunfortunate 8d ago
I think that's relatively emergent too though! BERT never really did that (at least to my recollection), presumably because its training was never sufficient for it to develop corrective reasoning in a chain of thought.
- mitxela 7d ago
  Back around late 2024 I think, LLMs were being trained to say "Wait," because that led to more correct answers in the end. It was actually a hack to inject "Wait," whenever the LLM tried to end a reasoning block.
- mitxela 8d ago
  BERT isn't a next token predictor. It predicts a single token based on the whole surrounding context in both directions.
  - howunfortunate 8d ago
    I mean sure, but BERT can be and often is used as a next token predictor / generator. I could have used any series of NLP examples, the point is this CoT behavior only emerges when you get to a certain scale (and training style, presumably).
- nairboon 8d ago
  What exactly do you mean with "emergent" here?
  - b112 7d ago
    I don't know what the parent poster means, but if you look at how LLMs are trained, and how it's trained on human communication, you should find it interesting that humans often correct themselves. They also often create facts out of nowhere, and do this crazy thing called dreaming. If you're an LLM trying to symbolically emulate human behavior, especially when coding, you're going to see git commits, where people write a bunch of code and then go, oh wait, I should fix that! From the perspective of an outside observer, making mistakes, and then correcting the mistakes, is how you code. In fact, an outside observer might think that this is how you teach people to code. You make a mistake, then you say, oh, here's the mistake I made, and here's why, and then you fix it. So I think true emergent behavior will be demonstrated, when an LLM says "Hey wait a minute! They're actually making real mistakes, and they're actually correcting, it's them not on purpose?!", and then stops hallucinating, and stops making mistakes as it codes. The same is true of bug trackers. You submit a bug request and you often have a patch. Then people discuss it. From a MLM's perspective, you're supposed to write poor code, or code that's not perfect, and then have a conversation. If I had the resources to train a large-scale LLM, I would clone, for example, GitHub. I would then remove everything where people are fixing broken code. The outcome would be very interesting. I've been looking at ways to make enhanced long-term memory stores for LLMs, and there's lots of problems with shifting symbolic relationships if you do it wrong, but definitely once there's true long-term memory, and adaptive behavio,r I think that's the only way you're going to get true real emergent behavior.
    - danielbarla 7d ago
      In general, I completely agree with you. Personally, as a human: I have fallible memory, beliefs that are not as sound as I may believe, and all of those notwithstanding, I may not correctly reason, even if my "inputs" were perfect. Let's not even get into cognitive biases, etc. I am acutely aware of my, and other humans' fallibilities. I think when people are irritated by the "hallucination" aspect of LLMs, they are often running into something of a slightly different nature. I mean, firstly, there seems to be a higher-than-normal ratio of these "little mistakes". And secondly, there are some pretty odd ones - e.g. in my team, Claude regularly just straight up makes up Jira ticket numbers, and then refers to them with high confidence. I guess what I'm saying is that a human would probably not just make up an id, and run with it (they may be off by one, or mix two up, etc). In my opinion, these can be successfully treated, but I guess you can never fully eliminate the tendency that irks people.

nightski 8d ago

It's a little different than that. Your bundle of nerves and meat is not static. It changes over time. To me the heart of the "next token predictor" is that the distributions are static. You can manipulate what you feed into it through context (and a lot of interesting engineering has been applied there through CoT and other techniques to manipulate the prompt). But these models as implemented will never be able try things and learn from mistakes or adapt. They are a set of weights frozen in time. A set of distributions derived from the original data that created them.

howunfortunate 8d ago
That's not quite true though. The fact that most models are in practice non-deterministic has been a huge point of contention. And there's nothing inherently stopping labs from continuously fine-tuning the weights after every new invocation. It's just a difficult (not to mention expensive) software and ML problem.
- deleted 8d ago
  [deleted]
- nightski 8d ago
  I was not saying that they are deterministic, rather that the distributions (aka weights) are fixed. A model as deployed today at anthropic/open ai/etc is not learning beyond the context as far as I know. What prevents continuous fine-tuning from what I understand is catastrophic forgetting. You can do things like RLHF which are built to minimize the damage but that is more about bringing out capabilities of the base model than incorporating new knowledge (at least from my understanding, I am obviously not a researcher at a lab).
  - howunfortunate 8d ago
    Yes, catastrophic forgetting is absolutely one of the problems that needs to be solved to enable something like this. My broader point is just that there's nothing inherent to the structure of LLMs that stops them from updating their weights and continuously learning from environmental feedback in the way humans do, and there's already solid templates for how they could push even further in that direction. But as an assessment of the current state, I agree with you, LLMs lag humans severely in ability to self-update.

redwood 7d ago

Right or put differently you can say that all computer stuff is reduced to ones and zeros. And if you say that in the right context you might convince someone that computers are therefore not very powerful or interesting but of course you're hiding the power of abstraction generalization scale and for lack of a better word entropy in action. Sometimes I feel we lack the words to describe what we're doing in a way that really conveys how all these things Ladder Up