19 ms·

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. What a great way of framing it. I've been trying to explain this to people, but this is a succinct version of what I was stumbling to convey.

jstrieb 1y ago
I have been explaining this to friends and family by comparing LLMs to actors. They deliver a performance in-character, and are only factual if it happens to make the performance better. https://jstrieb.github.io/posts/llm-thespians/ https://jstrieb.github.io/posts/llm-thespians/
- lagrange77 1y ago
  I'll steal that.
- red75prime 1y ago
  The analogy goes down the drain when a criterion for good performance is being objectively right. Like with Reinforcement Learning from Verifiable Rewards.
  - ACCount37 1y ago
    RLVR can also encourage hallucinations quite easily. Think of SAT: giving a random answer is right 20% of the time, giving "I don't know" is right 0% of the time. If you only reward for test score, you encourage guesswork. So good RL reward design is as important as ever. That being said, there are methods to train LLMs against hallucinations, and they do improve hallucination-avoidance. But anti-hallucination capabilities are fragile and do not fully generalize. There's no (known) way to train full awareness of its own capabilities into an LLM.
    - neonspark 1y ago
      I think what you say is true, and I think that this is exactly true for humans as well. There is no known way to completely eliminate unintentional bullshit coming from a human’s mouth. We have many techniques for reducing it, including critical thinking, but we are all susceptible to it and I imagine we do it many times a day without too much concern. We need to make these models much much better, but it’s going to be quite difficult to reduce the levels to even human levels. And the BS will always be there with us. I suppose BS is the natural side effect of any complex system, artificial or biological, that tries to navigate the problem space of reality and speak on it. These systems, sometimes called “minds”, are going to produce things that sound right but just are not true.
      - ACCount37 1y ago
        It's a feeling I can't escape: that by trying to build thinking machines, we glimpse more and more of how the human mind works, and why it works the way it does - imperfections and all. "Critical thinking" and "scientific method" feel quite similar to the "let's think step by step" prompt for the early LLMs. More elaborate directions, compensating for the more subtle flaws of a more capable mind.
  - jstrieb 1y ago
    Nobody that I'd be using this analogy with is currently using LLMs for tasks that are covered by RLVF. They're asking models for factual information about the real world (Google replacement), or to generate text (write a cover letter), not the type of outputs that are verifiable within formal systems—by definition the type of output that RLVF is intended to improve. The actor analogy is still helpful for providing intuition to non-technical people who don't know how to think about LLMs, but do use them. Also, unless I am mistaken, RLVF changes the training to make LLMs less likely to hallucinate, but in no way does it make hallucination impossible. Under the hood, the models still work the same way (after training), and the analogy still applies, no?
    - red75prime 1y ago
      > Under the hood, the models still work the same way (after training), and the analogy still applies, no? Under the hood we have billions of parameters that defy any simple analogies. Operations of a network are shaped by human data. But the structure of the network is not like the human brain. So, we have something that is human-like in some ways, but deviates from humans in ways, which are unlikely to be like anything we can observe in humans (and use as a basis for analogy).
  - jimbokun 1y ago
    But being "objectively right" is not the goal of an actor. Thus, why it's a good metaphor for the behavior of LLMs.
- ngc248 1y ago
  A better analogy is of an overconfident 5 year old kid, who never says that they don't know the answer and always has an "answer" for everything.
- bo1024 1y ago
  This is also related to the philosophical definition of bullshit[1]: speech intended to persuade or influence without any active intention to be either true or false. [1] https://en.wikipedia.org/wiki/On_Bullshit https://en.wikipedia.org/wiki/On_Bullshit
mohsen1 1y ago
Intelligence in a way is the ability to filter out useless information. Be it, thoughts or sensory information
tugberkk 1y ago
Yes, can't remember who said it but LLM's always hallucinate, it is just that they are 90 something percent right.
- OtomotO 1y ago
  Which totally depends on your domain and subdomain. E.g. Programming in JS or Python: good enough Programming in Rust: I can scrap over 50% of the code because it will a) not compile at all (I see this while the "AI" types) b) not meet the requirements at all
- ljm 1y ago
  If I was to drop acid and hallucinate an alien invasion, and then suddenly a xenomorph runs loose around the city while I’m tripping balls, does being right in that one instance mean the rest of my reality is also a hallucination? Because it seems the point being made multiple times that a perceptual error isn’t a key component of hallucinating, the whole thing is instead just a convincing illusion that could theoretically apply to all perception, not just the psychoactively augmented kind.
aitchnyu 1y ago
All models are wrong, some are merely useful - 1976/1933/earlier adage.
- pjmorris 1y ago
  Generally attributed to George Box
- lagrange77 1y ago
  Right, all models are inherently wrong. It's up to the user know about its limits / uncertainty. But i think this 'being wrong' is kind of confusing when talking about LLMs (in contrast to systems/scientific modelling). In what they model (language), the current LLMs are really good and acurate, except for example the occasional chinese character in the middle of a sentence. But what we mean by LLMs 'being wrong' most of the time is being factually wrong in answering a question, that is expressed as language. That's a layer on top of what the model is designed to model. EDITS: So saying 'the model is wrong' when it's factually wrong above the language level isn't fair. I guess this is essentially the same thought as 'all they do is hallucinate'.
deleted 1y ago
[deleted]

jstrieb 1y ago

I have been explaining this to friends and family by comparing LLMs to actors. They deliver a performance in-character, and are only factual if it happens to make the performance better. https://jstrieb.github.io/posts/llm-thespians/ https://jstrieb.github.io/posts/llm-thespians/

lagrange77 1y ago
I'll steal that.
red75prime 1y ago
The analogy goes down the drain when a criterion for good performance is being objectively right. Like with Reinforcement Learning from Verifiable Rewards.
- ACCount37 1y ago
  RLVR can also encourage hallucinations quite easily. Think of SAT: giving a random answer is right 20% of the time, giving "I don't know" is right 0% of the time. If you only reward for test score, you encourage guesswork. So good RL reward design is as important as ever. That being said, there are methods to train LLMs against hallucinations, and they do improve hallucination-avoidance. But anti-hallucination capabilities are fragile and do not fully generalize. There's no (known) way to train full awareness of its own capabilities into an LLM.
  - neonspark 1y ago
    I think what you say is true, and I think that this is exactly true for humans as well. There is no known way to completely eliminate unintentional bullshit coming from a human’s mouth. We have many techniques for reducing it, including critical thinking, but we are all susceptible to it and I imagine we do it many times a day without too much concern. We need to make these models much much better, but it’s going to be quite difficult to reduce the levels to even human levels. And the BS will always be there with us. I suppose BS is the natural side effect of any complex system, artificial or biological, that tries to navigate the problem space of reality and speak on it. These systems, sometimes called “minds”, are going to produce things that sound right but just are not true.
    - ACCount37 1y ago
      It's a feeling I can't escape: that by trying to build thinking machines, we glimpse more and more of how the human mind works, and why it works the way it does - imperfections and all. "Critical thinking" and "scientific method" feel quite similar to the "let's think step by step" prompt for the early LLMs. More elaborate directions, compensating for the more subtle flaws of a more capable mind.
- jstrieb 1y ago
  Nobody that I'd be using this analogy with is currently using LLMs for tasks that are covered by RLVF. They're asking models for factual information about the real world (Google replacement), or to generate text (write a cover letter), not the type of outputs that are verifiable within formal systems—by definition the type of output that RLVF is intended to improve. The actor analogy is still helpful for providing intuition to non-technical people who don't know how to think about LLMs, but do use them. Also, unless I am mistaken, RLVF changes the training to make LLMs less likely to hallucinate, but in no way does it make hallucination impossible. Under the hood, the models still work the same way (after training), and the analogy still applies, no?
  - red75prime 1y ago
    > Under the hood, the models still work the same way (after training), and the analogy still applies, no? Under the hood we have billions of parameters that defy any simple analogies. Operations of a network are shaped by human data. But the structure of the network is not like the human brain. So, we have something that is human-like in some ways, but deviates from humans in ways, which are unlikely to be like anything we can observe in humans (and use as a basis for analogy).
- jimbokun 1y ago
  But being "objectively right" is not the goal of an actor. Thus, why it's a good metaphor for the behavior of LLMs.
ngc248 1y ago
A better analogy is of an overconfident 5 year old kid, who never says that they don't know the answer and always has an "answer" for everything.
bo1024 1y ago
This is also related to the philosophical definition of bullshit[1]: speech intended to persuade or influence without any active intention to be either true or false. [1] https://en.wikipedia.org/wiki/On_Bullshit https://en.wikipedia.org/wiki/On_Bullshit

mohsen1 1y ago

Intelligence in a way is the ability to filter out useless information. Be it, thoughts or sensory information

tugberkk 1y ago

Yes, can't remember who said it but LLM's always hallucinate, it is just that they are 90 something percent right.

OtomotO 1y ago
Which totally depends on your domain and subdomain. E.g. Programming in JS or Python: good enough Programming in Rust: I can scrap over 50% of the code because it will a) not compile at all (I see this while the "AI" types) b) not meet the requirements at all
ljm 1y ago
If I was to drop acid and hallucinate an alien invasion, and then suddenly a xenomorph runs loose around the city while I’m tripping balls, does being right in that one instance mean the rest of my reality is also a hallucination? Because it seems the point being made multiple times that a perceptual error isn’t a key component of hallucinating, the whole thing is instead just a convincing illusion that could theoretically apply to all perception, not just the psychoactively augmented kind.

aitchnyu 1y ago

All models are wrong, some are merely useful - 1976/1933/earlier adage.

pjmorris 1y ago
Generally attributed to George Box
lagrange77 1y ago
Right, all models are inherently wrong. It's up to the user know about its limits / uncertainty. But i think this 'being wrong' is kind of confusing when talking about LLMs (in contrast to systems/scientific modelling). In what they model (language), the current LLMs are really good and acurate, except for example the occasional chinese character in the middle of a sentence. But what we mean by LLMs 'being wrong' most of the time is being factually wrong in answering a question, that is expressed as language. That's a layer on top of what the model is designed to model. EDITS: So saying 'the model is wrong' when it's factually wrong above the language level isn't fair. I guess this is essentially the same thought as 'all they do is hallucinate'.

deleted 1y ago

[deleted]