8 ms·

Researchers describe how to tell if ChatGPT is confabulating

56 points by glymor 2y ago

doe_eyes 2y ago
> LLMs aren't trained for accuracy This assertion in the article doesn't seem right at all. When LLMs weren't trained for accuracy, we had "random story generators" like GPT-2 or GPT-3. The whole breakthrough with RLHF was that we started training them for accuracy - or the appearance of it, as rated by human reviewers. This step both made the models a lot more useful and willing to stick to instructions, and also a lot better at... well, sounding authoritative when they shouldn't.
- fsmv 2y ago
  Isn't that the issue? Getting thumbs up from an underpaid human reviewer isn't the same as accurate facts.
  - golergka 2y ago
    It's not the same as completely accurate facts, but it's much closer to accurate facts than LLMs we had before.
  - gexla 2y ago
    The one person I know getting paid to review AI outputs gets paid anywhere from $25 / hour to $40 / hour. Not sure if that's underpaid. It may be a nice option when you can do it at any time to supplement your regular income.
    - drowsspa 2y ago
      Reviewing AI output or helping in training a LLM itself?
      - gexla 2y ago
        This person works through an interface which is similar to Mechanical Turk. You get a list of available projects you qualified for via an assessment. For the AI projects, many of them are comparing responses from two different models, answering questions, and selecting the best response. Other projects might be attempting to get the model to do something against the guidelines, or rating the model on certain capabilities. There's no requirements other than to pass the assessment. As with Mechanical Turk, you can work on your available projects at any time. This feedback is used for training.
dawatchusay 2y ago
Is confabulation different from hallucination? If not I do suppose this is a more accurate term for the phenomenon except that the exact definition isn’t common sense without looking it up whereas “hallucination” is more widely understood.
- curtis3389 2y ago
  When speaking about LLMs, confabulation and hallucination refer to the same thing. The term "confabulation" is just the most accurate description of what's happening, whereas the term "hallucination" refers to something LLMs are fundamentally incapable of.
- CGamesPlay 2y ago
  From the paper <https://www.nature.com/articles/s41586-024-07421-0 https://www.nature.com/articles/s41586-024-07421-0>: > Here we develop new methods grounded in statistics, proposing entropy-based uncertainty estimators for LLMs to detect a subset of hallucinations—confabulations—which are arbitrary and incorrect generations.
- amenhotep 2y ago
  Some people seem to get very angry about calling it "hallucination", because it's a computer, computers can't hallucinate! Stop anthropomorphising it!! So I suppose if you want to stay on the right side of those people - or you are one - you call it confabulation instead.
  - williamcotton 2y ago
    There’s also the position that a definition of confabulate… To fill in gaps in one's memory with fabrications that one believes to be facts. …is much more accurate. Since we’re talking about a technical process it helps to be more precise in our use of language.
    - joquarky 2y ago
      I like to think that it's always confabulating. It's just that usually the words it generates are accurate enough for my needs.
glymor 2y ago
TL;DR sample the top N results from the LLM and use traditional NLP to extract factoids, if the LLM is confabulating the factoids would have random distribution, but if it's not it will be heavily weighted towards one answer. A figure from the paper shows this better than my TL;DR: https://www.nature.com/articles/s41586-024-07421-0/figures/1 https://www.nature.com/articles/s41586-024-07421-0/figures/1
- visarga 2y ago
  Or we could just ask the same question on 3 different LLMs, ideally a large LLM, a RAG LLM and a small one, then use LLM again to rewrite the final answer. When models contradict each other there is likely hallucination going on, but correct answers tend to converge.
  - superb_dev 2y ago
    Why use an LLM to check the work of a different LLM? You could use the same technique that this paper describes to compare the answers each LLM gave. LLMs don’t have to be in opposition to traditional NLP techniques
- Terr_ 2y ago
  The LLM is already generating factoids: Things which resemble a fact without actually being one. (See also: Androids that resemble men but aren't, asteroids that resemble stars but aren't, meteoroids that resemble meteors but aren't...)
- cl42 2y ago
  Thank you! This is so helpful. It's also interesting to see what temperature value they use (1.0, 0.1 in some cases?)... I have a feeling using the actual raw probability estimates (if available) would provide a lot of information without having to rerun the LLM or sample quite as heavily.
zmmmmm 2y ago
> There are a number of reasons for this. The AI could have been trained on misinformation; the answer could require some extrapolation from facts that the LLM isn't capable of; or some aspect of the LLM's training might have incentivized a falsehood This article seems rather contrived. They present this totally broken idea of how LLMs work (that they are trained from the outset for accuracy on facts) and then proceed to present this research as it is a discovery that LLMs don't work like that.
ajuc 2y ago
Simplistic version of this is just asking the question in 2 ways - ask for confirmation that the answer is no, then ask for confirmation that the answer is yes :) If it's sure it won't confirm it both ways.
- 082349872349872 2y ago
  A corollary for natural intelligence: if you can prove that all foos are bar, and that all foos are not bar, that's a good time to suspect that no foos actually exist. (I still don't understand why everyone seems happy to conflate "intelligence" with fact-retrieval?)
  - persnickety 2y ago
    Because there is no single widely accepted definition of intelligence.
    - 082349872349872 2y ago
      No widely accepted definition, but fact retrieval is outside of any of the ones with which I'm familiar.
  - ajuc 2y ago
    > I still don't understand why everyone seems happy to conflate "intelligence" with fact-retrieval? Because it's useful and impossible till very recently.
- munchler 2y ago
  That works for yes/no questions, but not for informational questions, like "Where's the Eiffel Tower?".
derefr 2y ago
> But perhaps the simplest explanation is that an LLM doesn't recognize what constitutes a correct answer but is compelled to provide one Why is it compelled to provide one, anyway? Which is to say, why is the output of each model layer a raw softmax — thus discarding knowledge of the confidence each layer of the model had in its output? Why not instead have the output of each layer be e.g. softmax but rescaled by min(max(pre-softmax vector), 1.0)? Such that layers that would output higher than 1.0 just get softmax'ed normally; but layers that would output all "low-confidence" results (a vector all lower than 1.0) preserve the low-confidence in the output — allowing later decoder layers to use that info to build I-refuse-to-answer-because-I-don't-know text?
- Terr_ 2y ago
  Careful, I think there's a large difference here between: 1. An LLM's mathematical "confidence" of having a clear best-scoring candidate for the predicted next token when given a list of tokens. 2. A not-yet-invented AI that models the idea of different entities interacting, the concept of questions and answers, the concept of logical conflicts, and it's "confidence" that a proposition is compatible with other "true" propositions and incompatible with false ones. To help illustrate the difference, suppose you trained an LLM on texts where a particular question was always answered with "I don't know, I have zero confidence in anything anymore." Later the LLM will regurgitate similarly nihilistic text, and by all objective internal measures it will be extremely "confident" as it does so. > Why is it compelled to provide one, anyway It's following the patterns in its training data, which probably reflects a whole lot more people trying to provide answers (sometimes even deliberately wrong ones) as opposed to admitting uncertainty. This is especially true if developers put their thumb on the scale by injecting primer-text like "You are an intelligent computer eager to provide answers", as opposed to "behave like Socrates and help people understand that nothing is truly knowable."
  - mewpmewp2 2y ago
    Also to some questions it also is already overly cautious about answering. E.g. I give an image of a location when I am travelling and ask it to guess where the image is taken. It will not want to guess and it will at first provide a long disclaimer that it can't do it, but if I tell it, that it is a game and just make a guess for the fun of it, it is surprisingly accurate.
lokimedes 2y ago
What we lack is for these models to state their context for their response. We have focused on the inherent lack of input context, leading to wrong conclusions, but what about that 90B+ parameters universe, plenty of room for multiple contexts to associate any input to surprising pathways. In the olden days of MLPs we had the same problem with softmax basically squeezing N output scores into a normalized “probability”, where each output neuron actually was the sum of multiple weighted paths, which one winning the softmax made up the “true” answer, but there may as well have been two equally likely outcomes, with just the internal “context” as difference. In physics we have the path integral interpretation and I dare say, we humans too, may provide outputs that are shaped by our inner context.
gmerc 2y ago
So the same as SelfCheckGPT from several months ago?
techostritch 2y ago
This method seems to lean into the idea of LLM as fancy search engine rather than true intelligence. Isn’t the eventual goal of LLMs or ai that it’s smarter than humans. So I guess my questions are: Is it plausible that LLM’s get so smart that we can’t understand them. Do we spend like years trying to validate scientific theories confabulated by AI? In the run up to super-intelligence, it seems like we’ll have to tweak the creativity knobs up, like the whole goal will be to find novel patterns humans don’t find, is there a way to tweak those knobs that get us super genius and not super conspiracy theorist? Is there even a difference? Part of this might depend on whether or not we think we can feed LLM’s “all” the information. But in fact, assuming that Silicon Valley CEO’s are some of the smartest people in the world, I might argue that confabulation of a possible future is in fact their primary value. Not being allowed to confabulate is incredibly limiting.
- seeknotfind 2y ago
  Yes, I agree, scientific theories have their value from being new/confabulated. Though this isn't mutually exclusive from solving today's problems of confabulations. The proposed methodology could be used to mark semantic coherence, but it doesn't mean we have to hide confabulations.
- pornel 2y ago
  LLMs are language models, and I think it's best not to try to extrapolate them to general intelligence. They are universal language translators, and a lossy database of a lot of text. They might be a component of some bigger AI system in the future, but themselves they are not as intelligent as their marketing implies.

56 points by glymor 2y ago

> LLMs aren't trained for accuracy This assertion in the article doesn't seem right at all. When LLMs weren't trained for accuracy, we had "random story generators" like GPT-2 or GPT-3. The whole breakthrough with RLHF was that we started training them for accuracy - or the appearance of it, as rated by human reviewers. This step both made the models a lot more useful and willing to stick to instructions, and also a lot better at... well, sounding authoritative when they shouldn't.

fsmv 2y ago
Isn't that the issue? Getting thumbs up from an underpaid human reviewer isn't the same as accurate facts.
- golergka 2y ago
  It's not the same as completely accurate facts, but it's much closer to accurate facts than LLMs we had before.
- gexla 2y ago
  The one person I know getting paid to review AI outputs gets paid anywhere from $25 / hour to $40 / hour. Not sure if that's underpaid. It may be a nice option when you can do it at any time to supplement your regular income.
  - drowsspa 2y ago
    Reviewing AI output or helping in training a LLM itself?
    - gexla 2y ago
      This person works through an interface which is similar to Mechanical Turk. You get a list of available projects you qualified for via an assessment. For the AI projects, many of them are comparing responses from two different models, answering questions, and selecting the best response. Other projects might be attempting to get the model to do something against the guidelines, or rating the model on certain capabilities. There's no requirements other than to pass the assessment. As with Mechanical Turk, you can work on your available projects at any time. This feedback is used for training.

dawatchusay 2y ago

Is confabulation different from hallucination? If not I do suppose this is a more accurate term for the phenomenon except that the exact definition isn’t common sense without looking it up whereas “hallucination” is more widely understood.

curtis3389 2y ago
When speaking about LLMs, confabulation and hallucination refer to the same thing. The term "confabulation" is just the most accurate description of what's happening, whereas the term "hallucination" refers to something LLMs are fundamentally incapable of.
CGamesPlay 2y ago
From the paper <https://www.nature.com/articles/s41586-024-07421-0 https://www.nature.com/articles/s41586-024-07421-0>: > Here we develop new methods grounded in statistics, proposing entropy-based uncertainty estimators for LLMs to detect a subset of hallucinations—confabulations—which are arbitrary and incorrect generations.
amenhotep 2y ago
Some people seem to get very angry about calling it "hallucination", because it's a computer, computers can't hallucinate! Stop anthropomorphising it!! So I suppose if you want to stay on the right side of those people - or you are one - you call it confabulation instead.
- williamcotton 2y ago
  There’s also the position that a definition of confabulate… To fill in gaps in one's memory with fabrications that one believes to be facts. …is much more accurate. Since we’re talking about a technical process it helps to be more precise in our use of language.
  - joquarky 2y ago
    I like to think that it's always confabulating. It's just that usually the words it generates are accurate enough for my needs.

glymor 2y ago

TL;DR sample the top N results from the LLM and use traditional NLP to extract factoids, if the LLM is confabulating the factoids would have random distribution, but if it's not it will be heavily weighted towards one answer. A figure from the paper shows this better than my TL;DR: https://www.nature.com/articles/s41586-024-07421-0/figures/1 https://www.nature.com/articles/s41586-024-07421-0/figures/1

visarga 2y ago
Or we could just ask the same question on 3 different LLMs, ideally a large LLM, a RAG LLM and a small one, then use LLM again to rewrite the final answer. When models contradict each other there is likely hallucination going on, but correct answers tend to converge.
- superb_dev 2y ago
  Why use an LLM to check the work of a different LLM? You could use the same technique that this paper describes to compare the answers each LLM gave. LLMs don’t have to be in opposition to traditional NLP techniques
Terr_ 2y ago
The LLM is already generating factoids: Things which resemble a fact without actually being one. (See also: Androids that resemble men but aren't, asteroids that resemble stars but aren't, meteoroids that resemble meteors but aren't...)
cl42 2y ago
Thank you! This is so helpful. It's also interesting to see what temperature value they use (1.0, 0.1 in some cases?)... I have a feeling using the actual raw probability estimates (if available) would provide a lot of information without having to rerun the LLM or sample quite as heavily.

zmmmmm 2y ago

> There are a number of reasons for this. The AI could have been trained on misinformation; the answer could require some extrapolation from facts that the LLM isn't capable of; or some aspect of the LLM's training might have incentivized a falsehood This article seems rather contrived. They present this totally broken idea of how LLMs work (that they are trained from the outset for accuracy on facts) and then proceed to present this research as it is a discovery that LLMs don't work like that.

ajuc 2y ago

Simplistic version of this is just asking the question in 2 ways - ask for confirmation that the answer is no, then ask for confirmation that the answer is yes :) If it's sure it won't confirm it both ways.

082349872349872 2y ago
A corollary for natural intelligence: if you can prove that all foos are bar, and that all foos are not bar, that's a good time to suspect that no foos actually exist. (I still don't understand why everyone seems happy to conflate "intelligence" with fact-retrieval?)
- persnickety 2y ago
  Because there is no single widely accepted definition of intelligence.
  - 082349872349872 2y ago
    No widely accepted definition, but fact retrieval is outside of any of the ones with which I'm familiar.
- ajuc 2y ago
  > I still don't understand why everyone seems happy to conflate "intelligence" with fact-retrieval? Because it's useful and impossible till very recently.
munchler 2y ago
That works for yes/no questions, but not for informational questions, like "Where's the Eiffel Tower?".

derefr 2y ago

> But perhaps the simplest explanation is that an LLM doesn't recognize what constitutes a correct answer but is compelled to provide one Why is it compelled to provide one, anyway? Which is to say, why is the output of each model layer a raw softmax — thus discarding knowledge of the confidence each layer of the model had in its output? Why not instead have the output of each layer be e.g. softmax but rescaled by min(max(pre-softmax vector), 1.0)? Such that layers that would output higher than 1.0 just get softmax'ed normally; but layers that would output all "low-confidence" results (a vector all lower than 1.0) preserve the low-confidence in the output — allowing later decoder layers to use that info to build I-refuse-to-answer-because-I-don't-know text?

Terr_ 2y ago
Careful, I think there's a large difference here between: 1. An LLM's mathematical "confidence" of having a clear best-scoring candidate for the predicted next token when given a list of tokens. 2. A not-yet-invented AI that models the idea of different entities interacting, the concept of questions and answers, the concept of logical conflicts, and it's "confidence" that a proposition is compatible with other "true" propositions and incompatible with false ones. To help illustrate the difference, suppose you trained an LLM on texts where a particular question was always answered with "I don't know, I have zero confidence in anything anymore." Later the LLM will regurgitate similarly nihilistic text, and by all objective internal measures it will be extremely "confident" as it does so. > Why is it compelled to provide one, anyway It's following the patterns in its training data, which probably reflects a whole lot more people trying to provide answers (sometimes even deliberately wrong ones) as opposed to admitting uncertainty. This is especially true if developers put their thumb on the scale by injecting primer-text like "You are an intelligent computer eager to provide answers", as opposed to "behave like Socrates and help people understand that nothing is truly knowable."
- mewpmewp2 2y ago
  Also to some questions it also is already overly cautious about answering. E.g. I give an image of a location when I am travelling and ask it to guess where the image is taken. It will not want to guess and it will at first provide a long disclaimer that it can't do it, but if I tell it, that it is a game and just make a guess for the fun of it, it is surprisingly accurate.

lokimedes 2y ago

What we lack is for these models to state their context for their response. We have focused on the inherent lack of input context, leading to wrong conclusions, but what about that 90B+ parameters universe, plenty of room for multiple contexts to associate any input to surprising pathways. In the olden days of MLPs we had the same problem with softmax basically squeezing N output scores into a normalized “probability”, where each output neuron actually was the sum of multiple weighted paths, which one winning the softmax made up the “true” answer, but there may as well have been two equally likely outcomes, with just the internal “context” as difference. In physics we have the path integral interpretation and I dare say, we humans too, may provide outputs that are shaped by our inner context.

gmerc 2y ago

So the same as SelfCheckGPT from several months ago?

techostritch 2y ago

This method seems to lean into the idea of LLM as fancy search engine rather than true intelligence. Isn’t the eventual goal of LLMs or ai that it’s smarter than humans. So I guess my questions are: Is it plausible that LLM’s get so smart that we can’t understand them. Do we spend like years trying to validate scientific theories confabulated by AI? In the run up to super-intelligence, it seems like we’ll have to tweak the creativity knobs up, like the whole goal will be to find novel patterns humans don’t find, is there a way to tweak those knobs that get us super genius and not super conspiracy theorist? Is there even a difference? Part of this might depend on whether or not we think we can feed LLM’s “all” the information. But in fact, assuming that Silicon Valley CEO’s are some of the smartest people in the world, I might argue that confabulation of a possible future is in fact their primary value. Not being allowed to confabulate is incredibly limiting.

seeknotfind 2y ago
Yes, I agree, scientific theories have their value from being new/confabulated. Though this isn't mutually exclusive from solving today's problems of confabulations. The proposed methodology could be used to mark semantic coherence, but it doesn't mean we have to hide confabulations.
pornel 2y ago
LLMs are language models, and I think it's best not to try to extrapolate them to general intelligence. They are universal language translators, and a lossy database of a lot of text. They might be a component of some bigger AI system in the future, but themselves they are not as intelligent as their marketing implies.