6 ms·

> A probable guess will lower loss much better than "I don't know" or whatever equivalent. Guessing only reduces loss as much as the dataset allows -- a bad gu

> A probable guess will lower loss much better than "I don't know" or whatever equivalent. Guessing only reduces loss as much as the dataset allows -- a bad guess will give a higher loss. The model learns to assign probabilities to its guesses, just like we do. It seems to me all we need here is a measure of confidence for the result averaged over the entire answer. Low confidence is a guess/hallucination. > But also the dataset encourages it as well. There will be many many sentences that can't be completed accurate to source even with all the knowledge and understanding in the world. Many completions will have numerous sensible options. The dataset doesn't discriminate. Fiction, Fact, Opinion, Mistake. All the same. All given equal weight. This is an important issue but should be tackled as a distinctly different problem I think: it's the weighty concept of truth that humanity struggled with from day 1. Indeed, how do we discriminate? LLMs won't ever solve this via completions or dataset alone; instead successful models will use slow, step-by-step reasoning involving logical principles and rational heuristics in prompt space. Pretty much like we do.

bcherny 3y ago
The reason humans tend to tell the truth is if we don’t, other humans will call us out for it. I wonder if there’s a way to mimic this “bs penalty” for GPT. Maybe you could have a setup where GPT gives an answer, then a second GPT has to guess whether a human would know if that answer is true or not.
- dr_dshiv 3y ago
  This I think is the approach—a dialectic of LLMs that can critique and synthesize. Although, this will surely be a solved problem once we have TruthGPT, right? ;)
hakuseki 3y ago
> It seems to me all we need here is a measure of confidence for the result averaged over the entire answer. Low confidence is a guess/hallucination. Even if the model knows the exact answer to the question, there may be many distinct ways of phrasing the answer. This would also lead to low confidence in any particular phrasing.
- tempestn 3y ago
  Ah, interesting, that does begin to explain how this might be more difficult than it initially appears. Could there some way to define.. proximity of different possible responses, and sum the confidence for all the nearby possibilities?
- tshadley 3y ago
  That should be okay though, 10 good answers will still report the score of the best one chosen. I think the GPTs are using beam search which is projecting out a "beam" (looks more like a tree to me) of probable answers each of which has a score of accumulated token probabilities, and then just picking the highest. https://towardsdatascience.com/foundations-of-nlp-explained-visually-beam-search-how-it-works-1586b9849a24 https://towardsdatascience.com/foundations-of-nlp-explained-... In this case, it doesn't matter how wide the beam is or how many possible answers there are, the score is still the accumulated token possibilities of the best branch. However, others have noted in the thread that RLHF might hurt this approach severely by scoring polite responses high regardless of false answers (for example). Then you have to access the model pre-RLHF to get any idea of its true likelihood.
tempestn 3y ago
My knowledge in this area is very limited, but based on the high level descriptions I've seen of how LLMs work (including the OP), it seems like it would be fairly trivial to output, along with each response, a "confidence factor" of some sort for that response. While that might cause confusion for some users, it could be incredibly valuable to differentiate between confident responses and guesses, as you say.
- sgt101 3y ago
  The problem is that the models are already evaluating confidence on their answers and picking the best one... And that confidence is based on token generation....
  - tempestn 3y ago
    I don't think this is the problem. The confidence of the best answer won't always be the same. Sometimes there would be one answer that's significantly better than others, whereas other times there could be a lot of mediocre answers it's picking between. So having it spit out the confidence along with the answer could theoretically be useful. What would be a challenge is what others noted in reply, that sometimes there would be multiple good answers, so low confidence wouldn't necessarily be a sign of a poor answer. (Though I expect work could be done there.)
  - hnfong 3y ago
    AFAICT the tokens are probably the issue. Imagine the question "In which year was Donald Trump born?" The LLM would start the answer by either: "Donald Trump was born in ..." Or "I'm sorry I don't know" And for the vast majority of answers the first option looks more "probable", so it starts producing tokens with an affirmative answer, and if the model eventually sees a bunch of low probability answers when it tries to produce the year, it's already "too late" to backtrack in a naive GPT implementation. You could train LLM such that it responds with "I'm sorry I don't know" more often, but how do you predicate the response on "do this only if your 500B parameters don't encode the answer"? It requires self-referential logic on the model which isn't obvious to me how it would be done. Maybe some smart people have figured this out, but I can see how this makes it really hard to do.

bcherny 3y ago

The reason humans tend to tell the truth is if we don’t, other humans will call us out for it. I wonder if there’s a way to mimic this “bs penalty” for GPT. Maybe you could have a setup where GPT gives an answer, then a second GPT has to guess whether a human would know if that answer is true or not.

dr_dshiv 3y ago
This I think is the approach—a dialectic of LLMs that can critique and synthesize. Although, this will surely be a solved problem once we have TruthGPT, right? ;)

hakuseki 3y ago

> It seems to me all we need here is a measure of confidence for the result averaged over the entire answer. Low confidence is a guess/hallucination. Even if the model knows the exact answer to the question, there may be many distinct ways of phrasing the answer. This would also lead to low confidence in any particular phrasing.

tempestn 3y ago
Ah, interesting, that does begin to explain how this might be more difficult than it initially appears. Could there some way to define.. proximity of different possible responses, and sum the confidence for all the nearby possibilities?
tshadley 3y ago
That should be okay though, 10 good answers will still report the score of the best one chosen. I think the GPTs are using beam search which is projecting out a "beam" (looks more like a tree to me) of probable answers each of which has a score of accumulated token probabilities, and then just picking the highest. https://towardsdatascience.com/foundations-of-nlp-explained-visually-beam-search-how-it-works-1586b9849a24 https://towardsdatascience.com/foundations-of-nlp-explained-... In this case, it doesn't matter how wide the beam is or how many possible answers there are, the score is still the accumulated token possibilities of the best branch. However, others have noted in the thread that RLHF might hurt this approach severely by scoring polite responses high regardless of false answers (for example). Then you have to access the model pre-RLHF to get any idea of its true likelihood.

tempestn 3y ago

My knowledge in this area is very limited, but based on the high level descriptions I've seen of how LLMs work (including the OP), it seems like it would be fairly trivial to output, along with each response, a "confidence factor" of some sort for that response. While that might cause confusion for some users, it could be incredibly valuable to differentiate between confident responses and guesses, as you say.

sgt101 3y ago
The problem is that the models are already evaluating confidence on their answers and picking the best one... And that confidence is based on token generation....
- tempestn 3y ago
  I don't think this is the problem. The confidence of the best answer won't always be the same. Sometimes there would be one answer that's significantly better than others, whereas other times there could be a lot of mediocre answers it's picking between. So having it spit out the confidence along with the answer could theoretically be useful. What would be a challenge is what others noted in reply, that sometimes there would be multiple good answers, so low confidence wouldn't necessarily be a sign of a poor answer. (Though I expect work could be done there.)
- hnfong 3y ago
  AFAICT the tokens are probably the issue. Imagine the question "In which year was Donald Trump born?" The LLM would start the answer by either: "Donald Trump was born in ..." Or "I'm sorry I don't know" And for the vast majority of answers the first option looks more "probable", so it starts producing tokens with an affirmative answer, and if the model eventually sees a bunch of low probability answers when it tries to produce the year, it's already "too late" to backtrack in a naive GPT implementation. You could train LLM such that it responds with "I'm sorry I don't know" more often, but how do you predicate the response on "do this only if your 500B parameters don't encode the answer"? It requires self-referential logic on the model which isn't obvious to me how it would be done. Maybe some smart people have figured this out, but I can see how this makes it really hard to do.