5 ms·

Anecdotally this actually works. An LLM is trained on a wide distribution of data. "Just asking" it to not make mistakes makes it more likely to sample the par

Anecdotally this actually works. An LLM is trained on a wide distribution of data. "Just asking" it to not make mistakes makes it more likely to sample the part of the distribution that contains no mistakes.

marksomnian 3y ago
Doesn’t that predispose that it knows which parts of the distribution do and don’t have mistakes, and therefore that it knowingly makes mistakes unless you ask it not to? That doesn’t seem right to me and I’d be really surprised if this actually makes it stop hallucinating - seems more like something you’d put in the prompt without knowing why because it “seems to” produce better output (i.e. cargo cult prompt engineering).
- lmm 3y ago
  > Doesn’t that predispose that it knows which parts of the distribution do and don’t have mistakes, and therefore that it knowingly makes mistakes unless you ask it not to? Of course it does, to the extent it "knows" anything. It replies in a way that's average for the distribution. If you tell it this is an important task and not to make mistakes, it will give you a response that's more like that of someone who's been told this is an important task and not to make mistakes.
- avereveard 3y ago
  Well it does have concept of right and wrong, as evidenced here https://chat.openai.com/share/e117e800-f579-4eec-9192-e9912f17fbec https://chat.openai.com/share/e117e800-f579-4eec-9192-e9912f... you can ask it wrong answers only and it will oblige. Correctness is something it learns. I've read a few papers about hallucinations, and the jury is still out on whether a model knows when it's hallucinating, if we assume hallucinations are orthogonal to correctness Now this distinction isn't very useful in the grand scheme of things because in the end the output is wrong anyway, but it doesn't make asking to work along the axis of correctness cargo cult Further reads https://arxiv.org/abs/2304.13734 https://arxiv.org/abs/2304.13734 https://arxiv.org/abs/2305.18248 https://arxiv.org/abs/2305.18248

marksomnian 3y ago

Doesn’t that predispose that it knows which parts of the distribution do and don’t have mistakes, and therefore that it knowingly makes mistakes unless you ask it not to? That doesn’t seem right to me and I’d be really surprised if this actually makes it stop hallucinating - seems more like something you’d put in the prompt without knowing why because it “seems to” produce better output (i.e. cargo cult prompt engineering).

lmm 3y ago
> Doesn’t that predispose that it knows which parts of the distribution do and don’t have mistakes, and therefore that it knowingly makes mistakes unless you ask it not to? Of course it does, to the extent it "knows" anything. It replies in a way that's average for the distribution. If you tell it this is an important task and not to make mistakes, it will give you a response that's more like that of someone who's been told this is an important task and not to make mistakes.
avereveard 3y ago
Well it does have concept of right and wrong, as evidenced here https://chat.openai.com/share/e117e800-f579-4eec-9192-e9912f17fbec https://chat.openai.com/share/e117e800-f579-4eec-9192-e9912f... you can ask it wrong answers only and it will oblige. Correctness is something it learns. I've read a few papers about hallucinations, and the jury is still out on whether a model knows when it's hallucinating, if we assume hallucinations are orthogonal to correctness Now this distinction isn't very useful in the grand scheme of things because in the end the output is wrong anyway, but it doesn't make asking to work along the axis of correctness cargo cult Further reads https://arxiv.org/abs/2304.13734 https://arxiv.org/abs/2304.13734 https://arxiv.org/abs/2305.18248 https://arxiv.org/abs/2305.18248