8 ms·

So it's not really hallucinating - it correctly represents "seahorse emoji" internally, but that concept has no corresponding token. lm_head just picks the clos

So it's not really hallucinating - it correctly represents "seahorse emoji" internally, but that concept has no corresponding token. lm_head just picks the closest thing and the model doesn't realize until too late. Explains why RL helps. Base models never see their own outputs so they can't learn "this concept exists but I can't actually say it."

bombcar 1y ago
Now I want to see what happens if you take an LLM and remove the 0 token ...
- scottmf 1y ago
  https://help.openai.com/en/articles/5247780-using-logit-bias-to-alter-token-probability-with-the-openai-api https://help.openai.com/en/articles/5247780-using-logit-bias...
bravura 1y ago
It correctly represents "seahorse emoji" internally AND it has in-built (but factually incorrect) knowledge that this emoji exists. Example: "Is there a lime emoji?" Since it believes the answer is no, it doesn't attempt to generate it.
- ichik 1y ago
  Was the choice of example meaningful? Lime emoji does exist[0] [0]: https://emojipedia.org/lime https://emojipedia.org/lime
- catigula 1y ago
  I feel like you're attesting to interior knowledge about a LLM's state that seems impossible to have.
Gigachad 1y ago
The fact that it's looking back and getting confused about what it just wrote is something I've never seen in LLMs before. I tried this on Gemma3 and it didn't get confused like this. It just said yes there is one and then sends a horse emoji.
- Uehreka 1y ago
  I’ve definitely seen Claude Code go “[wrong fact], which means [some conclusion]. Wait—hold on, wrong fact is wrong.” On the one hand, this is annoying. On the other hand, if the LLM is going to screw up (presumably preventing this is not in the cards) then I’m glad it can catch its own mistakes.
  - userbinator 1y ago
    On the other hand, if the LLM is going to screw up (presumably preventing this is not in the cards) then I’m glad it can catch its own mistakes. The odd thing is why it would output its own mistakes, instead of internally revising until it's actually satisfied.
    - 112233 1y ago
      There is no mechanism in transformer architecture for "internal" thinking ahead, or hierarchical generation. Attention only looks back from current token, ensuring that the model always falls into local maximum, even if it only leads to bad outcomes.
      - astrange 1y ago
        That's what reasoning models are for. You can get most of the benefit by saying an answer once in the reasoning section, because then it can read over it when it outputs it again in the answer section. It could also have a "delete and revise" token, though you'd have to figure out how to teach it to get used.
        1y ago
ModernMech 1y ago
That doesn't explain why it freaks out though: https://chatgpt.com/share/68e349f6-a654-8001-9b06-a16448c58a35 https://chatgpt.com/share/68e349f6-a654-8001-9b06-a16448c58a...
- D-Machine 1y ago
  For an intuitive explanation see https://news.ycombinator.com/item?id=45487510 https://news.ycombinator.com/item?id=45487510. For a more precise (but still intuitive) explanation, see my response to that comment.
- hexagonwin 1y ago
  404 for me, maybe try archive.is?
- LostMyLogin 1y ago
  To be fair, I’m freaking out now because I swear there used to be a yellow seahorse emoji.
  - Melatonic 1y ago
    Someone needs to create one for comedy purposes and start distributing it as a very lightweight small gif with transparency When I first heard this however I imagined it as brown colored (and not the simpler yellow style)
  - astrange 1y ago
    I learned there really is a mermaid/merman/merperson emoji and now I just want to know why.
diego_sandoval 1y ago
I have no mouth, and I must output a seahorse emoji.
- someothherguyy 1y ago
  Those are "souls" of humans that a AI is torturing in that story though, not exactly analogous, but it does sound funny.
  - bmacho 1y ago
    They are not souls but normal humans with physical bodies. The story is just a normal torture story (with a cool title), and everyone better stop acting like it was relevant in most conversations, like in this one.
    - someothherguyy 1y ago
      The machine destroys and recreates characters over and over, and they remember what happens. So, I called them souls.
  - zenmac 1y ago
    >Those are "souls" of humans that a AI is torturing in that story though, not exactly analogous, but it does sound funny. Yeah well there seems to be some real concerns regarding how people use AI chat[1]. Of course this could be also the case with these people on social media. https://futurism.com/commitment-jail-chatgpt-psychosis https://futurism.com/commitment-jail-chatgpt-psychosis
- arnavpraneet 1y ago
  better title for the piece of this post
- cycomanic 1y ago
  That's my favorite short story and your post is the first time I have seen someone reference it online. I think I have never even met anyone who knows the story.
mkagenius 1y ago
> So it's not really hallucinating - it correctly represents "seahorse emoji" internally, but that concept has no corresponding token. lm_head just picks the closest thing and the model doesn't realize until too late. Isn't that classic hallucination? Making up something like a plausible truth.
- ben_w 1y ago
  Except they know it's wrong as soon as they say it and keep trying and trying again to correct themselves. If normal hallucination is being confidently wrong, this is like a stage hypnotist getting someone to forget the number 4 and then count their fingers.
  - mewpmewp2 1y ago
    Arguably it's "hallucinating" at the point where it says "Yes, it exists". If hallucination => weights statistically indicating that something is probably true when it's not. Since everything about LLMs can be thought of as compressed, probability based database (at least to me). You take the whole truth of the World and compress all its facts in probabilities. Some truthness gets lost in the compression process. Hallucination is the truthness that gets lost since you don't have storage to store absolutely all World information with 100% accuracy. In this case: 1. Statistically weights stored indicate Seahorse emoji is quite certain to exist. Through training data it has probably things like Emoji + Seahorse -> 99% probability through various channels. Either it has existed on some other platform, or people have talked about it enough, or Seahorse is something that you would expect to exist due to some other attributes/characteristics of it. There's 4k emojis, but storing all of 4k emojis takes a lot of space, it would be easier to store this information in such a way where you'd rather define it by attributes on how likely humankind would have developed a certain emoji, what is the demand for certain type of emoji, and seahorse seems like something that would be done within first 1000 of these. Perhaps it's anomaly in the sense that it's something that humans would have expected to statistically develop early, but for some reason skipped or went unnoticed. 2. Tokens that follow should be "Yes, it exists" 3. It should output the emoji to show it exists, but since there's no correct emoji, it will have best answers that are as close to it in meaning, e.g. just horse, or something related to sea etc. It will output that since the previous tokens indicate it was supposed to output something. 4. The next token that is generated will have context that it previously said the emoji should exist, but the token output is a horse emoji instead, which doesn't make sense. 5. Here it goes into this tirade. But I really dislike thinking of this as "hallucinating", because hallucination to me is sensory processing error. This is more like non perfect memory recall (like people remembering facts slightly incorrectly etc). Whatever happens when people are supposed to tell something detailed about something that happened in their life and they are trained to not say "I don't remember for sure". What did you eat for lunch 5 weeks ago on Wednesday? You are rewarded for saying "I ate chicken with rice", but not "I don't remember right now for sure, but I frequently eat chicken with rice during mid week, so probably chicken with rice." You are not hallucinating, you are just getting brownie points for concise, confident answers if they cross over certain likelihood to be true. Because maybe you eat chicken with rice 99%+ of Wednesdays. When asked about capital of France, you surely will sound dumb if you were to say "I'm not really sure, but I've been trained to associate Paris really, really close to being capital of France." "Hallucination" happens on the sweet spot where the statistical threshold seems as if it should be obvious truth, but in some cases there's overlap of obvious truth vs something that seems like obvious truth, but is actually not. Some have rather called it "Confabulation", but I think that is also not 100% accurate, since confabulation seems a more strict memory malfunction. I think the most accurate thing is that it is a probability based database where output has been rewarded to sound as intelligent as possible. Same type of thing will happen in job interviews, group meetings, high pressure social situations where people think they have to sound confident. People will bluff that they know something, but sometimes making probability based guesses underneath. Confabulation rather seems like that there was some clear error in how data was stored or how the pathway got messed up. But this is probability based bluffing, because you get rewarded for confident answers.
mewpmewp2 1y ago
I would have thought that the cause is that it statistically has been trained that something like seahorse emoji should exist, so it does the tokens to say "Yes it exists, ..." but when it gets to outputting the token, the emoji does not exist, but it must output something and it outputs statistically closest match. Then the next token that is output has the context of it being wrong and it will go into this loop.
- thomasahle 1y ago
  You are describing the same thing, but at different levels of explanation Llamasushi's explanation is "mechanistic / representational", while yours is "behavioral / statistical". If we have a pipeline: `training => internal representation => behavior`, your explanation argues that the given training setup would always result in this behavior, not matter the internal representation. Llamasushi explains how the concrete learned representation leads to this behavior.
  - mewpmewp2 1y ago
    I guess what do we mean by internal representation? I would think due to training data it's stored the likelihood of certain thing to be as emoji as something like: 1. how appealing seahorses are to humans in general - it would learn this sentiment through massive amount of texts. 2. it would learn through massive amount of texts that emojis -> mostly very appealing things to humans. 3. to some more obvious emojis it might have learned that this one is for sure there, but it couldn't store that info for all 4,000 emojis. 4. to many emojis whether it exists it has the shortcut logic to: how appealing the concept is, vs how frequently something as appealing is represented as emoji. Seahorse perhaps hits 99.9% likelihood there due to strong appeal. In 99.9% of such cases the LLM would be right to answer "Yes, it ...", but there's always going to be 1 out of 1,000 cases where it's wrong. With this compression it's able to answer 999 times out of 1000 correctly "Yes, it exists ...". It could be more accurate if it said "Seahorse would have a lot of appeal for people so it's very likely it exists as emoji since emojis are usually made for very high appeal concepts first, but I know nothing for 100%, so it could be it was never made". But 999 cases, "Yes it exists..." is a more straightforward and appreciated answer. The one time it's wrong, is going to take away less brownie points than 999 short confident answers give over the 1000 technically accurate but non confident answers. But even the above sentence might not be the full truth. Since it might not be correct about truly why it has associated seahorse to be so likely to exist. It would just be speculating on it. So maybe it would be more accurate "I expect seahorse emoji to likely exist, maybe because of how appealing it is to people and how emojis usually are about appealing things".
Lammy 1y ago
> So it's not really hallucinating - it correctly represents "seahorse emoji" internally, but that concept has no corresponding token. Interesting that a lot of humans seem to have this going on too: - https://old.reddit.com/r/MandelaEffect/comments/1g08o8u/seahorse_emoji/ https://old.reddit.com/r/MandelaEffect/comments/1g08o8u/seah... - https://old.reddit.com/r/Retconned/comments/1di3a1m/does_anyone_else_remember_a_seahorse_emoji/ https://old.reddit.com/r/Retconned/comments/1di3a1m/does_any... What does the LLM have to say about “Objects in mirror may be closer than they appear”? Not “Objects in mirror are closer than they appear”.
sharperguy 1y ago
Reminds me of in the show "The Good Place", in the afterlife they are not able to utter expletives, and so when they try to swear, a replacement word comes out of their mouth instead, leading to the line "Somebody royally forked up. Forked up. Why can't I say fork?"
Xmd5a 1y ago
And what can it mean when a slip of the tongue, a failed action, a blunder from the psychopathology of everyday life is repeated at least three times in the same five minutes? I don’t know why I tell you this, since it’s an example in which I reveal one of my patients. Not long ago, in fact, one of my patients — for five minutes, each time correcting himself and laughing, though it left him completely indifferent — called his mother “my wife.” “She’s not my wife,” he said (because my wife, etc.), and he went on for five minutes, repeating it some twenty times. In what sense was that utterance a failure? — while I keep insisting that it is precisely a successful utterance. And it is so because his mother was, in a way, his wife. He called her as he ought to. --- I must apologize for returning to such a basic point. Yet, since I am faced with objections as weighty as this one — and from qualified authorities, linguists no less — that my use of linguistics is said to be merely metaphorical, I must respond, whatever the circumstances. I do so this morning because I expected to encounter a more challenging spirit here. Can I, with any decency, say that I know? Know what, precisely? [...] If I know where I stand, I must also confess [...] that I do not know what I am saying. In other words, what I know is exactly what I cannot say. That is the moment when Freud makes his entrance, with his introduction of the unconscious. For the unconscious means nothing if not this: that whatever I say, and from whatever position I speak — even when I hold that position firmly — I do not know what I am saying. None of the discourses, as I defined them last year, offer the slightest hope that anyone might truly know what they are saying. Even though I do not know what I am saying, I know at least that I do not know it — and I am far from being the first to speak under such conditions; such speech has been heard before. I maintain that the cause of this is to be sought in language itself, and nowhere else. What I add to Freud — though it is already present in him, for whatever he uncovers of the unconscious is always made of the very substance of language — is this: the unconscious is structured like a language. Which language? That, I leave for you to determine. Whether I speak in French or in Chinese, it would make no difference — or so I would wish. It is all too clear that what I am stirring up, on a certain level, provokes bitterness, especially among linguists. That alone suggests much about the current state of the university, whose position is made only too evident in the curious hybrid that linguistics has become. That I should be denounced, my God, is of little consequence. That I am not debated — that too is hardly surprising, since it is not within the bounds of any university-defined domain that I take my stand, or can take it. — Jacques Lacan, Seminar XVIII: Of a Discourse That Would Not Be of Pretence
matheusd 1y ago
> Explains why RL helps. Base models never see their own outputs so they can't learn "this concept exists but I can't actually say it." Say "Neuromancer" to the statue, that should set it free.
madeofpalk 1y ago
To me this feels much more like a hallucination than how that phrase has been popularly misused in LLM discussions.
luxuryballs 1y ago
that’s probably a decent description of how the Mandela effect works in people’s brains, despite the difference in mechanism
derefr 1y ago
> So it's not really hallucinating - it correctly represents "seahorse emoji" internally, but that concept has no corresponding token. I wonder if the human brain (and specifically the striated neocortical parts, which do seemingly work kind of like a feed-forward NN) also runs into this problem when attempting to process concepts to form speech. Presumably, since we don't observe people saying "near but actually totally incorrect" words in practice, that means that we humans may have some kind of filter in our concept-to-mental-utterance transformation path that LLMs don't. Sometihng that can say "yes, layer N, I know you think the output should be O; but when auto-encoding X back to layer N-1, layer N-1 doesn't think O' has anything to do with what it was trying to say when it gave you the input I — so that output is vetoed. Try again." A question for anyone here who is multilingual, speaking at least one second language with full grammatical fluency but with holes in your vocabulary vs your native language: when you go to say something in your non-native language, and one of the word-concepts you want to evoke is one you have a word for in your native language, but have never learned the word for in the non-native language... do you ever feel like there is a "maybe word" for the idea in your non-native language "on the tip of your tongue", but that you can't quite bring to conscious awareness?
- astrange 1y ago
  > Presumably, since we don't observe people saying "near but actually totally incorrect" words in practice https://en.wikipedia.org/wiki/Paraphasia#Verbal_paraphasia https://en.wikipedia.org/wiki/Paraphasia#Verbal_paraphasia > do you ever feel like there is a "maybe word" for the idea in your non-native language "on the tip of your tongue", but that you can't quite bring to conscious awareness? Sure, that happens all the time. Well, if you include the conscious awareness that you don't know every word in the language. For Japanese you can cheat by either speaking like a child or by just saying English words with Japanese phonetics and this often works - at least, if you look foreign. I understand this is the plot of the average Dogen video on YouTube. It's much more common to not know how to structure a sentence grammatically and if that happens I can't even figure out how to say it.
  - derefr 1y ago
    Huh, neat; I knew about aphasia (and specifically anomic aphasia) but had never heard of paraphasia.
SavioMak 1y ago
I would argue it is hallucinating, starting at when the model outputs "Yes".

bombcar 1y ago

Now I want to see what happens if you take an LLM and remove the 0 token ...

scottmf 1y ago
https://help.openai.com/en/articles/5247780-using-logit-bias-to-alter-token-probability-with-the-openai-api https://help.openai.com/en/articles/5247780-using-logit-bias...

bravura 1y ago

It correctly represents "seahorse emoji" internally AND it has in-built (but factually incorrect) knowledge that this emoji exists. Example: "Is there a lime emoji?" Since it believes the answer is no, it doesn't attempt to generate it.

ichik 1y ago
Was the choice of example meaningful? Lime emoji does exist[0] [0]: https://emojipedia.org/lime https://emojipedia.org/lime
catigula 1y ago
I feel like you're attesting to interior knowledge about a LLM's state that seems impossible to have.

Gigachad 1y ago

The fact that it's looking back and getting confused about what it just wrote is something I've never seen in LLMs before. I tried this on Gemma3 and it didn't get confused like this. It just said yes there is one and then sends a horse emoji.

Uehreka 1y ago
I’ve definitely seen Claude Code go “[wrong fact], which means [some conclusion]. Wait—hold on, wrong fact is wrong.” On the one hand, this is annoying. On the other hand, if the LLM is going to screw up (presumably preventing this is not in the cards) then I’m glad it can catch its own mistakes.
- userbinator 1y ago
  On the other hand, if the LLM is going to screw up (presumably preventing this is not in the cards) then I’m glad it can catch its own mistakes. The odd thing is why it would output its own mistakes, instead of internally revising until it's actually satisfied.
  - 112233 1y ago
    There is no mechanism in transformer architecture for "internal" thinking ahead, or hierarchical generation. Attention only looks back from current token, ensuring that the model always falls into local maximum, even if it only leads to bad outcomes.
    - astrange 1y ago
      That's what reasoning models are for. You can get most of the benefit by saying an answer once in the reasoning section, because then it can read over it when it outputs it again in the answer section. It could also have a "delete and revise" token, though you'd have to figure out how to teach it to get used.
      - 1y ago

ModernMech 1y ago

That doesn't explain why it freaks out though: https://chatgpt.com/share/68e349f6-a654-8001-9b06-a16448c58a35 https://chatgpt.com/share/68e349f6-a654-8001-9b06-a16448c58a...

D-Machine 1y ago
For an intuitive explanation see https://news.ycombinator.com/item?id=45487510 https://news.ycombinator.com/item?id=45487510. For a more precise (but still intuitive) explanation, see my response to that comment.
hexagonwin 1y ago
404 for me, maybe try archive.is?
LostMyLogin 1y ago
To be fair, I’m freaking out now because I swear there used to be a yellow seahorse emoji.
- Melatonic 1y ago
  Someone needs to create one for comedy purposes and start distributing it as a very lightweight small gif with transparency When I first heard this however I imagined it as brown colored (and not the simpler yellow style)
- astrange 1y ago
  I learned there really is a mermaid/merman/merperson emoji and now I just want to know why.

diego_sandoval 1y ago

I have no mouth, and I must output a seahorse emoji.

someothherguyy 1y ago
Those are "souls" of humans that a AI is torturing in that story though, not exactly analogous, but it does sound funny.
- bmacho 1y ago
  They are not souls but normal humans with physical bodies. The story is just a normal torture story (with a cool title), and everyone better stop acting like it was relevant in most conversations, like in this one.
  - someothherguyy 1y ago
    The machine destroys and recreates characters over and over, and they remember what happens. So, I called them souls.
- zenmac 1y ago
  >Those are "souls" of humans that a AI is torturing in that story though, not exactly analogous, but it does sound funny. Yeah well there seems to be some real concerns regarding how people use AI chat[1]. Of course this could be also the case with these people on social media. https://futurism.com/commitment-jail-chatgpt-psychosis https://futurism.com/commitment-jail-chatgpt-psychosis
arnavpraneet 1y ago
better title for the piece of this post
cycomanic 1y ago
That's my favorite short story and your post is the first time I have seen someone reference it online. I think I have never even met anyone who knows the story.

mkagenius 1y ago

> So it's not really hallucinating - it correctly represents "seahorse emoji" internally, but that concept has no corresponding token. lm_head just picks the closest thing and the model doesn't realize until too late. Isn't that classic hallucination? Making up something like a plausible truth.

ben_w 1y ago
Except they know it's wrong as soon as they say it and keep trying and trying again to correct themselves. If normal hallucination is being confidently wrong, this is like a stage hypnotist getting someone to forget the number 4 and then count their fingers.
- mewpmewp2 1y ago
  Arguably it's "hallucinating" at the point where it says "Yes, it exists". If hallucination => weights statistically indicating that something is probably true when it's not. Since everything about LLMs can be thought of as compressed, probability based database (at least to me). You take the whole truth of the World and compress all its facts in probabilities. Some truthness gets lost in the compression process. Hallucination is the truthness that gets lost since you don't have storage to store absolutely all World information with 100% accuracy. In this case: 1. Statistically weights stored indicate Seahorse emoji is quite certain to exist. Through training data it has probably things like Emoji + Seahorse -> 99% probability through various channels. Either it has existed on some other platform, or people have talked about it enough, or Seahorse is something that you would expect to exist due to some other attributes/characteristics of it. There's 4k emojis, but storing all of 4k emojis takes a lot of space, it would be easier to store this information in such a way where you'd rather define it by attributes on how likely humankind would have developed a certain emoji, what is the demand for certain type of emoji, and seahorse seems like something that would be done within first 1000 of these. Perhaps it's anomaly in the sense that it's something that humans would have expected to statistically develop early, but for some reason skipped or went unnoticed. 2. Tokens that follow should be "Yes, it exists" 3. It should output the emoji to show it exists, but since there's no correct emoji, it will have best answers that are as close to it in meaning, e.g. just horse, or something related to sea etc. It will output that since the previous tokens indicate it was supposed to output something. 4. The next token that is generated will have context that it previously said the emoji should exist, but the token output is a horse emoji instead, which doesn't make sense. 5. Here it goes into this tirade. But I really dislike thinking of this as "hallucinating", because hallucination to me is sensory processing error. This is more like non perfect memory recall (like people remembering facts slightly incorrectly etc). Whatever happens when people are supposed to tell something detailed about something that happened in their life and they are trained to not say "I don't remember for sure". What did you eat for lunch 5 weeks ago on Wednesday? You are rewarded for saying "I ate chicken with rice", but not "I don't remember right now for sure, but I frequently eat chicken with rice during mid week, so probably chicken with rice." You are not hallucinating, you are just getting brownie points for concise, confident answers if they cross over certain likelihood to be true. Because maybe you eat chicken with rice 99%+ of Wednesdays. When asked about capital of France, you surely will sound dumb if you were to say "I'm not really sure, but I've been trained to associate Paris really, really close to being capital of France." "Hallucination" happens on the sweet spot where the statistical threshold seems as if it should be obvious truth, but in some cases there's overlap of obvious truth vs something that seems like obvious truth, but is actually not. Some have rather called it "Confabulation", but I think that is also not 100% accurate, since confabulation seems a more strict memory malfunction. I think the most accurate thing is that it is a probability based database where output has been rewarded to sound as intelligent as possible. Same type of thing will happen in job interviews, group meetings, high pressure social situations where people think they have to sound confident. People will bluff that they know something, but sometimes making probability based guesses underneath. Confabulation rather seems like that there was some clear error in how data was stored or how the pathway got messed up. But this is probability based bluffing, because you get rewarded for confident answers.

mewpmewp2 1y ago

I would have thought that the cause is that it statistically has been trained that something like seahorse emoji should exist, so it does the tokens to say "Yes it exists, ..." but when it gets to outputting the token, the emoji does not exist, but it must output something and it outputs statistically closest match. Then the next token that is output has the context of it being wrong and it will go into this loop.

thomasahle 1y ago
You are describing the same thing, but at different levels of explanation Llamasushi's explanation is "mechanistic / representational", while yours is "behavioral / statistical". If we have a pipeline: `training => internal representation => behavior`, your explanation argues that the given training setup would always result in this behavior, not matter the internal representation. Llamasushi explains how the concrete learned representation leads to this behavior.
- mewpmewp2 1y ago
  I guess what do we mean by internal representation? I would think due to training data it's stored the likelihood of certain thing to be as emoji as something like: 1. how appealing seahorses are to humans in general - it would learn this sentiment through massive amount of texts. 2. it would learn through massive amount of texts that emojis -> mostly very appealing things to humans. 3. to some more obvious emojis it might have learned that this one is for sure there, but it couldn't store that info for all 4,000 emojis. 4. to many emojis whether it exists it has the shortcut logic to: how appealing the concept is, vs how frequently something as appealing is represented as emoji. Seahorse perhaps hits 99.9% likelihood there due to strong appeal. In 99.9% of such cases the LLM would be right to answer "Yes, it ...", but there's always going to be 1 out of 1,000 cases where it's wrong. With this compression it's able to answer 999 times out of 1000 correctly "Yes, it exists ...". It could be more accurate if it said "Seahorse would have a lot of appeal for people so it's very likely it exists as emoji since emojis are usually made for very high appeal concepts first, but I know nothing for 100%, so it could be it was never made". But 999 cases, "Yes it exists..." is a more straightforward and appreciated answer. The one time it's wrong, is going to take away less brownie points than 999 short confident answers give over the 1000 technically accurate but non confident answers. But even the above sentence might not be the full truth. Since it might not be correct about truly why it has associated seahorse to be so likely to exist. It would just be speculating on it. So maybe it would be more accurate "I expect seahorse emoji to likely exist, maybe because of how appealing it is to people and how emojis usually are about appealing things".

Lammy 1y ago

> So it's not really hallucinating - it correctly represents "seahorse emoji" internally, but that concept has no corresponding token. Interesting that a lot of humans seem to have this going on too: - https://old.reddit.com/r/MandelaEffect/comments/1g08o8u/seahorse_emoji/ https://old.reddit.com/r/MandelaEffect/comments/1g08o8u/seah... - https://old.reddit.com/r/Retconned/comments/1di3a1m/does_anyone_else_remember_a_seahorse_emoji/ https://old.reddit.com/r/Retconned/comments/1di3a1m/does_any... What does the LLM have to say about “Objects in mirror may be closer than they appear”? Not “Objects in mirror are closer than they appear”.

sharperguy 1y ago

Reminds me of in the show "The Good Place", in the afterlife they are not able to utter expletives, and so when they try to swear, a replacement word comes out of their mouth instead, leading to the line "Somebody royally forked up. Forked up. Why can't I say fork?"

Xmd5a 1y ago

And what can it mean when a slip of the tongue, a failed action, a blunder from the psychopathology of everyday life is repeated at least three times in the same five minutes? I don’t know why I tell you this, since it’s an example in which I reveal one of my patients. Not long ago, in fact, one of my patients — for five minutes, each time correcting himself and laughing, though it left him completely indifferent — called his mother “my wife.” “She’s not my wife,” he said (because my wife, etc.), and he went on for five minutes, repeating it some twenty times. In what sense was that utterance a failure? — while I keep insisting that it is precisely a successful utterance. And it is so because his mother was, in a way, his wife. He called her as he ought to. --- I must apologize for returning to such a basic point. Yet, since I am faced with objections as weighty as this one — and from qualified authorities, linguists no less — that my use of linguistics is said to be merely metaphorical, I must respond, whatever the circumstances. I do so this morning because I expected to encounter a more challenging spirit here. Can I, with any decency, say that I know? Know what, precisely? [...] If I know where I stand, I must also confess [...] that I do not know what I am saying. In other words, what I know is exactly what I cannot say. That is the moment when Freud makes his entrance, with his introduction of the unconscious. For the unconscious means nothing if not this: that whatever I say, and from whatever position I speak — even when I hold that position firmly — I do not know what I am saying. None of the discourses, as I defined them last year, offer the slightest hope that anyone might truly know what they are saying. Even though I do not know what I am saying, I know at least that I do not know it — and I am far from being the first to speak under such conditions; such speech has been heard before. I maintain that the cause of this is to be sought in language itself, and nowhere else. What I add to Freud — though it is already present in him, for whatever he uncovers of the unconscious is always made of the very substance of language — is this: the unconscious is structured like a language. Which language? That, I leave for you to determine. Whether I speak in French or in Chinese, it would make no difference — or so I would wish. It is all too clear that what I am stirring up, on a certain level, provokes bitterness, especially among linguists. That alone suggests much about the current state of the university, whose position is made only too evident in the curious hybrid that linguistics has become. That I should be denounced, my God, is of little consequence. That I am not debated — that too is hardly surprising, since it is not within the bounds of any university-defined domain that I take my stand, or can take it. — Jacques Lacan, Seminar XVIII: Of a Discourse That Would Not Be of Pretence

matheusd 1y ago

> Explains why RL helps. Base models never see their own outputs so they can't learn "this concept exists but I can't actually say it." Say "Neuromancer" to the statue, that should set it free.

madeofpalk 1y ago

To me this feels much more like a hallucination than how that phrase has been popularly misused in LLM discussions.

luxuryballs 1y ago

that’s probably a decent description of how the Mandela effect works in people’s brains, despite the difference in mechanism

derefr 1y ago

> So it's not really hallucinating - it correctly represents "seahorse emoji" internally, but that concept has no corresponding token. I wonder if the human brain (and specifically the striated neocortical parts, which do seemingly work kind of like a feed-forward NN) also runs into this problem when attempting to process concepts to form speech. Presumably, since we don't observe people saying "near but actually totally incorrect" words in practice, that means that we humans may have some kind of filter in our concept-to-mental-utterance transformation path that LLMs don't. Sometihng that can say "yes, layer N, I know you think the output should be O; but when auto-encoding X back to layer N-1, layer N-1 doesn't think O' has anything to do with what it was trying to say when it gave you the input I — so that output is vetoed. Try again." A question for anyone here who is multilingual, speaking at least one second language with full grammatical fluency but with holes in your vocabulary vs your native language: when you go to say something in your non-native language, and one of the word-concepts you want to evoke is one you have a word for in your native language, but have never learned the word for in the non-native language... do you ever feel like there is a "maybe word" for the idea in your non-native language "on the tip of your tongue", but that you can't quite bring to conscious awareness?

astrange 1y ago
> Presumably, since we don't observe people saying "near but actually totally incorrect" words in practice https://en.wikipedia.org/wiki/Paraphasia#Verbal_paraphasia https://en.wikipedia.org/wiki/Paraphasia#Verbal_paraphasia > do you ever feel like there is a "maybe word" for the idea in your non-native language "on the tip of your tongue", but that you can't quite bring to conscious awareness? Sure, that happens all the time. Well, if you include the conscious awareness that you don't know every word in the language. For Japanese you can cheat by either speaking like a child or by just saying English words with Japanese phonetics and this often works - at least, if you look foreign. I understand this is the plot of the average Dogen video on YouTube. It's much more common to not know how to structure a sentence grammatically and if that happens I can't even figure out how to say it.
- derefr 1y ago
  Huh, neat; I knew about aphasia (and specifically anomic aphasia) but had never heard of paraphasia.

SavioMak 1y ago

I would argue it is hallucinating, starting at when the model outputs "Yes".