6 ms·

I realized if someone were to assign me the ticket for fixing this behavior, I would have no idea where to begin with solving it even with this blog post explai

by zten 1y ago

I realized if someone were to assign me the ticket for fixing this behavior, I would have no idea where to begin with solving it even with this blog post explaining the problem, so I'm very curious to know what the most practical solution is. (They obviously aren't adding "If someone asks you about a seahorse emoji, there isn't one available yet, no matter how strongly you believe one exists." to the system prompt.)

deleted 1y ago
[deleted]
maxbond 1y ago
Petition the Unicode consortium to include a seahorse emoji.
- beeflet 1y ago
  It takes a strong man to deny what's right in front of him, and if the truth is undeniable... you create your own
- Sharlin 1y ago
  If a LLM is wrong about facts, the facts have to change.
Gigachad 1y ago
I bet they probably are adding that to the system prompt at least in the short term while people are paying attention before looking for a longer term answer. The system prompts I've seen are absolutely massive.
- serced 1y ago
  I find interesting that their blog post on prompt/context engineering kind of stands against their ultra long system prompt. Maybe it is not too specific as in their visual example (too specific - just right - too vague). https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents https://www.anthropic.com/engineering/effective-context-engi... and the system prompt https://docs.claude.com/en/release-notes/system-prompts#september-29-2025 https://docs.claude.com/en/release-notes/system-prompts#sept...
  - catlifeonmars 1y ago
    > This attention scarcity stems from architectural constraints of LLMs. LLMs are based on the transformer architecture, which enables every token to attend to every other token across the entire context. This results in n² pairwise relationships for n tokens. The n² time complexity smells like it could be reduced by algorithm engineering. Maybe doing a preprocessing pass to filter out attending to tokens (not sure what the right term of art is here) that do not contribute significantly to the meaning of the input. Basically some sort of context compression mechanism.
username332211 1y ago
Couldn't you just add a large number of repetitions of "There is no seahorse emoji." to the training set? Edit: Come to think of it, training on a Q&A format is probably better - "Is there a seahorse emoji? No, there isn't."
- layman51 1y ago
  Something about how you have to keep repeating "There is no seahorse emoji" or something similar reminded me of the Local 58 horror web series where it seems like the program is trying to get you to repeat "There are no faces" while showing the viewer faces: https://www.youtube.com/watch?v=NZ-vBhGk9F4&t=221 https://www.youtube.com/watch?v=NZ-vBhGk9F4&t=221
- Jaxan 1y ago
  If you had to do this for every falsity in the LLM, there wouldn’t be an end to it.
  - username332211 1y ago
    I agree, but I still suspect OpenAI and other LLM companies do stuff like that, when an example of a hallucination becomes popular. If I see some example of an LLM saying dumb stuff here, I know it's going to be fixed quickly. If I encounter an example myself and refuse to share it, it may be fixed with a model upgrade in a few years. Or it may still exist.
classified 1y ago
> the most practical solution Maybe there is none, and this is just one example of a fundamental LLM limitation.
- TillE 1y ago
  People really really want LLMs to output a highly reliable finished product, and I suspect we're probably never gonna get there. Lots of progress over the past couple years, but not on that. I think it's much more interesting to focus on use cases which don't require that, where gen AI is an intermediate step, a creator of input (whether for humans or for other programs).
catlifeonmars 1y ago
I think you’re putting the cart before the horse. The article is a good starting point for a root cause analysis but is ultimately speculation. Before coming up with the solution, I think you’d need to understand the problem much more deeply.
jfyi 1y ago
"This behavior is a function of the core AI technology we use, we are unable to resolve this issue with a standard software patch or update at this time. For the time being this issue can be mitigated by not asking about seahorse emoji. We are closing this support ticket as the issue is an inherent limitation of the underlying technology and not a bug in our specific implementation."

by zten 1y ago

deleted 1y ago

[deleted]

maxbond 1y ago

Petition the Unicode consortium to include a seahorse emoji.

beeflet 1y ago
It takes a strong man to deny what's right in front of him, and if the truth is undeniable... you create your own
Sharlin 1y ago
If a LLM is wrong about facts, the facts have to change.

Gigachad 1y ago

I bet they probably are adding that to the system prompt at least in the short term while people are paying attention before looking for a longer term answer. The system prompts I've seen are absolutely massive.

serced 1y ago
I find interesting that their blog post on prompt/context engineering kind of stands against their ultra long system prompt. Maybe it is not too specific as in their visual example (too specific - just right - too vague). https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents https://www.anthropic.com/engineering/effective-context-engi... and the system prompt https://docs.claude.com/en/release-notes/system-prompts#september-29-2025 https://docs.claude.com/en/release-notes/system-prompts#sept...
- catlifeonmars 1y ago
  > This attention scarcity stems from architectural constraints of LLMs. LLMs are based on the transformer architecture, which enables every token to attend to every other token across the entire context. This results in n² pairwise relationships for n tokens. The n² time complexity smells like it could be reduced by algorithm engineering. Maybe doing a preprocessing pass to filter out attending to tokens (not sure what the right term of art is here) that do not contribute significantly to the meaning of the input. Basically some sort of context compression mechanism.

username332211 1y ago

Couldn't you just add a large number of repetitions of "There is no seahorse emoji." to the training set? Edit: Come to think of it, training on a Q&A format is probably better - "Is there a seahorse emoji? No, there isn't."

layman51 1y ago
Something about how you have to keep repeating "There is no seahorse emoji" or something similar reminded me of the Local 58 horror web series where it seems like the program is trying to get you to repeat "There are no faces" while showing the viewer faces: https://www.youtube.com/watch?v=NZ-vBhGk9F4&t=221 https://www.youtube.com/watch?v=NZ-vBhGk9F4&t=221
Jaxan 1y ago
If you had to do this for every falsity in the LLM, there wouldn’t be an end to it.
- username332211 1y ago
  I agree, but I still suspect OpenAI and other LLM companies do stuff like that, when an example of a hallucination becomes popular. If I see some example of an LLM saying dumb stuff here, I know it's going to be fixed quickly. If I encounter an example myself and refuse to share it, it may be fixed with a model upgrade in a few years. Or it may still exist.

classified 1y ago

> the most practical solution Maybe there is none, and this is just one example of a fundamental LLM limitation.

TillE 1y ago
People really really want LLMs to output a highly reliable finished product, and I suspect we're probably never gonna get there. Lots of progress over the past couple years, but not on that. I think it's much more interesting to focus on use cases which don't require that, where gen AI is an intermediate step, a creator of input (whether for humans or for other programs).

catlifeonmars 1y ago

I think you’re putting the cart before the horse. The article is a good starting point for a root cause analysis but is ultimately speculation. Before coming up with the solution, I think you’d need to understand the problem much more deeply.

jfyi 1y ago

"This behavior is a function of the core AI technology we use, we are unable to resolve this issue with a standard software patch or update at this time. For the time being this issue can be mitigated by not asking about seahorse emoji. We are closing this support ticket as the issue is an inherent limitation of the underlying technology and not a bug in our specific implementation."