11 ms·

GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

• Anoian 3mo ago
  I don't understand how this did not reach frontpage. Great post!
• solid_fuel 3mo ago
  > It’s been proven that when a model is trained on large volumes of highly factual and non-theoretical data, it learns to always have an answer. DeepSeek V4 Pro (1.6T params, 49B active, 44 AA Intelligence Index score) has a ludicrous 94% hallucination score on the AA-Omniscience benchmark, meaning on questions that it couldn’t figure out, it only stated that it didn’t know around 6% of the time, and the rest it confidently hallucinated an answer. GLM-5.2 scored a 28% hallucination rate, Opus 4.8 was 36%, Fable 5 was 48%, and GPT-5.5 was 86%. Wow! I already knew from previous research shared here that hallucinations are a fundamental problem for LLMs and likely to be unfixable, just like prompt injection, but I didn't realize the hallucination rates were so bad! Everyone has been acting like the best models only hallucinate in edge cases, but even the best performing one mentioned here - GLM-5.2 - has a hallucination rate of 28% when it doesn't "know" the answer to something. That said, I think the title on the blog - "Bigger models are not the way" is probably more fitting and touches on what should be even bigger news. If bigger models and bigger training sets have already stopped producing proportional returns, then it seems likely we are already near the top of the S-curve. That's huge news, considering the valuation of companies like OpenAI and xAI is largely based around the (absurd) idea of ever increasing scaling from these models.
  • oshrimpton 3mo ago
    Agreed on the title, my bad! But yeah, I've had some truly terrible experiences using these "frontier" models in coding agents especially, where they just fabricate facts about codebases.
  • SeriousM 3mo ago
    There is no concept of "knowledge" in LLM as it is on Wikipedia. The question-tokens define the answer-tokens. That's it. The art relies in clustering the relevant weights together.
    • tempaccount420 3mo ago
      > There is no concept of "knowledge" in LLM as it is on Wikipedia. There can be, you don't know if the closed source models aren't using something like DeepSeek's Engram.
      • HarHarVeryFunny 3mo ago
        The name "Engram" (n-gram) says it all - this is just another type of statistical word association, not a factual knowledge store. While DeepSeek describe this as "knowledge lookup", what Engram is really trying to do is separate dynamic reasoning from static pattern recall, with the static patterns just being word-level n-gram statistics, not declarative facts/knowledge. Just because 2-3 words often appear together in a sequence doesn't mean they represent a fact or truth (or falsehood) - it is just an n-gram statistical regularity. If Engram helps reduce LLM GPU memory and FLOP requirements then that is great, but it's not a solution for Hallucination.
    • baq 3mo ago
      If it were that simple we’d all be talking with sql and yet this isn’t happening. Circuits which emerge in the layers during training are much more complicated than a simple Bayesian relation.
    • solid_fuel 3mo ago
      Correct, LLMs are not ontologically capable of “knowing”. That is why I put “know” in quotes.
• aesthesia 3mo ago
  Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. I'd also hesitate to attribute this difference in hallucination rates purely to model size. Yes, GLM-5.2 hallucinates much less frequently than DeepSeek-V4 Pro with twice as many parameters, but DeepSeek-V4 Flash is less than half the size of GLM-5.2 and tops the AA-Omniscience hallucination index. Opus 4.8, which is likely larger than DeepSeek-V4 Pro, has a 36% hallucination rate on the index, above GLM-5.2's 28%, but way below the DeepSeek numbers. Opus also has a 47% accuracy rate vs GLM-5.2's 25%. If you use these numbers to calculate the absolute hallucination rate (i.e., the number of hallucinated responses divided by the total number of responses), you get 19% for Opus and 21% for GLM-5.2. So yes, all else equal larger models may be more prone to hallucination in scenarios where they don't know the answer, but there are a lot of other factors that affect hallucination rates, and it's not totally clear that this is the main metric that's worth tracking.
  • in-silico 3mo ago
    Additionally, maybe it's easier for a model to realize that it doesn't know the answer when the question is easier. If Opus gets all but the hardest questions right, it might have a higher hallucination rate because the questions it gets wrong are the questions where verification or hallucination detection are the most difficult
  • grayhatter 3mo ago
    > Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. Do you have a cite for this? If a human makes up some bullshit lie, I wouldn't accuse them of making it up only if they actually knew the correct answer. If you don't know, the only correct answer is I don't know. Any other answer is made up bullshit. Why is it only a hallucination if and only if the LLM contains the answer? If you make something up it's still wrong. It shouldn't matter if you could give the correct answer. You didn't, and instead invented some bullshit instead? Follow up question, how can I apply this rule set to the next test I have to take? I'd love to be able to use "I didn't know" as the excuse for why I made something up. edit: > and it's not totally clear that this is the main metric that's worth tracking. I don't know, the rate at which some model is willing to make up something feels useful. If the argument I see repeated on HN so much is that it's impossible to completely get rid of hallucinations; being able to choose a model that's less likely to invent some lie seems like a positive trait, no? Either way, I'm happy to agree that a restrictive definition, where a lie doesn't count as a hallucination iff the model doesn't know the answer feels strictly, infinitely less useful than an exact error rate. What percentage of emitted tokens are misleading would be useful for me. Anyone know any group that's attempted to quantify the global error rate?
• Ozzie-D 3mo ago
  [flagged]
• cwillu 3mo ago
  Please don't editorialize titles unless the original title is misleading.
• nextaccountic 3mo ago
  >GPT-5.5 and DeepSeek V4 Pro are two of the clearest hallucination leaders, despite being absolutely huge. Because of their immense size they simply did not learn how to say “I don’t know” or recognize intricate logical and technical fallacies. While it is true that a multi-trillion parameter model will always beat a lightweight consumer model on paper (today at least), the commoditization of these huge models is blurring the line between benchmark performance and actual real-world truthfulness and accuracy. What about using two models, with a smaller model used for this kind of negative reasoning?
  • bastawhiz 3mo ago
    Now you need a third model to decide if the two other models disagree
    • deleted 3mo ago
      [deleted]
• frankohn 3mo ago
  I think hallucination rates are not a matter of model size but depends on the training of the model. They have been trained on a huge corpus of material that had overwhelmingly well formed questions and we'll formulated and correct answers. This is typically the case of books where the material is highly curated by experts in the field. In a book you never see a question which admit no answer and the book just reasoning and explaining why and how the question has no answer. Neither you will see a good question and the book explaining candidly it doesn't know the answer , because the way the book material is curated the author will omit discussing the question for which it has no answers. In addition, I think that during HFRL, the labs has a bias for interesting answers that admit a solution and under represent the "bad" questions that admit no good answer. In addition they probably do less effort to HFRL on questions the model should admit it doesn't know. As humans we have been trained all our lives, in the real world, to be confronted with questions we don't know the response right away and we learned to very quickly assess that we don't know or that we are not sure about the answer. Another thing we have and LLM have not is fear. We have an amygdala in our brain, separated from the logic thinking part, that can raise a signal of fear so that we get much more carefully about what we say. On the other LLM has no fear organ like the amygdala and just learn to respond based on the patterns in it's training corpus. It never "fears" looking bad or being fired because it gave a wrong answer so it can merrily give perfectly wrong answers. So, we see hallucination rates can be improved with training but currently the lab are not optimizing for that because there is an high stake race to get the most intelligent and capable model. Alternatively I can see creating a separate amygdala-like organ for an LLM and that organ may asynchronously fires signal, based on the user prompt and the LLM thinking trace, to inject into the LLM reasoning a fear signal so that it can steer it's answer to something more safe.
  • oshrimpton 3mo ago
    I'd definitely agree that it isn't directly model size, but there is the fact that a larger model in terms of parameter count needs a large amount of training data to not overfit or underfit. So I think this race to the top of "max training data size" has kind of led to unintentional overfitting, not catastrophically, but enough to trigger this perceived omniscience within the model
  • 3mo ago
• taffydavid 3mo ago
  > For the non technical, this is like asking a delivery driver to drop off packages at 3 houses at the same time without ever stopping the truck. I'm already hallucinating about how this could work and it involves catapults
  • m3h 3mo ago
    Or we could simply hallucinate that the packages are there at the three houses. Hallucinations all the way down...
    • sigmoid10 3mo ago
      In the end it's just Boltzmann brains. https://en.wikipedia.org/wiki/Boltzmann_brain https://en.wikipedia.org/wiki/Boltzmann_brain
    • boofus 3mo ago
      Nobody said the 3 houses needed to be on separate properties. Just throw the 3 packages from the moving truck at the one address where all 3 live. Being an LLM is easy!
  • Lionga 3mo ago
    [flagged]
• EbNar 3mo ago
  The fact that a huge amount uf parameters may lead to worse hallucinations is something I didn't think of. Would this somewhat imply that DeepSeek V4 flash should be less prone tho these issues?
  • oshrimpton 3mo ago
    Surprisingly not! It is the biggest hallucinator on the AA Omniscience Index just 2pp away from V4 Pro. I think this is partially due to the fact that Flash was trained on >32T tokens just like Pro deapite being almost 10x smaller - it seems somewhat likely it was overfit.
  • fuck_google 3mo ago
    [dead]
• spwa4 3mo ago
  Why is everyone expecting LLMs to be like the Star Trek computer? I wonder if anyone's ever measured what the hallucination rate of a human is.
  • oshrimpton 3mo ago
    I would be so curious to find a comprehensive benchmark on this, humans do have an unfortunate ahem Dunning-Kruger effect ahem tendency to do this
  • bravetraveler 3mo ago
    Marketing, essentially
  • master-lincoln 3mo ago
    Yeah it has been looked at e.g. in [0]. They separate that from lying, but I think for the LLM context it should be included. To me the difference is humans do not bullshit at the same rate and I can find out over time who tends to bullshit more and exclude that persons info from my pool. > Why is everyone expecting LLMs to be like the Star Trek computer? Because they are often marketed as magic AIs, not as mere language models. [0] https://bpspsychub.onlinelibrary.wiley.com/doi/10.1111/bjso.12379 https://bpspsychub.onlinelibrary.wiley.com/doi/10.1111/bjso....
  • flexagoon 3mo ago
    Because AI company executives and devoted vibecoders constantly make egregious claims like "programming is fully solved" and even straight up "hallucinations don't exist on frontier models"
    • verdverm 3mo ago
      We don't have to listen to these people and can form our own perspectives. Following bad leaders is something to avoid
      • flexagoon 3mo ago
• anchorapi 3mo ago
  [dead]
• abracadobre 3mo ago
  This is where I asked GPT 5.5 "they say u hallucinate 3x more than GLM 5.2, whats your comeback to this? do i need to dump u? $article"
• xlii 3mo ago
  My anecdotal experience differs (though I hold ground that LLM evaluations are highly subjective and benchmarks are just as useful for LLMs as they are for dating websites users). GLM 5.2 tends to stray way more than and 5.1. It also hallucinates you things subtly: morphs requirements, makes unfounded conclusions. This output is not something I experienced in any model I seen so far. In coding it's especially annoying because it steers whole request. E.g. I give instruction: "make we a Rust-WASM-Canvas app" and GLM 5.2 goes like "Oh user surely doesn't mean that. I'll better build Dioxus app instead".
  • oshrimpton 3mo ago
    Yeah the benchmark for sure isn't perfect and without super rigid prompting it is far too easy for it to get off course. 28% hallucination rate isn't nothing either
  • LaurensBER 3mo ago
    GLM 5.2 is great but it heavily detoriates once the context window gets past 200k tokens. I've had more success with creating a plan first and then implementing it in (short-lived) sub-agents. Ironically good software architecture patterns (small functions, single responsibility) heavily impact the performance of these models as well. They do surprisingly well in well architectured codebases. They do very poorly in anything that's a mess where Opus and GPT 5.5 still get reasonable performance.
• cws_ai_buddy 3mo ago
  [flagged]
• raincole 3mo ago
  > meaning on questions that it couldn’t figure out, it only stated that it didn’t know around 6% of the time, and the rest it confidently hallucinated an answer. From how they measure it, a model that simply answers "I don't know." to any prompt would be the one hallucinates the least. So it's not surprising at all that a smaller model can perform better.
• jingpostmedia 3mo ago
  [flagged]
• wiether 3mo ago
  Purely anecdotal, but when OpenAI removed Codex-5.3 from the ChatGPT sub and forced me to move to GPT-5.5, the result was far worse than what I was enjoying with Codex. And, of course, it was burning 10 times more tokens for this output.
  • oshrimpton 3mo ago
    Yeah they are 100% in the wrong for removing the fine tuned codex models. It makes sense why they wouldn't want to allocate so many resources towards fine tuning but still the enshittification of GPT models is real
    • fuck_google 3mo ago
      [dead]
    • embedding-shape 3mo ago
      Huh, the fine-tuned "codex" variants always seemed like "quick specific edit" prototypes that weren't meant for real use. They worked OK when you were very specific, but besides that, nowhere close to GPT5.X and the other "real" models.
      • wiether 3mo ago
        Since Codex-5.3 came out it was my daily driver for everything: quick scripting, greenfield projects, new features on old projects... Idk if it was the harness (OpenCode), my AGENT or my prompts, but I was getting exactly what I wanted, and quickly. With GPT-5.5 it tries to play smart, takes much more times and is often stuck on basic stuff that DeepSeek solves oneshot.
        • embedding-shape 3mo ago
          > With GPT-5.5 it tries to play smart, takes much more times and is often stuck on basic stuff that DeepSeek solves oneshot. You have any session logs or similar that shows this thing? Never once, since I started using the codex TUI when it became available, has GPT models gotten stuck on something another model breeze through, I quite literally run every prompt I do through multiple providers, this would be very visible very quickly for me. I remember trying every -codex variant of the models and could never get them to be productive for tasks taking longer than 5-10 minutes, compared to GPT 5.5 which quite literally worked through the night day (with the /goal feature), and actually had something valuable and useful in the end this morning that wasn't exploding in LOC and complexity. I don't think any of the -codex variants would have been able to do this at all, based on how they worked when I last used them.
• stalfie 3mo ago
  One thing I wonder about hallucinations, is that it seems on the surface that it is an easy problem for RLVR to target. Since you're already generating enormous amounts of reasoning traces which are verified by correct answers, just have "don't know" as an option as a valid answer, and on problems where none of the thousands of reasoning traces led to a correct answer, just promote the traces that led to the "don't know" answer as training data. Essentially teaching the model that "I don't know" is a valid answer. Sam Altman himself had a blog post about this a while ago that seemed to suggest this thought, so I guess it's obvious to everyone. But if that is so I assume it's just not as easy in practice.
  • cyanydeez 3mo ago
    the problem is the null answer will stop the "markov" chain. so, thats all.
    • BDPW 3mo ago
      You dont have to literally send a null token. Train it to generate text that summarizes the evidence that is there but the uncertainty of the final answer to a prompt.
    • make3 3mo ago
      Transformers are not Markovian, their whole point is arguably to be the reverse of Markovian, to efficiently make it so the new tokens are a function of all previous tokens
  • amelius 3mo ago
    But if an LLM says "I don't know" should you pay for the tokens?
    • guerrilla 3mo ago
      Why not? It did the work. Why should you expect it to be omniscient? We can rank them based on how much they know and people will gravitate towards those that do know more. It's a market after all.
• Naveja 3mo ago
  loving glm 5.2 personally
• aubanel 3mo ago
  > Bigger is not better The article uses the example of GLM being smaller than DeepSeek, yet better on hallucinations as "smaller can be good too" But the GLM family itself is scaling up fast: GLM-5.x family is 754B, double the previous generation of GLM-4.x > comes within just 4 points of GPT-5.5 and 9 points of Fable 5 9 percentage points IS a big difference
  • CuriouslyC 3mo ago
    If we're hand waiving how an open source model from a Chinese lab that you can use a nearly unlimited amount for <100/mo's 9% difference from the premier, unavailable, expensive when it was available American frontier model, we've already lost.
• metalspot 3mo ago
  hallucination is good for tasks that have an external oracle like computer programming
  • dgellow 3mo ago
    Could you explain what you mean? That feels like a waste of processing to me. Yes the model will correct itself once it eventually run a compiler/linter. But that's still wasted time and compute
  • sometimelurker 3mo ago
    ehh ur right but there's a lot of nuance here. if you have a system that doesn't hallucinate a ton and is still very "creative" that's great, and probably much better than a hallucinating system regardless of its creativity. I'm reminded of theoremproving LLMs working in lean producing millions of slop proofs until one works, but if you have something like that simple RLVR should fix it (external oracle can be the judge for the RL.
• flexagoon 3mo ago
  [dead]
• EbNar 3mo ago
  The fact that a huge amount uf parameters may lead to worse hallucinations is something I didn't think of. Would this somewhat imply that DeepSeek V4 flash should be less prone to these issues?
  • verdverm 3mo ago
    small models cannot encode so many facts, they will hallucinate more out-of-box a key method to help with hallucinations is to provide good sources when asking questions (context engineering / knowledge base)
• deleted 3mo ago
  [deleted]
  • embedding-shape 3mo ago
    > This is a terrible line of thought They're basing this all on public benchmarks which stopped being a reliable indicator of anything the last 2-3 years. Of course it'll be filled with more terrible lines of thoughts. People really need to stop placing such importance on public benchmarks. They're valuable for comparing very close models, useful to evaluate if quantization and similar have negative impact, but you're not gonna be able to tell if one model is better than the other based on one scoring a few percentage points higher than a completely different model.
• anArbitraryOne 3mo ago
  It's fine if it hallucinates, as long as it sounds overconfident
• remix2000 3mo ago
  Calling llm slop "hallucinating" is so counter-productive imo. After all, LLMs are just a variant of markov chains and as such this technology isn't able to discern falsehoods from truths. It's like trying to use a barometer to tell the time.
  • hit8run 3mo ago
    You are also just a variant of markov chains wired in your brain. So what you complaining about?
    • __natty__ 3mo ago
      And often it’s not perfect either. Just because one is true it doesn’t dismiss the other
    • remix2000 3mo ago
      Well the difference here is that you're overly simplifying complex biology and many other factors whereas llms are in fact actually simple mathematical models. As always, the devil lies in the details. Dismissing intricacies is a useful tool for daydreamers, not so much for engineers.
      • sometimelurker 3mo ago
        LLMs actually aren't simple Markov chains tho, your also simplifying. and LLMs trained with RLVR aren't just optimized over the space of functions (like gpt2 was), they're optimized over the space of programs (programs under some length). You find the ideal algorithm that can do the task you need it to.
        • qwery2 3mo ago
          RLVR is a process which updates the Markov chain
        • deleted 3mo ago
          [deleted]
• wolttam 3mo ago
  > it is clear that actual intelligence has plateaued significantly. > Moving forward, the industry cannot continue to train bigger and bigger models since their intelligence not only plateaus but often will get worse These are wild claims - why are we concluding that bigger models and more data = more hallucination? That’s actually the opposite of what’s been happening over the last couple years. Some models may still hallucinate more but they all hallucinate much less than the original 175B ChatGPT which was smaller and trained on (much) less data than anything current. Edit: My mention of data comes from this quote: > A shift is happening among major AI labs, who are becoming increasingly skeptical of endless parameter count and training data scaling My take on the current situation: it seems clear that the industry has seen that there is still a lot left to squeeze out of sub-1T models. But for that you do need more, high-quality data in the distribution which you want to unlock capabilities for.
  • madduci 3mo ago
    Isn't that the case of over fitting? You have more data, but when you ask something that's not in that data, hallucinations happen
  • an0malous 3mo ago
    > why are we concluding that bigger models and more data = more hallucination? That’s not what your quotes said. They said bigger models = plateau in intelligence, nothing about more data or increased hallucinations The relevant quote for what you’re talking about would be: > It’s been proven that when a model is trained on large volumes of highly factual and non-theoretical data, it learns to always have an answer. So there’s two separate claims: 1) bigger models have plateauing results 2) models trained on larger amounts of factual data have a higher hallucination rate I’m pretty sure #1 is well known, I think OpenAI’s own research on scaling laws showed diminishing returns on parameter count and training data volume years ago. I don’t know what the support for #2 is besides for the actual post contents.
    • jmalicki 3mo ago
      I find these internet arguments talking about LLMs as if they are trained by reading the internet to be wild. Yes, pretraining still exists. But for the past few years, pretraining by reading the internet is just the initial bootstrapping of LLM training. The RL training they get from bespoke training data, with very very different characteristics than what these armchair analyses claim, dominates these days.
• andai 3mo ago
  > GPT-5.5 and DeepSeek V4 Pro are two of the clearest hallucination leaders, despite being absolutely huge. Because of their immense size they simply did not learn how to say “I don’t know” or recognize intricate logical and technical fallacies. This implies that bigger models are more likely to hallucinate? That doesn't match my experience.
  • npilk 3mo ago
    I think it implies they are more likely to hallucinate if they don't know the answer. So a big model will return the correct answer more often than a small one, but in the cases where it doesn't, it will be more likely to make something up instead of saying "I don't know".
• brown_munda 3mo ago
  GLM 5.2 is really impressive at design as well. Overall loving it.
• zuzuen_1 3mo ago
  I think we need better classification and taxonomy on erroneous LLM behaviors than the catch-all term "hallucinate"..
• stevenhubertron 3mo ago
  The more I have been using 5.2 the more I have been impressed with it. And I’ve just been using the usually neutered ollama version.
• chinallm_ai 3mo ago
  [dead]
• czk 3mo ago
  if you're benchmaxxing then maybe bigger doesnt always mean better, but for general intelligence and big model smell, that couldn't be further from the truth the oss models are impressive but it's pretty clear how quickly they fall off when you try to use them outside of a narrow set of problems they benchmarked well on when compared to opus/5.5
• dgellow 3mo ago
  > One of the biggest models in the world was banned because a single jailbreak was too much of a risk. We really don't know what the actual reason is given the politics at play. I would bet more on the Trump administration looking for any excuse to punish Anthropic
• deleted 3mo ago
  [deleted]
• giancarlostoro 3mo ago
  I wonder if this is what a “Minimally Viable LLM” looks like. I often wonder how much of an LLM do you need before you can just shove a bigger context Window and any dynamic knowledge content to it like a PDF or markdown file to give it knowledge outside of its training data. I feel like LLMs don’t need more data they just need to be refined.
  • x3cca 3mo ago
    You might be interested in this model. It's a densely trained on math whuch let's it punch way higher than it should https://github.com/WeiboAI/VibeThinker https://github.com/WeiboAI/VibeThinker
    • giancarlostoro 3mo ago
      Cant open the link without an account is it private or is that just GitHub being annoying?
• orbital-decay 3mo ago
  DS v4 is an undertrained snapshot, which is mentioned in their model card. The full version is supposed to be released later and have multimodal input. That said, hallucination rate likely depends on the training policy and different optimization tradeoffs a lot more than on the scale.
• corlinp 3mo ago
  [flagged]
• hyperpape 3mo ago
  > A shift is happening among major AI labs, who are becoming increasingly skeptical of endless parameter count and training data scaling. The limits of this paradigm were put on the world’s stage when Claude Fable 5 was restricted by the US government just three days after its release, marking the first US AI ban stemming from national security. One of the biggest models in the world was banned because a single jailbreak was too much of a risk. Such a weird thing to start with. The legal status of Fable does not mean that it's not intelligent. If anything, the problem is the opposite, someone thinks it's too intelligent (and/or that Anthropic wouldn't share its last gen intelligent models on the terms the government demanded).
• hereme888 3mo ago
  Artificial Analysis says GPT-5.5 xhigh scores highest on AA-Omniscience accuracy. The article focuses on rate instead of overall accuracy. Those are different things: a model can answer more questions correctly overall while still being worse at abstaining when wrong. Curiously, this post and article is the only submission and interaction the OP has made, and these claims support the product he's intending to release.
• gcanyon 3mo ago
  > it is clear that actual intelligence has plateaued significantly N=1, but I disagree strongly. I'm writing a hard-science science fiction story, and the physics of it is at (and frankly, beyond) my skillset. The story's plot has had to change over a dozen times as I realized errors in my application of physics in the story. Throughout, I've been reviewing the physics with LLMs, mainly Gemini 3.1 Pro Preview, but also with Claude and OpenAI. Often I have the LLMs debate each other -- "My friend [another model] said XYZ about the physics, is that right or wrong?" In almost all cases, Gemini explains why the other models are wrong, and when I send its explanation to them, they concede it is right and they are wrong. As I said, I did the above checks literally dozens of times as I wrote the story. And everything was dialed in: no further issues claimed by anyone, me or the LLMs. Not with Fable. I managed to get it to review the story while it was running, and it listed out something like ten issues: some minor, some general knowledge-based, and two that were impressive: 1. It pointed out where Gemini (and I, and other LLMs) had missed a , resulting in values about 152 times larger than they should have been. I sent that to Gemini and it fully conceded that it had been wrong all along. 2. It pointed out a simple inconsistency in the application of special relativity (I thought I had that at least dialed in, but no :-/ ) that affected a very specific plot point. The story is novella-length, about 28,000 words long, and this is a point that was mentioned in the first two pages, and then not again until the very last page. And it's obvious, once you realize it. And I missed it. Gemini missed it. Claude and ChatGPT missed it. Only Fable found it. Again, N=1, but that was a remarkable run I got out of it in the couple days it was available.
  • Bolwin 3mo ago
    Hah, I noticed the same thing writing fiction with fable. Most models seem to go into a sort of "storytelling mode" where they forget their PhD level smarts. I had a character who is doing repair on a satellite. Most models would give you a half-baked explanation with some technical terms - half of them right half of them wrong. Fable gave a description so deep that even I couldn't figure out what was going on and had to ask it to give me a simpler explanation.
    • gcanyon 3mo ago
• nathan_compton 3mo ago
  Synthesizing a bunch of stuff I've read here lately, it seems like if OpenAI and Claude have actually found product market fit (generating code) then the question of hallucination is going to get less attention in the future. If the real money is in code generation (where there is a relatively clear acceptance criteria of at least "it runs and does what I wanted as far as I can tell") then there doesn't seem to be a lot of juice in pulling ones hair out on hallucination of facts. It seems like for agentic coding, just making sure the AI can find the relevant documentation to establish a ground truth is probably sufficient. Note that I'm distinguishing here between hallucination of what you might call "free facts" and hallucination of material which deviates from what is in the context itself. The latter seems both a tractable problem and one which will improve coding agent functionality. But the former seems like its no longer on the critical path, probably because its hard.
• ecommerceguy 3mo ago
  It's very much looking like OpenAI will be bailed out, along with all the other Capex'ers. I say this because the trump admin (I feel partially at fault because I voted for him) has indicated they will be bailing out the entire ai stargate from intel and amd to amazon and anthropic. I know alot of everyday folks that absolutely hate - passionately HATE - anything and everything tech bro. Downvote all you want, that's the reality. They see Palantir et al as evil and demonic.
• chazeon 3mo ago
  GPT-5.5 must have serious issues; it is fast, but quality-wise, it is just not good. It read one LaTeX paper (which is not long) and can spell my name wrong. This is GPT-5.5-high.
• balgaly 3mo ago
  [flagged]
• gitaarik 3mo ago
  This reminds me of the Missing Dollar Riddle [1], where the listener is deliberately put on a wrong thinking path, to fool it. With your own logical thinking you might never come to this confusion, and if you never heard this riddle before, you might be tricked by it. But as we grow in life, and get experience, we learn about these riddles and aren't fooled as easily anymore. Maybe it'll work like that for LLMs too? [1]: https://en.wikipedia.org/wiki/Missing_dollar_riddle https://en.wikipedia.org/wiki/Missing_dollar_riddle
• Tanxsinxlnx 3mo ago
  [flagged]
• nghnam 3mo ago
  I’d be careful about reading too much into these numbers. The test only looks at cases where the model doesn’t know the answer, so it doesn’t show how often users will actually see hallucinations.
• metalman 3mo ago
  to paraphrase the title, "in the land of the insane, those who are meerly delusional will rule"
• stcg 3mo ago
  In the referenced benchmark GLM-5.2 (max) got 25% of all questions correct. GPT-5.5 (xhigh) got 57% correct. https://artificialanalysis.ai/evaluations/omniscience https://artificialanalysis.ai/evaluations/omniscience I'd much rather have some answer that I can verify than no answer to verify. I don't want a model that says "I don't know", because I will verify the answer anyway.
  • bwfan123 3mo ago
    > I don't want a model that says "I don't know", because I will verify the answer anyway. Few people actually review answers or code. Because they have been sold the myth that these models can do it all. The main problem is that LLMs dont have causal models, and as a result, their reasoning is a high probability word salad and not a logically sound argument. Particularly on tricky corner cases which it hasnt encountered. I would still agree with you that sometimes hallucinations are actually useful as it provides a strawman, and having even a hallucinated answer to spar with is better than a "dont know".
• firemelt 3mo ago
  my exp with gpt is the model tend to mention file that not even exist