6 ms·

Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seem

by nsingh2 2mo ago

Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and return the wrong result. When it ends up using 6000-8000 thinking tokens it returns the correct result. Maybe some issue with adaptive thinking? Another point for local models I guess, don't have to worry about silent server side changes. Edit: To follow up, it seems to happen quite often. Out of 10 runs of the exact same prompt, 4/10 had this 516 thinking token issue, and every one of these had the wrong solution. So nearly half the time, 5.5 xhigh could be short circuiting and degrading performance. Granted the sample size is small.

deleted 2mo ago
[deleted]
dannyw 2mo ago
I wonder if testing during different time/days show patterns? For example, whether the short circuiting happens more often during workday peak hours.
postalcoder 2mo ago
You still have to worry about misconfigured local models. Even the professionals get it wrong, which is why local model performance is uneven across providers.
- deleted 2mo ago
  [deleted]
- jdiff 2mo ago
  But in that case you have nobody but yourself to blame, and you can stabilize things yourself at any time by refraining from making any changes. You won't be surprised by a provider. Honestly? That's not just valuable—it's essential.
  - LiamPowell 2mo ago
    > Honestly? That's not just valuable—it's essential. I'm curious if you wrote this or had a LLM write it. I'm genuinely curious to be clear as I don't see why anyone would bother to go through a LLM to write such a short reply. Have we reached the point where Claudeisms that are this obnoxious have become part of regular speech?
    - UqWBcuFx6NV4r 2mo ago
      Or they were making a joke[0]. [0]: https://en.wikipedia.org/wiki/Joke https://en.wikipedia.org/wiki/Joke (…just like that)
      - LiamPowell 2mo ago
        Sure, but it doesn't really fit there as a joke, it looks like it's just meant to be part of what they were trying to say.
        subscribed 2mo ago
        I also think it's a joke, it starts with the response / argument, and then flows into tongue-in-cheek joke about the core issue of the post (LLM)
    - topynate 2mo ago
      I've noticed them trying to creep into my writing. It doesn't help that I was a heavy em-dash user ten years before GPT-3.
      - mdgld 2mo ago
        I’ve always loved em dashes…very sad they’re a hallmark of LLM slop now. That and trios in arguments. Maybe I’m part LLM?
        embedding-shape 2mo ago
- miki123211 2mo ago
  And to add insult to injury, some providers will ride on the good reputation of some local model, selling you a terrible quant instead. With OpenAI, at least my gpt-5.5 is the same as your gpt-5.5. You can't say that about glm for example.
  - embedding-shape 2mo ago
    > And to add insult to injury, some providers will ride on the good reputation of some local model, selling you a terrible quant instead. I just started using OpenRouter for some control testing of local models and what surprises me the most isn't that there are different providers providing different quantization levels, that makes sense, but I can't seemingly find a way of seeing what provider+model+quantization is actually used?! https://openrouter.ai/models https://openrouter.ai/models shows the models, then say https://openrouter.ai/moonshotai/kimi-k2.7-code https://openrouter.ai/moonshotai/kimi-k2.7-code shows the providers but when I go to https://openrouter.ai/moonshotai/kimi-k2.7-code?endpoint=e7a9a3c1-6f00-4524-9dd4-ca1ae87050d6 https://openrouter.ai/moonshotai/kimi-k2.7-code?endpoint=e7a... for example, why on earth is it not showing the actual details about the actual weights they're serving?! Give me details! It does have a "Precision" value that is sometimes filled out, but that seems to be a guess at best, even providers with the same values there have wildly different quality responses. I like the idea about OpenRouter but holy hell does the implementation seem very far off from what it needs to be, in order to be useful.
    - jermaustin1 2mo ago
      There are properties on the API call you can pass for specific providers, so you test which providers you like the output, then add them to the list in ranked order if you want one by default, then to fall back to the other. There might be something in the response, or in a followup API call for the session, that you get better details. I think I've seen the details in the dashboard, so they do exist.
  - beacon294 2mo ago
    That's not a real equivalency. They are not necessarily the same (testing in production, hello!) And most importantly you do not have a local model because openai is not open!
mohsen1 2mo ago
I have a philosophical problem with adaptive thinking. It’s a dumb guess for how much thinking budget to allocate ahead of thinking. At least in the context of LLMs there is probably no way of knowing how much thinking (token generation) is needed. The problem space is infinity vast, similarly of two prompts is not going to help any LLM decide how much thinning is needed. Models already stop thinking before hitting the thinking budget. Why there is so much effort in making adaptive thinking happen and don’t we train models to produce the end of thinning token better? Feels like a bandaid. We need models to be trained to do a reasonable amount of reasoning (no pub intended): reason estimate remaining uncertainty continue? reason more repeat
- breakingcups 2mo ago
  Kind of feels like a variation on P versus NP
- aspenmartin 2mo ago
  Right now we have a LOT of band aids. You want to optimize compute and thinking to a particular problem, sort of like we do. Yes you cannot perfectly predict this but you can do decently well and save a ton of tokens at the cost of this band aid being sort of leaky and gross. But the larger problem is sound, and the answer is something jointly optimized (idk how they do the routing) but it’s hard to shoehorn it into the current paradigm.
- wongarsu 2mo ago
  At least there should be a tool call that's the equivalent of saying "wow, this is more complicated than I thought". Humans are also often prone to under-allocating reasoning time and coming to wrong conclusions because their reasoning ends up too shallow. But the best humans are great at mentally mapping the problem space and readjusting on the fly
  - solarkraft 2mo ago
    Deepseek regularly does this for me, at least when actively exploring something (multi-turn). It massively varies its reasoning budget based on the task.
varispeed 2mo ago
And people pay for those wasted tokens? If that's the case, it is probably good idea to ask for refunds.
bentoner 2mo ago
I've made a passive workaround: a pair of Codex CLI hooks that detect the truncation from the local session transcript and warn — in the TUI at turn end, and via a message injected into the model's context on your next message. See https://github.com/bentoner/codex-516-hook https://github.com/bentoner/codex-516-hook
nsingh2 2mo ago
This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide updates. Removing that entirely makes all runs succeed [1]. I wonder if it's somehow getting confused between what's supposed to be an intermediate update vs the final result. [1] https://github.com/openai/codex/issues/30364#issuecomment-4886510189 https://github.com/openai/codex/issues/30364#issuecomment-48...

by nsingh2 2mo ago

deleted 2mo ago

[deleted]

dannyw 2mo ago

I wonder if testing during different time/days show patterns? For example, whether the short circuiting happens more often during workday peak hours.

postalcoder 2mo ago

You still have to worry about misconfigured local models. Even the professionals get it wrong, which is why local model performance is uneven across providers.

deleted 2mo ago
[deleted]
jdiff 2mo ago
But in that case you have nobody but yourself to blame, and you can stabilize things yourself at any time by refraining from making any changes. You won't be surprised by a provider. Honestly? That's not just valuable—it's essential.
- LiamPowell 2mo ago
  > Honestly? That's not just valuable—it's essential. I'm curious if you wrote this or had a LLM write it. I'm genuinely curious to be clear as I don't see why anyone would bother to go through a LLM to write such a short reply. Have we reached the point where Claudeisms that are this obnoxious have become part of regular speech?
  - UqWBcuFx6NV4r 2mo ago
    Or they were making a joke[0]. [0]: https://en.wikipedia.org/wiki/Joke https://en.wikipedia.org/wiki/Joke (…just like that)
    - LiamPowell 2mo ago
      Sure, but it doesn't really fit there as a joke, it looks like it's just meant to be part of what they were trying to say.
      - subscribed 2mo ago
        I also think it's a joke, it starts with the response / argument, and then flows into tongue-in-cheek joke about the core issue of the post (LLM)
  - topynate 2mo ago
    I've noticed them trying to creep into my writing. It doesn't help that I was a heavy em-dash user ten years before GPT-3.
    - mdgld 2mo ago
      I’ve always loved em dashes…very sad they’re a hallmark of LLM slop now. That and trios in arguments. Maybe I’m part LLM?
      - embedding-shape 2mo ago
miki123211 2mo ago
And to add insult to injury, some providers will ride on the good reputation of some local model, selling you a terrible quant instead. With OpenAI, at least my gpt-5.5 is the same as your gpt-5.5. You can't say that about glm for example.
- embedding-shape 2mo ago
  > And to add insult to injury, some providers will ride on the good reputation of some local model, selling you a terrible quant instead. I just started using OpenRouter for some control testing of local models and what surprises me the most isn't that there are different providers providing different quantization levels, that makes sense, but I can't seemingly find a way of seeing what provider+model+quantization is actually used?! https://openrouter.ai/models https://openrouter.ai/models shows the models, then say https://openrouter.ai/moonshotai/kimi-k2.7-code https://openrouter.ai/moonshotai/kimi-k2.7-code shows the providers but when I go to https://openrouter.ai/moonshotai/kimi-k2.7-code?endpoint=e7a9a3c1-6f00-4524-9dd4-ca1ae87050d6 https://openrouter.ai/moonshotai/kimi-k2.7-code?endpoint=e7a... for example, why on earth is it not showing the actual details about the actual weights they're serving?! Give me details! It does have a "Precision" value that is sometimes filled out, but that seems to be a guess at best, even providers with the same values there have wildly different quality responses. I like the idea about OpenRouter but holy hell does the implementation seem very far off from what it needs to be, in order to be useful.
  - jermaustin1 2mo ago
    There are properties on the API call you can pass for specific providers, so you test which providers you like the output, then add them to the list in ranked order if you want one by default, then to fall back to the other. There might be something in the response, or in a followup API call for the session, that you get better details. I think I've seen the details in the dashboard, so they do exist.
- beacon294 2mo ago
  That's not a real equivalency. They are not necessarily the same (testing in production, hello!) And most importantly you do not have a local model because openai is not open!

mohsen1 2mo ago

I have a philosophical problem with adaptive thinking. It’s a dumb guess for how much thinking budget to allocate ahead of thinking. At least in the context of LLMs there is probably no way of knowing how much thinking (token generation) is needed. The problem space is infinity vast, similarly of two prompts is not going to help any LLM decide how much thinning is needed. Models already stop thinking before hitting the thinking budget. Why there is so much effort in making adaptive thinking happen and don’t we train models to produce the end of thinning token better? Feels like a bandaid. We need models to be trained to do a reasonable amount of reasoning (no pub intended): reason estimate remaining uncertainty continue? reason more repeat

breakingcups 2mo ago
Kind of feels like a variation on P versus NP
aspenmartin 2mo ago
Right now we have a LOT of band aids. You want to optimize compute and thinking to a particular problem, sort of like we do. Yes you cannot perfectly predict this but you can do decently well and save a ton of tokens at the cost of this band aid being sort of leaky and gross. But the larger problem is sound, and the answer is something jointly optimized (idk how they do the routing) but it’s hard to shoehorn it into the current paradigm.
wongarsu 2mo ago
At least there should be a tool call that's the equivalent of saying "wow, this is more complicated than I thought". Humans are also often prone to under-allocating reasoning time and coming to wrong conclusions because their reasoning ends up too shallow. But the best humans are great at mentally mapping the problem space and readjusting on the fly
- solarkraft 2mo ago
  Deepseek regularly does this for me, at least when actively exploring something (multi-turn). It massively varies its reasoning budget based on the task.

varispeed 2mo ago

And people pay for those wasted tokens? If that's the case, it is probably good idea to ask for refunds.

bentoner 2mo ago

I've made a passive workaround: a pair of Codex CLI hooks that detect the truncation from the local session transcript and warn — in the TUI at turn end, and via a message injected into the model's context on your next message. See https://github.com/bentoner/codex-516-hook https://github.com/bentoner/codex-516-hook

nsingh2 2mo ago

This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide updates. Removing that entirely makes all runs succeed [1]. I wonder if it's somehow getting confused between what's supposed to be an intermediate update vs the final result. [1] https://github.com/openai/codex/issues/30364#issuecomment-4886510189 https://github.com/openai/codex/issues/30364#issuecomment-48...