10 ms·

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leave sessions idle for hours or days and use the capability to pick it back up with full context and power. The default thinking level seems more forgivable, but the churn in system prompts is something I'll need to figure out how to intentionally choose a refresh cycle.

seizethecheese 5mo ago
It's also a bit of a fishy explanation for purging tokens older than an hour. This happens to also be their cache limit. I doubt it is incidental that this change would also dramatically drop their cost.
- cma 5mo ago
  They moved it to 5m around the same timeframe though: https://www.reddit.com/r/ClaudeAI/comments/1sk3m12/followup_anthropic_quietly_switched_the_default/ https://www.reddit.com/r/ClaudeAI/comments/1sk3m12/followup_...
tadfisher 5mo ago
It astounds me that a company valued in the hundreds-of-billions-of-dollars has written this. One of the following must be true: 1. They actually believed latency reduction was worth compromising output quality for sessions that have already been long idle. Moreover, they thought doing so was better than showing a loading indicator or some other means of communicating to the user that context is being loaded. 2. What I suspect actually happened: they wanted to cost-reduce idle sessions to the bare minimum, and "latency" is a convenient-enough excuse to pass muster in a blog post explaining a resulting bug.
- retinaros 5mo ago
  they just vibecoded a fix and didnt think about the tradeoff they were making and their always yes-man of a model just went with it
- someguyiguess 5mo ago
  It’s definitely a cost / resource saving strategy on their end.
- billywhizz 5mo ago
  what's even more amazing is it took them two weeks to fix what must have been a pretty obvious bug, especially given who they are and what they are selling.
- raincole 5mo ago
  It's very weird that they frame caching as "latency reduction" when it comes to a cloud service. I mean, yes, technically it reduces latency, but more importantly it reduces cost. Sometimes it's more than 80% of the total cost. I'm sure most companies and customers will consider compromising quality for 80% cost reduction. If they just be honest they'll be fine.
- sekai 5mo ago
  The same company that claims they have models that are too "dangerous" to release btw.
- adam_patarino 5mo ago
  It’s certainly #2. They have shown over dozens of decisions they move very quickly, break stuff, then have to both figure out what broke and how to explain it.
bcherny 5mo ago
Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for users. In an extreme case, if you had 900k tokens in your context window, then idled for an hour, then sent a message, that would be >900k tokens written to cache all at once, which would eat up a significant % of your rate limits, especially for Pro users. We tried a few different approaches to improve this UX: 1. Educating users on X/social 2. Adding an in-product tip to recommend running /clear when re-visiting old conversations (we shipped a few iterations of this) 3. Eliding parts of the context after idle: old tool results, old messages, thinking. Of these, thinking performed the best, and when we shipped it, that's when we unintentionally introduced the bug in the blog post. Hope this is helpful. Happy to answer any questions if you have.
- PeterStuer 5mo ago
  At least for me, option 2 seems far favorable to the others. Give me the info, then let me decide.
- fidrelity 5mo ago
  Just wanted to say I appreciate your responses here. Engaging so directly with a highly critical audience is a minefield that you're navigating well. Thank you.
  - shimman 5mo ago
    Very easy to do when you stand to make tens of millions when your employer IPOs. Let's not maybe give too much praise and employ some critical thinking here.
    - simplify 5mo ago
      What is the purpose of this mindset? Should we encourage typical corporate coldness instead?
sockaddr 5mo ago
Yeah this is actually quite shocking. In my earlier uses of CC I might noodle on a problem for a while, come back and update the plan, go shower, think, give CC a new piece of advice, etc. Basically treating it like a coworker. And I thought that it was a static conversation (at least on the order of a day or so). An hour is absurd IMO and makes me want to rethink whether I want to keep my anthropic plan.
zmmmmm 5mo ago
Seems like it would interact very badly with the time based usage reset. If lots of people are hitting their limit and then letting the session idle until they can come back, this wouldn't be an exception. It would almost be the default behaviour.
deleted 5mo ago
[deleted]
Aperocky 5mo ago
Wow, I always thought the context is always stored locally and this is something I have control over. Glad I use kiro-cli which doesn't do this.
- Bishonen88 5mo ago
  you might be biased due to your employment :)
  - Aperocky 5mo ago
    Objectively speaking, I want control of context and when I compact it. That wouldn't change with employment.
deleted 5mo ago
[deleted]
greatgib 5mo ago
In addition with the bug, a big part of the issue is that this change was done secretly by Anthropic and not communicated to the users. If that was done, users could have been mindful of the change and figure out more easily that their problems could have come from that.

seizethecheese 5mo ago

It's also a bit of a fishy explanation for purging tokens older than an hour. This happens to also be their cache limit. I doubt it is incidental that this change would also dramatically drop their cost.

cma 5mo ago
They moved it to 5m around the same timeframe though: https://www.reddit.com/r/ClaudeAI/comments/1sk3m12/followup_anthropic_quietly_switched_the_default/ https://www.reddit.com/r/ClaudeAI/comments/1sk3m12/followup_...

tadfisher 5mo ago

It astounds me that a company valued in the hundreds-of-billions-of-dollars has written this. One of the following must be true: 1. They actually believed latency reduction was worth compromising output quality for sessions that have already been long idle. Moreover, they thought doing so was better than showing a loading indicator or some other means of communicating to the user that context is being loaded. 2. What I suspect actually happened: they wanted to cost-reduce idle sessions to the bare minimum, and "latency" is a convenient-enough excuse to pass muster in a blog post explaining a resulting bug.

retinaros 5mo ago
they just vibecoded a fix and didnt think about the tradeoff they were making and their always yes-man of a model just went with it
someguyiguess 5mo ago
It’s definitely a cost / resource saving strategy on their end.
billywhizz 5mo ago
what's even more amazing is it took them two weeks to fix what must have been a pretty obvious bug, especially given who they are and what they are selling.
raincole 5mo ago
It's very weird that they frame caching as "latency reduction" when it comes to a cloud service. I mean, yes, technically it reduces latency, but more importantly it reduces cost. Sometimes it's more than 80% of the total cost. I'm sure most companies and customers will consider compromising quality for 80% cost reduction. If they just be honest they'll be fine.
sekai 5mo ago
The same company that claims they have models that are too "dangerous" to release btw.
adam_patarino 5mo ago
It’s certainly #2. They have shown over dozens of decisions they move very quickly, break stuff, then have to both figure out what broke and how to explain it.

bcherny 5mo ago

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for users. In an extreme case, if you had 900k tokens in your context window, then idled for an hour, then sent a message, that would be >900k tokens written to cache all at once, which would eat up a significant % of your rate limits, especially for Pro users. We tried a few different approaches to improve this UX: 1. Educating users on X/social 2. Adding an in-product tip to recommend running /clear when re-visiting old conversations (we shipped a few iterations of this) 3. Eliding parts of the context after idle: old tool results, old messages, thinking. Of these, thinking performed the best, and when we shipped it, that's when we unintentionally introduced the bug in the blog post. Hope this is helpful. Happy to answer any questions if you have.

PeterStuer 5mo ago
At least for me, option 2 seems far favorable to the others. Give me the info, then let me decide.
fidrelity 5mo ago
Just wanted to say I appreciate your responses here. Engaging so directly with a highly critical audience is a minefield that you're navigating well. Thank you.
- shimman 5mo ago
  Very easy to do when you stand to make tens of millions when your employer IPOs. Let's not maybe give too much praise and employ some critical thinking here.
  - simplify 5mo ago
    What is the purpose of this mindset? Should we encourage typical corporate coldness instead?

sockaddr 5mo ago

Yeah this is actually quite shocking. In my earlier uses of CC I might noodle on a problem for a while, come back and update the plan, go shower, think, give CC a new piece of advice, etc. Basically treating it like a coworker. And I thought that it was a static conversation (at least on the order of a day or so). An hour is absurd IMO and makes me want to rethink whether I want to keep my anthropic plan.

zmmmmm 5mo ago

Seems like it would interact very badly with the time based usage reset. If lots of people are hitting their limit and then letting the session idle until they can come back, this wouldn't be an exception. It would almost be the default behaviour.

deleted 5mo ago

[deleted]

Aperocky 5mo ago

Wow, I always thought the context is always stored locally and this is something I have control over. Glad I use kiro-cli which doesn't do this.

Bishonen88 5mo ago
you might be biased due to your employment :)
- Aperocky 5mo ago
  Objectively speaking, I want control of context and when I compact it. That wouldn't change with employment.

deleted 5mo ago

[deleted]

greatgib 5mo ago

In addition with the bug, a big part of the issue is that this change was done secretly by Anthropic and not communicated to the users. If that was done, users could have been mindful of the change and figure out more easily that their problems could have come from that.