4 ms·

So to defend a litte, its a Cache, it has to go somewhere, its a save state of the model's inner workings at the time of the last message. so if it expires, it

So to defend a litte, its a Cache, it has to go somewhere, its a save state of the model's inner workings at the time of the last message. so if it expires, it has to process the whole thing again. most people don't understand that every message the ENTIRE history of the conversion is processed again and again without that cache. That conversion might of hit several gigs worth of model weights and are you expecting them to keep that around for /all/ of your conversions you have had with it in separate sessions?

3836293648 5mo ago
No? It's not because it's a cache, it's because they're scared of letting you see the thinking trace. If you got the trace you could just send it back in full when it got evicted from the cache. This is how open weight models work.
- eknkc 5mo ago
  I’m not familiar with the Claude API but OpenAI has an encrypted thking messages option. You get something that you can send back but it is encrypted. Not available on Anthropic?
- reactordev 5mo ago
  They are sending it back to the cache, the part you are missing is they were charging you for it.
  - eknkc 5mo ago
    The blog post says they prune them now not to charge you. That’s the change they implemented.
    - reactordev 5mo ago
      right. they were charging you for it, now they aren't because they are just dropping your conversation history.
- mpyne 5mo ago
  The trace goes back fine, that's not the issue. The issue is that if they send the full trace back, it will have to be processed from the start if the cache expired, and doing that will cause a huge one-time hit against your token limit if the session has grown large. So what Boris talked about is stripping things out of the trace that goes back to regenerate the session if the cache expires. Doing this would help avert burning up the token limit, but it is technically a different conversation, so if CC chooses poorly on stripping parts of the context then it would lead to Claude getting all scatter-brained.
  - charcircuit 5mo ago
    >and doing that will cause a huge one-time hit against your token limit if the session has grown large. Anthropic already profited from generating those tokens. They can afford subsidize reloading context.
    - pixl97 5mo ago
      No they can't, that's what you don't seem to get. Reloading those tokens takes around the same effort as processing them in the first place. It's ok to be ignorant of how the infrastructure for LLMs work, just don't be proud of it.
      - charcircuit 5mo ago
        They literally can. They could make the API free to use if they wanted. There is no law that states that costs have to equal the cost it takes to process the request.
deleted 5mo ago
[deleted]
rsfern 5mo ago
It seems like an opportunity for a hierarchical cache. Instead of just nuking all context on eviction, couldn’t there be an L2 cache with a longer eviction time so task switching for an hour doesn’t require a full session replay?
- sfink 5mo ago
  Living where? If it's in the GPU, then it's still taking up precious space that could be used for serving other sessions. If it's not in the GPU, then it doesn't help.
CjHuber 5mo ago
No of course it’s unrealistic for them to hold the cache indefinitely and that’s not the point. You are keeping the session data yourself so you can continue even after cache expiry. The point I‘m making is that it made me very angry that without any announcement they changed behavior to strip the old thinking even when you have it in your session file. There is absolutely no reason to not ask the user about if they want this And it’s part of a larger problem of unannounced changes it‘s just like when they introduced adaptive thinking to 4.6 a few weeks ago without notice. Also they seem to be completely unaware that some users might only use Claude code because they are used to it not stripping thinking in contrast to codex. Anyway I‘m happy that they saw it as a valid refund reason
cyanydeez 5mo ago
what matters isn't that it's a cache; what matter is it's cached _in the GPU/NPU_ memory and taking up space from another user's active session; to keep that cache in the GPU is a nonstarter for an oversold product. Even putting into cold storage means they still have to load it at the cost of the compute, generally speaking because it again, takes up space from an oversold product.

3836293648 5mo ago

No? It's not because it's a cache, it's because they're scared of letting you see the thinking trace. If you got the trace you could just send it back in full when it got evicted from the cache. This is how open weight models work.

eknkc 5mo ago
I’m not familiar with the Claude API but OpenAI has an encrypted thking messages option. You get something that you can send back but it is encrypted. Not available on Anthropic?
reactordev 5mo ago
They are sending it back to the cache, the part you are missing is they were charging you for it.
- eknkc 5mo ago
  The blog post says they prune them now not to charge you. That’s the change they implemented.
  - reactordev 5mo ago
    right. they were charging you for it, now they aren't because they are just dropping your conversation history.
mpyne 5mo ago
The trace goes back fine, that's not the issue. The issue is that if they send the full trace back, it will have to be processed from the start if the cache expired, and doing that will cause a huge one-time hit against your token limit if the session has grown large. So what Boris talked about is stripping things out of the trace that goes back to regenerate the session if the cache expires. Doing this would help avert burning up the token limit, but it is technically a different conversation, so if CC chooses poorly on stripping parts of the context then it would lead to Claude getting all scatter-brained.
- charcircuit 5mo ago
  >and doing that will cause a huge one-time hit against your token limit if the session has grown large. Anthropic already profited from generating those tokens. They can afford subsidize reloading context.
  - pixl97 5mo ago
    No they can't, that's what you don't seem to get. Reloading those tokens takes around the same effort as processing them in the first place. It's ok to be ignorant of how the infrastructure for LLMs work, just don't be proud of it.
    - charcircuit 5mo ago
      They literally can. They could make the API free to use if they wanted. There is no law that states that costs have to equal the cost it takes to process the request.

deleted 5mo ago

[deleted]

rsfern 5mo ago

It seems like an opportunity for a hierarchical cache. Instead of just nuking all context on eviction, couldn’t there be an L2 cache with a longer eviction time so task switching for an hour doesn’t require a full session replay?

sfink 5mo ago
Living where? If it's in the GPU, then it's still taking up precious space that could be used for serving other sessions. If it's not in the GPU, then it doesn't help.

CjHuber 5mo ago

No of course it’s unrealistic for them to hold the cache indefinitely and that’s not the point. You are keeping the session data yourself so you can continue even after cache expiry. The point I‘m making is that it made me very angry that without any announcement they changed behavior to strip the old thinking even when you have it in your session file. There is absolutely no reason to not ask the user about if they want this And it’s part of a larger problem of unannounced changes it‘s just like when they introduced adaptive thinking to 4.6 a few weeks ago without notice. Also they seem to be completely unaware that some users might only use Claude code because they are used to it not stripping thinking in contrast to codex. Anyway I‘m happy that they saw it as a valid refund reason

cyanydeez 5mo ago

what matters isn't that it's a cache; what matter is it's cached _in the GPU/NPU_ memory and taking up space from another user's active session; to keep that cache in the GPU is a nonstarter for an oversold product. Even putting into cold storage means they still have to load it at the cost of the compute, generally speaking because it again, takes up space from an oversold product.