6 ms·
Doesn't that mean these 33k tokens can be cached, since they don't depend on the input? The model can just start generation on the 33k+1th token.
by IX-103 2mo ago
Doesn't that mean these 33k tokens can be cached, since they don't depend on the input? The model can just start generation on the 33k+1th token.
- 0-_-0 2mo agoYes, but all subsequent tokens do depend on them, so they are still being charged for
- mixermachine 2mo agoright, but the cache retention time is very short for Anthropics LLMs. 5 minutes or 1 hour (with additional costs). So you have to prompt basically non stop to not get a cache eviction. Anthropic even changed this silently: https://www.reddit.com/r/ClaudeAI/comments/1sk3m12/followup_anthropic_quietly_switched_the_default/ https://www.reddit.com/r/ClaudeAI/comments/1sk3m12/followup_...
- NiloCK 2mo agoA const prompt across all of Anthropic's subscribers could draw from a global cache rather than per-user? Although saying that out loud makes me question it - each per-user chat and growing cache would need eventually to own its own ~contiguous memory block.
- mixermachine 2mo agoIf there are no changes to the system message, yes this is possible and also likely done by Anthropic. When there are additional local MCPs, reusability will be lower. I think that Anthropic will bill you in any case :D