8 ms·

Seems like a pretty pointless post that still centers around output tokens. In agentic coding, cached input tokens is 90% of the API "cost". It doesn't require

Seems like a pretty pointless post that still centers around output tokens. In agentic coding, cached input tokens is 90% of the API "cost". It doesn't require GPU compute, and DeepSeek has shown that it can be done 50~100x cheaper with MLA/CSA/HCA, and a whole bunch of disks. This should collapse the margin.

nozzlegear 2mo ago
Aren't the American AI labs desperately struggling to find a market beyond just agentic coding?
- le-mark 2mo ago
  I have heard but don’t have first hand knowledge that at least one company (financial services BPO) has moved most of their previously manual processing to llms. The person I talked to wasn’t forthcoming with any detail. This is what we’d expect to see though.
- dgellow 2mo ago
  All AI labs. Not just Americans
throwdbaaway 2mo ago
The current top comment in https://lobste.rs/s/ua1gxl/glm_5_2_coming_ai_margin_collapse https://lobste.rs/s/ua1gxl/glm_5_2_coming_ai_margin_collapse correctly zoomed into cached input tokens, but landed on the opposite conclusion: > That is, for your $100/month fee, you get $3600 equivalent of API usage. This is presumably because Anthropic has figured out some clever things to do with model routing and input caching, and also can subsidize with investor money and take a hit on their operating margins. My take: this is exactly what Anthropic wants everyone to think. In reality, 90% of that $3600 are for cached input tokens, that can be made to cost next to nothing, as shown by DeepSeek.
- throwdbaaway 2mo ago
  While we are all speculating, Boris kindly provided some guidance in https://news.ycombinator.com/item?id=47880089 https://news.ycombinator.com/item?id=47880089 > The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for users. In an extreme case, if you had 900k tokens in your context window, then idled for an hour, then sent a message, that would be >900k tokens written to cache all at once, which would eat up a significant % of your rate limits, especially for Pro users. Using the current Opus pricing, that pre-lunch 900k tokens should roughly consist of: 720k input tokens = 0.72 x $5 = $3.6 180k output tokens = 0.18 x $25 = $4.5 900k 1h cached writes = 0.9 x $10 = $9 500M cached input tokens = 500 x $0.5 = $250 $267.1 in total, with 93.6% from cached input tokens. The portion that requires GPU compute is about 3% of the total. Post-lunch, the 900k tokens should consist of: 900k input tokens = 0.9 x $5 = $4.5 900k 1h cached writes = 0.9 x $10 = $9 So Anthropic is fine with the $267.1 accumulated over 3~4 hours before lunch, but not fine with the $13.5 incurred immediately after lunch. Why? The only plausible explanation is that the actual cost of caching is way less than the API pricing. If you use a coding plan, Anthropic doesn't really care about your cached input tokens usage. Indeed they want you to show your ccusage screenshots. On the other hand, if you pay by API tokens, the margin is huge for cached input tokens. Only when you do something that requires a lot of FLOPs, e.g. the post-lunch 900k input tokens, the cost becomes real.
  - nl 2mo ago
    I wouldn't be too fixated on the specific numbers in that post. Anthropic was extremely capacity constrained at that point. They still are but not to that extent. I'd note that OpenAI offers 24 hour caching. I'd be surprised if Anthropic hasn't optimised their caching for Claude code too. SemiAnalysis recently posted that their actual Opus usage works out at $0.99 because of caching. The principles remain though.
  - Roark66 2mo ago
    Recently I started getting messages from Clause Code (on a plan). "You're restoring an old session are you sure you don't want to compress the context? This will use a substantial amount of your usage quota" So it seems they do care.
stingraycharles 2mo ago
> MLA/CSA/HCA Aren’t these techniques all “lossy” compression, and one of the reasons people complain about loss in quality as the context size grows larger?
- throwdbaaway 2mo ago
  Indeed they are all lossy. Not sure how much they contribute to the quality loss in long context though. I got a 700k session with DSV4 Pro (official API), and the model was still coherent and didn't make any tool call error.
  - stingraycharles 2mo ago
    That’s a low bar though, and the least I would expect.
    - throwdbaaway 2mo ago
      Well I wouldn't call it a low bar, since some of the edits were quite complex. And 1M context in less than 6GB of VRAM is truly impressive, but somehow this gets way less attention than the crappy turbo quant from Google.
      - scope2093 2mo ago
        I'd like to understand this please. Why would the 1M context be kept in VRAM if you're using DSV4 Pro through the API? Or did you refer to different sessions?
        throwdbaaway 2mo ago
        Different sessions. With https://github.com/fairydreaming/llama.cpp/tree/dsv4 https://github.com/fairydreaming/llama.cpp/tree/dsv4, 1M context with DSV4 Flash takes less than 6GB of VRAM. I can't run DSV4 Pro, but it should take less than 9GB of VRAM for 1M context, based on the numbers shared in https://arxiv.org/html/2606.19348v1 https://arxiv.org/html/2606.19348v1.

nozzlegear 2mo ago

Aren't the American AI labs desperately struggling to find a market beyond just agentic coding?

le-mark 2mo ago
I have heard but don’t have first hand knowledge that at least one company (financial services BPO) has moved most of their previously manual processing to llms. The person I talked to wasn’t forthcoming with any detail. This is what we’d expect to see though.
dgellow 2mo ago
All AI labs. Not just Americans

throwdbaaway 2mo ago

The current top comment in https://lobste.rs/s/ua1gxl/glm_5_2_coming_ai_margin_collapse https://lobste.rs/s/ua1gxl/glm_5_2_coming_ai_margin_collapse correctly zoomed into cached input tokens, but landed on the opposite conclusion: > That is, for your $100/month fee, you get $3600 equivalent of API usage. This is presumably because Anthropic has figured out some clever things to do with model routing and input caching, and also can subsidize with investor money and take a hit on their operating margins. My take: this is exactly what Anthropic wants everyone to think. In reality, 90% of that $3600 are for cached input tokens, that can be made to cost next to nothing, as shown by DeepSeek.

throwdbaaway 2mo ago
While we are all speculating, Boris kindly provided some guidance in https://news.ycombinator.com/item?id=47880089 https://news.ycombinator.com/item?id=47880089 > The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for users. In an extreme case, if you had 900k tokens in your context window, then idled for an hour, then sent a message, that would be >900k tokens written to cache all at once, which would eat up a significant % of your rate limits, especially for Pro users. Using the current Opus pricing, that pre-lunch 900k tokens should roughly consist of: 720k input tokens = 0.72 x $5 = $3.6 180k output tokens = 0.18 x $25 = $4.5 900k 1h cached writes = 0.9 x $10 = $9 500M cached input tokens = 500 x $0.5 = $250 $267.1 in total, with 93.6% from cached input tokens. The portion that requires GPU compute is about 3% of the total. Post-lunch, the 900k tokens should consist of: 900k input tokens = 0.9 x $5 = $4.5 900k 1h cached writes = 0.9 x $10 = $9 So Anthropic is fine with the $267.1 accumulated over 3~4 hours before lunch, but not fine with the $13.5 incurred immediately after lunch. Why? The only plausible explanation is that the actual cost of caching is way less than the API pricing. If you use a coding plan, Anthropic doesn't really care about your cached input tokens usage. Indeed they want you to show your ccusage screenshots. On the other hand, if you pay by API tokens, the margin is huge for cached input tokens. Only when you do something that requires a lot of FLOPs, e.g. the post-lunch 900k input tokens, the cost becomes real.
- nl 2mo ago
  I wouldn't be too fixated on the specific numbers in that post. Anthropic was extremely capacity constrained at that point. They still are but not to that extent. I'd note that OpenAI offers 24 hour caching. I'd be surprised if Anthropic hasn't optimised their caching for Claude code too. SemiAnalysis recently posted that their actual Opus usage works out at $0.99 because of caching. The principles remain though.
- Roark66 2mo ago
  Recently I started getting messages from Clause Code (on a plan). "You're restoring an old session are you sure you don't want to compress the context? This will use a substantial amount of your usage quota" So it seems they do care.

stingraycharles 2mo ago

> MLA/CSA/HCA Aren’t these techniques all “lossy” compression, and one of the reasons people complain about loss in quality as the context size grows larger?

throwdbaaway 2mo ago
Indeed they are all lossy. Not sure how much they contribute to the quality loss in long context though. I got a 700k session with DSV4 Pro (official API), and the model was still coherent and didn't make any tool call error.
- stingraycharles 2mo ago
  That’s a low bar though, and the least I would expect.
  - throwdbaaway 2mo ago
    Well I wouldn't call it a low bar, since some of the edits were quite complex. And 1M context in less than 6GB of VRAM is truly impressive, but somehow this gets way less attention than the crappy turbo quant from Google.
    - scope2093 2mo ago
      I'd like to understand this please. Why would the 1M context be kept in VRAM if you're using DSV4 Pro through the API? Or did you refer to different sessions?
      - throwdbaaway 2mo ago
        Different sessions. With https://github.com/fairydreaming/llama.cpp/tree/dsv4 https://github.com/fairydreaming/llama.cpp/tree/dsv4, 1M context with DSV4 Flash takes less than 6GB of VRAM. I can't run DSV4 Pro, but it should take less than 9GB of VRAM for 1M context, based on the numbers shared in https://arxiv.org/html/2606.19348v1 https://arxiv.org/html/2606.19348v1.