6 ms·
You still pay cache token costs on API calls. Cache cost/token are 90% lower, but you pay it every single turn. I’m not sure if they let you skip the cache wri
by yojo 2mo ago
You still pay cache token costs on API calls. Cache cost/token are 90% lower, but you pay it every single turn.
I’m not sure if they let you skip the cache write cost on the first turn. That would imply cross-user caching infrastructure or special casing the default system prompt to give you a discount. Maybe? Away from the computer but you could try a “hello” in a fresh session and see what was billed.
- cyanydeez 2mo agoIts inexpensive to reserve KV cache for the first turn and would benefit users, given the first turn already requires costly locating and allocating a model slot for a user. So yeah, when they banned tgird party harnesses there was a technical and $ case to have.