5 ms·
> I have a feeling that the price of 1 million tokens transmitted over the internet is more expensive than cache hit. And this kinda makes sense. What is cheap
by ponyous 6d ago
> I have a feeling that the price of 1 million tokens transmitted over the internet is more expensive than cache hit.
And this kinda makes sense. What is cheaper few KB of disk space or internet bandwidth?
- k9294 6d ago100%, but this means we are going to move to stateful APIs on the AI provider's end (like OpenAI already does with Codex and Responses API) to make this work.