5 ms·
Incentives to optimize cache-usage are only aligned between anthropic and you, dear user, when there is not enough compute available to serve the tokens fresh.
by Phemist 1mo ago
Incentives to optimize cache-usage are only aligned between anthropic and you, dear user, when there is not enough compute available to serve the tokens fresh. Apparently enough compute is now available that frugal usage is no longer a requirement and user cost/profit maximization is now the name of the game. I would expect to see more and more "cache-busting" strategies implemented in order to both eek out marginal performance increases and to heavily increase profit.