5 ms·
Aren't token prices fixed? The cost is growing quadratically in the large context situation so if they can reduce the number of tokens, they actually profit of
by imtringued 15d ago
Aren't token prices fixed?
The cost is growing quadratically in the large context situation so if they can reduce the number of tokens, they actually profit off the fixed token pricing because they can set the pricing based on some average context length with CoT tokens but the actual context length is shorter now.
Ok, so I thought about this a bit more and the true answer is that the companies have an incentive to make you fill up the context until the marginal cost per token is reached and then they want you to quit the session.
- simianwords 15d agothis is not the point - we were previously billed by the number of reasoning tokens produced. now we aren't. so this means, even if Astra uses a lot of reasoning we may not be billed for it because they aren't real tokens. what they could be doing is billing us by virtual tokens meaning number of loops? but even then the more interesting part is context rot - previously conversation you might have 50k tokens spent on reasoning. the next turn takes all the previous tokens as well (if you wanna preserve prompt caching) which is not ideal. this new method skips that so you get more free context until compaction kicks in.