5 ms·
> With 1m context window there is no reason for a single task to require compaction Only if money is no object. Cache reads are cheap (10% of uncached input co
by wgjordan 2mo ago
> With 1m context window there is no reason for a single task to require compaction
Only if money is no object. Cache reads are cheap (10% of uncached input costs) but definitely not free, and cached reads dominate session costs at long context lengths. A prompt at 20k context with $0.01 in cached reads would cost $0.40 in cached reads at 800k context, that quickly adds up for long sessions.
- brookst 2mo agoI’m not following the implication that these economics argue for jsing compaction instead of just clearing context?
- wgjordan 2mo agoYour 'with 1m context window' implied that some manually-curated task 'chunks' would overflow smaller context windows (eg 200-400k tokens). If you're instead curating chunks small enough to avoid getting burned on long-context cache-read costs, you're not using a 1m context window at all. At that point, compaction is a convenience over manually chunking and clearing context for fine-grained bits of work. I think the stronger claim is: there is no reason for a single task to require a 1m context window.