I'm not an LLM engineer, but I just got an idea of how it could work, combining CoT + checkpoints : - use checkpoints to save KV cache before trigger CoT - tr

I'm not an LLM engineer, but I just got an idea of how it could work, combining CoT + checkpoints : - use checkpoints to save KV cache before trigger CoT - trigger CoT, save result as a summary - go back to previous checkpoint - instead of generating tokens, add result of CoT summary as input tokens - continue normally For the price of twice the KV cache memory, the context stays perpetually small, allowing smarter sessions. You can even apply that continually by summarizing tool calls, etc. This idea is free. I'm not sure it's that advantageous though: it consumes more memory, and the sessions are already quite long at 1M+ tokens. One would need to run the economics down, and just test if the shorter sessions are actually smarter with the continuous summarization.