5 ms·

With prefix caching, you get checkpoints for free. Do you mean that?

• vidarh 2mo ago
  No. Prefix caching is just an optimisation on the server side. What Kimi CLI does is insert <system> tags that include a checkpoint marker with an id. The model is then given a tool that allows the model to decide to roll back to a checkpoint + a message containing any additional useful information. It's specifically instructed to use that tool[1] in cases like when it has inadvertendly read a large file where most of the content is not relevant to the task, or after a web search where it's found what it's looking for but most of the content isn't needed, or when it's written code that didn't work as expected, or similar. It basically lets the model backtrack and "forget" irrelevant details at the end of the context but give itself hints on how it should continue from the checkpoint. Though, interestingly they seem to be abandoning it in their new CLI (kimi-code), unless it's been folded into other functionality. Not sure if they just feel it's not needed any more with their newer models or if it just didn't work as well as they expected. [1] named "D-Mail", or "DeLorean Mail" in a reference to Steins;Gate, which again references Back To The Future. See https://github.com/MoonshotAI/kimi-cli/blob/main/src/kimi_cli/tools/dmail/dmail.md https://github.com/MoonshotAI/kimi-cli/blob/main/src/kimi_cl... and https://steins-gate.fandom.com/wiki/D-Mail https://steins-gate.fandom.com/wiki/D-Mail
  • DelightOne 2mo ago
    Prefix caching automatically detects priorly read tokens -> more fained grained checkpoints then. With prefix caching you don't have to ask for a checkpoint, its automatic. One way or the other, a prefix is saved. Only the additional info how to continue is new.