7 ms·

It's not exactly this. It's something like it. But it's not a token that erases a previous token. Not that I know how that would work or how you'd get training

It's not exactly this. It's something like it. But it's not a token that erases a previous token. Not that I know how that would work or how you'd get training data (edit histories of internet comments seem too few). (You could have a token that hides previous tokens, but that'd be rather closer to CoT.)

gspr 24d ago
Backtracking, of sorts. The LLM descends a seemingly fruitful path, and all of a sudden that path seems less so. So you backtrack a few tokens ("erasing"/"undoing") and resample the probability distribution a bit back in the stream.
- krautsauer 23d ago
  Would it retain some memory of the fact that it backtracked? That path becoming less likely somehow?
  - gspr 23d ago
    Yeah that's what I was imagining. No idea how though, so this is just rambling on my part :)
dpkirchner 24d ago
You could burn a few tokens by prompting the LLM to summarize the omit all of the "wrong" answers (where it overrode itself), after the user accepts the response. It'd probably reduce token use in the end, especially for long sessions.
Aissen 24d ago
I'm not sure how that would work. You'd need to be able to reverse the operations in the KV cache, and I'm not sure if it's doable or not. And then you'd need to pick another path, or "rebalance" the probabilities or you'd go the same token path. CoT is probably easier to implement (but not necessarily better).
- StilesCrisis 23d ago
  You could set up checkpoints where you store off the current state, and roll back to those checkpoints?
  - Aissen 23d ago
    Definitely, this is the easy "bruteforce" way, but it would require more memory, and would not resolve the second part of the problem (i.e "how to pick a better path").
    - Aissen 23d ago
      I'm not an LLM engineer, but I just got an idea of how it could work, combining CoT + checkpoints : - use checkpoints to save KV cache before trigger CoT - trigger CoT, save result as a summary - go back to previous checkpoint - instead of generating tokens, add result of CoT summary as input tokens - continue normally For the price of twice the KV cache memory, the context stays perpetually small, allowing smarter sessions. You can even apply that continually by summarizing tool calls, etc. This idea is free. I'm not sure it's that advantageous though: it consumes more memory, and the sessions are already quite long at 1M+ tokens. One would need to run the economics down, and just test if the shorter sessions are actually smarter with the continuous summarization.

gspr 24d ago

Backtracking, of sorts. The LLM descends a seemingly fruitful path, and all of a sudden that path seems less so. So you backtrack a few tokens ("erasing"/"undoing") and resample the probability distribution a bit back in the stream.

krautsauer 23d ago
Would it retain some memory of the fact that it backtracked? That path becoming less likely somehow?
- gspr 23d ago
  Yeah that's what I was imagining. No idea how though, so this is just rambling on my part :)

dpkirchner 24d ago

You could burn a few tokens by prompting the LLM to summarize the omit all of the "wrong" answers (where it overrode itself), after the user accepts the response. It'd probably reduce token use in the end, especially for long sessions.

Aissen 24d ago

I'm not sure how that would work. You'd need to be able to reverse the operations in the KV cache, and I'm not sure if it's doable or not. And then you'd need to pick another path, or "rebalance" the probabilities or you'd go the same token path. CoT is probably easier to implement (but not necessarily better).

StilesCrisis 23d ago
You could set up checkpoints where you store off the current state, and roll back to those checkpoints?
- Aissen 23d ago
  Definitely, this is the easy "bruteforce" way, but it would require more memory, and would not resolve the second part of the problem (i.e "how to pick a better path").
  - Aissen 23d ago
    I'm not an LLM engineer, but I just got an idea of how it could work, combining CoT + checkpoints : - use checkpoints to save KV cache before trigger CoT - trigger CoT, save result as a summary - go back to previous checkpoint - instead of generating tokens, add result of CoT summary as input tokens - continue normally For the price of twice the KV cache memory, the context stays perpetually small, allowing smarter sessions. You can even apply that continually by summarizing tool calls, etc. This idea is free. I'm not sure it's that advantageous though: it consumes more memory, and the sessions are already quite long at 1M+ tokens. One would need to run the economics down, and just test if the shorter sessions are actually smarter with the continuous summarization.