5 ms·
I don't understand why changing effort levels busts the cache. Couldn't effort levels be a decoding-only thing where they just change the probability of the <en
by lovasoa 1mo ago
I don't understand why changing effort levels busts the cache. Couldn't effort levels be a decoding-only thing where they just change the probability of the <end of thought> token? Are they literally adding a hidden system prompt that says "effort level: $level" ?
- Traubenfuchs 1mo agoObviously it‘s something being put in the context that can not be taken out of it anymore.
- Philpax 1mo ago> Are they literally adding a hidden system prompt that says "effort level: $level" ? Yes. https://magazine.sebastianraschka.com/p/controlling-reasoning-effort-in-llms https://magazine.sebastianraschka.com/p/controlling-reasonin...
- namjh 1mo agoIn that way the autoregressive nature of LLM won't let itself "plan" to reason with the intended budget. It doesn't "look ahead".