7 ms·
It should burn N + len(answer), because you have to re-cache the whole answer without the prompt stack. Perhaps more persnickety, it pushes the LLM out of dist
by thatguymike 26d ago
It should burn N + len(answer), because you have to re-cache the whole answer without the prompt stack.
Perhaps more persnickety, it pushes the LLM out of distribution - if it’s unnatural for it to write in plain language without the prompt stack, your prefix will be an unnatural conversation which can reduce intelligence in hard to measure ways, especially over long conversations.
Not saying don’t do it, clarity is perhaps worth the intelligence hit, but it’s not going to be a free lunch.
- bitexploder 25d agoYes, maybe. Evidence around caveman shows this isn’t a big deal for token consumption (forcing it to respond in a way it was not tuned) and I don’t see a big difference either way. And maybe it increases intelligence in hard to measure ways. Lots of parameters in these models. I feel I fight them less with this setup. They get so lost in their own invented bullshit they stop being useful pretty often without it. So I would take bets on that :)