6 ms·
Sure, you're right. But it's a policy learned from a next-token prediction task. You could also call it an inferrer or generator or whatever. The point is that
by gwerbin 11d ago
Sure, you're right.
But it's a policy learned from a next-token prediction task. You could also call it an inferrer or generator or whatever. The point is that it takes as input a sequence of preceding tokens and emits one more token to continue the sequence.
- danielmarkbruce 11d agoThe policy is not learned token by token during RLHF and RLVR. The reward model doesn't score token by token.
- gwerbin 10d agoSo? tokens are emitted one at a time according to the output distribution & sampler algorithm, and the next token distribution is a function of the preceding token sequence only. The process by which the output distribution is shaped doesn't change the core mental model, and doesn't reduce its value. It's a prediction in the jargonic sense that an inference about future values of a time series is broadly called "prediction", and it's relevant for reasoning about LLMs because they are fundamentally limited to converting tokens sequences into next-token predictive distributions, and that bears on how they can do what they do and what their limitations are. Nothing about the training process changes that.