6 ms·
The policy is not learned token by token during RLHF and RLVR. The reward model doesn't score token by token.
by danielmarkbruce 11d ago
The policy is not learned token by token during RLHF and RLVR. The reward model doesn't score token by token.
- gwerbin 10d agoSo? tokens are emitted one at a time according to the output distribution & sampler algorithm, and the next token distribution is a function of the preceding token sequence only. The process by which the output distribution is shaped doesn't change the core mental model, and doesn't reduce its value. It's a prediction in the jargonic sense that an inference about future values of a time series is broadly called "prediction", and it's relevant for reasoning about LLMs because they are fundamentally limited to converting tokens sequences into next-token predictive distributions, and that bears on how they can do what they do and what their limitations are. Nothing about the training process changes that.