8 ms·

Does an LLM have any idea of what "best" is? I think it doesn't, and just predicts the range of most statistically likely next tokens based on its training dat

Does an LLM have any idea of what "best" is? I think it doesn't, and just predicts the range of most statistically likely next tokens based on its training data, and picks one of those.

greggoB 14d ago
In ML, "best" is what minimises the loss function, so in a certain (very basic) sense it does.
- eru 13d ago
  You are mixing up levels. When you are asking it a question (like which of these two texts is the best), the output is also just picked by minimising that loss function. There's no guarantee that answering "Text B is better" aligns with text B minimising the loss function. (And they aren't really minimising loss functions during inference. They sample from a distribution. During training they minimise the loss function of the distribution.)
  - greggoB 13d ago
    Point taken, I was possibly overly curt in my response, leading to it being ambiguous, both re when the loss function is being minimised and that it is a stochastic process. So to OPs question: I guess LLMs do have an "idea" of what is best (conditioned on minimising a loss function during training), however they may not always output that (because stochasticity), which maybe represents a degree of uncertainty in that "idea"?
    - eru 13d ago
      My point is that when you asked them to 'judge' two texts, they are just going to output the tokens that have the highest probability (or a sample from that distribution). But depending on their training data, these tokens might say 'correct horse battery staple', and not necessarily 'text A is better' or 'text B is better'. Even if text A would have been more likely to be produced by the LLM.
dspillett 14d ago
> Does an LLM have any idea of what "best" is? "Best" here isn't being used to imply a conscious decision, but at each stage which token has the best score coming out of the model, so the overall best is the sequence of those best tokens. When judging another output it is essentially running the numbers the same way. It is a bit more complicated than that as the output tokens become part of the context for the next choice, but I think that simplified way of thinking about it holds water.

Does an LLM have any idea of what "best" is? I think it doesn't, and just predicts the range of most statistically likely next tokens based on its training data, and picks one of those.

greggoB 14d ago

In ML, "best" is what minimises the loss function, so in a certain (very basic) sense it does.

eru 13d ago
You are mixing up levels. When you are asking it a question (like which of these two texts is the best), the output is also just picked by minimising that loss function. There's no guarantee that answering "Text B is better" aligns with text B minimising the loss function. (And they aren't really minimising loss functions during inference. They sample from a distribution. During training they minimise the loss function of the distribution.)
- greggoB 13d ago
  Point taken, I was possibly overly curt in my response, leading to it being ambiguous, both re when the loss function is being minimised and that it is a stochastic process. So to OPs question: I guess LLMs do have an "idea" of what is best (conditioned on minimising a loss function during training), however they may not always output that (because stochasticity), which maybe represents a degree of uncertainty in that "idea"?
  - eru 13d ago
    My point is that when you asked them to 'judge' two texts, they are just going to output the tokens that have the highest probability (or a sample from that distribution). But depending on their training data, these tokens might say 'correct horse battery staple', and not necessarily 'text A is better' or 'text B is better'. Even if text A would have been more likely to be produced by the LLM.

dspillett 14d ago

> Does an LLM have any idea of what "best" is? "Best" here isn't being used to imply a conscious decision, but at each stage which token has the best score coming out of the model, so the overall best is the sequence of those best tokens. When judging another output it is essentially running the numbers the same way. It is a bit more complicated than that as the output tokens become part of the context for the next choice, but I think that simplified way of thinking about it holds water.