5 ms·

Is there any way to do this with the frontier LLM's?

• red75prime 10mo ago
  Ask them to mark low confidence words.
  • seidleroni 10mo ago
    interesting... I'll give that a shot
  • akoboldfrying 10mo ago
    Do they actually have access to that info "in-band"? I would guess not. OTOH it should be straightforward for the LLM program to report this -- someone else commented that you can do this when running your own LLM locally, but I guess commercial providers have incentives not to make this info available.
    • red75prime 10mo ago
      Naturally, their "confidence" is represented as activations in layers close to output, so they might be able to use it. Research ([0], [1], [2], [3]) shows that results of prompting LLMs to express their confidence correlate with their accuracy. The models tend to be overconfident, but in my anecdotal experience the latest models are passably good at judging their own confidence. [0] https://ieeexplore.ieee.org/abstract/document/10832237 https://ieeexplore.ieee.org/abstract/document/10832237 [1] https://arxiv.org/abs/2412.14737 https://arxiv.org/abs/2412.14737 [2] https://arxiv.org/abs/2509.25532 https://arxiv.org/abs/2509.25532 [3] https://arxiv.org/abs/2510.10913 https://arxiv.org/abs/2510.10913
• criemen 10mo ago
  It used to be that the answer was logprobs, but it seems that is no longer available.