5 ms·

Topk sampling with temp = 0 should be pretty much deterministic (ignoring floating-point errors)

by randomgermanguy 10mo ago

AJRF 10mo ago
> Ignoring floating point errors. I think you mean non-associativity. And you can’t ignore that.
- sunrunner 10mo ago
  Ignoring floating point errors, assuming a perfectly spherical cow, and taking air resistance as zero.
  - AJRF 10mo ago
    Imagine you are predicting the next token, you have two tokens very close in probability in the distribution, kernel execution is not deterministic because of floating point non-associativity - the token that gets predicted impacts the tokens later in the prediction stream - so it's very consequential which one gets picked. This isn't some hypothetical - it happens all the time with LLM's - it isn't some freak accident that isn't probable
    - randomgermanguy 10mo ago
      Okay yes, but would you really say that the main part of non-determinism in LLM-usage stems from this ? No its obviously the topk sampling. I don't think my tech-lead was trying to suggest the floating-point error/non-associativity was the real source.
      - AJRF 10mo ago
        > Would you really say that the main part of non-determinism in LLM-usage stems from this Yes I would because it causes exponential divergence (P(correct) = (1-e)^n) and doesn't have a widely adopted solution. The major labs have very expensive researchers focused on this specific problem. There is a paper from Thinking Machines from September around Batch Invariant kernels you should read, it's a good primer on this issue of non-determinism in LLM's, you might learn something from it! Unfortunately the method has quite a lot of overhead, but promising research all the same.
        randomgermanguy 10mo ago
        Alright fair enough. I dont think this is relevant to the main-point, but it's definitely something I wasn't aware of. I would've thought it might have an impact on like O(100)th token in some negligible way, but glad to learn.
- deleted 10mo ago
  [deleted]

Topk sampling with temp = 0 should be pretty much deterministic (ignoring floating-point errors)

AJRF 10mo ago

> Ignoring floating point errors. I think you mean non-associativity. And you can’t ignore that.

sunrunner 10mo ago
Ignoring floating point errors, assuming a perfectly spherical cow, and taking air resistance as zero.
- AJRF 10mo ago
  Imagine you are predicting the next token, you have two tokens very close in probability in the distribution, kernel execution is not deterministic because of floating point non-associativity - the token that gets predicted impacts the tokens later in the prediction stream - so it's very consequential which one gets picked. This isn't some hypothetical - it happens all the time with LLM's - it isn't some freak accident that isn't probable
  - randomgermanguy 10mo ago
    Okay yes, but would you really say that the main part of non-determinism in LLM-usage stems from this ? No its obviously the topk sampling. I don't think my tech-lead was trying to suggest the floating-point error/non-associativity was the real source.
    - AJRF 10mo ago
      > Would you really say that the main part of non-determinism in LLM-usage stems from this Yes I would because it causes exponential divergence (P(correct) = (1-e)^n) and doesn't have a widely adopted solution. The major labs have very expensive researchers focused on this specific problem. There is a paper from Thinking Machines from September around Batch Invariant kernels you should read, it's a good primer on this issue of non-determinism in LLM's, you might learn something from it! Unfortunately the method has quite a lot of overhead, but promising research all the same.
      - randomgermanguy 10mo ago
        Alright fair enough. I dont think this is relevant to the main-point, but it's definitely something I wasn't aware of. I would've thought it might have an impact on like O(100)th token in some negligible way, but glad to learn.
deleted 10mo ago
[deleted]