5 ms·

I think so, specifically lossy compression though. A modern version of the book would include an extra section in the 'Lossy compression' chapter - 'Text' (alo

by briansm 3mo ago

I think so, specifically lossy compression though. A modern version of the book would include an extra section in the 'Lossy compression' chapter - 'Text' (alongside Images/Video/Audio) that would discuss LLM's.

eru 3mo ago
No, it's not for lossy compression only. An LLM can give you a probability distribution for the next token. You can pair that with arithmetic coding to get a lossless compression/decompression algorithm. See https://en.wikipedia.org/wiki/Arithmetic_coding https://en.wikipedia.org/wiki/Arithmetic_coding
- adrian_b 3mo ago
  In the way that you say, you can do lossless data compression, but then the LLM is used in a very distinct way than it is used in applications like chat or coding assistance. In the latter applications, you do queries which aim to extract information from the training data set, but which may return hallucinated content instead of correct content. If you use an LLM just to provide an estimation for the frequencies of tokens in an input data stream, and then you use the estimated frequencies to encode the input data, then you do not care about which were the tokens predicted by the LLM, because they are not used. The worst effect of any wrong predictions by the LLM is a slightly worse data compression ratio than the optimum. When it is said that LLMs do a lossy data compression, that refers to the compression from the training data set to sequences of output tokens.
  - eru 3mo ago
    > If you use an LLM just to provide an estimation for the frequencies of tokens in an input data stream, [...] Why would you use an LLM for that? The whole point is to encode contextual probabilities. So basically: given this prefix of text, what's are the probabilities for next tokens? You can use this conditional probability distribution to sample from to create plausible text, or you can use it for lossless compression. The math is very similar.

by briansm 3mo ago

eru 3mo ago

No, it's not for lossy compression only. An LLM can give you a probability distribution for the next token. You can pair that with arithmetic coding to get a lossless compression/decompression algorithm. See https://en.wikipedia.org/wiki/Arithmetic_coding https://en.wikipedia.org/wiki/Arithmetic_coding

adrian_b 3mo ago
In the way that you say, you can do lossless data compression, but then the LLM is used in a very distinct way than it is used in applications like chat or coding assistance. In the latter applications, you do queries which aim to extract information from the training data set, but which may return hallucinated content instead of correct content. If you use an LLM just to provide an estimation for the frequencies of tokens in an input data stream, and then you use the estimated frequencies to encode the input data, then you do not care about which were the tokens predicted by the LLM, because they are not used. The worst effect of any wrong predictions by the LLM is a slightly worse data compression ratio than the optimum. When it is said that LLMs do a lossy data compression, that refers to the compression from the training data set to sequences of output tokens.
- eru 3mo ago
  > If you use an LLM just to provide an estimation for the frequencies of tokens in an input data stream, [...] Why would you use an LLM for that? The whole point is to encode contextual probabilities. So basically: given this prefix of text, what's are the probabilities for next tokens? You can use this conditional probability distribution to sample from to create plausible text, or you can use it for lossless compression. The math is very similar.