6 ms·
There is an excellent talk by Jack Rae called “compression for AGI”, where he shows (what I believe to be) a little known connection between transformers and co
by angusturner 1y ago
There is an excellent talk by Jack Rae called “compression for AGI”, where he shows (what I believe to be) a little known connection between transformers and compression;
In one view, you can view LLMs as SOTA lossless compression algorithms, where the number of weights don’t count towards the description length. Sounds crazy but it’s true.
- Workaccount2 1y agoA transformer that doesn't hallucinate (or knows what is a hallucination) would be the ultimate compression algorithm. But right now that isn't a solved problem, and it leaves the output of LLMs too untrustworthy to use over what are colloquially known as compression algorithms.
- Nevermark 1y agoIt is still task related. Compressing a comprehensive command line reference via model might introduce errors and drop some options. But for many people, especially new users, referencing commands, and getting examples, via a model would delivers many times the value. Lossy vs. lossless are fundamentally different, but so are use cases.
- swyx 1y agohis talk here https://www.youtube.com/watch?v=dO4TPJkeaaU https://www.youtube.com/watch?v=dO4TPJkeaaU and his last before departing for Meta Superintelligence https://www.youtube.com/live/U-fMsbY-kHY?si=_giVEZEF2NH3lgxI&t=1944 https://www.youtube.com/live/U-fMsbY-kHY?si=_giVEZEF2NH3lgxI...