6 ms·
GPT is a very impressive technical achievement. But that technical achievement is more in the field of compression rather than intelligence.
by dcolkitt 3y ago
GPT is a very impressive technical achievement. But that technical achievement is more in the field of compression rather than intelligence.
- sebzim4500 3y agoI don't think compression and intelligence can be disentangled in this way.
- cjbprime 3y agoAgree: compression is straightforwardly prediction. Prediction is intelligence when you're able to do it well across disparate novel tasks. (Of course definitions are idiosyncratic, but I'm curious if anyone disagrees with these ones.)
- cubefox 3y agoIt seems that intelligence also is about explanation, apart from prediction. Humans not just try to predict future evidence from current evidence, or use the current evidence to confirm given hypotheses, but they also try to find an hypothesis which best explains this evidence. It's not quite clear how explanation would relate to compression.
- kfarr 3y agoExplanation is decompression, using a limited set of tokens to model a more complex output
- IIAOPSW 3y agoExplanation is when the number of bits in your model is smaller than the number of bits in the system. To understand is to have a compression good enough to store in your working memory.
- Matumio 3y ago> Prediction is intelligence Depends on your definition of "intelligence". The big missing part is the ability to explore, try new things, to act (enactivism). Basically to become part of the environment, instead of being a sealed box with frozen weights. By predicting characters, the system had to master, digest, maybe even understand, all the cultural human knowledge it got in text form. Now let's aim for the process that generated this knowledge in the first place.