6 ms·
The weights aren’t compressed. So there are interpretable redundancies in practice.
by canjobear 10d ago
The weights aren’t compressed. So there are interpretable redundancies in practice.
- rhelz 10d agoIf the weights arn't compressed, then a smaller set of weights would perform as well. Sure, you can always induce as much symmetry and patterns as you want by bloating the data set, but that hardly gives us insight into how a set of weights which is "as full as it can be" of information.
- canjobear 10d agoThe point of TFA is that there are regularities you can exploit in the actually existing weights of machine learning systems, not in some hypothetically maximally efficient weights. The maximally efficient weights would indeed have no structure, but that’s not what anyone is working with.