6 ms·
IMHO from guy working with heavily quantized models and trying to get results. I do think quantization at less than 4-bit requires more advanced mathematics tha
by nikhizzle 2mo ago
IMHO from guy working with heavily quantized models and trying to get results. I do think quantization at less than 4-bit requires more advanced mathematics than is usually used. See Google Scholar Babak Hassibi (founder of Prism ML). Generally am amazed about the resiliency of these new data structures to approximation.
- cubefox 2mo agoThere was also a NeurIPS paper which successfully trained 1-bit models directly without floating point weights or any quantization: https://arxiv.org/abs/2405.16339 https://arxiv.org/abs/2405.16339 I didn't see any further work in this direction however. The paper seems to have been mostly overlooked.
- adrian_b 2mo agoWhile what can be done with less bits per parameter may be interesting, this is not really the important number for an LLM. Much more important is the total amount of memory needed by an LLM, which can be decreased either by reducing the number of bits per parameter or the number of parameters. Arithmetic circuits are already cheap so for the inference cost the amount of memory transfers is more important. It is not clear yet which number of bits per parameter allows the minimization of the total memory requirement, at a given LLM quality.
- cubefox 2mo agoI thought the multiplications during training (backpropagation) are quite computationally expensive. The method from the paper above doesn't need any multiplications.