9 ms·
How does the quantization happen? Are the weights preprocessed before loading the model?
by stri8ed 3y ago
How does the quantization happen? Are the weights preprocessed before loading the model?
- sebzim4500 3y agoYes, but to my knowledge it doesn't do any of the complicated optimization stuff that SOTA quantisation methods use. It basically is just doing a bunch of rounding. There are advantages to simplicity, after all.
- brucethemoose2 3y agoIts not so simple anymore, see https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684
- ggerganov 3y agoThe weights are preprocessed into integer quants combined with scaling factors in various configurations (4, 5, 8-bits and recently more exotic 2, 3 and 6-bit quants). At runtime, we use efficient SIMD implementations to perform the matrix multiplication at integer level, carefully optimizing for both compute and memory bandwidth. Similar strategies are applied when running GPU inference - using custom kernels for fast Matrix x Vector multiplications