mobicham

10 ms·

1.
by mobicham 2y ago
The extreme quant buys you potentially 70x more efficient matmul via binary/ternary operations. You still have a group-size of 64 in 4-bit fyi.And even if you keep the meta-data on-device, provided that the quality is high (which is th
2.
by mobicham 2y ago
Yes, correct, and that fetching operation is a non-blocking operation, once we dequantize the weights we discard it before moving to the next layer. Technically, you can do it for the weights as well. But that wouldn't work in many sit
3.
by mobicham 2y ago
Hey Daniel! The VRAM is still the same as a pure n-bit model would take. Because we only need meta-data for a single nn.Linear at a time, you only need an additional (3GB-1.7GB)/224 = 5.8MB. If we compress the meta-data as well that wo
4.
by mobicham 2y ago
Sure, I can give you detailed answers: 1- The answer is still ~1.7GB. You only need meta-data of a single nn.Linear at a time. There are 32x(4+3) = 224 layers quantized, so you need an additional (3GB - 1.7GB)/224 = 1.3GB/224 ~ 5.
5.
by mobicham 2y ago
That's correct. Only the dequantization is done on CUDA, the matmul is done with Pytorch. If they put their kernels open-source we could re-use them!
6.
by mobicham 2y ago
Thank you, very glad to hear that!
7.
by mobicham 2y ago
At least you can copy 16 times more data to the shared memory with binary weights.
8.
by mobicham 2y ago
LoRA training should benefit from the same speed-up, because the 1-bit weights will be frozen and all you need for both the forward and backward pass is a binary matmul, then maybe cast after to get more stable gradients.
9.
by mobicham 2y ago
Hello, I am the main author, would love to clarify a couple of things: All the linear-quantization methods have meta-data, including the 1.58bit paper. You can control the quality vs. memory usage by reducing the group-size. However, the me
10.
Half-Quadratic Quantization of Large Machine Learning Models(mobiusml.github.io)
2 points by mobicham 3y ago|1 comments
11.
by mobicham 3y ago
Very excited to share our latest work on model quantization. No data calibration needed, extremely fast , works on both language and vision models! Code: https://lnkd.in/dM_NgSCQ Models: https://lnkd.in/dyw