7 ms·

Here is one question I have not seen answered yet: All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16 and then fur

Here is one question I have not seen answered yet: All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16 and then further quantizing to int4. Clearly this quantized model still outputs something resembling human language, at the very least. But I haven't seen anyone show what effect this quantizing has on the quality of the output. If the quality of the output is bad, it's unclear if it's because the model needs to be finetuned (as Stanford did here) or if it's because the quanitizing reduced the quality, or both. If this fine-tuned Stanford model still has excellent output after quantizing it to run on a Raspberry Pi 4GB, that would be awesome!

gamegoblin 4y ago
Decrease in accuracy is negligible and decreases as model size increases. That is, larger models quantize even better than smaller models. https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323
- byteknight 4y ago
  Is this because averages are weighed less (less sensitive) as the total sample size increases?
  - londons_explore 4y ago
    Yes. In a dense everything to everything neural network layer, the number of 'inputs' to a node is proportional to the square root of the number of weights. Therefore, assuming quantization noise is uncorrelated, as the number of weights doubles, the number of inputs goes up by sqrt(2), and the (normalized) noise goes down by a factor of 2*(sqrt(2)). So, as a rule of thumb, you can remove 1 bit of precision of the weights for every 4x increase in the number of weights. All this assumes weights and activations are uncorrelated random variables - which may not hold true.
    - londons_explore 4y ago
      Something is wrong with this math... by your logic I could scale the network up big enough that I could quantize the weights down to zero bits...
      - Robotbeat 4y ago
        Rules of thumb typically are just first order approximations which by definition are not guaranteed to hold far from their point of interest (or point of tangency).
      - sebzim4500 4y ago
        Having fewer than 1 bit per weight is not absurd. E.g. you can use 2 bits to represent 3 'weights' if you insist that at most one of the weights is allowed to exist. If you try to order nodes so that adjacent nodes are uncorrelated the performance loss might be manageable. People are already doing stuff like this (see sparsification) so it is conceivable to me that this is just what networks will look like in a few years.
        magicalhippo 4y ago
        > If you try to order nodes so that adjacent nodes are uncorrelated the performance loss might be manageable. shower thought In graphics we use barycentric coordinates to encode the position within an arbitrary triangle using two coordinates (u,v), with the third being constrained to be 1-u-v. If you order nodes to be correlated, could you use a similar trick to encode three weights for the price of two?
        hgsgm 4y ago
        Yes, it's the same thing.
- deleted 4y ago
  [deleted]
dmw_ng 4y ago
For 10 billion+ parameter models, the effects of quantization are relatively small, for smaller models like Llama 7B the effect becomes more dramatic, but there is ongoing research on new quantization methods (like GPTQ) that preserve significant performance even on the lower end. Quantization isn't the only technique available for downsizing a model, Llama itself is already the result of sizing the model and input data according to "Chinchilla optimality", a very recent (as in 2022) result that e.g. GPT-3 predates. The result is that Llama-13B performs in benchmarks similarly with GPT-3 175B despite the tremendous size difference. There are separately also a variety of pruning methods to further eliminate inactive weights present in the trained model (I think this is also active research) Finally even on something like a Raspberry Pi, implementations for inference (like llama.cpp) are nowhere near mature yet. There are already a multitude of runtimes available for inference making large tradeoffs between performance and flexibility (e.g. many models running on PyTorch vs ONNX report 5-10x speedups running under ONNX) I think the really exciting part of Alpaca is the size and budget of the team - 5 students with $100 scraping OpenAI put this model together in a couple of hours of training. Any notions of premium persisting in the AI space for much longer seem fantastic at best, for all intents and purposes it has already been commoditized. And that's scary considering the size of the dent ChatGPT has put in my Google traffic
- koheripbal 4y ago
  The difference is small, UNTIL you get to 4 bit quantization, where the model is noticeably dumber. 8 bits, imo, is the minimum.
  - magicalhippo 4y ago
    Some parameters would be more sensitive than others I suppose? So could you use 4 bits for most, and 8 bits, or even 16, for the remaining?
    - sour-taste 4y ago
      I know nothing about this so my opinion means little, but I imagine it's hard to know which parameters are important enough to use more bits for. I do wonder if it would be possible to have the model determine during training how important each parameter is, while maybe rewarding it for having more small parameters?
gpm 4y ago
> All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16 LLaMa weights are f16s to start out with, no lowering necessary to get to there. You can stream weights from RAM to the GPU pretty efficiently. If you have >= 32GB ram and >=2GB vram my code here should work for you: https://github.com/gmorenz/llama/tree/gpu_offload https://github.com/gmorenz/llama/tree/gpu_offload There's probably a cleaner version of it somewhere else. Really you should only need >= 16 GB ram, but the (meta provided) code to load the initial weights is completely unnecessarily making two copies of the weights in RAM simultaneously. You could also lower vram requirements a bit more with a bit more work (I just made the smallest change possible to make it work)
- nextaccountic 4y ago
  > the (meta provided) code to load the initial weights is completely unnecessarily making two copies of the weights in RAM simultaneously This is the kind of thing that the stable diffusion community optimized the shit out
underlines 4y ago
See: https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 Q: Doesn't 4bit have worsen output performance than 8bit or 16bit? A: GPTQ doesn't quantize linearly. While RTN 8bit does reduce output quality, GPTQ 4bit has effectively little output quality loss compared to baseline uncompressed fp16. https://i.imgur.com/xmaNNDd.png https://i.imgur.com/xmaNNDd.png https://i.imgur.com/xmaNNDd.png https://i.imgur.com/xmaNNDd.png
- bippingchip 4y ago
  This is really interesting, thank you for the reference! Having worked more with images based NN than language models before, I wonder: are LLM inherently more suited to aggressive quantisation, due to their very large size? I see people suggesting here 4b is pretty good, and 3b should be the target. I remember ResNets etc can of course also be quantized, and up to 8-6b you get pretty good results with very little effort, with low-ish degradation in performance. Trying to go down to 4b is more challenging, though this paper claims with quantisation aware training 4b is possible indeed, but that means a lot of dedicate training compute needed to get to 4b (not just finetuning post-training): https://arxiv.org/abs/2105.03536 https://arxiv.org/abs/2105.03536

gamegoblin 4y ago

Decrease in accuracy is negligible and decreases as model size increases. That is, larger models quantize even better than smaller models. https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323

byteknight 4y ago
Is this because averages are weighed less (less sensitive) as the total sample size increases?
- londons_explore 4y ago
  Yes. In a dense everything to everything neural network layer, the number of 'inputs' to a node is proportional to the square root of the number of weights. Therefore, assuming quantization noise is uncorrelated, as the number of weights doubles, the number of inputs goes up by sqrt(2), and the (normalized) noise goes down by a factor of 2*(sqrt(2)). So, as a rule of thumb, you can remove 1 bit of precision of the weights for every 4x increase in the number of weights. All this assumes weights and activations are uncorrelated random variables - which may not hold true.
  - londons_explore 4y ago
    Something is wrong with this math... by your logic I could scale the network up big enough that I could quantize the weights down to zero bits...
    - Robotbeat 4y ago
      Rules of thumb typically are just first order approximations which by definition are not guaranteed to hold far from their point of interest (or point of tangency).
    - sebzim4500 4y ago
      Having fewer than 1 bit per weight is not absurd. E.g. you can use 2 bits to represent 3 'weights' if you insist that at most one of the weights is allowed to exist. If you try to order nodes so that adjacent nodes are uncorrelated the performance loss might be manageable. People are already doing stuff like this (see sparsification) so it is conceivable to me that this is just what networks will look like in a few years.
      - magicalhippo 4y ago
        > If you try to order nodes so that adjacent nodes are uncorrelated the performance loss might be manageable. shower thought In graphics we use barycentric coordinates to encode the position within an arbitrary triangle using two coordinates (u,v), with the third being constrained to be 1-u-v. If you order nodes to be correlated, could you use a similar trick to encode three weights for the price of two?
        hgsgm 4y ago
        Yes, it's the same thing.
deleted 4y ago
[deleted]

dmw_ng 4y ago

For 10 billion+ parameter models, the effects of quantization are relatively small, for smaller models like Llama 7B the effect becomes more dramatic, but there is ongoing research on new quantization methods (like GPTQ) that preserve significant performance even on the lower end. Quantization isn't the only technique available for downsizing a model, Llama itself is already the result of sizing the model and input data according to "Chinchilla optimality", a very recent (as in 2022) result that e.g. GPT-3 predates. The result is that Llama-13B performs in benchmarks similarly with GPT-3 175B despite the tremendous size difference. There are separately also a variety of pruning methods to further eliminate inactive weights present in the trained model (I think this is also active research) Finally even on something like a Raspberry Pi, implementations for inference (like llama.cpp) are nowhere near mature yet. There are already a multitude of runtimes available for inference making large tradeoffs between performance and flexibility (e.g. many models running on PyTorch vs ONNX report 5-10x speedups running under ONNX) I think the really exciting part of Alpaca is the size and budget of the team - 5 students with $100 scraping OpenAI put this model together in a couple of hours of training. Any notions of premium persisting in the AI space for much longer seem fantastic at best, for all intents and purposes it has already been commoditized. And that's scary considering the size of the dent ChatGPT has put in my Google traffic

koheripbal 4y ago
The difference is small, UNTIL you get to 4 bit quantization, where the model is noticeably dumber. 8 bits, imo, is the minimum.
- magicalhippo 4y ago
  Some parameters would be more sensitive than others I suppose? So could you use 4 bits for most, and 8 bits, or even 16, for the remaining?
  - sour-taste 4y ago
    I know nothing about this so my opinion means little, but I imagine it's hard to know which parameters are important enough to use more bits for. I do wonder if it would be possible to have the model determine during training how important each parameter is, while maybe rewarding it for having more small parameters?

gpm 4y ago

> All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16 LLaMa weights are f16s to start out with, no lowering necessary to get to there. You can stream weights from RAM to the GPU pretty efficiently. If you have >= 32GB ram and >=2GB vram my code here should work for you: https://github.com/gmorenz/llama/tree/gpu_offload https://github.com/gmorenz/llama/tree/gpu_offload There's probably a cleaner version of it somewhere else. Really you should only need >= 16 GB ram, but the (meta provided) code to load the initial weights is completely unnecessarily making two copies of the weights in RAM simultaneously. You could also lower vram requirements a bit more with a bit more work (I just made the smallest change possible to make it work)

nextaccountic 4y ago
> the (meta provided) code to load the initial weights is completely unnecessarily making two copies of the weights in RAM simultaneously This is the kind of thing that the stable diffusion community optimized the shit out

underlines 4y ago

See: https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 Q: Doesn't 4bit have worsen output performance than 8bit or 16bit? A: GPTQ doesn't quantize linearly. While RTN 8bit does reduce output quality, GPTQ 4bit has effectively little output quality loss compared to baseline uncompressed fp16. https://i.imgur.com/xmaNNDd.png https://i.imgur.com/xmaNNDd.png https://i.imgur.com/xmaNNDd.png https://i.imgur.com/xmaNNDd.png

bippingchip 4y ago
This is really interesting, thank you for the reference! Having worked more with images based NN than language models before, I wonder: are LLM inherently more suited to aggressive quantisation, due to their very large size? I see people suggesting here 4b is pretty good, and 3b should be the target. I remember ResNets etc can of course also be quantized, and up to 8-6b you get pretty good results with very little effort, with low-ish degradation in performance. Trying to go down to 4b is more challenging, though this paper claims with quantisation aware training 4b is possible indeed, but that means a lot of dedicate training compute needed to get to 4b (not just finetuning post-training): https://arxiv.org/abs/2105.03536 https://arxiv.org/abs/2105.03536