5 ms·

To be more exact, LoRA adds two matrices `A` and `B` to any layers that contain trainable weights. The original weights (`W_0`) have the shape `d × k`. These ar

To be more exact, LoRA adds two matrices `A` and `B` to any layers that contain trainable weights. The original weights (`W_0`) have the shape `d × k`. These are frozen. Matrix `A` has dimensions `d × <rank>` (`rank` is configurable) and matrix `B` has the shape `<rank> × k`. A and B are then multiplied and added to `W_0` to get altered weights. The benefit here is that the extra matrices are small compared to `W_0`, which means less parameters need to be optimized, so less activations need to be stored in memory.

grph123dot 3y ago
Your explanation is crystal clear. I suppose it works well in practice, but is there any reason it works that well?
- stu2b50 3y ago
  Per the original paper, empirically it’s been found that neural network weights often have low intrinsic rank. It follows, then, that the change in the weights as you train also have low intrinsic rank, which means that you should be able represent them with a lower rank matrix.
  - grph123dot 3y ago
    Since we are in ELI5, it seems that the concept of low rank approximation is required to understand this method. (1) https://en.wikipedia.org/wiki/Low-rank_approximation https://en.wikipedia.org/wiki/Low-rank_approximation Edited: By the way, it seems to me that there is an error in the wikipedia page because if the Low-rank approximation takes a larger rank then the bound of the error should decrease, and in this page the error increases.
  - grph123dot 3y ago
    >> that the change in the weights as you train also have low intrinsic rank It seems that the initial matrix of weights has a low rank approximation A and this implies that the difference E = W - A is small, also it seems that PCA fails when E is sparse because PCA is designed to be optimum when the error is gaussian.
    - stu2b50 3y ago
      In terms of PCA, PCA is also quite expensive computationally. Additionally, you'd probably have to do SVD instead. Since the weights are derived from gradient descent, yeah we don't really know what the distributions would be. A random projection empirically works quite well for very high dimensions, and is of course very cheap computationally.
  - seydor 3y ago
    Does this mean the matrices are highly compressible?
    - loxias 3y ago
      kinda/yes. To translate to more intuitive concepts: the matrices don't contain much variance in as many degrees of freedom as they could. Think of a point cloud of a piece of paper floating in the wind. It would be a 3xn list of points, but "really" it's a 2d piece of paper. Just like I can rewrite the number 27 as 333 or 8+19 or (2^3)+(2^4)+3.. Given a single matrix one can find myriad ways to rewrite it as a sequence of matrices that have the same (or similar) numeric value, but with interesting or desirable properties. :D My favorite example (which is used in signal processing) is to take your ugly matrix and rewrite it as a set of smaller matrices where most of the elements are zero, or a power of 2. It turns out, computers can multiply by zeros and powers of two very fast
twic 3y ago
Ah, so the resulting model contains both the large matrix of original weights, and also the two small matrices of alterations? But this is smaller than the alternative of a model which contains the large matrix of original weights, and an equally large matrix of alterations. Why is fine-tuning done with separate alterations, rather than by mutating the original weights?
- stu2b50 3y ago
  > But this is smaller than the alternative of a model which contains the large matrix of original weights, and an equally large matrix of alterations. It's actually larger. If you just have two equally large matrices of the same dimension, one original, and one of "altercations"... then you can just add them together. > Why is fine-tuning done with separate alterations, rather than by mutating the original weights? Then you'd have to compute the gradients for the whole network, which is very expensive when the model has 7b, 65b, 165b parameters. The intent is to make that cheaper by only computing gradients for a low rank representation of the change in the weight matrix from training.
  - gliptic 3y ago
    Correct me if I'm wrong, but I think you still need to compute gradients of non-trained weights in order to compute the gradients of the LoRA weights. What you don't have to do is store and update the optimizer state for all those non-trained weights.
    - stu2b50 3y ago
      I mean the derivative of a constant is 0. So if all of the original weights are considered constants, then computing their gradients is trivial, since they’re just zero.
      - jprafael 3y ago
        Computing gradients is easy/cheap. What this technique solves is that you no longer need to store the computed values of the gradient until the backpropagation phase, which saves on expensive GPU RAM, allowing you to use commodity hardware.
seydor 3y ago
Can rank decomposition be used to reduce the original weight matrices as well? Or are they assumed to be compressed already?
- metanonsense 3y ago
  Those fully trained networks are usually considered full-rank. At least that is what they say in the LoRA paper.

grph123dot 3y ago

Your explanation is crystal clear. I suppose it works well in practice, but is there any reason it works that well?

stu2b50 3y ago
Per the original paper, empirically it’s been found that neural network weights often have low intrinsic rank. It follows, then, that the change in the weights as you train also have low intrinsic rank, which means that you should be able represent them with a lower rank matrix.
- grph123dot 3y ago
  Since we are in ELI5, it seems that the concept of low rank approximation is required to understand this method. (1) https://en.wikipedia.org/wiki/Low-rank_approximation https://en.wikipedia.org/wiki/Low-rank_approximation Edited: By the way, it seems to me that there is an error in the wikipedia page because if the Low-rank approximation takes a larger rank then the bound of the error should decrease, and in this page the error increases.
- grph123dot 3y ago
  >> that the change in the weights as you train also have low intrinsic rank It seems that the initial matrix of weights has a low rank approximation A and this implies that the difference E = W - A is small, also it seems that PCA fails when E is sparse because PCA is designed to be optimum when the error is gaussian.
  - stu2b50 3y ago
    In terms of PCA, PCA is also quite expensive computationally. Additionally, you'd probably have to do SVD instead. Since the weights are derived from gradient descent, yeah we don't really know what the distributions would be. A random projection empirically works quite well for very high dimensions, and is of course very cheap computationally.
- seydor 3y ago
  Does this mean the matrices are highly compressible?
  - loxias 3y ago
    kinda/yes. To translate to more intuitive concepts: the matrices don't contain much variance in as many degrees of freedom as they could. Think of a point cloud of a piece of paper floating in the wind. It would be a 3xn list of points, but "really" it's a 2d piece of paper. Just like I can rewrite the number 27 as 333 or 8+19 or (2^3)+(2^4)+3.. Given a single matrix one can find myriad ways to rewrite it as a sequence of matrices that have the same (or similar) numeric value, but with interesting or desirable properties. :D My favorite example (which is used in signal processing) is to take your ugly matrix and rewrite it as a set of smaller matrices where most of the elements are zero, or a power of 2. It turns out, computers can multiply by zeros and powers of two very fast

twic 3y ago

Ah, so the resulting model contains both the large matrix of original weights, and also the two small matrices of alterations? But this is smaller than the alternative of a model which contains the large matrix of original weights, and an equally large matrix of alterations. Why is fine-tuning done with separate alterations, rather than by mutating the original weights?

stu2b50 3y ago
> But this is smaller than the alternative of a model which contains the large matrix of original weights, and an equally large matrix of alterations. It's actually larger. If you just have two equally large matrices of the same dimension, one original, and one of "altercations"... then you can just add them together. > Why is fine-tuning done with separate alterations, rather than by mutating the original weights? Then you'd have to compute the gradients for the whole network, which is very expensive when the model has 7b, 65b, 165b parameters. The intent is to make that cheaper by only computing gradients for a low rank representation of the change in the weight matrix from training.
- gliptic 3y ago
  Correct me if I'm wrong, but I think you still need to compute gradients of non-trained weights in order to compute the gradients of the LoRA weights. What you don't have to do is store and update the optimizer state for all those non-trained weights.
  - stu2b50 3y ago
    I mean the derivative of a constant is 0. So if all of the original weights are considered constants, then computing their gradients is trivial, since they’re just zero.
    - jprafael 3y ago
      Computing gradients is easy/cheap. What this technique solves is that you no longer need to store the computed values of the gradient until the backpropagation phase, which saves on expensive GPU RAM, allowing you to use commodity hardware.

seydor 3y ago

Can rank decomposition be used to reduce the original weight matrices as well? Or are they assumed to be compressed already?

metanonsense 3y ago
Those fully trained networks are usually considered full-rank. At least that is what they say in the LoRA paper.