edwardjhu

7 ms·

1.
by edwardjhu 3y ago
Yes, the plan is to keep the original weights in VRAM and merge/unmerge LoRA weights on the fly. You can even cache a large library of LoRA ckpts in RAM. Yup, I am!
2.
by edwardjhu 3y ago
If latency is irrelevant, I don't think there is a strong practical reason to prefer one over another. (LoRA is more elegant in my biased opinion because you roughly recover finetuning with a large r.) In practice, you see one do a lit
3.
by edwardjhu 3y ago
Adapters are extra layers inserted between existing layers, so they can't be parallelized. LoRA reparametrizes the weight updates and is easily parallelized or merged with the original weights during inference. Also, if you let the ran
4.
by edwardjhu 3y ago
This paper tries something like that https://arxiv.org/pdf/2202.13914.pdf The gain isn't that significant. We don't understand what these low-rank updates represent, and they might not correspond to "ski
5.
by edwardjhu 3y ago
Yup. That's exactly what happened.
6.
by edwardjhu 3y ago
Good question! I came up with the name because the idea is best described as low-rank adaptation. I know very little about radio communication and didn't anticipate the visibility my repo has today :)
7.
by edwardjhu 3y ago
> Merged means you are modifying the model weights, which means you are stuck with that one model on that device (though, this usually applies for most implementations for the unmerged versions too). If one is careful with floating point
8.
by edwardjhu 3y ago
Hi! I'm the author of the repo. The insight is that we don't need to modify a lot of parameters to get a generally competent model to do well on specific tasks. When you have a linear layer with a weight matrix of dimension d_in x
9.
Finetuning GPT-3 175B efficiently with low-rank adaptation(twitter.com)
1 points by edwardjhu 5y ago|0 comments
10.
Wide Neural Networks Are Feature Learners, Not Kernel Machines(arxiv.org)
2 points by edwardjhu 6y ago|0 comments
11.
by edwardjhu 6y ago
The claim here is a bit misleading, as already pointed out by other comments, since the kernel is an evolving one that is essentially learned after seeing the data. Contrary to many related works that compare wide neural networks to kernel