5 ms·
Has anyone figured out a way to fine tune this with 24gb of vram? I have tried with deepspeed etc but no luck. Seems to be just out of reach for fine tuning req
by fbodz 3y ago
Has anyone figured out a way to fine tune this with 24gb of vram? I have tried with deepspeed etc but no luck. Seems to be just out of reach for fine tuning requiring 26gb.
- csdvrx 3y agoHave you tried quantization? It's often a cheap and simple way to reduce the VRAM requirements. What hardware are you using? (CPU,RAM,GPU,VRAM) Have you considered using llama.cpp for a mixed CPU+GPU use (if you have enough RAM)
- fbodz 3y agoYeah I am using the default training script with int8 quantisation. It uses peft with lora but this still requires 26gb
- freeqaz 3y agoIs it possible to offload some layers to CPU and still train in a reasonable amount of time?
- generalizations 3y agoThere’s also that pruning tool that was on hn in the last couple weeks. It seemed to work really well on the larger models, and could reduce size by 30-50%
- int_19h 3y agoI'm not sure about this model specifically, but training with 4-bit quantization has been a thing with LLaMA for a while now, although the setup involves manual hacks of various libraries.
- nl 3y agoYou probably don't want to fine-tune a quantized model. They are fine for inference but not great for training.
- mirekrusin 3y agoPeople should be training model sizes that fit-and-fill consumer GPUs, ie: 2x 24G - for dual GPU ~ 28B model 1x 24G ~ 14B model etc.