5 ms·
A simple guide to fine-tuning Llama 2
- marcopicentini 3y agoAnyone has calculate the break even point (as number of token per month) between self-hosted LLAMA and OpenAI GPT-3.5 API?
- syntaxing 3y agoCan someone share a good tutorial how to prepare the data? And for fine tuning, does a 3090 have enough VRAM? I want to do what the author mentioned by fine tuning the model on my personal data but I’m not sure how to prepare the data. I tried using vector search + LLM but I find the results very subpar when using a local LLM.
- samlhuillier 3y agoWorking on this now!
- syntaxing 3y agoI’m looking forward to this! Are you using an adapter (I don’t see it mentioned in your article)? I was under the impression you cannot fit 7B at 4 bit since it’ll take 25GB of VRAM or so.
- samlhuillier 3y agoYes using the qlora adapter that hugging face provides with peft
- syntaxing 3y agoahh, I was on my phone before so I must of glimpsed over it, I see it on the last section. Thanks!
- jawerty 3y agoI just streamed this last night https://m.youtube.com/watch?v=TYgtG2Th6fI&t=3998s https://m.youtube.com/watch?v=TYgtG2Th6fI&t=3998s I’ve been live streaming myself fine tuning llama on my GitHub data (to code like me)
- jeremycarter 3y agoFantastic job! Very easy to follow
- jawerty 3y agoThank you! I have some other streams where I do little projects like these check them out
- notpublic 3y agoAs mentioned in the OP's blog post, checkout https://github.com/facebookresearch/llama-recipes.git https://github.com/facebookresearch/llama-recipes.git. specifically files in ft_datasets directory. I am able to finetune meta-llama/Llama-2-13b-chat-hf on a 3090 using instructions from quickstart.ipynb.
- syntaxing 3y agoOh interesting, I didn’t know the documentation expanded so much in this past month.
- eachro 3y agoI've veen a bit out of the loop on this area but would like to get back into it given how much has changed in the LLM landscape in the last 1-2 yrs. What models are small enough to play with on Collab? Or am I going to have to spin up my own gpu box on aws to be able to mess around with these models?
- naderkhalil 3y agoHey, you could use a template on brev.dev to spin up a gpu box with the model and Jupyter notebook. Alternatively, the falcon 7b model should be small enough for colab
- deleted 3y ago[deleted]
- nmitchko 3y agoThis is a pretty useless post. You could also follow the same 1000x tutorials about llama and use the already uploaded hugging face formats that are on hugging face... Here are some actually useful links https://blog.ovhcloud.com/fine-tuning-llama-2-models-using-a-single-gpu-qlora-and-ai-notebooks/ https://blog.ovhcloud.com/fine-tuning-llama-2-models-using-a... https://huggingface.co/meta-llama/Llama-2-70b-hf https://huggingface.co/meta-llama/Llama-2-70b-hf https://huggingface.co/meta-llama/Llama-2-7b-hf https://huggingface.co/meta-llama/Llama-2-7b-hf
- jeremycarter 3y agoThanks!
- onlypositive 3y agoIs it really "useless" if I didn't even know about llama? And look, now I have 3 more links to dive into. This is the opposite of useless.
- mciancia 3y agoWell it's quite possible that it is useless for you since you didn't hear about llama by now ;)
- hoten 3y agoI guess this was meant as a tongue-in-cheek joke comment, but it comes off as needless gatekeeping.
- deleted 3y ago[deleted]
- onlypositive 3y agoI dunno what's worse, the pointless commentary, needless gatekeeping, the superfluous white knighting or the fact we're getting upvotes for all this nonsense.
- m00dy 3y agoWhich dataset would be good to fine-tune for developing sales assistant like chatbot ?
- ShamelessC 3y agoYou could try using a transcript of The Wolf of Wall Street, maybe throw in Glengarry Glen Ross for good measure? /s
- CamperBob2 3y agoFirst prize is an 80 GB H100. Second prize is a 4090. Third prize is a PIP.
- treprinum 3y agoIs there any tutorial on how to use HuggingFace LLaMA 2-derived models? They don't have checkpoint files of the original LLaMA and can't be used by the Meta's provided inference code, instead they use .bin files. I am only interested in Python code so no llama.cpp.
- lolinder 3y agoI'd reconsider your rejection of llama.cpp if I were you. You can always call out to it from Python, but llama.cpp is by far the most active project in this space, and they've gotten the UX to the point where it's extremely simple to use. This user on HuggingFace has all the models ready to go in GGML format and quantized at various sizes, which saves a lot of bandwidth: https://huggingface.co/TheBloke https://huggingface.co/TheBloke
- treprinum 3y agoI understand, I use llama.cpp for my own personal stuff but can't override the policy on the project I want to plug it in, which is python-only.
- pests 3y agoThere was a post yesterday about a 500 line single-file C implmenetation of llama2 with no dependencies. The llama2 architecture is hard coded. It shouldn't be too hard to port to python. Found the repo, couldn't easily find the HN thread. https://github.com/karpathy/llama2.c https://github.com/karpathy/llama2.c
- zoogeny 3y agoWhat I'd like to do is create a website where: 1. There is a list of open source fine-tuning datasets on millions of topics. Like, anime, lord of the rings, dnd, customer service responses, finance, code in many programming languages, children's books, religions, philosophies, etc. I mean, on every topic imaginable sort of like a Wikipedia or Reddit of fine-tuning data sets. 2. Users can select one or more available datasets as well as upload their own private datasets 3. Users can turn-key fine-tune llama 2 or other pre-trained models Right now, doing this kind of thing is way beyond the capability of the common user.
- brucethemoose2 3y agoThis sounds like a great fit for Cerebras, if they can set up the text database front end. They could host the text database for free, and then offer a "oh look, you can train llama on this text right now for cheaper than a Nvidia box" button on every listing. Then charge through the nose for private business training (kinds like they do now, but charging more.)
- zoogeny 3y agoI agree that it would be almost impossible to defend this kind of business, especially if you stayed committed to open-source datasets. It would come down to the UX and the community if you hoped to survive. Probably long-term you would either have to get into your own pre-trained models, fight the commodity hosting business or aim to get acquired.
- brucethemoose2 3y agoWell civitai is basically what you are describing. Its very doable. But a big difference is that (for now) Stable Diffusion finetuning is much easier than LLaMA.
- IAmNotACellist 3y agoI personally don't see a future where common users will ever have to know the phrase "fine-tuning" or worry about it. The most I can see is "Do you consent to share your information with Apple/Meta/X/Microsoft/OpenAI's knowledge engine?" and if you agree, everything they have on you will power an extremely powerful all-encompassing knowledge engine. Probably with some daily recommendations to integrate a new domain into it, like, "We noticed you're into Lord of the Rings, so we went ahead and made your knowledge engine familiar with the collected works of Tolkein, all historical academic and modern interpretations and criticisms, transcripts of the movies, and generative AI fan fiction capabilities."
- moneywoes 3y agoAny fine tuning success stories? Or real world use cases
- bvm 3y agoSure. I worked at a company that produced tens of thousands of human written summaries of news data a year. This was costly and slow but our clients really valued them. Back in 2019 we fine tuned an LLM to help, we put a lot of effort into creating a human-in-the-loop experience, highlighting parts of speech that were commonly hallucinated and ensuring that we were allowing humans to focus on things that humans are good at. We also released some of the data as a free dataset with a commercial option for all of it. This was more successful than I thought it would be and was hoovered up by the kind of people that buy these datasets. It will have been surpassed by recent developments now but it was an incredibly enjoyable project.
- rmbyrro 3y agoWhat kind of clients value news summaries that much?
- bvm 3y agolarge corporates, financial services. Use cases were needle-in-a-haystack style searching, internal comms, following research topics over time, external newsletters, that kinda stuff. It wasn't particularly high margin but it was a fun business.