9 ms·
Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs
- xatalytic 3y ago15,000 instruction tuning records generated by Databricks employees in seven of the behavior categories outlined in the InstructGPT paper (predecessor to ChatGPT). Coincides with the release of Dolly 2.0, which is trained exclusively on this dataset and demonstrates high quality (but not state-of-the-art) instruction-following behavior. The data and models are licensed for commercial use, setting them apart from recent releases trained on data from OpenAI.
- nickthegreek 3y ago>Coincides with the release of Dolly 2.0, which is trained exclusively on this dataset and demonstrates high quality (but not state-of-the-art) instruction-following behavior. This is not correct. It was fine-tuned with this data set, but the model itself is the 12B Eleuther AI pythia model.
- mrtranscendence 3y agoThere are two, a 6B parameter model fine-tuned on GPT-J and a 12B parameter model fine-tuned on Pythia.
- anentropic 3y agothe GPT-J-6B one is Dolly 1.0, previously released Dolly 2.0 is Pythia-12B fine-tuned on this new dataset on their hugging face page [1] they admit the performance may not be much or any better than the original model (I am guessing this may be a weakness of Pythia-12B, which was intended for model-training research rather than best results) the main point of Dolly 2.0 is the new dataset is unencumbered legally [2] whereas Alpaca et al were trained on ChatGPT transcripts, so commercialising those models would contradict OpenAI licensing terms [1] https://huggingface.co/databricks/dolly-v2-12b https://huggingface.co/databricks/dolly-v2-12b [2] https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm https://www.databricks.com/blog/2023/04/12/dolly-first-open-...
- cma 3y agoI think there's probably nothing wrong with training on others' ChatGPT transcripts posted on the open web. OpenAI trains on source-available projects with non-commercial terms, so their lawyers have already been over a similar case and decided it should be fine.
- ralfn 3y agoNot just that: Imagine OpenAI going to court and establishing the legal precedent that makes their own product illegal. So OpenAI can claim whatever they like, there is no way they will ever pursue legal actions, unless their intent is to (intentionally) lose the court case to establish the precedent that it is okay to train on random data you scraped from the internet. We would also get into a weird situation anyhow where it is hard/impossible to prove whether all/some/none of the information in a dataset is curated by humans. So in the worst case, we will have companies work with human curators (but secretly supplement with gray sourced materials) during their training. Just like how its hard to get 100% slave free coffee beans or cacao.
- anentropic 3y agoI don't think it's about things being illegal per se But that they can sue you because, by making a competing product with data obtained by using their product, you contravened their terms & conditions for using their product
- ralfn 3y agoBut so did they when they scraped the web for content. That's not within anyone's terms and conditions except Wikipedia. That's what I mean with precedent. If OpenAI would win that they would be sued in term by Bloomberg for example.
- falaki 3y agoThis is the blog post with more details and background: https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm https://www.databricks.com/blog/2023/04/12/dolly-first-open-... Disclosure: I work at Databricks.
- falaki 3y agoWe also open sourced the Dolly model itself with a license that allows commercial use.
- choppaface 3y agocan you compare your dolly offering with https://github.com/microsoft/DeepSpeedExamples/blob/master/applications/DeepSpeed-Chat/README.md https://github.com/microsoft/DeepSpeedExamples/blob/master/a...
- oidar 3y agoHow hard would it be to get dolly running on llama.cpp?
- anentropic 3y agoit's probably simple for Dolly v1 (?) since it was a fine-tuned version of GPT-J https://github.com/ggerganov/ggml/tree/master/examples/gpt-j https://github.com/ggerganov/ggml/tree/master/examples/gpt-j AFAIK there is no .cpp version of Pythia-12B yet
- ankitmathur 3y agoHey there! I worked on Dolly, and I work on Model Serving at Databricks. DollyV1 is GPT-J-based, so it'll run easily on llama.cpp. DollyV2 is Pythia-based, which is built with the GPT-NeoX library GPT-NeoX is not that different than GPT-J (it also has the rotary embeddings, which llama.cpp supports for GPT-J). I would imagine it's not too heavy of a lift to add NeoX architecture support
- 3y ago
- mydpy 3y agoBenchmarks here: https://huggingface.co/databricks/dolly-v2-12b#benchmark-metrics https://huggingface.co/databricks/dolly-v2-12b#benchmark-met...
- omneity 3y ago> As outlined above, these results demonstrate that dolly-v2-12b is not state of the art, and in fact underperforms dolly-v1-6b in some evaluation benchmarks. We believe this owes to the composition and size of the underlying fine tuning datasets, but a robust statement as to the sources of these variations requires further study. Taking a moment to appreciate the integrity of the team.
- ingenieroariel 3y agoDitto, this is release early release often without necessarily meaning move fast and break things. Other teams can do the equivalent of Alpaca to Llama and we can all learn for the next round.
- xatalytic 3y agoOne of the creators here - yeah, the thing we have our eyes on is the vector not the point. It’s astounding how adaptable these open models are, even with just a quarter of the Alpaca data. We’re a team of machine learning engineers and hackers, not an AI science lab, but that’s kind of the point frankly - this whole exercise appears to be far easier that it might at first seem.
- itake 3y agoWhy are they not doing metrics against GPT-3.5 and GPT-4? My understanding is Dolly performs significantly worse.
- thewataccount 3y agoI haven't played with the model just yet - but just eye balling it's performance it's significantly worse. I'm surprised they don't have Pythia on there as that's what they're based on from my understanding. At their performance level it's the most important to compare to GPT-neoX, and I do appreciate they aren't making the "95% of GPT4" claims that some fine-tuned llama models are. EDIT: For databricks people: I'd love to see this compared with Pythia, LLaMa, Alpaca, and vicuna/gpt4all if possible.
- __debugger__ 3y agoPrevious [flagged] discussion: https://news.ycombinator.com/item?id=35539085 https://news.ycombinator.com/item?id=35539085
- mrg3_2013 3y agoHow does this compare to openai ? Curious if anyone has any anecdotes.
- falaki 3y agoWe don't expect this to be as good as the latest OpenAI GPT release. This is just to demonstrate that developing a conversation agent using an existing foundation model is not as hard as some may assume. Take a foundation model that is not capable of Q&A and tune it with a fairly small Q&A data and you get your in-house ChatGPT. Disclaimer: I work at Databricks.
- mrg3_2013 3y agoThanks for the feedback. The potential edge with Dolly is huge. Building a firewalled model with custom corpus is a big deal. I have been experimenting with openai and even with public data (but really limiting to the domain), yields great improvements (openai may be stale because of cut off data). I am excited to see where Dolly goes.
- mrtranscendence 3y agoDolly appears to fundamentally be a tech demo advertising how you can use Databricks for compute. I honestly wouldn't expect them to take it that much further, particularly in the context of larger models that would be significantly more expensive to fine-tune. But I'm happy to be proven wrong.
- m3kw9 3y agoI’m not seeing how 15k q/a training can get you much other than the simplest things. Maybe that’s the point, get the ball rolling for people to add more training data?
- swid 3y agoIt's used for fine tuning a pre-trained model. This takes an LLM that is already capable of emulating lots of different kinds of personalities, and narrows it down to act more like the examples. Since the heavy lifting has already been done, 15k examples of a chatbot following instructions they way you want has a significant effect.
- gamegoblin 3y agoWhat reasons do you have for believing that is true? It seems plausible to me that a general autoregressive LLM that is capable of completing text wouldn't take that much fine-tuning to shift it from "text completion" to "instruction following". After all, the raw GPT3 model can be made to follow instructions with just a few examples. Consider the prompt: What is the capital of France? Raw GPT3, not the newer instruction-tuned variants, does not understand it's being asked a question. It offers the completion: What is the capital of France? If a student answers with a word, she is asked to identify the word. She is not asked whether the capital of France is Paris. On the other hand, if the student answers by pointing to a map, she is asked to identify the capital of France. She is not asked whether it is Paris. It just starts appending to the text. But if you give it a few examples, it happily gets into instruction following mode: The following is a transcript between a human and a helpful AI assistant who answers questions and obeys commands. Human: How many eggs are in a dozen? AI: 12 Human: Say "hello" 3 times AI: hello hello hello Human: What is the capital of France? AI: GPT3 completes "Paris" here. If you can get decent instruction/question following behavior out of a 2-shot example prompt, why do you think 15k is small for this?
- dontupvoteme 3y agoN-shot at inference-time is fundamentally different from training/fine-tuning which is inherently pre-inference-time. Though it would be interesting to know if OpenAI has a few generic multishot inputs before the prompt. It's all extremely cryptic what the actual context window and system prompt (assuming chatgpt even is using the same API the proles are given) is with them
- stuartjohnson12 3y agoI don't think these upvotes are organic.
- iaw 3y agoWhy do you say that?
- nickthegreek 3y agoprobably based on the situation 3hrs ago - https://news.ycombinator.com/item?id=35539085 https://news.ycombinator.com/item?id=35539085
- dimitrios1 3y agoThere's a big difference between employees who got excited to see their work on hacker news and upvoted it and premeditated shill / astroturf campaign. We should pretty much assume that a San Fransisco based company is going to have significant readership / membership here. One can easily see how a message over a company communicator could result in a surge of upvotes.
- nickthegreek 3y agoAgreed. I was just providing the context that the user asked for.
- visarga 3y agoMine was.
- catchnear4321 3y agoI don’t think you fully appreciate the value of the training corpus.
- Havoc 3y agoGreat to see more releases under open licenses!
- simonw 3y agoHere's a link to open up and explore that training data in Datasette Lite: https://lite.datasette.io/?json=https://github.com/databrickslabs/dolly/blob/master/data/databricks-dolly-15k.jsonl#/data/databricks-dolly-15k?_facet=category https://lite.datasette.io/?json=https://github.com/databrick...
- jarek83 3y agoCan someone help me to understand why categories for these two differ? row #51 "Think of some family rules to promote a healthy family relationship" - brainstorsming [1] row #68 "What is the future for human?" - general_qa [2] In nature they both are brainstorming to me - does the question mark is what assigned the #68 as _qa? [1] https://lite.datasette.io/?json=https://github.com/databrickslabs/dolly/blob/master/data/databricks-dolly-15k.jsonl#/data/databricks-dolly-15k/51 https://lite.datasette.io/?json=https://github.com/databrick... [2] https://lite.datasette.io/?json=https://github.com/databrickslabs/dolly/blob/master/data/databricks-dolly-15k.jsonl#/data/databricks-dolly-15k/68 https://lite.datasette.io/?json=https://github.com/databrick...
- gpm 3y agoThe labelling doesn't seem to be entirely consistent to me, but I think the idea is that 51 is inviting you to brainstorm, while 68 is asking a question that just happens to be open ended.
- ankitmathur 3y agoHey! Worked on this here at Databricks: the blog post goes into the dataset collection design a bit (https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm https://www.databricks.com/blog/2023/04/12/dolly-first-open-...). In summary, you're right - brainstorming and GeneralQA will have overlap because the taxonomy naturally has some overlap
- rnosov 3y agoI'm going through the dataset with your datasette tool and it looks like it might be a good idea to clean things up a bit. There are many duplicates[1], creepypastas[2] and other strange things in there. [1] https://lite.datasette.io/?json=https%3A%2F%2Fraw.githubusercontent.com%2Fdatabrickslabs%2Fdolly%2Fmaster%2Fdata%2Fdatabricks-dolly-15k.jsonl#/data?sql=select+count%28instruction%29%2C+instruction%2C+group_concat%28context%2C+%27%0A%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%0A%27%29+as+c%2C+group_concat%28response%2C+%27%0A%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%0A%27%29+as+r%2C+group_concat%28category%2C+%27%0A%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%0A%27%29+as+cat++from+%5Bdatabricks-dolly-15k%5D+group+by+instruction+having+count%28instruction%29%3E1+order+by+count%28instruction%29desc+limit+500 https://lite.datasette.io/?json=https%3A%2F%2Fraw.githubuser... [2] https://lite.datasette.io/?json=https://github.com/databrickslabs/dolly/blob/master/data/databricks-dolly-15k.jsonl#/data/databricks-dolly-15k?_filter_column=instruction&_filter_op=like&_filter_value=%25Imagine+you+are+the+last+person+on+Earth%25&_sort=rowid&_facet=category https://lite.datasette.io/?json=https://github.com/databrick... EDIT: Maybe I'm passing link wrong, the query I'm using is select count(instruction), instruction, group_concat(context, ' ============= ') as c, group_concat(response, ' ============= ') as r, group_concat(category, ' ============= ') as cat from [databricks-dolly-15k] group by instruction having count(instruction)>1 order by count(instruction)desc limit 100 [databricks-dolly-15k] should be the name of dataset, first column is the number of instruction duplicates Creepypastas are responses to instruction: Imagine you are the last person on Earth. Write a diary entry describing your thoughts and feelings.
- simonw 3y agoI got this model working on a GPU instance, notes here: https://til.simonwillison.net/llms/dolly-2 https://til.simonwillison.net/llms/dolly-2 Anyone managed to run it on an M1/M2 Mac yet?
- brianjking 3y agoI'm sure we'll see this by the end of the day or two.
- gavi 3y agoNot on M1/M2 yet, but my response time seems pretty fast on Tesla V100-SXM2-16GB
- mrtranscendence 3y agoI attempted using the Transformers library but failed. Not sure, might be a VRAM issue; I'm going to try on my far beefier personal MacBook Pro later tonight.
- rnk 3y agoHow much ram is likely needed on an apple arm for models like this? And for general use, 64, 96, 128? Trying to decide how large I should go for a new laptop.
- aldarisbm 3y agosame same
- Szpadel 3y agoAFAIK current models can run even with 64GB, but I would assume that we will very likely have bigger models very soon so I guess the answer is as much as you can afford
- rnk 3y agoThe next question is m1 or m2, and the impact of the various number of gpu units between pro, max, ultra skews. I'm really tempted to buy a "refurbished m1 studio" with 128gb because I think the ram is the key. Have not seen any benchmarks with diff # of gpus/aka diff skews.
- kumarski 3y agoAmazing. Love databricks.
- mrtranscendence 3y agoDatabricks is fine. I wasn't happy using it until they implemented the ability to work in a git repo, with proper file support, but that's gone some way to making it more usable to me. The interface sucks pretty hard, slowing down and using a significant amount of memory with only modestly high number of cells (where a Jupyterlab notebook would remain very snappy). I also wish there were a better story for local development; they've addressed this to some degree recently but I'm not sold on their solution. It's certainly better than what we did prior to Databricks, which was roll our own in-house provisioning and notebook solution. I won't/can't go into too many details, but not only was it cumbersome and very buggy, but it was as if they designed it to encourage data scientists to spend as much money on compute as possible (only to panic at the millions they were spending). They dropped it for cost reasons, which is hilarious given how expensive Databricks is. I do appreciate the work Databricks have done improving Spark. Capabilities like adaptive query execution have made optimization significantly easier.
- sandkoan 3y agoWhen you say you wish they had a "better story for local development," what do you mean? What do you wish for?
- dreaminvm 3y agoHappy to see this type of work that is truly open source and commercially usable. Is this the entire corpus or a subset? Do you intend to release any new iterations? I've been thinking of starting similar efforts at another BigCorp by hosting a UL2 or GPT-J instance.
- pwendell 3y ago15k is the entire corpus we have right now. Hopefully others can join up in releasing additional samples that can be merged in over time. We'll definitely keep iterating on Dolly and releasing everything openly.
- zan2434 3y agoAnyone wanna convert this to GGML so we can run it with LLaMa.cpp?
- itamarcode 3y ago"dolly-v2-12b is not a state-of-the-art generative language model and, though quantitative benchmarking is ongoing, is not designed to perform competitively with more modern model architectures or models subject to larger pretraining corpuses." from: https://huggingface.co/databricks/dolly-v2-12b https://huggingface.co/databricks/dolly-v2-12b
- nomilk 3y ago> instruction: Why mobile is bad for human > response: We are always engaged one phone which is not good. Curious about the poor grammar in the response. Is it intentionally mimicking the style of the input instruction?