4 ms·
Today Databricks announced [0] 6b parameter model from EleutherAI finetuned on Alpaca dataset. According to their CEO[1], training took 3 hours, and costed $30.
by romanzubenko 3y ago
Today Databricks announced [0] 6b parameter model from EleutherAI finetuned on Alpaca dataset. According to their CEO[1], training took 3 hours, and costed $30. They didn't release any details on how it was trained, but likely with LoRa.
[0] https://www.databricks.com/blog/2023/03/24/hello-dolly-democratizing-magic-chatgpt-open-models.html https://www.databricks.com/blog/2023/03/24/hello-dolly-democ...
[1] https://twitter.com/alighodsi/status/1639251347777388544 https://twitter.com/alighodsi/status/1639251347777388544
- m3affan 3y agoLet the revolutionbbegin
- numlocked 3y agoInteresting. I wonder what the training cost was for: https://huggingface.co/EleutherAI/gpt-neox-20b https://huggingface.co/EleutherAI/gpt-neox-20b Perhaps it’s in the paper…
- michaelhartm 3y agoThey used the 6b GPT4-J, not 20B. That's what's interesting, it's a smallish large language model :).
- dragonwriter 3y agoGPT-J, not GPT4-J.
- int_19h 3y agoThere are also some LLaMA LoRAs that are trained on the Anthropic dataset specifically for chat: https://huggingface.co/serpdotai https://huggingface.co/serpdotai I haven't done any formal tests on this yet, but with llama-13b, the overall structure of its responses definitely becomes much more ChatGPT-like. It would be very interesting to see how the 65B model performs.