patelajay285

7 ms·

1.
Wharton and Harvard Business School: LLMs on business decisions and tasks(business-ai-benchmark.github.io)
2 points by patelajay285 2mo ago|0 comments
2.
by patelajay285 2y ago
When you train bidirectionally only, you don't get a generative model, that would be the downside. However, you can train on a mixture of causal and bidirectional objectives as some LLM pre-training has done. As far as I am aware, ther
3.
by patelajay285 2y ago
It's not at all expected. T5 models are not generative models by default and they were not thought to be able to perform generation, let alone in-context learning. Remember these models were released before any of the existing LLMs a
4.
by patelajay285 2y ago
We found the same result a few years ago in our ICLR paper: https://arxiv.org/pdf/2209.14500 We found Google's T5 models which were released in 2019, pre-GPT-3, were "secretly" capable of in-context lear
5.
by patelajay285 2y ago
We've been working on a Python framework where one of the use cases is easy distillation from larger models to smaller open-source models and smaller-closed source models (where you don't have to still use / pay for the close
6.
by patelajay285 3y ago
Thanks for clarifying, for the record, I generally agree with you. I think we just disagree on the snippets and how in-depth they need to be. Our library is built on HF libraries (we don't implement the training code ourselves), which
7.
by patelajay285 3y ago
I think you're interpreting hostility where there is none, so I don't have much to say other than it's an infrastructure library, a demonstration snippet doesn't need to show how to train a production grade model. I appr
8.
by patelajay285 3y ago
It's a demo snippet of how to setup the workflow, it's not meant to be a working production example a self-rewarding model or a faithful reproduction of the original paper. Whether self-rewarding LLMs are a good idea or not, it&#x
9.
by patelajay285 3y ago
Collecting data is hard, but the library is also a synthetic data generation library, so for example you can create the data for DPO fully synthetically, check out the self-rewarding LLMs example: https://datadreamer.dev/doc
10.
by patelajay285 3y ago
Yes it is :), but the library is also a synthetic data generation library, so for example you can create the data for DPO fully synthetically, check out the self-rewarding LLMs example: https://datadreamer.dev/docs/late
11.
by patelajay285 3y ago
That's totally fair and good feedback, it's hard to support everyone's use cases simultaneously, but from my own research and other researchers we collaborate with, this solves and streamlines the right set of problems, but w
12.
by patelajay285 3y ago
I think the discussion in the other comment thread discusses this well. They are different techniques, but the line between RL & SL is quite fuzzy. The DPO authors advertise this as a "non-RL" technique to precisely get away f
13.
by patelajay285 3y ago
Thanks for the question. This is built for ML researchers, so in examples we use the defacto source for datasets researchers often use, HF Hub. However, there is a lot of documentation on the site to help guide users. This documentation pag
14.
by patelajay285 3y ago
This was discussed in another comment, DPO is pretty much strictly better than RLHF + PPO, and far more stable when training. Yes, DPO is not technically "RL", but it's semantics for the most part. DataDreamer does support PP
15.
by patelajay285 3y ago
This is built for ML researchers out of an academic lab. There's a ton of functionality in the library (beyond RLHF and alignment) that ML researchers do every day to write papers and run experiments that the library helps abstract and
16.
by patelajay285 3y ago
I tend to agree @espadrine, it's semantics for the most part
17.
by patelajay285 3y ago
That’s totally valid and something we would even encourage! This project is for researchers so if there is a point where the abstraction is no longer useful, by all means configure, or subclass, or copy code.
18.
by patelajay285 3y ago
Thanks! It makes it easier to run with the existing run scripts I have on our large university GPU cluster. :) no other reason
19.
by patelajay285 3y ago
Thanks, appreciate the feedback, will update when I get a chance!
20.
by patelajay285 3y ago
Yep, DPO is not technically “RL” and implicitly uses the LLM itself as a reward model, but training with DPO is far more stable for that reason.
21.
by patelajay285 3y ago
Fair, DPO is considered a fairly well established technique now that is far more stable in training than PPO, but also helps align LLMs from human feedback. The package also helps do PPO, so you can do traditional RLHF, but figured more peo
22.
by patelajay285 3y ago
Appreciate the feedback on the title, this is developed for ML researchers, so I assure there is a lot it’s doing under the hood to make this process easier (for example introducing automatic caching and resumability). However, we also trie
23.
by patelajay285 3y ago
This is developed for researchers, so I assure it’s very hackable and configurable. ;-) but appreciate the feedback on the title!
24.
by patelajay285 3y ago
Hi everyone, there are no easy tools for synthetic data generation or training and aligning LLMs simply in Python. Most of the stuff out there are messy adhoc scripts. DataDreamer is an open source Python package with a nice API from the Un
25.
RLHF a LLM in <50 lines of Python(datadreamer.dev)
223 points by patelajay285 3y ago|66 comments
26.
by patelajay285 3y ago
We are planning to expand this in the future! Thanks for the feedback
27.
Distill OpenAI GPT-4 to a cheaper, faster GPT-3.5 model(datadreamer.dev)
1 points by patelajay285 3y ago|2 comments
28.
by patelajay285 3y ago
We've been working on a project that provides this sort of easy swapping between open source (via HF, VLLM) & commercial models (OpenAI, Google, Anthropic, Together) in Python: https://github.com/datadreamer-dev&#x2
29.
by patelajay285 3y ago
We've built a prompting, synthetic data generation, and training library called DataDreamer: https://github.com/datadreamer-dev/DataDreamer
30.
by patelajay285 5y ago
Check out Magnitude, we built it to solve that problem: https://github.com/plasticityai/magnitude It's still loaded from a file, but heavily uses memory-mapping and caching to be speedy and not overload your RAM i

More ›