7 ms·
The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next
by akelly 4y ago
The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this:
1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet
2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst
3. The GPT model needs a massive amount of training data, it would be cost prohibitive to get enough human feedback to fine tune GPT manually. So you train another model, called the reward model, to predict how the humans will rate each response. Then you train the GPT model against the reward model millions of times
5. Feed a small percentage of the output from that training process back to the human reviewers to continue training the reward model, based on heuristics like reward model uncertainty which predict how helpful the human feedback will be towards improving the reward model
6. Release ChatGPT to the public, and use user feedback like response upvotes/downvotes to further optimize the reward model, while continuing to train ChatGPT against the reward model
https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/
https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/ https://openai.com/blog/deep-reinforcement-learning-from-hum...
- amelius 4y agoThat's not genius, that's called unsupervised learning and it is an entire subfield.
- dsr3 4y agoI think number 3 is a description of Generative Adversarial Network (GAN).
- amelius 4y agoOk, regardless, it is not really new. ML researchers are doing these kinds of things all the time. By the way, according to some people, GANs are also a kind of unsupervised learning: https://stackoverflow.com/questions/44445778/are-gans-unsupervised-or-supervised https://stackoverflow.com/questions/44445778/are-gans-unsupe...
- nsb1 4y agoYou forgot the fresh cup of really hot tea :) https://hitchhikers.fandom.com/wiki/Infinite_Improbability_Drive https://hitchhikers.fandom.com/wiki/Infinite_Improbability_D...
- anthropodie 4y agoI want to know if it will ever be possible to run this kind of AI at home once its training is complete. I dont need all the knowledge just subset that I'm interested in. Actually I'm more interested in its ability to transform things. For example I can ask it to convert docker-compose to docker run command, it can manipulate JSON, it can sort numbers in table when prompted. I'm more interested in these abilities rather than just getting answers for which I already have Google
- hackernewds 4y agoyou can do that today in the free release?
- xvector 4y agoI think he wants to self host. It sucks to have no ownership of such a powerful tool I would pay upwards of $3000 to be able to self host something like this.
- gillesjacobs 4y agoRest assured someone is working on a self-hosted (distilled) model. Stable Diffusion has shown there is a viable market for open, consumer-hardware inferencable models.
- DeWilde 4y agoIt uses GPT-3 under the hood which requires about 350 gigabytes of GPU VRAM (back of the envelope calc, likely more) to perform these inferences.
- bozhark 4y ago7x NVIDIA RTX A6000's so ~$32,550
- KaoruAoiShiho 4y ago
- soulofmischief 4y agoI don't mean to downplay how incredible the tech is, but I'm not sure I'd call this approach genius as it's the industry standard.
- sillysaurusx 4y agoThen why were they the first ones to exploit it so effectively? I don’t think it was standard for GPT models.
- gentoo 4y agoI don't think anything about high-performance GPT models is standard, since they are only a couple years old and only a handful of organizations have developed them
- soulofmischief 4y agoThe technique in question has little to do with GPT itself; it involves using ML to generate more training data in an automated fashion, creating a generative training loop, which as another commenter mentioned, is also the basis behind general adversarial networks.
- mdp2021 4y agoCompare it to Generative Adversarial Networks. (There are parallels - pun not initially intended.)
- 411111111111111 4y agoI think the issue here was with the term genius, which makes it sounds like what was a completely new paradigme and revolutionary. OpenAIs success mainly stems from extremely well executed previous concepts while mostly ignoring cost. And as they're pretty much the most successful public player in this domain, they've got the first-mover advantage which they're currently very succesfully leveraging. At least thats how it looks from the perspecitve of an armchair analysts, which wouldn't have been able to achieve the same -- even if I had the same resources and time. The actual result is absolutely incredible however, regardless wherever the road to this end was genius or not
- dzink 4y agoChatGPT seems to be/result in some amount of caching of responses - there is very little variation when to asking the same question multiple times. CharacterAI produces a lot more variety in comparison, making it more helpful for brainstorming. That said ChatGPT is likely closer to the truth, even if not perfect, for searches. The innovation happening lately is incredible.
- ravi-delia 4y agoThere's definitely some live pruning happening, but another factor is that the temperature is turned way down. Obviously at a low temp it's just a totally deterministic function, and if it's doing it's job you'd hope that similar questions would be mapped very close together in the configuration space
- deleted 4y ago[deleted]
- djmips 4y agoDid you try to vary your question or add modifiers or elaborate?
- ofrzeta 4y ago> Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst Recently I saw an image where Indian women sat in front of computers and the caption said they were classifying "AI" responses. I guess that's true and this kind of work is the new outsourced cheap labour in the AI age.
- dotancohen 4y agoThat Indian woman's idea of acceptable and not acceptable AI responses surely vary from that of a San Fransisco tech worker, or Cape Town motorcycle mechanic, or an English teacher from Liverpool. I really doubt the mechanical turk method is applicable or even useful for the current state of AI-generated text.
- xp84 4y agoi actually disagree a lot with this. Sure, if you asked something with heavy cultural baggage that would frequently be a real concern, but when you are primarily trying to bridge the machine-human chasm, our cultural differences among the examples you gave are trivial in comparison. For instance, if you offered an AI personal assistant but the catch was that it would (at least starting out) only have the perspective of an average middle-class Indian person, it would still beat the absolute crap out of "first generation" technology like Siri or Alexa!
- weird-eye-issue 4y agoIt could be a first pass
- hcks 4y agoI personally worked as a « human trainer » for the fine tuning of ChatGPT. The pay was 50$ per hour, which is not bad for a side job as a student.
- jacknews 4y agoI'd say. Where do you apply for this kind of work?
- starik36 4y agoOk, but how does it take my code and fixes the bug? It's my own code, no one has seen it besides me and the model wasn't trained on it.
- ravi-delia 4y agoIn learning to predict the next token, the model has to pick up lots of little bits of world knowledge. I'm sure someone would disagree with the phrasing of "understand", but it certainly operates with more complexity than, say, a markov chain. It has seen lots of python, and in order to predict better, it has developed internal models of how python works. Think of how much better you'd do predicting the next character of python code compared to random noise- there's a lot of structure there.
- zorr 4y agoIn my (limited) experience it seems to perform even better for typed languages (for example Kotlin/Java/Swift) compared to Python. The Python code it provided often had subtle type issues when working with dates. While the Kotlin date-related code it provided was more accurate and correct in terms of types. Which makes sense since the additional type information likely leads to a much better "internal model of how Kotlin works" What surprised me was the level of "understanding" it seems to do when providing it with some of my own sample code. It can analyze the code, explain how it works/what it does, use libraries, suggest improvements and apply those improvements. Have a look at this conversation: https://imgur.com/a/ZtViC3d https://imgur.com/a/ZtViC3d While the end result isn't perfect, it's still highly impressive and while I was an AI-skeptic before, I now see the possible benefits of AI assistants for programming. Some other prompts with very impressive results: * "Write an implementation for the following Kotlin repository interface: <insert-interface-with-full-type-signatures>." * (followup) "Add save/load methods that store the backing map in a JSON file" * (followup) "Replace Gson with Jackson for JSON serialization" * "Write an Android layout xml for a login form with username/password/loginbutton" * (followup) "Provide the Kotlin activity code for this layout" * "Write a Kotlin function that parses a semver input string into a data class"
- 4y ago
- mclightning 4y agoThat's super interesting. When GPT-3 came out, I wrote an article inspired by it. That we could one day build an AI that acts like AGI, by a crazy vast amount of multimedia training data, collected by willing users to participate in ever improving AI interactions; https://medium.com/swlh/bicameral-mind-humanoid-robot-with-gpt-3-as-the-story-teller-dcd06d5c7f8d https://medium.com/swlh/bicameral-mind-humanoid-robot-with-g...
- tluyben2 4y agoWhen I was a first year AI student beginning of the 90s I asked my professor what would happen if we just made a massive neural network and trained it with all information in the world. He said it cannot happen as it it impossible.
- namaria 4y agoThis reminds me of a major US newspaper declaring heavier-than-air flying machines a million years away mere months before the Wright brothers experiments.
- samus 4y agoConsidering the computational resources available at the time, he was not that wrong. Research into artificial neuronal networks has always been held back by available computational power.
- tluyben2 4y agoAgreed, but as a professor I believe one needs to be looking in the future. It was not that far out, but yeah it was an AI winter. We were stuck until 2012 basically. That's a long time.
- Al-Khwarizmi 4y agoEasier said than done. I'm a professor in an AI field, and I can tell you that neither myself nor the colleagues I regularly have scientific discussions with could imagine ten years ago that something like ChatGPT would be possible in 2022. I suppose there might be a minority who called it, but recent advances in deep learning absolutely whooshed past the predictions of the overwhelming majority of people in the field.
- hanniabu 4y agoHow does step 1 work? It seems incredibly inefficient to check your word combo against every single segment of text they have. How does it do this efficiently?
- seydor 4y agohttps://en.wikipedia.org/wiki/Transformer_(machine_learning_model) https://en.wikipedia.org/wiki/Transformer_(machine_learning_...
- deleted 4y ago[deleted]
- visarga 4y ago> 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst Step 2 is not that. It's manually writing responses for a few tasks. > A labeller demonstrates the desired output behavior. (left side on https://cdn.openai.com/chatgpt/draft-20221129c/ChatGPT_Diagram.svg https://cdn.openai.com/chatgpt/draft-20221129c/ChatGPT_Diagr...) So it is supervised training in this stage. Ranking is the next stage, for training the reward model. This is not the reward model, it's a model to generate sample responses to be used by the reward model. So there are two kinds of manual work involved here - manually demonstrating how to solve tasks, and ranking responses. There is even talk about how much effort to invest in the first vs the second and what is the trade-off.
- akelly 4y agoRight I intentionally left off Step 1 from that chart to simplify the explanation, since it didn't seem necessary. Is Step 1 just for creating the ChatGPT content blocker?
- tnzk 4y ago> 6. Release ChatGPT to the public, and use user feedback like response upvotes/downvotes to further optimize the reward model, while continuing to train ChatGPT against the reward model Can someone provide a pointer to an article that elaborate this part?