5 ms·
Doesn't it cost hundreds of thousands of dollars just to train GPT-3 ? If so, that seems like a good reason to use a "managed" GPT-3.
by ritz_labringue 5y ago
Doesn't it cost hundreds of thousands of dollars just to train GPT-3 ?
If so, that seems like a good reason to use a "managed" GPT-3.
- dave_sullivan 5y agoYes, but they didn't release the model after training and you can't take your weights with you if you finetune their model. GPT Neo was trained at similar expense, and they released the weights. Use that.
- ritz_labringue 5y agoThanks ! Didn't know that. Isn't it also very expensive to run ?
- damvigilante 5y agoFirst part is correct, the second part is not. GPT Neo is a 2.7B param model, the largest GPT is 175B (they have various flavours, up to 175B). I appreciate the sentiment and what ElutherAI is doing with GPT Neo, but there is no open source equivlenet of the full GPT-3 available for the public to use. Hopefuly it's just a matter of time.
- priansh 5y agoGPT-J is 6B and comes pretty close. Also practically I haven’t noticed a difference. Keep in mind there are also closed source alternatives: for example, AI21’s Jurassic-1 models are comparable, cheaper, and technically larger (albeit somewhat comically, 178B instead of 175B parameters).