8 ms·
GPT-2: 6-Month Follow-Up
- minimaxir 7y agoFor finetuning GPT-2 on custom text, my gpt-2-simple package (https://github.com/minimaxir/gpt-2-simple https://github.com/minimaxir/gpt-2-simple) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test. Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text compared to the 345M which is slow even on a GPU. How exactly the large GPT-2 models are deployed is a mystery I really wish was open-sourced more.
- gwern 7y agoI've already tried training with nshepperd's codebase. Sampling works, but even with the memory checkpointing and freezing the embedding and using SGD rather than Adam, it OOMs on a 1080ti's 11GB. Either additional tricks or CPU training are going to be required.
- p1esk 7y agoAre you using FP16?
- gwern 7y agoNo. We weren't sure if that'd be a good idea since it wasn't trained with low-precision, and 345M thankfully didn't require going that far. 744M might, though. (Another option is model parallelism since I have 2 GPUs and that might be enough, perhaps freezing more layers and training incrementally, or reducing the 1024 token window to smaller ones like 700.)
- slashcom 7y agofp16 saves a lot of memory and is worth doing. I've not had trouble fine tuning all these models with fp16.
- sdan 7y agoHave you fine tuned 774 successfully using a single GPU?
- p1esk 7y agoI recommend Nvidia Apex, it offers several ways to mix precision.
- The_rationalist 7y agoYou maybe should try tensorflow automatic mixed precision! https://github.com/zihangdai/xlnet/pull/200 https://github.com/zihangdai/xlnet/pull/200
- JonathanFly 7y agoOn the NVIDIA GPT- 2 implementation: >What would be the largest model one could train across 2x 2080Ti? >~800M gpt2. this is largely due to the memory required to House parameters + optimizer states. If one uses a smaller optimizer than Adam training something larger should be possible. Make sure to turn on activation checkpointing with —checkpoint-activations https://twitter.com/TheRealRPuri/status/1161322580126470145 https://twitter.com/TheRealRPuri/status/1161322580126470145
- gwern 7y agoThey haven't released such models, though, and I don't know if it would be drop-in compatible with the OA GPT-2-774M checkpoint (they're training their own GPT-2s using their own webtext corpus).
- JonathanFly 7y agoI haven't look into at all myself, but he also said: >We do provide training code that should work out of the box for gpt2 117M/345M https://twitter.com/TheRealRPuri/status/1161319745259393024 https://twitter.com/TheRealRPuri/status/1161319745259393024
- p1esk 7y agoIt would take forever (or $$$) to train even 117M model from scratch.
- JonathanFly 7y agoI read that meaning you can start with the actual pre-trained GPT-2 models but I never got an answer when I specifically asked if that was the case.
- minimaxir 7y agoFigured. I'll make changes to allow sampling from the default model more easily.
- newhaus1994 7y agoI'm the lead researcher on the Middlebury Institute project looking at fine-tuning the bigger models, and I got OOM on 745M and 1.5B originally. I had to get an Azure instance with 24GB VRAM to handle it (using nshepperd's codebase). It works, but takes a while (~500 epochs takes 12 hours on a 100k word training dataset).
- gwern 7y agoOuch! So 11GB is nowhere close to being enough, then. I wonder if even switching to FP16 will be adequate?
- newhaus1994 7y agoMight be able to get 745M down to work on a single GPU. I'm definitely not using all 24GB, so fp16 might be able to get it down enough.
- sdan 7y agoHow would you use fp16 to get it to work on a single GPU? And if you did, what GPU should you use?
- the8472 7y agoPossibly a stupid question, but does AMD lift such restrictions on models with its unified memory, by allowing the GPU to "page out" chunks of vram to system ram?
- minimaxir 7y agoNo idea how modeling works on AMD. (most discussions are about NVidia/CUDA)
- p1esk 7y agoMy guess is it would be much slower, because GPU processor would wait for data. Compare bandwidth - system RAM to GPU memory (PCIe): 16GBps vs GPU memory to GPU processor: 900GBps.
- vimy 7y agoIsn't that a macOS specific feature?
- AdamDKing 7y ago>How exactly the large GPT-2 models are deployed is a mystery I really wish was open-sourced more. TalkToTransformer.com uses preemptible P4 GPUs on Google Kubernetes Engine. Changing the number of workers and automatically restarting them when they're preempted is easy with Kubernetes. To provide outputs incrementally rather than waiting for the entire sequence to be generated, I open a websocket to a a worker and have it do a few tokens at a time, sending the output back as it goes. GPT-2 tokens can end partway through a multi-byte character, so to make this work you need to send the raw UTF-8 bytes to the browser and then have it concatenate them _before_ decoding the string. While my workers can batch requests from multiple users, the modest increase in performance is probably not worth the complexity in most cases.
- jcims 7y agoAny thoughts on the larger model? Doesn't seem materially better than the last one. Maybe the fine tuning exercises will show the benefit?
- deleted 7y ago[deleted]
- wyldfire 7y agoAre there any applications for the GPT-2 models beyond text synthesis? Inference, question-answering, NER detection/disambiguation, anything like this?
- gwern 7y agoAll of those were part of the original benchmarks GPT-2 was evaluated on.
- make3 7y agoBERT and its descendants do better at all of this, and are the industry standard now https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805
- deleted 7y ago[deleted]
- The_rationalist 7y agoExcept that BERT is now obscoleted by https://github.com/zihangdai/xlnet https://github.com/zihangdai/xlnet (but xlnet would never have existed without BERT)
- ivalm 7y agoKind of, there are a bunch of transformers that might perform better than BERT (Ernie 2.0 being stronger than xlnet, for example), but often this is a function of training size (xlnet trained on 10x more data than original BERT). Realistically there are now BERTs released finetuned for special corpa (biobert, clinical bert, etc) so if you want to work on those kind of texts you are better off starting with a BERT that was previously fine tuned to something close to your task (and then fine tune it more yourself).
- The_rationalist 7y agoWell you comment was really interesting to me because I didn't know ERNIE 2.0 and it's concept of continual learning seems to be really a step forward! But some of you statements seems incorrect: Ernie 2.0 being stronger than xlnet XLnet is the neural net with the biggest number of first places on benchmarck leaderboards. Cf: https://paperswithcode.com/paper/xlnet-generalized-autoregressive-pretraining https://paperswithcode.com/paper/xlnet-generalized-autoregre... While ernie 2.0 has currently 0 first place on paperswithcode.com https://paperswithcode.com/paper/ernie-20-a-continual-pre-training-framework https://paperswithcode.com/paper/ernie-20-a-continual-pre-tr... xlnet trained on 10x more data than original BERT No, I've read on a github issue of xlnet that xlnet base is same size as bert base and xlnet large is same size as bert large. (I don't know for ernie 2) Well your point on finetuned bert vs non finetuned xlnet is interesting. ROBERTA is so fine tuned it beat XLnet on some tasks. But generally xlnet non finetuned beat BERT finetuned and there are more and more xlnet finetuned each week. But your point does apply for Roberta, and for the few tasks where bert as been applyed but xlnet hasn't yet.
- lucidrains 7y agoHmm, no mention of Megatron in their timeline? https://nv-adlr.github.io/MegatronLM https://nv-adlr.github.io/MegatronLM
- ivalm 7y agoThey do mention the 8b+ "GPT-2" model trained by nVidia. Which is their reference to megatron.
- lucidrains 7y agoOh! You are right! How did I miss that..
- The_rationalist 7y ago<rant> Are there any real use case for GPT-2? Does it solve any problem? I've read almost all state of the art leaderboards of all Nlp tasks of paperswithcode.com and truth is except text generation, openAI has not one state of the art, they are not even visible in leaderboards. OpenAI is maybe the AI research center with the biggest funding and comparatively to other well known (Microsoft, Facebook, Google or even zalando..) they are the ones with the least results. From my observations most SOTAs come from chineses researchers by far, followed by deepmind. BTW isn't that a sad truth that not even one of all major AI actors has a draft of an AGI architecture, something comparable to CYC or opencog. https://wiki.opencog.org/w/CogPrime_Overview https://wiki.opencog.org/w/CogPrime_Overview Two other observations I would like to share: Many important NLP tasks have almost nobody publicly working on them it seems, on paperswithcode.com or NLP-progress (from github) some tasks have only one or two papers... And many others have not evolved since 2016. Most of the time it seems trivial to beat the old state of the art, just use BERT or XLnet on a task where nobody applied it before and hop, free state of the art for you! Yet researchers don't seems to chase those low hanging, high returns fruits. Also researchers seems to work a lot in isolation, many new generic improvements like new optimizers (RAdam for example) and new activation functions (Swish) allow to beat most of older state of the art on almost all task just by using them. Yet researchers will take years before using them because of an absurd inertia. Also unlike an open source program, BERT and XLnet have very low response and activity on github despite major open issues... </rant>
- p1esk 7y agoMany important NLP tasks have almost nobody publicly working on them Well, then perhaps you should go work on them, instead of ranting here.
- The_rationalist 7y agoWhy the ad hominem? I am pointing a problem of allocation of ressources on the AI research field. It's not to me to fixe that, but yes I am actively working on a logical fallacies detector which is the first of human history and works for the 256 possible forms of syllogisms, I'm expanding it to other logical forms such as modus ponens/tollens.
- brentsch 7y agoI'm curious about the "fine-tuning based detection" mentioned in the report ("Fine-tunes a language model to 'detect itself'... over a range of available settings"). Does anyone know good articles/papers (or have an off-the-top tl;dr) to get a high-level grasp of "self-detection" for generative models?
- mappingbabeljc 7y agoHiya, I work at OpenAI. I think the Grover paper is a good place to read about some of this:https://arxiv.org/abs/1905.12616 https://arxiv.org/abs/1905.12616 We're likely publishing more on detecting fine-tuned outputs in the future, also.
- brentsch 7y agoMany thanks! Looking forward to reading the OpenAI research when it comes out as well.
- gambler 7y agoHopefully someone will make a working demo of it, like Adam King did for 345M. People should be able to experiment with this stuff without relying on the hype of press releases: https://medium.com/@VictorBanev/interrogating-gpt-2-345m-aaff8dcc516d https://medium.com/@VictorBanev/interrogating-gpt-2-345m-aaf... Not sure why open AI doesn't do this themselves. That fully aligns with their stated mission.
- nwah1 7y agohttps://talktotransformer.com/ https://talktotransformer.com/
- pixelHD 7y agoI made a discord chatbot for interacting with gpt 2: https://github.com/itsmehemant123/gpt2-discord-bot https://github.com/itsmehemant123/gpt2-discord-bot
- minimaxir 7y agoIt appears TalkToTransformer has been updated for 774M: https://twitter.com/AdamDanielKing/status/1163879500716941314 https://twitter.com/AdamDanielKing/status/116387950071694131...
- enkiv2 7y agoIt's not particularly hard to check out the source & run it on your own machine. For those who don't know how to use git or are afraid of the command line, there's TalkToTransformer.
- gambler 7y ago"Cornell University is studying human susceptibility to digital disinformation generated by language models." "The Middlebury Institute of International Studies Center on Terrorism, Extremism, and Counterterrorism (CTEC) is exploring how GPT-2 could be misused by terrorists and extremists online." "The University of Oregon is developing a series of “bias probes” to analyze bias within GPT-2." But apparently no university studies the social and economic impact of using terabytes of public data to train algorithms that for all practical reasons end up being inaccessible to an average person. If things go on the way they're going right now, in 20 years millions of people will be "mechanical turked". Most of information processing tools will be mediated exclusively through companies like Google and Amazon. They will be less like normal tools (e.g. word processors) and more like systems you have to be a part of. Can you imagine the levels of inequality involved? The hyper-centralization of power? This is the foremost challenge presented by AI, not some hypothetical nonsense involving terrorists using a text generator. And it's not like there aren't any solutions. Douglas Engelbart, for example, pointed out a great way of introducing technology into society without screwing most of the society over: http://dougengelbart.org/content/view/138 http://dougengelbart.org/content/view/138 We kind of followed his vision for a while, with good results, but AI seems to be going in an entirely different direction.
- comradesmith 7y agohttp://talktotransformer.com http://talktotransformer.com Seems accessible to me
- postalrat 7y agoThat isn't running the full model.
- enkiv2 7y agoIt's running the latest model to be released, which is 774M. Since the initial announcement, larger models have been released every few months, so we're on track to have the full model released by 2020. (This is what OP is literally about -- the roadmap for releasing larger subsets of the model.)
- zitterbewegung 7y agoI was able to take all of Donald Trumps tweets and using GPT2 to make a program that would mimic his tweets. I found that it might be very effective. I have the test at https://docs.google.com/forms/d/1p7tlobl5y5plBCu_enK4KawR7B8_4Yyb-wCUh6vr9A0/edit https://docs.google.com/forms/d/1p7tlobl5y5plBCu_enK4KawR7B8... I got the information from trumptwitterarchive.com I also explored creating a system that could recognize fake tweets from real ones and I believe I got 94% accuracy. It was a Bayes classifier but I think I have to double check my work.
- The_rationalist 7y ago"I also explored creating a system that could recognize fake tweets from real ones and I believe I got 94% accuracy. It was a Bayes classifier but I think I have to double check my work." Is it open source? This interest me a lot!
- rovyko 7y ago>As part of our staged release strategy, our current plan is to release the 1558M parameter model in a few months, but it’s plausible that findings from a partner, or malicious usage of our 774M model, could change this. This seems naive but I think it's a misdirection. Of course the model will have malicious users. Propaganda teams started testing its integration as soon as it was released. It's likely that OpenAI is counting on this for insights into HOW the model can be used maliciously. It's also possible that the model results have inherent trackable markers and OpenAI can later say that X% of social media posts were made using this model. So what are the positive applications, aside from prettifying data like sports and weather reports? Even with Skyrim's 800+ books, you frequently ran into the same book. Imagine libraries filled with plausible text that hides nuggets of lore seeded by developers. Along with more realistic text-to-speech this can allow games to support a large diversity of NPCs that have true radiant dialogue and sound more realistic than "I saw a mudcrab the other day". With some modifications, I think models like this can outweigh even their nefarious applications: Defense against text decomposition analysis. The model can be used to obfuscate writing patterns that can reveal a person's identity, either by randomizing form or standardizing it. Take your post and run it through the formatter to get the same idea and intent, but in a style that can't be traced to your other writing. Or you reform it into style of Ernest Hemmingway, like thousands of others. Realtime plausible deniability encryption. Messages in a monitored chat can look like mundane conversation but contain encrypted messages. This would require the model accept seeds and work partially in reverse to diff two sets of text to reveal the hidden message. In it's current form it doesn't look like it can do any of those things, but there's the potential.
- lxe 7y agoAnyone wired a "talktotransformer"-style system to this one yet? Would like to see how it works without going through the steps of setting it up. EDIT: Looks like https://talktotransformer.com/ https://talktotransformer.com/ already uses the 774M one!
- deleted 7y ago[deleted]
- revel 7y agoThe OpenAI approach to managing the release of the larger dataset strikes me as totally flawed and upside down. The biggest concern the team seem to have is that the fully trained GPT2 model will be used to spread propaganda and misinformation. They also imply that the biggest hurdle to training a similar model is money needed to pay for the training resources. The problem with this approach is that the users most likely to be malicious users of GPT2 are state actors. China, for example, already spends millions on an immense propaganda factory. Money is not a serious obstacle for a state. Given that other research entities are, by the sound of things, already far along with development of similar models it seems unlikely that China and the US don't already have functional models internally. On the other hand, legitimate business and research is clearly hamstrung by withholding the full model. What we have is the maximum degree of inconvenience and the minimum degree of security. It feels almost perfectly analogous to ban on liquids in airports. The motivation for that ban was that existing security measures couldn't detect liquids, but simply announcing a ban was to be enforced didn't change the fact that liquids were undetectable. Instead millions of travelers were pointlessly inconvenienced at great cost. Release the kraken already!
- Judgmentality 7y agoThey do seem to have a bit of a mismatch between what they say and what they do. They wanted to be the non-profit benefiting humanity with their advanced research, but had to raise money because that wasn't working. And now they claim to have these impressive models, but also claim it's not safe to release them to the public. Okay, so what exactly are you developing that's good for anybody?
- skybrian 7y agoUh, you're forgetting about spammers and malware authors.
- jcims 7y agoI’d recommend re-reading the original GPT2 announcement, particularly this section regarding their release policy: This decision, as well as our discussion of it, is an experiment: while we are not sure that it is the right decision today, we believe that the AI community will eventually need to tackle the issue of publication norms in a thoughtful way in certain research areas. This release approach is an experiment used to force the conversation around a release strategy before we actually and unambiguously need it.
- baalimago 7y agoEven if GPT-2 were released, very very few would have the hardware to run it because of gpu ram running out (and doing some sort of load-unload system would make training times unfeasibly long). And those who have the hardware to run it, has probably already made a version of their own or reasons not to. So I'm wondering if this GPT-2 hype is a genuine concern of openai, or if it's mostly a PR flex to say 'Look at us, we made a good model!'. As an example, look here by Nvidia https://devblogs.nvidia.com/training-bert-with-gpus/ https://devblogs.nvidia.com/training-bert-with-gpus/ who made GPT-2 8B, which is ~5 times as large as GPT-2.