craffel

6 ms·

1.
When will language models be good enough?(colinraffel.com)
2 points by craffel 2mo ago|1 comments
2.
by craffel 5y ago
(different author, not Stella) To your first question: Unpublished experiments done by the BigScience architecture and scaling WG suggest that training on book corpus yields a boost of 10-15% accuracy on LAMBADA. To your second question: LA
3.
by craffel 5y ago
(author here) The paper/model/code was just made public today. This may be why no one is talking about it yet. Regarding whether the size is a hassle: It's possible to run inference on a single Google Cloud TPU v3-8 device or
4.
ICLR Workshop on Enormous Language Models – May 7th, 2021 (Livestream)(welmworkshop.github.io)
1 points by craffel 5y ago|0 comments
5.
by craffel 7y ago
Yes, that's me! Sorry if I'm being overeager, I like talking about my research!
6.
by craffel 7y ago
Agreed! The interesting thing is that basic unsupervised pre-training seems to produce a model which functions not only as a knowledge base but also an NLU system which can effectively query the knowledge base using natural text questions.
7.
by craffel 7y ago
It's slightly more than that -- it also involves lowercasing and removing articles before testing for string equality.
8.
by craffel 7y ago
We include MASS in our empirical survey (see e.g. section 3.3.2 of our paper, https://arxiv.org/pdf/1910.10683.pdf ). FWIW, people were pre-training Transformers before MASS, e.g. "Improving Language Understanding
9.
by craffel 7y ago
The blogpost has a summary of our paper from October (a bit late, sorry!) but also has some (fun?) new results on closed-book question answering and fill-in-the-blank text generation.
10.
by craffel 7y ago
Thanks, fixed!
11.
by craffel 7y ago
Yes, unfortunately we have to rely on the very brittle "exact match" method of evaluating whether an answer is correct. FWIW and perhaps surprisingly, this is the primary way question-answering systems are evaluated in common benc
12.
by craffel 7y ago
It actually can be more pernicious than that: https://arxiv.org/abs/1802.08232 However note that the dataset used to train GPT-2 is about 20x smaller than C4. I'm not 100% sure how many times the training set was
13.
by craffel 7y ago
To be clear, here is the claim we make in the paper (we did not write the title of this post to HN): > For SuperGLUE, we improved upon the state-of-the-art by a large margin (from an average score of 84.6 [Liu et al., 2019c] to 88.9). Su
14.
by craffel 7y ago
Hi, one of the paper authors here. Indeed this is a good question. A couple of comments: - Common Crawl overall is a sparse web dump, it is unlikely that the month we used includes any of the data that are in any of the test sets. - In orde
15.
by craffel 7y ago
Hi, one of the paper's authors here. We didn't submit our model's predictions for the AX-b task yet, we just copied over the predictions from the example submission. We will submit predictions for AX-b in the next few days.