stephenroller

6 ms·

1.
by stephenroller 1y ago
with hf transformers, you still need to manage GPUs
2.
by stephenroller 2y ago
Why did the share of childless women 35-44 suddenly drop 5 points around 2012? Such a massive difference must be a change in measurement methodology?
3.
by stephenroller 3y ago
The llama1 team did not have a validation set. I don’t know what the Llama2 team did - I left before seeing any of the details. My guess is Llama2 upsamples Wikipedia a good bit, but given they didn’t report any information about training
4.
by stephenroller 3y ago
One noteworthy thing is that no one is posting validation curves, only training curves. All these models will happily bring training loss eventually to near zero with infinite compute, as the model overfits to the dataset -- there are no re
5.
by stephenroller 4y ago
Updating the model happens offline to ensure protections of this. As part of the research on BlenderBot 3 we also have improved detections for safety and can learn from "trolls." https://parl.ai/projects/troll
6.
by stephenroller 4y ago
We trained on Reddit comments and HackerNews comments.
7.
by stephenroller 5y ago
I run a fairly large open source project ( https://github.com/facebookresearch/ParlAI/ ) and we use mypy. Our experience has been that it can be quite difficult to placate, so we usually treat it only as a warning.
8.
by stephenroller 6y ago
https://github.com/pytorch/fairseq/pull/3331 and https://github.com/pytorch/fairseq/pull/3327
9.
by stephenroller 6y ago
Support for this was also added to [Fairscale]( https://fairscale.readthedocs.io/en/latest/ ) and [Fairseq]( https://github.com/pytorch/fairseq ) last week. In particular, the Fairscale implement
10.
by stephenroller 6y ago
They don't. But half the people got a placebo and half got the vaccine. 185 in the placebo group got covid, with 30 having severe covid, but 11 with covid and 0 severe in the vaccine group got it. So do you really think that totally by
11.
by stephenroller 6y ago
Streets would be much more useful than a true north compass.
12.
by stephenroller 6y ago
Perhaps you are outside the United States? PBS is the public television channel, so perhaps they geolock :(
13.
by stephenroller 6y ago
The dataset can be obtained around the web. It's mostly CommonCrawl, Reddit, Toronto Book Corpus, and Wikipedia. You can find a very comparable corpus open sourced and easy to use on the [T5 repo]( https://github.com/goo
14.
by stephenroller 6y ago
No, they aren't releasing the weights. They are releasing it as ML as a service. Right now it's in free beta, but it will open up for commercial usage in the future. On another note: At 175B parameters, with float16 representation
15.
by stephenroller 6y ago
I would not encourage using the model for anything other than AI research -- we're still in the early days of dialogue, and there are a lot of unexplored avenues. There are still nuances around safety, controlling generation, consisten
16.
by stephenroller 6y ago
(BlenderBot original developer here) You can read the paper here, which contains much more technical details of how we built it: https://arxiv.org/abs/2004.13637
17.
by stephenroller 6y ago
Hi there, thanks for asking! It's very cool to make research more accessible. Would you be able to rename your experience to "Modified Small BlenderBot without Safety Layer"? And I think it’d be helpful to call out what chang
18.
by stephenroller 6y ago
Hi there. One of the original authors of the BlenderBot paper here. This cocohub website is not what we released. It appears that cocohub has removed our safety layer and provided other modifications to the bot. We released our model open s
19.
by stephenroller 7y ago
Embeddings tables aren't hard on the GPU (being only a lookup table), and the output softmax still requires you do the full matrix-multiply. The label may be sparse, but the computation is far from sparse.
20.
Recent advances in low-resource machine translation(ai.facebook.com)
7 points by stephenroller 7y ago|0 comments
21.
by stephenroller 7y ago
We limited the scope of our study to the PersonaChat dataset, which is a standard benchmark in dialogue. In PersonaChat, both participants are presented with Personas and asked to roleplay as that person. (e.g. "I love the beach; Horse
22.
by stephenroller 7y ago
Yes, that's a great response for one of the models to have. :)
23.
by stephenroller 7y ago
We originally set out to see whether we could use conditional generation to sort of adjust the behavior of these neural generative dialogue systems. Much of the original idea was about sort of "planning" our response to improve th
24.
by stephenroller 7y ago
Co-author of the original paper, if you have any questions.