6 ms·
This guy doesn't have any idea what he is talking about. He consistently posts such bullshit on twitter. Mostly copy paste with added spice mix.
by shahules 3y ago
This guy doesn't have any idea what he is talking about. He consistently posts such bullshit on twitter. Mostly copy paste with added spice mix.
- mk_stjames 3y agoI noted several things that don't seem consistent with what people have been assuming from before. For instance - MoE yes, but 16 experts at 111B parameters? Doesn't make sense. GPT 3 had 175B parameters. I doubt they would go less on base models from now on. The number that makes more sense is ~220B parameters per model and 8 expert models. That is the same inference cost in total. The 13T tokens of training data seems pulled from thin air.
- mt_ 3y agoIt's Twitter, why would you think otherwise?
- s3p 3y agoI'm tired of people just saying 'oh, internet' every time there is a factual inaccuracy somewhere. Yes, we know this is a social media network. Now can we get back to discussing the topic at hand?