6 ms·
What are the geohot leaks?
by biofunsf 3y ago
What are the geohot leaks?
- ZunarJ5 3y agohttps://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/ https://the-decoder.com/gpt-4-architecture-datasets-costs-an... Not op, but this is where a cheeky google got me.
- 1vuio0pswjnm7 3y ago"The idea is nearly 30 years old and has been used for large language models before, such as Google's Switch Transformer." Innovation! :)
- bodecker 3y agoI assume comments like these, "GPT-4: 8 x 220B experts trained with different data/task distributions and 16-iter inference." https://twitter.com/soumithchintala/status/1671267150101721090 https://twitter.com/soumithchintala/status/16712671501017210... https://archive.li/rfFlW https://archive.li/rfFlW I'm not sure the most canonical paper on mixture of experts but here's one possible: https://arxiv.org/pdf/1701.06538.pdf https://arxiv.org/pdf/1701.06538.pdf
- arthurcolle 3y agoI think when ppl refer to MoE they are referring generally to the Google GLaM paper actually
- ta988 3y agoGeorge Hotz (pseudo geohot) in his recent interview with Lex Fridman gave some info on the probable structure of gpt4.