Yes. EleutherAI is doing it, probably one of many: https://www.eleuther.ai/projects/gpt-neox/ https://www.eleuther.ai/projects/gpt-neox/ https://github.com/El

Yes. EleutherAI is doing it, probably one of many: https://www.eleuther.ai/projects/gpt-neox/ https://www.eleuther.ai/projects/gpt-neox/ https://github.com/EleutherAI/gpt-neox https://github.com/EleutherAI/gpt-neox https://arxiv.org/abs/2204.06745 https://arxiv.org/abs/2204.06745 They have a 20B parameter model. I think the primary dataset for these open models is The Pile: https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027 (web scrape, pubmed, arxiv, github, wikipedia, etc. There is a nice diagram on page 2 that summarizes the contents.)