kbwal7

6 ms·

1. 1.▲
   by kbwal7 12d ago

   I'm honestly not too aware of the true scene of which optimizers are used nowadays by which labs (though I think Moonshot has shown Muon's ability to scale to large models, and Muon is definitely more used now). That being said,
2. 2.▲
   One geometric idea that explains SGD, AdamW, and Muon(kbwal.github.io)

   6 points by kbwal7 12d ago|4 comments
3. 3.▲
   by kbwal7 12d ago

   I do think there are now open weight models that are on par with (or beating) Opus 4.5 by now (e.g. Kimi K3, GLM5.3). But yeah obviously the frontier closed source models seem to have pulled away once again, so open weight seems to be a few
4. 4.▲
   by kbwal7 12d ago

   Olmo is one truly open source model. https://allenai.org/blog/olmo3
5. 5.▲
   by kbwal7 12d ago

   I think it's just a harness that uses proprietary (or maybe IBM is hosting open weight?) models. They say on the site (in the FAQs) that you can't even choose which model! So it seems to be a harness that auto-routes your queries
6. 6.▲
   by kbwal7 13d ago

   It's pretty incredible how robust neural networks are to even things like architectural changes mid-run. The team at Meta when training their own copy of GPT-3 (called OPT-3) even changed optimizers mid-run (from AdamW -> SGD ->