5 ms·
One geometric idea that explains SGD, AdamW, and Muon
- kushalfan87 12d agoWow awesome!!
- crystalskies 12d agoQuite fascinating!
- crystalskies 12d agoWhat are your thoughts on some of the disadvantages of using Muon over AdamW?
- kbwal7 12d agoI'm honestly not too aware of the true scene of which optimizers are used nowadays by which labs (though I think Moonshot has shown Muon's ability to scale to large models, and Muon is definitely more used now). That being said, obviously orthogonalization is pretty expensive! I looked it up and it's roughly 5-15% slower per step, but it converges in fewer steps fwiw. I'm guessing it's also less battle-tested than AdamW is, so AdamW still might be the safer option (and you're gonna have to use AdamW for your 1D params anyways).