6 ms·
I haven't heard of this before. Has Muon dethroned Adam and AdamW as the standard general purpose optimizer for deep learning?
by ComplexSystems 11mo ago
I haven't heard of this before. Has Muon dethroned Adam and AdamW as the standard general purpose optimizer for deep learning?
- spyder 11mo agoIt's for hidden layers and not for every parameter: From Keller's Muon github page: "Muon is an optimizer for the hidden weights of a neural network. Other parameters, such as embeddings, classifier heads, and hidden gains/biases should be optimized using standard AdamW." And I just looked into this nanochat repo and it's also how it's used here. https://github.com/karpathy/nanochat/blob/dd6ff9a1cc23b38ce69ddc119fb220f9ee96cedd/scripts/base_train.py#L129 https://github.com/karpathy/nanochat/blob/dd6ff9a1cc23b38ce6...