9 ms·
Outrageously Small Neural Networks: 6,616 tok/s on One Intel AMX Core [pdf]
- gdiamos 11d agoI think we should revisit outrageously small neural nets. I needed a cheap model that runs at over 10k token/sec on a single CPU core for some data processing. So I gave Anthropic claude code a pile of tokens to build one. It made three discoveries that I thought were interesting: 1) One Intel AMX core can train a 3M active parameter MoE foundation model at 6,616 tok/s on 4.91B NVIDIA Nemotron tokens in a few days. 2) That model shows emergent in-context copying, positional analogies, and basic arithmetic after about 250M tokens. 3) The foundation model gives large gains in downstream SFT, and the training & eval loss keep going down all the way through 4.91B (and likely beyond). Claude is not as good as a great MLE at debugging MoE. It made a bunch of bone headed mistakes, but it got there in the end. I asked it to write a paper about it's work, and it produced this. Claude Co-Authored Paper: https://huggingface.co/gdiamos/amx-reasoning-v1-instruct/blob/main/paper.pdf https://huggingface.co/gdiamos/amx-reasoning-v1-instruct/blo... I read through it and it sounds a bit LLMy, but the main points and experiment results are correct. Some of the models are published on HF: https://huggingface.co/gdiamos/amx-reasoning-v1-instruct https://huggingface.co/gdiamos/amx-reasoning-v1-instruct
- gdiamos 11d ago[dead]