6 ms·
Could someone please share how such open source micro-LLMs might have been created? Do the creators take something like DeepSeek, and then delete most of the n
by profsummergig 1mo ago
Could someone please share how such open source micro-LLMs might have been created?
Do the creators take something like DeepSeek, and then delete most of the neurons to whittle down the size?
- HenryNdubuaku 1mo agoTechnically, you could do that, but we trained this one from the ground up!
- profsummergig 1mo agoThat sounds like an enormously expensive exercise.
- genxy 1mo agoYou can train a model of this size on your laptop in a day.
- profsummergig 1mo agoSo is the challenge, now, getting the right sort of data?
- ronsor 1mo agoAt <50M parameters, training costs are completely trivial. You'll spend a lot more on your rent this month.
- HenryNdubuaku 1mo agoHaha, my rent is cheap lol
- salamo 1mo agoAs someone who's done something similar (https://blog.lukesalamone.com/posts/creating-tiny-semantic-search/ https://blog.lukesalamone.com/posts/creating-tiny-semantic-s...) the expensive part wasn't the training itself but the data curation and evaluation post-training. For this, getting a reasonable distribution of tool calls when the tool call can be anything isn't easy. Once you have that, the model is small enough batch sizes are probably enormous and training can probably be done on a consumer-grade GPU in a week or less. Or even faster on a bigger GPU.
- kadoban 1mo agoTraining scales pretty badly, so smaller models like this are really not that bad in terms of cost.
- ComputerGuru 1mo agoAt this size, it certainly doesn’t have to be.
- hgoel 1mo agoAnother option for something this small and narrowly specialized could be to get traditional LLMs to synthesize the training data. Model collapse is probably less of an issue at this size relative to terabyte sized models.
- HenryNdubuaku 1mo agoYour thinking is correct haha
- anigbrowl 1mo agoThere's a Manning book on creating your own LLM from scratch which answers your question exactly. There's another book from the same publisher specifically about small language models for specialty purposes.