5 ms·
Ask HN: Small LM or API?
Is small language models still worth it in 2026, or are most people just using APIs now?
- amoriodi 6mo ago[dead]
- politelemon 6mo agoDepends on what you're using it for, a small model could be viable as long as you're willing to absorb the maintenence overheads of running and deploying your own inference. A simple API would be much more cost effective especially if there are scaling requirements and time constraints.
- ostefani 6mo agoUse for support chat bot. I see a lot of open source models. Not sure if it's worth it. Reply via API from LLM should be better?
- yours3lf 6mo ago[flagged]
- JaceDev 6mo ago[flagged]
- ostefani 6mo agoBut you need to host it? Small model will provide worse results?
- ok_computer_ 6mo agoGemma 4 dropped two days ago and it's a pretty direct answer to this question. Google DeepMind built it explicitly for local deployment, the 26B MoE activates only 3.8B parameters during inference (so it runs at roughly 4B cost while hitting near-31B benchmark quality), and the smaller E4B variant runs fully offline on an 8GB laptop. The 31B Dense currently ranks third among all open models on the Arena AI leaderboard. The quality-per-parameter gap between local and cloud is closing faster than most people expected. That said, "worth it" still depends heavily on your hardware. A 4070 Ti gets you a very different answer than a 3060. Disclosure: I'm building localllm-advisor.com, free and client-side, which also helps answer these types of questions. It shows which models fit your GPU with quantization options and estimated tok/s, or which GPU you'd need to run a specific model. Relevant to the question so I'm mentioning it, but take it for what it is.
- ostefani 6mo agoI've just tried Gemma-4 26b. It's very good. I would say it's the best from what I tried so far. But I wouldn't say it runs at 4B cost. Thank you for sharing your resource :)
- Intelligent_Fox 6mo ago[flagged]
- ostefani 6mo agoThank you for sharing your thoughts about it :)