6 ms·

Genuinely curious: what are people using these smaller local models for? They are getting decently capable, but they are still small enough that I don't trust t

by JosNun 2mo ago

Genuinely curious: what are people using these smaller local models for? They are getting decently capable, but they are still small enough that I don't trust them for "real" work outside of a handful of fun toy projects. Are people actually using them in coding agents? Or are they mostly using them for other things?

teaearlgraycold 2mo ago
They're great at helping me look up web dev stuff when I don't have internet access.
kgeist 2mo ago
We've shipped some code generated by Qwen3.6 27B to production (under OpenCode). It lacks the breadth of knowledge of models like Opus, but if a change is fully inferable from the prompt and the surrounding code, it works very well. It won't be able to write something from scratch that requires niche knowledge (say, a performant inference engine tailored to Blackwell GPUs), but if it's just a PR adding a new use case to an existing project (which is usually just "load from the DB, do some invariant checks, modify the entities, store them back"), it works as well as Sonnet (provided you have the correct configuration, like recommended temperature and top-p settings, the model isn't over-quantized, you have at least 150k tokens of context available, etc.).
jiqiren 2mo ago
there is plenty of grunt work these smaller models can do. update dependencies, fix merge conflicts, write --help, markdown, or readme files for existing code. etc. sometimes they fail but undo is just a "git restore" or if automated, rejecting a PR and having a better model take a crack at it.
mips_avatar 2mo ago
Qwen35ba3b can do a huge amount of data cleaning work on pretty modest hardware. Already have run about 100 billion tokens on it using 2x3090 gpus.
- febed 2mo ago
  What exactly do you mean by data cleaning
  - mips_avatar 2mo ago
    The coolest project I’ve got this running on is improving the depicts metadata for photos on Wikipedia. A lot of times they won’t have the landmarks tagged correctly in a photo. So I will load in all the metadata that exists from each photo and the pixels of those photos and give a small qwen agent access to Wikipedia search as well as a geocoder. It does a great job of figuring out what is depicted and tagging it with the correct depicts field. Im still early on but I have been able to double the number of places that have a photo attached to them on wikimedia
    - febed 2mo ago
      Interesting project. So I guess the search tool is to crawl Wikipedia for articles? And how do you ensure that the tagging stays within the Wikidata taxonomy? How exactly are you using a geocoder? Sorry, just curious
      - mips_avatar 2mo ago
        So like oftentimes the picture will be of a church and there’s geographic coordinates for where the photo was taken. My qwen will use the geocoder to search for “church” at the coordinates of the photo and then read the Wikipedia articles about all the churches nearby and see if any of them could plausibly be the church. So far I have parsed about 2 million photos and have tagged about 800k places. My goal is to do the whole 40 million places to create a world map of open places with photos. The tool I’m using is topoloop for the geocoding
- noja 2mo ago
  are you using it differently to OpenRefine?
  - 2mo ago
ActorNightly 2mo ago
[flagged]
- kridsdale1 2mo ago
  Wow what a hater. You know what else is thousands of dollars and doesn’t even include a monitor? An ATX case with 2 3090s in it. And that will use a kilowatt or more to do what the Studio (which is excellent) does with about 250 W.
  - ActorNightly 2mo ago
    Compute is compute. You need power to make LLMs go brrr. And you can get a case with 2 3090s for about 5k, which is still less than what a "comparable" top of the line Mac Studio costs.
- invictati 2mo ago
  Apple would not waste money astroturfing on HN to sell O(1000) Macs.
  - ActorNightly 2mo ago
    No, they would spend money on creating a trend, like "Macs are good for AI". They literally did this before AI with Macs being good for programming, because "they are Unix like", "have the best trackpad" and other bullshit, and as a result, new CS grads were convinced that Macs are the best for programming.
netghost 2mo ago
I don't use them as coding agents, but they can be very useful for things like text transformation, summarizing, or text extraction. That said, if you have a subscription to a paid model already, you're not necessarily winning out on anything except perhaps privacy, which isn't nothing.
efficax 2mo ago
I use a gemma4 model locally to extract content from messages to a personal agent I'm building for its memory graph (to break the message up into the topic, source (assistant or owner), facts, entities, etc. in the message content (all getting thrown into a magma-esque graph using NLEmbeddings for memory search). This is for a custom personal agent that targets deepseek-v4 flash. The local model is too slow in my setup for a chat agent, but for memory extraction it works pretty well, saving API usage on every chat turn.

by JosNun 2mo ago

teaearlgraycold 2mo ago

They're great at helping me look up web dev stuff when I don't have internet access.

kgeist 2mo ago

We've shipped some code generated by Qwen3.6 27B to production (under OpenCode). It lacks the breadth of knowledge of models like Opus, but if a change is fully inferable from the prompt and the surrounding code, it works very well. It won't be able to write something from scratch that requires niche knowledge (say, a performant inference engine tailored to Blackwell GPUs), but if it's just a PR adding a new use case to an existing project (which is usually just "load from the DB, do some invariant checks, modify the entities, store them back"), it works as well as Sonnet (provided you have the correct configuration, like recommended temperature and top-p settings, the model isn't over-quantized, you have at least 150k tokens of context available, etc.).

jiqiren 2mo ago

there is plenty of grunt work these smaller models can do. update dependencies, fix merge conflicts, write --help, markdown, or readme files for existing code. etc. sometimes they fail but undo is just a "git restore" or if automated, rejecting a PR and having a better model take a crack at it.

mips_avatar 2mo ago

Qwen35ba3b can do a huge amount of data cleaning work on pretty modest hardware. Already have run about 100 billion tokens on it using 2x3090 gpus.

febed 2mo ago
What exactly do you mean by data cleaning
- mips_avatar 2mo ago
  The coolest project I’ve got this running on is improving the depicts metadata for photos on Wikipedia. A lot of times they won’t have the landmarks tagged correctly in a photo. So I will load in all the metadata that exists from each photo and the pixels of those photos and give a small qwen agent access to Wikipedia search as well as a geocoder. It does a great job of figuring out what is depicted and tagging it with the correct depicts field. Im still early on but I have been able to double the number of places that have a photo attached to them on wikimedia
  - febed 2mo ago
    Interesting project. So I guess the search tool is to crawl Wikipedia for articles? And how do you ensure that the tagging stays within the Wikidata taxonomy? How exactly are you using a geocoder? Sorry, just curious
    - mips_avatar 2mo ago
      So like oftentimes the picture will be of a church and there’s geographic coordinates for where the photo was taken. My qwen will use the geocoder to search for “church” at the coordinates of the photo and then read the Wikipedia articles about all the churches nearby and see if any of them could plausibly be the church. So far I have parsed about 2 million photos and have tagged about 800k places. My goal is to do the whole 40 million places to create a world map of open places with photos. The tool I’m using is topoloop for the geocoding
noja 2mo ago
are you using it differently to OpenRefine?
- 2mo ago

ActorNightly 2mo ago

[flagged]

kridsdale1 2mo ago
Wow what a hater. You know what else is thousands of dollars and doesn’t even include a monitor? An ATX case with 2 3090s in it. And that will use a kilowatt or more to do what the Studio (which is excellent) does with about 250 W.
- ActorNightly 2mo ago
  Compute is compute. You need power to make LLMs go brrr. And you can get a case with 2 3090s for about 5k, which is still less than what a "comparable" top of the line Mac Studio costs.
invictati 2mo ago
Apple would not waste money astroturfing on HN to sell O(1000) Macs.
- ActorNightly 2mo ago
  No, they would spend money on creating a trend, like "Macs are good for AI". They literally did this before AI with Macs being good for programming, because "they are Unix like", "have the best trackpad" and other bullshit, and as a result, new CS grads were convinced that Macs are the best for programming.

netghost 2mo ago

I don't use them as coding agents, but they can be very useful for things like text transformation, summarizing, or text extraction. That said, if you have a subscription to a paid model already, you're not necessarily winning out on anything except perhaps privacy, which isn't nothing.

efficax 2mo ago

I use a gemma4 model locally to extract content from messages to a personal agent I'm building for its memory graph (to break the message up into the topic, source (assistant or owner), facts, entities, etc. in the message content (all getting thrown into a magma-esque graph using NLEmbeddings for memory search). This is for a custom personal agent that targets deepseek-v4 flash. The local model is too slow in my setup for a chat agent, but for memory extraction it works pretty well, saving API usage on every chat turn.