5 ms·

I don't get this obsession with smaller models. I've been using Claude and GPT models for years and have had zero issues with them. I see absolutely no benefit

I don't get this obsession with smaller models. I've been using Claude and GPT models for years and have had zero issues with them. I see absolutely no benefit to me as a end user for a local model which is going to take up more of my CPU and memory and slow down my machine. I almost always have Internet and if I don't then not having access to a AI model is the least of my concerns.

adam_arthur 3mo ago
The entire universe of automation projects that can be run effectively for free relative to SoTA models? I don't think many realize that most LLM embedded automation, pipelines, products will soon be able to run extremely cheaply on models < 100B parameters. Frontier models will be used for coding/creation use cases, yes. But for all the pseudo-deterministic, pipeline, analysis style things there will be no practical benefit to running frontier models, only additional cost. Gemma 4 26B outperforms most 100-200B models that I've tested for reasoning and structured output. Gemma 4 12B can consistently select where to click on browser images given a minimal prompt, and do so very quickly.
- steno132 3mo ago
  Practically if you're running a small personal automation project you're not going to want to waste a lot of time configuring and tuning a local model. You want to build the automation and move on. If you're building a automation as a company you definitely won't want to take on the long term maintenance overhead of running your own models for some automation project.
  - adam_arthur 3mo ago
    These small models exist in the cloud and are/will be priced commensurately to their size. Your claim is effectively that companies don't care about operational/cloud costs. Even pre-LLM, companies regularly assessed and tried to pare down cloud spend.
- dofm 3mo ago
  The 26B model is really surprising, and it is impressively concise — it spends a lot less time dithering than Qwen3.6.
Zambyte 3mo ago
I like using my computer.
- steno132 3mo ago
  Exactly, thank you, we are on the same page! It's great to be able to use our own devices and not have their compute coopted by a third party. I'd rather not have intensive compute needed shifted onto my personal machine which I want to use for something else.
  - Zambyte 3mo ago
    I am not a "third party" on my own computer.
  - satvikpendem 3mo ago
    By that logic, any software you run that isn't fully built by yourself is "third party" therefore you shouldn't run anything at all on your machine, thus obviating the need for it entirely.
    - steno132 3mo ago
      But practically AI inference requires substantial local computing resources. It's not some web app, it's a order of magnitude more compute needed
      - deleted 3mo ago
        [deleted]
      - Zambyte 3mo ago
        Hopefully now you understand why people want smaller models.
      - satvikpendem 3mo ago
        Not really, I run a production service on a basic server using these Gemma models, the server is weaker than my MacBook. Most people's laptops and even phones actually can run local models, most simply don't know how. Run Unsloth Studio and you'll see how easy it is. As the sibling says this is why people want smaller but still performant models.
user2722 3mo ago
There is tinfoil.sh as well but honestly running this stuff on an airgapped server allows a better peace of mind about the data being used for something else.
- steno132 3mo ago
  What's wrong with the data being used for something else? Someone is providing digital intelligence to us, saving us many hours a week, so the least we can do is provide them a little data so they are able to improve their service. It would be selfish and unethical not to in my view. And ultimately the data is just being used in order to improve the models and benefit us, not for anything nefarious.
  - NicuCalcea 3mo ago
    If sharing our data is the least we can do, they shouldn't also ask us for our money. Otherwise, it's more than the least.
mikeocool 3mo ago
> I've been using Claude and GPT models for years All 3 years?
- steno132 3mo ago
  GPT1 was released in 2018, so yes, since then.
  - victorbjorklund 3mo ago
    GPT1 was way worse than small Gemma’s are now.
mannanj 3mo ago
I don't like the gaslighting of paying Anthropic or Open(Closed)AI and it being said its unsustainable for them to take my payment while simultaneously they take my data (edit: which is incredibly valuable) and I cannot opt out of that. The obsession is for leaving hostile and abusive entities, the corporations or the people who fund them that have a horrible track record in regards to ethicality, rights and respect & human dignity.
- steno132 3mo ago
  My view is, if you're going to use the service - you should give the data. It's like using Gmail and expecting them not to train their AI models on your data - how can you expect that when they're giving you a secure, reliable, highly functional email client completely for free? The digital economy only works if everyone pays their fair share. If you don't want to give your data then you are really harming everyone by slowing down AI development for everyone else.
  - klardotsh 3mo ago
    Because we pay for the models. If I pay you for a service, what implicit right should you have to then continue to profit in perpetuity by storing the data I paid you to process? If LLMs were free your Gmail analogy might hold up. They aren’t, and so it doesn’t. AI development can continue with the data folks opt into, or with the data AI companies incessantly scrape with reckless disregard for polite system loads. AI development does not require retaining all user inputs forever.
  - mannanj 3mo ago
    However, you didn't actually get what I meant down, so you ended up inadvertently Straw Manning me. My disinterest is in sharing my intellectual IP. Most people up to now, have never shared this much of their intellectual IP with a company. Name one product through human history before that got this much data and insight into human thinking and now can use your most intimate conversations, ideas and needs for non-training purposes? You can't even opt out of that! At least for the training data you can opt-out.
sowbug 3mo ago
Whatever you're doing, try doing 500 or 1,000 of it in a batch. You'll exhaust any subscription quota you have, or if you're paying per token, you will probably find it too expensive. That's when you'll start to ask "how smart a model do I really need for this job?", and you'll investigate running a small but sufficiently capable model on your own PC, churning overnight through your 1,000 tasks.

adam_arthur 3mo ago

The entire universe of automation projects that can be run effectively for free relative to SoTA models? I don't think many realize that most LLM embedded automation, pipelines, products will soon be able to run extremely cheaply on models < 100B parameters. Frontier models will be used for coding/creation use cases, yes. But for all the pseudo-deterministic, pipeline, analysis style things there will be no practical benefit to running frontier models, only additional cost. Gemma 4 26B outperforms most 100-200B models that I've tested for reasoning and structured output. Gemma 4 12B can consistently select where to click on browser images given a minimal prompt, and do so very quickly.

steno132 3mo ago
Practically if you're running a small personal automation project you're not going to want to waste a lot of time configuring and tuning a local model. You want to build the automation and move on. If you're building a automation as a company you definitely won't want to take on the long term maintenance overhead of running your own models for some automation project.
- adam_arthur 3mo ago
  These small models exist in the cloud and are/will be priced commensurately to their size. Your claim is effectively that companies don't care about operational/cloud costs. Even pre-LLM, companies regularly assessed and tried to pare down cloud spend.
dofm 3mo ago
The 26B model is really surprising, and it is impressively concise — it spends a lot less time dithering than Qwen3.6.

Zambyte 3mo ago

I like using my computer.

steno132 3mo ago
Exactly, thank you, we are on the same page! It's great to be able to use our own devices and not have their compute coopted by a third party. I'd rather not have intensive compute needed shifted onto my personal machine which I want to use for something else.
- Zambyte 3mo ago
  I am not a "third party" on my own computer.
- satvikpendem 3mo ago
  By that logic, any software you run that isn't fully built by yourself is "third party" therefore you shouldn't run anything at all on your machine, thus obviating the need for it entirely.
  - steno132 3mo ago
    But practically AI inference requires substantial local computing resources. It's not some web app, it's a order of magnitude more compute needed
    - deleted 3mo ago
      [deleted]
    - Zambyte 3mo ago
      Hopefully now you understand why people want smaller models.
    - satvikpendem 3mo ago
      Not really, I run a production service on a basic server using these Gemma models, the server is weaker than my MacBook. Most people's laptops and even phones actually can run local models, most simply don't know how. Run Unsloth Studio and you'll see how easy it is. As the sibling says this is why people want smaller but still performant models.

user2722 3mo ago

There is tinfoil.sh as well but honestly running this stuff on an airgapped server allows a better peace of mind about the data being used for something else.

steno132 3mo ago
What's wrong with the data being used for something else? Someone is providing digital intelligence to us, saving us many hours a week, so the least we can do is provide them a little data so they are able to improve their service. It would be selfish and unethical not to in my view. And ultimately the data is just being used in order to improve the models and benefit us, not for anything nefarious.
- NicuCalcea 3mo ago
  If sharing our data is the least we can do, they shouldn't also ask us for our money. Otherwise, it's more than the least.

mikeocool 3mo ago

> I've been using Claude and GPT models for years All 3 years?

steno132 3mo ago
GPT1 was released in 2018, so yes, since then.
- victorbjorklund 3mo ago
  GPT1 was way worse than small Gemma’s are now.

mannanj 3mo ago

I don't like the gaslighting of paying Anthropic or Open(Closed)AI and it being said its unsustainable for them to take my payment while simultaneously they take my data (edit: which is incredibly valuable) and I cannot opt out of that. The obsession is for leaving hostile and abusive entities, the corporations or the people who fund them that have a horrible track record in regards to ethicality, rights and respect & human dignity.

steno132 3mo ago
My view is, if you're going to use the service - you should give the data. It's like using Gmail and expecting them not to train their AI models on your data - how can you expect that when they're giving you a secure, reliable, highly functional email client completely for free? The digital economy only works if everyone pays their fair share. If you don't want to give your data then you are really harming everyone by slowing down AI development for everyone else.
- klardotsh 3mo ago
  Because we pay for the models. If I pay you for a service, what implicit right should you have to then continue to profit in perpetuity by storing the data I paid you to process? If LLMs were free your Gmail analogy might hold up. They aren’t, and so it doesn’t. AI development can continue with the data folks opt into, or with the data AI companies incessantly scrape with reckless disregard for polite system loads. AI development does not require retaining all user inputs forever.
- mannanj 3mo ago
  However, you didn't actually get what I meant down, so you ended up inadvertently Straw Manning me. My disinterest is in sharing my intellectual IP. Most people up to now, have never shared this much of their intellectual IP with a company. Name one product through human history before that got this much data and insight into human thinking and now can use your most intimate conversations, ideas and needs for non-training purposes? You can't even opt out of that! At least for the training data you can opt-out.

sowbug 3mo ago

Whatever you're doing, try doing 500 or 1,000 of it in a batch. You'll exhaust any subscription quota you have, or if you're paying per token, you will probably find it too expensive. That's when you'll start to ask "how smart a model do I really need for this job?", and you'll investigate running a small but sufficiently capable model on your own PC, churning overnight through your 1,000 tasks.