6 ms·

Nice setup; but, for simple tasks or questions, AI is currently free? And it will probably stay free, as I don't see Google starting to charge for using AI on i

by bambax 15d ago

Nice setup; but, for simple tasks or questions, AI is currently free? And it will probably stay free, as I don't see Google starting to charge for using AI on its search engine? So costs can't be a motivation for running small models locally? For more complex or important tasks, costs, autonomy and privacy matter, but then so does performance/quality. So I'm not completely convinced it's really worth it; but it's tempting!

Den_VR 15d ago
Only “Free as in free beer.”
- gwd 14d ago
  Maybe, "Free as in free WiFi?" Like WiFi, the models you can use for free online aren't the highest quality, and can be pulled any time. The models used in TFA are halfway in between the traditional "free as in beer" software. Open weight means once you download it, it continues to work forever; and you can also do your own RL on them; but you can't really see what went into their training, nor train a new one yourself from scratch.
RugnirViking 14d ago
I figure most free AI is free as in free electricity in the coffee shop. You're welcome to use it for small reasonable loads, but try to build anything off of it and you'll soon find yourself barred from the establishment. And that's probably good, otherwise the free ai would just be unavailable for everyone else
ajb 14d ago
It's free like ads are free. Or certain kinds of advice.
nobodyandproud 14d ago
At some point and for some tasks, predictability is important if not critical. I’d rather use a tool where I know the limitations, over a tool where the limitations and strengths keep changing. This way I know where in the process I ought to step in and pay attention.
phazonoverload 14d ago
I'm the author - hello! I talk about it in the blog post - knowing what's being run, knowing where it's being run, and not having anyone else control it.
nzxt210 14d ago
Traditional search is “free” too, but you see ads. If something looks free, then you are the product.
- DennisP 14d ago
  OpenAI has started doing ads, but for Anthropic the free tier is still just a loss leader. It's basically an ad for their pay tiers.
numpad0 14d ago
Getting undeserved safety filter refusals, watching LLM accidentally reading what you might not find comfortable showing, and hitting free tier/basic subscription quotas, are not fun. Going local removes somewhat irrational anxiety that come from those.
ActorNightly 14d ago
You are correct. In large part, the cost of something like Gemini on a very basic Google AI plan provides far more utility than local LLMs for coding assistance. There are 2 main reasons for running local LLMS. 1. Process private data/work with uncensored models. 2. Use a large amount of inference that would quickly blow through rate limits and/or run up API costs. The thing that is critical for 2 is that a) you have to have a sweetspot between a pretty good model, which means largest parameter counts, and fast enough token generation where you can run agentic loops. The latter is needed because you aren't going go get the "intelligence" of larger models to form shell commands and run tools to figure stuff out, so the only way around that is to have custom agentic loops to force the model into doing what you want, which results in more text processing. From my testing, Gemma4:31b is basically the only local model that can be relied upon to produce accurate results. Qwen models chase benchmarks, which results in MoE models (thus the A3B in the model, i.e 3 billion parameters are only active during inference). In general, these are good for very specific tasks, but fail to be accurate in considering cross task data, whereas Gemma, being fully active does a much better job. If you only need to do a very specific deterministic task, those models are pretty good. As an aside though, if your task involves pure text processing (for example take html data, make it into a markdown document), you can also additive train Gemma270M quite easily all on CPU, and on a decent CPU it gets like 50-100 tok/sec, no need for any extra hardware. The thing with Macs is that while they can run those models and larger models no problem, the tok/sec is very slow. This limits effectively what you can do with the models. On the M4 that the poster mentioned, Gemma:31b will run about 20 tok/sec. That means that when you wants to write a whole code file or process large context, you have to wait for it to do things. Compared to workflow with larger models, where file generation often takes like <10 seconds, it takes a while to adapt. The only benefit of using Macs is the price for Mini and cheaper studios. However, once you reach the total cost of about 2.5k (note that the M4 statedin the article us about 2k), building a gfx card rig is the way to go. You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models.
- bambax 14d ago
  > You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models I have a beefy Linux box with a 4090 but never took the time to set it up properly beyond simple testing; any tutorial you would recommend?

by bambax 15d ago

Den_VR 15d ago

Only “Free as in free beer.”

gwd 14d ago
Maybe, "Free as in free WiFi?" Like WiFi, the models you can use for free online aren't the highest quality, and can be pulled any time. The models used in TFA are halfway in between the traditional "free as in beer" software. Open weight means once you download it, it continues to work forever; and you can also do your own RL on them; but you can't really see what went into their training, nor train a new one yourself from scratch.

RugnirViking 14d ago

I figure most free AI is free as in free electricity in the coffee shop. You're welcome to use it for small reasonable loads, but try to build anything off of it and you'll soon find yourself barred from the establishment. And that's probably good, otherwise the free ai would just be unavailable for everyone else

ajb 14d ago

It's free like ads are free. Or certain kinds of advice.

nobodyandproud 14d ago

At some point and for some tasks, predictability is important if not critical. I’d rather use a tool where I know the limitations, over a tool where the limitations and strengths keep changing. This way I know where in the process I ought to step in and pay attention.

phazonoverload 14d ago

I'm the author - hello! I talk about it in the blog post - knowing what's being run, knowing where it's being run, and not having anyone else control it.

nzxt210 14d ago

Traditional search is “free” too, but you see ads. If something looks free, then you are the product.

DennisP 14d ago
OpenAI has started doing ads, but for Anthropic the free tier is still just a loss leader. It's basically an ad for their pay tiers.

numpad0 14d ago

Getting undeserved safety filter refusals, watching LLM accidentally reading what you might not find comfortable showing, and hitting free tier/basic subscription quotas, are not fun. Going local removes somewhat irrational anxiety that come from those.

ActorNightly 14d ago

You are correct. In large part, the cost of something like Gemini on a very basic Google AI plan provides far more utility than local LLMs for coding assistance. There are 2 main reasons for running local LLMS. 1. Process private data/work with uncensored models. 2. Use a large amount of inference that would quickly blow through rate limits and/or run up API costs. The thing that is critical for 2 is that a) you have to have a sweetspot between a pretty good model, which means largest parameter counts, and fast enough token generation where you can run agentic loops. The latter is needed because you aren't going go get the "intelligence" of larger models to form shell commands and run tools to figure stuff out, so the only way around that is to have custom agentic loops to force the model into doing what you want, which results in more text processing. From my testing, Gemma4:31b is basically the only local model that can be relied upon to produce accurate results. Qwen models chase benchmarks, which results in MoE models (thus the A3B in the model, i.e 3 billion parameters are only active during inference). In general, these are good for very specific tasks, but fail to be accurate in considering cross task data, whereas Gemma, being fully active does a much better job. If you only need to do a very specific deterministic task, those models are pretty good. As an aside though, if your task involves pure text processing (for example take html data, make it into a markdown document), you can also additive train Gemma270M quite easily all on CPU, and on a decent CPU it gets like 50-100 tok/sec, no need for any extra hardware. The thing with Macs is that while they can run those models and larger models no problem, the tok/sec is very slow. This limits effectively what you can do with the models. On the M4 that the poster mentioned, Gemma:31b will run about 20 tok/sec. That means that when you wants to write a whole code file or process large context, you have to wait for it to do things. Compared to workflow with larger models, where file generation often takes like <10 seconds, it takes a while to adapt. The only benefit of using Macs is the price for Mini and cheaper studios. However, once you reach the total cost of about 2.5k (note that the M4 statedin the article us about 2k), building a gfx card rig is the way to go. You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models.

bambax 14d ago
> You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models I have a beefy Linux box with a 4090 but never took the time to set it up properly beyond simple testing; any tutorial you would recommend?