6 ms·

I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40

by Akuehne 4mo ago

I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. With modern Apple, Intel, and AMD silicon all shipping dedicated AI acceleration, this is the where IMO the focus should be heading. How many Flops or whatever can your phone do? I bet it's enough to paint the walls of your living room, or draw a pretty good pelican on a bike.

andybak 4mo ago
> draw a pretty good pelican on a bike. You mean the famously hard task? The one picked because it stretches frontier models to their limits?
- daveguy 4mo ago
  https://simonwillison.net/2026/Apr/22/qwen36-27b/ https://simonwillison.net/2026/Apr/22/qwen36-27b/ Maybe this is an example of training overfit. But it won't be too long before local models chew through the "famously hard tasks". Except possibly ARC-AGI. That's one benchmark that is still developing with capabilities. And every time a new ARC-AGI benchmark is released it make the SOTA LLMs look pathetic. Because there is very little understanding or transferability with LLMs. But in terms of benchmark-able micro tasks, the local LLMs are improving.
- quantummagic 4mo ago
  In fairness, that isn't due to a lack of compute.
- munk-a 4mo ago
  It was a famously hard task. It was an ingenious idea for an unexpected task that falls outside of the bounds of predictable normal input but is still readily comprehended by the public. Unfortunately, as soon as it's a famously hard task trainers know they need to succeed at it and it loses a lot of the power to detect correctness.
deleted 4mo ago
[deleted]
AdeelKhamisa 4mo ago
I was writing just about this last week for fun: AI + hardware team-up to build localized AI with specialized functions to your organization. Ex. Adoble Studio AI in an on premise Box, made by Apple and powered by something like Cohere with privacy: https://www.notion.so/adeelkhamisa/Cohere-s-next-steps-to-become-the-dominant-AI-company-for-sovereignty-and-privacy-35ba5de2f8018096a4d8e35206b75abb?source=copy_link https://www.notion.so/adeelkhamisa/Cohere-s-next-steps-to-be...
henry_kang 4mo ago
Actually you can do way more things than that. We have optimized it to process 2TB of high def videos on a M5 MBP in under 24 hours, including everything such as speech understanding, face recog, LLM and VLM. Super fun.
- sanderjd 4mo ago
  Is this project public or have you written about it anywhere?
  - deleted 4mo ago
    [deleted]
  - henry_kang 4mo ago
    Yeah, we've recently made it public. You can check it out here: https://clipto.com https://clipto.com Be aware that it is still a beast that sucks in a lot of memory. Oh, one more thing ;) remember to keep your Mac plugged in...
- jimbokun 4mo ago
  If Steve Jobs was alive Apple would have already demoed this as a new line of Macs with open weight models pre-installed with hooks into all of their existing content creation software. And he would have the audience believing all the demos were running through third party AI providers, until at the last moment explaining “actually all of that ran on device with no connection to any external services.”
  - henry_kang 4mo ago
    hhh, "one more thing"
DrewADesign 4mo ago
And this is exactly what the LLM provider industry is fighting tooth-and-nail. It’s not only because it doesn’t directly contribute to their bottom line, it also directly opposes the idea that LLMs are going to replace entire workers rather than enhance the abilities of individual workers. What we’re headed towards would have been a killer product and probably still shifted a bunch of capital to the bazillionaires had these companies set more realistic goals rather than banking that they’d be the ones that won the war that “changed everythingTM”.
- achenet 4mo ago
  mainframe industry vs personal computers. If their product is "access to a big model running on a really big computer" (if we can count 'multiple data-centers' as a single enormous distributed computer), then the product "small, accessible device that everyone has" risks killing their cash cow. Ironically enough, the first company to really focus on "an LLM in every phone" will have a good shot at actually being the ones that "changed everythingTM", in the way Microsoft changed the world from IBM mainframes to PCs, or Apple made smartphones a thing.
  - DrewADesign 4mo ago
    As an aside, the mainframe industry was profitable for decades before PCs took over. It’s not like they spent a zillion dollars ramping up at the same time.
    - albuic 4mo ago
      The mainframe industry IS profitable.
- voncheese 4mo ago
  As long as Apple and Google put reasonable AI capabilities on device, then software engineers will use those capabilities when it makes sense (the article gives lots of good examples of capabilities that make sense to run locally). As the author notes, it's cheaper and more reliable to run these things locally. That also doesn't preclude LLM services from being massively successful, they'll just have to justify the pricing and complexity that comes with their adoption, just like any other product.
zozbot234 4mo ago
A phone makes a very crappy AI inference rig. It's battery powered and can't even really run at 100% utilization on an ongoing basis due to how challenging the thermals are.
- fridder 4mo ago
  at the moment yes. The one possible silver lining with all of the current hardware crunch is that it _should_ force some hardware advancements. The last couple years hardware has been kinda boring. My m1max is still zippy as all hell and doesn't really need to be upgraded, unless I am committing to local AI inference.
  - zozbot234 4mo ago
    I kinda assume phones are going to be battery powered for the foreseeable future. "Gaming" phones with better cooling do exist, but they are a tiny niche. Most local AI users will want to serve their inference needs through a very different kind of system.
    - array_key_first 4mo ago
      Yes, but the battery tech itself is improving. We're already seeing new phones approach 8000 mAH internal batteries, which is large enough that you can splurge on compute and still have some left over at the end of the day.
  - virgil_disgr4ce 4mo ago
    > it _should_ force some hardware advancements I'm very curious what kind of hardware advancements you're imagining. Because we're already kind of near a physical wall regarding heat dissipation on phones. I mean hey, maybe foundational physics will surprise the world with a radical breakthrough that disappears heat into a black hole or something, but I sure wouldn't hold my breath
    - csheaff 4mo ago
tootie 4mo ago
I just did something exactly like this. I have a self-hosted personal dashboard and one of the APIs I'm reading gives slightly too verbose of an output. So I added a feature to summarize the text using Qwen 3.5 2B which happily runs on a CPU. I've never clocked the tokens per second because I only generate like 100 tokens an hour in a very narrow domain of knowledge and speed isn't critical.

by Akuehne 4mo ago

andybak 4mo ago

> draw a pretty good pelican on a bike. You mean the famously hard task? The one picked because it stretches frontier models to their limits?

daveguy 4mo ago
https://simonwillison.net/2026/Apr/22/qwen36-27b/ https://simonwillison.net/2026/Apr/22/qwen36-27b/ Maybe this is an example of training overfit. But it won't be too long before local models chew through the "famously hard tasks". Except possibly ARC-AGI. That's one benchmark that is still developing with capabilities. And every time a new ARC-AGI benchmark is released it make the SOTA LLMs look pathetic. Because there is very little understanding or transferability with LLMs. But in terms of benchmark-able micro tasks, the local LLMs are improving.
quantummagic 4mo ago
In fairness, that isn't due to a lack of compute.
munk-a 4mo ago
It was a famously hard task. It was an ingenious idea for an unexpected task that falls outside of the bounds of predictable normal input but is still readily comprehended by the public. Unfortunately, as soon as it's a famously hard task trainers know they need to succeed at it and it loses a lot of the power to detect correctness.

deleted 4mo ago

[deleted]

AdeelKhamisa 4mo ago

I was writing just about this last week for fun: AI + hardware team-up to build localized AI with specialized functions to your organization. Ex. Adoble Studio AI in an on premise Box, made by Apple and powered by something like Cohere with privacy: https://www.notion.so/adeelkhamisa/Cohere-s-next-steps-to-become-the-dominant-AI-company-for-sovereignty-and-privacy-35ba5de2f8018096a4d8e35206b75abb?source=copy_link https://www.notion.so/adeelkhamisa/Cohere-s-next-steps-to-be...

henry_kang 4mo ago

Actually you can do way more things than that. We have optimized it to process 2TB of high def videos on a M5 MBP in under 24 hours, including everything such as speech understanding, face recog, LLM and VLM. Super fun.

sanderjd 4mo ago
Is this project public or have you written about it anywhere?
- deleted 4mo ago
  [deleted]
- henry_kang 4mo ago
  Yeah, we've recently made it public. You can check it out here: https://clipto.com https://clipto.com Be aware that it is still a beast that sucks in a lot of memory. Oh, one more thing ;) remember to keep your Mac plugged in...
jimbokun 4mo ago
If Steve Jobs was alive Apple would have already demoed this as a new line of Macs with open weight models pre-installed with hooks into all of their existing content creation software. And he would have the audience believing all the demos were running through third party AI providers, until at the last moment explaining “actually all of that ran on device with no connection to any external services.”
- henry_kang 4mo ago
  hhh, "one more thing"

DrewADesign 4mo ago

And this is exactly what the LLM provider industry is fighting tooth-and-nail. It’s not only because it doesn’t directly contribute to their bottom line, it also directly opposes the idea that LLMs are going to replace entire workers rather than enhance the abilities of individual workers. What we’re headed towards would have been a killer product and probably still shifted a bunch of capital to the bazillionaires had these companies set more realistic goals rather than banking that they’d be the ones that won the war that “changed everythingTM”.

achenet 4mo ago
mainframe industry vs personal computers. If their product is "access to a big model running on a really big computer" (if we can count 'multiple data-centers' as a single enormous distributed computer), then the product "small, accessible device that everyone has" risks killing their cash cow. Ironically enough, the first company to really focus on "an LLM in every phone" will have a good shot at actually being the ones that "changed everythingTM", in the way Microsoft changed the world from IBM mainframes to PCs, or Apple made smartphones a thing.
- DrewADesign 4mo ago
  As an aside, the mainframe industry was profitable for decades before PCs took over. It’s not like they spent a zillion dollars ramping up at the same time.
  - albuic 4mo ago
    The mainframe industry IS profitable.
voncheese 4mo ago
As long as Apple and Google put reasonable AI capabilities on device, then software engineers will use those capabilities when it makes sense (the article gives lots of good examples of capabilities that make sense to run locally). As the author notes, it's cheaper and more reliable to run these things locally. That also doesn't preclude LLM services from being massively successful, they'll just have to justify the pricing and complexity that comes with their adoption, just like any other product.

zozbot234 4mo ago

A phone makes a very crappy AI inference rig. It's battery powered and can't even really run at 100% utilization on an ongoing basis due to how challenging the thermals are.

fridder 4mo ago
at the moment yes. The one possible silver lining with all of the current hardware crunch is that it _should_ force some hardware advancements. The last couple years hardware has been kinda boring. My m1max is still zippy as all hell and doesn't really need to be upgraded, unless I am committing to local AI inference.
- zozbot234 4mo ago
  I kinda assume phones are going to be battery powered for the foreseeable future. "Gaming" phones with better cooling do exist, but they are a tiny niche. Most local AI users will want to serve their inference needs through a very different kind of system.
  - array_key_first 4mo ago
    Yes, but the battery tech itself is improving. We're already seeing new phones approach 8000 mAH internal batteries, which is large enough that you can splurge on compute and still have some left over at the end of the day.
- virgil_disgr4ce 4mo ago
  > it _should_ force some hardware advancements I'm very curious what kind of hardware advancements you're imagining. Because we're already kind of near a physical wall regarding heat dissipation on phones. I mean hey, maybe foundational physics will surprise the world with a radical breakthrough that disappears heat into a black hole or something, but I sure wouldn't hold my breath
  - csheaff 4mo ago

tootie 4mo ago

I just did something exactly like this. I have a self-hosted personal dashboard and one of the APIs I'm reading gives slightly too verbose of an output. So I added a feature to summarize the text using Qwen 3.5 2B which happily runs on a CPU. I've never clocked the tokens per second because I only generate like 100 tokens an hour in a very narrow domain of knowledge and speed isn't critical.