5 ms·
It would be interesting to see how these models impact battery life. I’ve tried a few local LLMs on my iPhone 15 Pro via the PrivateLLM app, and the battery cha
by epipolar 2y ago
It would be interesting to see how these models impact battery life. I’ve tried a few local LLMs on my iPhone 15 Pro via the PrivateLLM app, and the battery charge plummets just after a few minutes of usage.
- urbandw311er 2y agoLikely they’ll be able to take advantage of the hardware neural engine and be far more power efficient. Apple has demonstrated this is something it takes pretty seriously.
- brcmthrowaway 2y agoSo iOS LLM Apps dont use the neural engine? Lol
- deleted 2y ago[deleted]
- renewiltord 2y agoProbably not. The CoreML LLM stuff only works on Macs AFAIK. Probably the phone app uses the GPU.
- hmottestad 2y agoIf they use Llama.cpp they probably run on the GPU. Apple hasn’t published much about their neural engine, so you kinda have to use it through CoreML. I assume they have some aces up their sleeves for running LLMs efficiently that haven’t told anyone yet.
- woadwarrior01 2y agoNone of the current iOS and macOS LLM Apps use the Neural Engine. They use the CPU and the GPU. nb: I'm the author of a fairly popular app in that category.
- jjtheblunt 2y agoHow would you know none of the apple apps use the neural engine? Is the key in the statement “LLM”?
- woadwarrior01 2y agoYes, I specifically meant autoregressive LLMs. BERT style encoder only models, ViTs and CNNs ran perfectly fine. Yesterday's coremltools update[1] changes that. [1]: https://github.com/apple/coremltools/pull/2232 https://github.com/apple/coremltools/pull/2232
- l33t7332273 2y agoWhy do they not?
- wpm 2y agoAFAIK there is no general purpose, "do this on the ANE" API. You have to be using specific higher level APIs like CoreML or VisionKit in order for it to end up on the ANE.
- bradly 2y agoDuring my time at Apple the bigger issue with personalized, on-device models was the file size. At the time, each model was a significant amount of data to push to a device, and with lots of teams wanting an on-device model and the desire to update them regularly, it was definitely a big discussion.
- hmottestad 2y agoThey’ve gone with a single 3B model and several “adapters” for each use case. One adapter is good at summarising while another good a generating message replies.
- onesociety2022 2y agoAI noob here. Is every single model in iOS really just a thin adapter on top of one base model? Can everything they announced today really be built on top of one base LLM model with a specific type of architecture? What about image generation? What about text-to-speech? If they’re obviously different models, they can’t load them all at once into RAM. If they have to load from storage every time an app is opened, how will they do this fast enough to maintain low latency?
- wmf 2y agoThe main LLM is only 1.5 GB so it should only take a half second to load. Or they could keep it loaded. The other models may be even smaller.
- glial 2y agoMaybe they use the "Siri is waking up and the screen wabbles" animation time for loading the model. That would be clever.
- mholm 2y agoThey'll have plenty of time to load the model; It still needs to wait for the user to actually voice/type their request. Invoking Siri happens well before the request is ready.
- jamesy0ung 2y agoIt looks like PrivateLLM uses the GPU for inferencing, from what I can tell, Apple is using the ANE on the A17 Pro. For M1 and above, I'd presume they are using the GPU since the ANE in M series isn't great.
- woadwarrior01 2y agoI'm the author of Private LLM. Looks like it's just become possible[1] to run quantized LLM inference using the ANE with iOS 18. I think there are some major efficiency gains on the table now. [1]: https://github.com/apple/coremltools/pull/2232 https://github.com/apple/coremltools/pull/2232