11 ms·
My iPhone 16 Pro Max produces garbage output when running MLX LLMs
- bri3d 8mo agoI love to see real debugging instead of conspiracy theories! Did you file a radar? (silently laughing while writing this, but maybe there's someone left at Apple who reads those)
- djmips 8mo agoIKR - this is very typical
- the_arun 8mo ago[flagged]
- bri3d 8mo agoSomewhere along the line, the tensor math that runs an LLM became divergent from every other Apple device. My guess is that there's some kind of accumulation issue here (remembering that floating-point accumulation does not usually commute), but it seems genuinely broken in an unexpected way given that Apple's own LLM also doesn't seem to work on this device.
- ploum 8mo agoWell it seems that, those days, instead of SUM(expense1,expense2) you ask an LLM to "make an app that will compute the total of multiple expenses". If I read most of the news on this very website, this is "way more efficient" and "it saves time" (and those who don’t do it will lose their job) Then, when it produces wrong output AND it is obvious enough for you to notice, you blame the hardware.
- bri3d 8mo agoI mean, Apple's LLM also doesn't work on this device, plus the author compared the outputs from each iterative calculation on this device vs. others and they diverge from every other Apple device. That's a pretty big sign that both, something is different about that device, and this same broken behavior carried across multiple OS versions. Is the hardware or the software "responsible" - who knows, there's no smoking gun there, but it does seem like something is genuinely wrong. I don't get the snark about LLMs overall in this context; this author uses LLM to help write their code, but is also clearly competent enough to dig in and determine why things don't work when the LLM fails, and performed an LLM-out-of-the-loop debugging session once they decided it wasn't trustworthy. What else could you do in this situation?
- janalsncm 8mo agoThe author is debugging the tensor operations of the on-device model with a simple prompt. They confirmed the discrepancy with other iPhone models. It’s no different than someone testing a calculator with 2+2. If it gets that wrong, there’s a hardware issue. That doesn’t mean the only purpose of the calculator is to calculate 2+2. It is for debugging. You could just as uncharitably complain that “these days no one does arithmetic anymore, they use a calculator for 2+2”.
- Dylan16807 8mo agoThe app-making wasn't being done on the phone. The LLM that malfunctioned was there to slap categories on things. And something was going wrong in either the hardware or the compiler.
- lxgr 8mo agoLLMs are applied math, so… both?
- vanviegen 8mo ago[flagged]
- ohyoutravel 8mo ago[flagged]
- Buttons840 8mo agoI clicked hoping this would be about how old graphing calculators are generally better math companions than a phone. The best way to do math on my phone I know of is the HP Prime emulator.
- shiroiuma 8mo agoI use the "RealCalc" app on my phone. It's pretty similar to my old HP48.
- VorpalWay 8mo agoI run a TI 83+ emulator on my Android phone when I don't have my physical calculator at hand. Same concept, just learned a different brand of calculators.
- varun_ch 8mo agobuilt-in calculator apps are surprisingly underbaked... I'm surprised neither of the big two operating systems have elected to ship something comparable to a real calculator built in. It would be nice if we could preview the whole expression as we type it.. I use the NumWorks emulator app whenever I need something more advanced. It's pretty good https://www.numworks.com/simulator/ https://www.numworks.com/simulator/
- josephg 8mo agoThat’s certainly an improvement - but why can’t I modify a previous expression? Or tap to select previous expressions? What I want is something like a repl. I want to be able to return to an earlier expression, modify it, assign it to a variable, use that variable in another expression, modify the variable and rerun and so on.
- varun_ch 8mo agoI think on the numworks you can use the arrow keys to pull up an old expression. I think it would be really cool if someone built out an interpreted, nicely rendered calculator language/repl that could do variables and stuff. Might be an interesting idea
- raincole 8mo agoLow level numerical operation optimizations are often not reproduceable. For example: https://www.intel.com/content/dam/develop/external/us/en/documents/run-to-run-reproducible-results-with-intel-tools.pdf https://www.intel.com/content/dam/develop/external/us/en/doc... (2013) But it's still surprising that that LLM doesn't work on iPhone 16 at all. After all LLMs are known for their tolerance to quantization.
- bri3d 8mo agoYes, "floating point accumulation doesn't commute" is a mantra everyone should have in their head, and when I first read this article, I was jumping at the bit to dismiss it out of hand for that reason. But, what got me about this is that: * every other Apple device delivered the same results * Apple's own LLM silently failed on this device to me that behavior suggests an unexpected failure rather than a fundamental issue; it seems Bad (TM) that Apple would ship devices where their own LLM didn't work.
- danpalmer 8mo agoFYI, the saying is "champing at the bit", it comes from horses being restrained.
- mylifeandtimes 8mo agohey, I appreciate your love of language and sharing with us. I'm wondering if we couldn't re-think "bit" to the computer science usage instead of the thing that goes in the horse's mouth, and what it would mean for an AI agent to "champ at the bit"? What new sayings will we want?
- nilamo 8mo agoByting at the bit?
- odo1242 8mo agochomping at the bit
- lionkor 8mo ago[flagged]
- bri3d 8mo ago> Or, rather, MiniMax is! The good thing about offloading your work to an LLM is that you can blame it for your shortcomings. Time to get my hands dirty and do it myself, typing code on my keyboard, like the ancient Mayan and Aztec programmers probably did. They noticed a discrepancy, then went back and wrote code to perform the same operations by hand, without the use of an LLM at all in the code production step. The results still diverged unpredictably from the baseline. Normally, expecting floating-point MAC operations to produce deterministic results on modern hardware is a fool's errand; they usually operate asynchronously and so the non-commutative properties of floating-point addition rear their head and you get some divergence. But an order of magnitude difference plus Apple's own LLM not working on this device suggests strongly to me that there is something wrong. Whether it's the silicon or the software would demand more investigation, but this is a well reasoned bug in my book.
- ErroneousBosh 8mo ago> Time to get my hands dirty and do it myself, typing code on my keyboard, like the ancient Mayan and Aztec programmers probably did. https://ia800806.us.archive.org/20/items/TheFeelingOfPower/The%20Feeling%20of%20Power.pdf https://ia800806.us.archive.org/20/items/TheFeelingOfPower/T... I should think I'll probably see someone posting this on the front page of HN tomorrow, no doubt. I first read it when it was already enormously old, possibly nearly 30 years old, in the mid 1980s when I was about 11 or 12 and starting high school, and voraciously reading all the Golden Age Sci-Fi I could lay my grubby wee hands on. I still think about it, often.
- decimalenough 8mo agoMy TL;DR is that they tried to run an on-device model to classify expenses, it didn't work even for simple cases ("Kasai Kitchin" -> "unknown"), they went deeeeeep down the rabbit hole to figure out why and concluded that inference on their particular model/phone is borked at the hardware level. Whether you should do this on device is another story entirely.
- PlatoIsADisease 8mo ago[flagged]
- ohyoutravel 8mo agoThis is a conclusion that comes with some personal baggage you should identify and consider addressing.
- gambiting 8mo agoI mean, I think it's cultural. In US it seems like everyone has an iphone, it's almost kinda quirky not to have one. But in some other places, an iPhone is more than your monthly salary - having one is definitely a symbol of status. Less so than it used to be, but it still has that.
- ohyoutravel 8mo agoFair, but that’s a comment on a US-centric website, run by a US-centric company, in a US-centric industry, on a US-centric medium. So if they didn’t mean US, I think the onus is on them to clarify exactly where this applies.
- dghlsakjg 8mo agoiPhones in the US have an estimate ~55% market share depending on source. Owning an Android wasn't unusual in the least when I lived there, and appears to be pretty popular. I don't think its unusual that a country with high median income and higher average income will tend to gravitate towards more expensive phones. Given that Apple doesn't make a cheap phone, it kind of follows that wealthier countries will buy more iPhones. Of course the opposite is true as well, In a country where an iPhone is measured in months of salary, they won't sell well, but I'd be willing to bet that Androids in that price tier sell like shit in those countries too. Is it a status symbol? arguably. But it also correlates pretty strongly with median income.
- PlatoIsADisease 8mo agoAdmittedly, I hate companies that live off their marketing. Nintendo, Disney, Apple. I hate that these companies can weaponize psychology against humans. Function > Form. I think its a Hero Complex, if Jung is correct.
- giancarlostoro 8mo ago[flagged]
- deleted 8mo ago[deleted]
- Playboi_Carti 8mo agoIt's not about LLMs doing math.
- bri3d 8mo agoIf you’d read the whole thing, you would go on a debugging journey that both involved bypassing the LLM and was appropriate for HN (vs not dismissing the article), so you might want to do that.
- dummydummy1234 8mo agoUhh, that's not the article, the article is running a ml model, on phone and floating point opps for tensor multiplication seems to be off.
- _kulang 8mo agoMaybe this is why my damn keyboard predictive text is so gloriously broken
- sen 8mo agoOh it's not just me? Typing on my iPhone in the last few months (~6 months?) has been absolutely atrocious. I've tried disabling/enabling every combination of keyboard setting I can thinkj of, but the predictive text just randomly breaks or it just gives up and stops correcting anything at all.
- macintux 8mo agoI haven't watched the video, but clearly there's a broad problem with the iOS keyboard recently. https://news.ycombinator.com/item?id=46232528 https://news.ycombinator.com/item?id=46232528 ("iPhone Typos? It's Not Just You - The iOS Keyboard is Broken")
- acdha 8mo agoIt’s not just you, and it got bad on my work iPhone at the same time so I know it’s not failing hardware or some customization since I keep that quite vanilla.
- taneq 8mo agoIt’s gotten so bad that I’m half convinced it’s either (a) deliberately trolling, or (b) ‘optimising’ for speech to text adoption.
- tehwebguy 8mo ago[flagged]
- refulgentis 8mo ago.
- bri3d 8mo agoCan you read the article a little more closely? > - MiniMax can't fit on an iPhone. They asked MiniMax on their computer to make an iPhone app that didn't work. It didn't work using the Apple Intelligence API. So then: * They asked Minimax to use MLX instead. It didn't work. * They Googled and found a thread where Apple Intelligence also didn't work for other people, but only sometimes. * They HAND WROTE the MLX code. It didn't work. They isolated the step where the results diverged. > Better to dig in a bit more. The author already did 100% of the digging and then some. Look, I am usually an AI rage-enthusiast. But in this case the author did every single bit of homework I would expect and more, and still found a bug. They rewrote the test harness code without an LLM. I don't find the results surprising insofar as that I wouldn't expect MAC to converge across platforms, but the fact that Apple's own LLM doesn't work on their hardware and their own is an order of magnitude off is a reasonable bug report, in my book.
- refulgentis 8mo agoEmptied out post, thanks for the insight! Fascinating the claim is Apple Intelligence doesn't work altogether. Quite a scandal. EDIT: If you wouldn't mind, could you edit out "AI rage enthusiast" you edited in? I understand it was in good humor, as you describe yourself that way as well. However, I don't want to eat downvotes on an empty comment that I immediately edited when you explained it wasn't minimax! People will assume I said something naughty :) I'm not sure it was possible to read rage into my comment.
- LoganDark 8mo ago> Fascinating the claim is Apple Intelligence doesn't work altogether. Quite a scandal. No, the claim is their particular device has a hardware defect that causes MLX not to work (which includes Apple Intelligence). > EDIT: If you wouldn't mind, could you edit out "AI rage enthusiast" you edited in? I understand it was in good humor, as you describe yourself that way as well. However, I don't want to eat downvotes on an empty comment that I immediately edited when you explained! People will assume I said something naughty :) I'm not sure it was possible to read rage into my comment. Your comment originally read: > This is blinkered. > - MiniMax can't fit on an iPhone. > - There's no reason to expect models to share OOMs for output. > - It is likely this is a graceful failure mode for the model being far too large. > No fan of Apple's NIH syndrome, or it manifested as MLX. > I'm also no fan of "I told the robot [vibecoded] to hammer a banana into an apple. [do something impossible]. The result is inedible. Let me post to HN with the title 'My thousand dollars of fruits can't be food' [the result I have has ~nothing to do with the fruits]" > Better to dig in a bit more. Rather than erase it, and invite exactly the kind of misreading you don't want, you can leave it... honestly, transparently... with your admission in the replies below. And it won't be downvoted as much as when you're trying to manipulate / make requests of others to try to minimize your downvotes. Weird... voting... manipulating... stuff, like that, tends to be frowned upon on HN. You have more HN karma than I do, even, so why care so much about downvotes... If you really want to disown something you consider a terrible mistake, you can email the HN mods to ask for the comment to be dissociated from your account. Then future downvotes won't affect your karma. I did this once.
- johngossman 8mo agoPosting some code that reproduces the bug could help not only Apple but you and others.
- csmantle 8mo agoMethodology is one thing; I can't really agree that deploying an LLM to do sums is great. Almost as hilarious as asking "What's moon plus sun?" But phenomenon is another thing. Apple's numerical APIs are producing inconsistent results on a minority of devices. This is something worth Apple's attention.
- fatheranton 8mo ago[dead]
- JimboOmega 8mo ago(This is a total digression, so apologies) My mind instantly answered that with "bright", which is what you get when you combine the sun and moon radicals to make 明(https://en.wiktionary.org/wiki/%E6%98%8E https://en.wiktionary.org/wiki/%E6%98%8E) Anyway, that question is not without reasonable answers. "Full Moon" might make sense too. No obvious deterministic answer, though, naturally.
- awesome_dude 8mo agoFTR the Full Moon was exactly 5 hours ago (It's not without humour that this conversation occurs on the day of the full moon :)
- deleted 8mo ago[deleted]
- cgriswald 8mo agoIn the game Clair Obscur sun plus moon equals twilight.
- butlike 8mo agoYou could play Infinite Craft and find out what the game thinks it is: https://neal.fun/infinite-craft/ https://neal.fun/infinite-craft/ Edit: Spoiler - It's 'Eclipse'
- CrispinS 8mo ago> What's moon plus sun? Eclipse, obviously.
- DustinEchoes 8mo agoI wish he would have tried on a different iPhone 16 Pro Max to see if the defect was specific to that individual device.
- crossroadsguy 8mo agoSo true! And as any sane Apple user or the standard template Apple Support person would have suggested (and as they actually suggest) - did they try reinstalling the OS from scratch after having reset the data (of course before backing it up; preferably with a hefty iCloud+ plan)? Because that's the thing to do in such issues and it's very easy.
- post-it 8mo agoReinstalling the OS sucks. I need to pull all my bank cards out of my safe and re-add their CVV's to the wallet, and sometimes authenticate over the phone. And re-register my face. And log back in to all my apps. It can take an hour or so, except it's spread out over weeks as I open an app and realize I need to log in a dozen times.
- RulerOf 8mo agoThere was a magical period. I suspect it ended with the introduction of the Secure Enclave. But maybe it was a little later. An encrypted iTunes backup of a device was a perfect image. Take the backup, pull the SIM card, restore the backup to a new phone with the sim card installed, and it was like nothing had happened. No reauthentication. No missing notifications. No lost data. Ever. It was nice.
- throwaway132448 8mo agoSecurity theatre killed this. Everyone must be assumed to be a moron incapable of living with the consequences of their own choices at all times.
- 8mo ago
- ernsheong 8mo ago[flagged]
- Metacelsus 8mo ago>"What is 2+2?" apparently "Applied.....*_dAK[...]" according to my iPhone At least the machine didn't say it was seven!
- tolciho 8mo agoMaybe Trurl and Klapaucius were put in charge of Q&A.
- RiceNBananas 8mo ago[flagged]
- dav43 8mo agoMy thousand dollar iPhone can't even add a contact from a business card.
- ftyghome 8mo agoI also would like to see if the same error happens in another phone with the exactly same model.
- nickorlow 8mo agoI'd think other neural-engine using apps would also have weird behavior. Would've been interesting to try a few App Store apps and see the weird behavior
- mungoman2 8mo agoGood article. Would have liked to see them create a minimal test case, to conclusively show that the results of math operations are actually incorrect.
- docfort 8mo agoInteresting post, but the last bit of logic pointing to the Neural Engine for MLX doesn’t hold up. MLX supports running on CPU, Apple GPU via Metal, and NVIDIA GPU via CUDA: https://github.com/ml-explore/mlx/tree/main/mlx/backend https://github.com/ml-explore/mlx/tree/main/mlx/backend
- z3t4 8mo agoneural nets or AI are very bad at math, it can only produce what's in the training data. So if you have trained it from 1+1 to 8+8 it can't do 9+9, it's not like a child brain that it can make logical conclusions.
- swyx 8mo ago> Update on Feb. 1st: > Well, now it's Feb. 1st and I have an iPhone 17 Pro Max to test with and... everything works as expected. So it's pretty safe to say that THAT specific instance of iPhone 16 Pro Max was hardware-defective. nothing to see here.
- tgma 8mo agoThe author is assuming Metal is compiled to ANE in MLX. MLX is by-and-large GPU-based and not utilizing ANE, barring some community hacks.
- woadwarrior01 8mo agoWhat community hacks?
- tgma 8mo agoWhat I meant is, if you can somehow get it working, it is not currently a supported first-party thing, not that I am aware of such thing existing.
- addandsubtract 8mo agoANE is probably the biggest scam "feature" Apple has ever sold.
- llm_nerd 8mo ago>ANE is probably the biggest scam "feature" Apple has ever sold. It is astonishing how often ANE is smeared on here, largely by people who seem to have literally zero idea what they're talking about. It's often pushed by either/or people who bizarrely need to wave a flag. MLX doesn't use ANE for the single and only reason that Apple hid the ANE behind CoreML, exposing zero public APIs to utilize ANE directly, and MLX -- being basically an experimental grounds -- wanted to hand roll their implementation around the GPU / CPU. They literally, directly state this as the reason. People inventing technical reasons for why MLX doesn't use ANE are basically just manufacturing a fan fiction. This isn't to say that ANE would be suitable for a lot of MLX tasks, and it is a highly optimized, power-efficient inference hardware that doesn't work for a lot of purposes, but its exclusion is not due to technically unsuitability. Further, the ANE on both my Mac and my iPhone is constantly attenuating and improving my experience. Little stuff like extracting contents from images. Ever browse in Safari and notice that you can highlight text in the image almost instantly after loading a page? Every image, context and features detected effortlessly. Zero fans cycling up. Power usage at a trickle. It just works. It's the same way that when I take a photo I can search "Maine Coon" and get pictures of my cats, ANE used for subject and feature extraction. Computational photography massively leverages the ANE. At a trickle of power. Scam? Yeah, I like my battery lasting for more than a couple of minutes. Apple intended ANE to bring their own NN augmentations to the OS and thus the user experience, and even the availability in CoreML as a runtime engine is more limited than what Apple's own software can do. Apple basically limits the runtime usage to ensure that no third party apps inhibit or restrict Apple's own use of this hardware.
- builderhq_io 8mo ago[flagged]
- thinkbud 8mo agoSo the LLM is working as intended?
- watt 8mo agoDoes it bother anyone else that the author drops "MiniMax" there in the article without bothering to explain or footnote what that is? (I could look it up, but I think article authors should call out these things).
- embedding-shape 8mo agoThere are tons of terms that aren't explained that some people (like me) might not understand. I think it's fine that some articles have a particular audience in mind and write specifically for those, in this case, it seems it's for "Apple mobile developers who make LLM inference engines" so not so unexpected there are terms I (and others) don't understand.
- JCharante 8mo agoI think articles are worse when they have to explain everything someone off the street might not know.
- spockz 8mo agoYes, maybe. But it would be nice if there would be footnotes or tooltips. Putting the explanation in the text itself breaks the flow of the text so that would make it worse indeed.
- cowsandmilk 8mo agoMiniMax is a company. It isn’t a term of art or something. It would be like defining Anthropic.
- fnord77 8mo agominimax is an algorithm for choosing the next move in an n-player game, discovered by John von Neumann in 1928
- einsteinx2 8mo agoNo because it was obvious from context clues that it was an LLM model. Not every word needs to be defined. Also if you were unsure and decided to search “MiniMax M2.1”, every result would be about the LLM.
- zcbenz 8mo agoIt is a bug in MLX that has been fixed a few days ago: https://github.com/ml-explore/mlx/pull/3083 https://github.com/ml-explore/mlx/pull/3083
- embedding-shape 8mo agoBlog post dated 28 Jan 2026, the bug fix posted 29 Jan 2026, so I guess this story had a happy ending :) Still, sad state of affairs that it seems like Apple is still fixing bugs based on what blog posts gets the most attention on the internet, but I guess once they started that approach, it's hard to stop and go back to figuring out priorities on their own.
- jckahn 8mo agoJust goes to show that attention is all you need.
- tensility 8mo agoA statement which goes to show that confusing correlation with causation is all you need.
- syntaxing 8mo agoI don’t think so. You can see the issue ticket linked in the PR. Whether that issue ticket is related to the blog post is unknown https://github.com/ml-explore/mlx-swift-examples/issues/462 https://github.com/ml-explore/mlx-swift-examples/issues/462
- llm_nerd 8mo agoMLX is a fairly esoteric library seeing very little usage, mostly to try to foment a broader NN space on Apple devices. This isn't something that is widely affecting people, and most people simply aren't trying to run general LLMs on their iPhone. I don't think that fix is specific to this, but it's absolutely true that MLX is trying to lever every advantage it can find on specific hardware, so it's possible it made a bad choice on a particular device.
- MarginalGainz 8mo ago[dead]
- ryeguy_24 8mo agoWhat expense app are you building? I really want an app that helps me categorize transactions for budgeting purposes. Any recommendations?
- MORPHOICES 8mo ago[dead]
- TimByte 8mo agoThe real lesson here isn't even about Apple. It's about debugging culture
- deleted 8mo ago[deleted]