18 ms·
Where did you see the matmul acceleration support? I couldn't find this detail online.
by sercand 1y ago
Where did you see the matmul acceleration support? I couldn't find this detail online.
- aurareturn 1y agoApple calls it "Neural Accelerators". It's all over their A19 marketing.
- kridsdale3 1y agoWhat a ridiculous way to market "linear algebra transistor array".
- jimbokun 1y agoWhich means what, exactly, to someone whose not a machine learning researcher?
- jacquesm 1y agoHey man, it helps you think different. You just never knew your neurons needed accelerating.
- kridsdale1 1y agoI accelerate them every morning with an Americano.
- liamwire 1y agoI have to ask out of curiosity, why is your first comment made with one account, and the reply with a similarly-named alt?
- kmarc 1y agoTo confuse all those neural accelerators scraping this conversation.
- liamwire 1y agoThat seems incredibly prescient for accounts created before even GPT-1. Obviously broad data scraping existed before then, but even amongst this crowd I find it hard to believe that’s the real motivator.
- fennecfoxy 1y agoAccount on laptop, account on mobile.
- deleted 1y ago[deleted]
- butlike 1y agoI really hope someone got fired for this blunder
- kamranjon 1y agoDon’t all of the M series chips contain neural cores?
- aurareturn 1y agoYes, they do. They're called Neural Engine, aka NPUs. They aren't being used for local LLMs on Macs because they are optimized for power efficiency running much smaller AI models. Meanwhile, the GPU is powerful enough for LLMs but has been lacking matrix multiplication acceleration. This changes that.
- cchance 1y agoThese are different these are built into the GPU Cores
- mrheosuper 1y agoI thought 1 of the reason we do ML on GPU is fast Matrix multiplication ? So the new engine is accelerator for matmul accelerator ?
- wtallis 1y agoFrom a compute perspective, GPUs are mostly about fast vector arithmetic, with which you can implement decently fast matrix multiplication. But starting with NVIDIA's Volta architecture at the end of 2017, GPUs have been gaining dedicated hardware units for matrix multiplication. The main purpose of augmenting GPU architectures with matrix multiplication hardware is for machine learning. They aren't directly useful for 3D graphics rendering, but their inclusion in consumer GPUs has been justified by adding ML-based post-processing and upscaling like NVIDIA's various iterations of DLSS.
- astrange 1y agoThe neural engine is used for the built-in LLM that does text summaries etc., just not third party LLMs. And there's an official port of Stable Diffusion to it: https://github.com/apple/ml-stable-diffusion https://github.com/apple/ml-stable-diffusion
- emchammer 1y agoDoes this mean that equivalent logic for what has been called Neural Engine is now integrated into each CPU core?
- rmccue 1y agoEach GPU core, but yes, this was part of what they announced today - it’s now integral rather than separate.