5 ms·
Thank you, found some benchmarks and they look really promising. To my understanding those Neural Accelerators are like AMX but for GPUs. With those accelerator
by Lwrless 22d ago
Thank you, found some benchmarks and they look really promising. To my understanding those Neural Accelerators are like AMX but for GPUs. With those accelerators the GPU performance on a M5 Max in LLM inferencing would totally be on par with a 5090, that's quite impressive!
- bigyabai 22d agoWhich inference benchmark are you looking at? The prefill speeds should be comparable on some LLMs, but the 5090 has much a higher theoretical max decode speed.
- Lwrless 22d agoI found some on Reddit, and here are the two benchmark results from review sites: - against a desktop 5090: https://nanoreview.net/en/gpu-compare/geforce-rtx-5090-vs-apple-m5-max-gpu-40-core https://nanoreview.net/en/gpu-compare/geforce-rtx-5090-vs-ap... - against a laptop 5090: https://nanoreview.net/en/gpu-compare/geforce-rtx-5090-mobile-vs-apple-m5-max-gpu-40-core https://nanoreview.net/en/gpu-compare/geforce-rtx-5090-mobil... With larger (also really fast) unified memory on the chip, it could easily load larger weights, and the thing is that the quantized results on the M5s are really good, I believe that for most local inferencing, users would be using INT4 (maybe more bits per weight sometimes) quantized models, that might be where the Neural Accelerators kick in. In raw power, a desktop 5090 easily outperforms the M5 Max, but for this specific use case, I believe that the M5 Max is good enough.