9 ms·
IDK, prefill speed is a bigger concern for most wokflows, like agent coding, and I heard that this is quite low on macs?
by irthomasthomas 22d ago
IDK, prefill speed is a bigger concern for most wokflows, like agent coding, and I heard that this is quite low on macs?
- smcleod 22d agoThat was mainly before the M4 generation when they didn't have matmul instructions.
- jasonjmcghee 22d agoM5 prefill is much faster than M4. I've seen benchmarks that show 4-5x faster of M5 Max vs. M4 Max. For local models you're likely using M5 Max, prefill is low thousands of tokens per second, as opposed to, say high hundreds with M4 Max. For larger dense models, some fraction of that, but similar multiple.
- smcleod 22d agoYes, I have the M5 Max. But there was no matmul acceleration before the M4 which made things a lot slower.