7 ms·
How's the compute side now, I wonder? Because while the Ultras have impressive memory bandwidth for inference, processing prompts still takes a dog's age on my
by petercooper 23d ago
How's the compute side now, I wonder? Because while the Ultras have impressive memory bandwidth for inference, processing prompts still takes a dog's age on my M3 Ultra. I heard the M5 makes some strides forward in this area, though, and the M7 in particular promises to go a lot further.
- deleted 23d ago[deleted]
- dannyw 23d agoM5 is excellent, they’ve finally gotten their own tensor cores. Good for inference; however if you like to train, data format support and effective performance is limited (M5 Pro). Some hardware features are not exposed or extremely slow. You’ll be fine for inference, but pales in comparison to what a RTX 6000 Pro can do for compute/matmuls/training.
- aurareturn 22d ago4x faster prompt processing than M3 Ultra.