6 ms·
GPUs will be better optimized for large matrix multiplications than CPUs, by design. And you need to the inference again and again, not just a single time (lik
by rerx 3y ago
GPUs will be better optimized for large matrix multiplications than CPUs, by design.
And you need to the inference again and again, not just a single time (like your training).
- dist-epoch 3y agoDesktop CPUs have integrated GPUs, so it's more complicated. If I infer on the GPU inside my CPU, how do you count that?
- rerx 3y agoTrue. Memory bandwidth may be the most limiting factor.