6 ms·
Inference speed is heavily dependent on memory read/write speed versus size. As long as you can fit the model in memory, what’ll determine functionality is the
by ata_aman 2y ago
Inference speed is heavily dependent on memory read/write speed versus size. As long as you can fit the model in memory, what’ll determine functionality is the mem bandwidth.
- menaerus 2y agoThis is not universally true although I see this phrase being repeated here too often. And it is especially not true with the small models. Small models are compute-bound.