6 ms·
It's kinda funny how nowadays an AI with 8 billion parameters is something "small". Specially when just two years back entire racks were needed to run something
by mrfinn 2y ago
It's kinda funny how nowadays an AI with 8 billion parameters is something "small". Specially when just two years back entire racks were needed to run something giving way worst performance.
- atemerev 2y agoIDK, 8B-class quantized models run pretty fast on commodity laptops, with CPU-only inference. Thanks to the people who figured out quantization and reimplemented everything in C++, instead of academic-grade Python.
- actualwitch 2y agoA solid chunk of python is just wrappers around C/C++, most tensor frameworks included.
- atemerev 2y agoI know, and yet early model implementations were quite unoptimized compared to the modern ones.