6 ms·
I've been wanting a local LLM appliance.
by dariusj18 2y ago
I've been wanting a local LLM appliance.
- brookst 2y agoTech is evolving too quickly; every year the hardware will be much more powerful at the same price (as LLM optimizations reach hardware), so you’d end up replacing the device frequently.
- deleted 2y ago[deleted]
- readthenotes1 2y agoLike phones?
- nextts 2y agoNot convinced. Are CPUs and GPUs killing it %/$ wise each year like it's 1996? Models are killing it but that is just an "ollama run" command away.
- brookst 2y agoGPUs and NPUs are gaining optimizations for the transformer architecture. It’s not “GPU is 3x faster this year”, it’s “GPU has gates specifically designed to accelerate your LLM workload” See for instance [0], which is just starting to appear in commercial parts. This is continuing; pretty much every low cost SoC maker is racing to build and extend ML optimizations. 0. https://www.synopsys.com/blogs/chip-design/best-edge-ai-processor-vision-systems.html https://www.synopsys.com/blogs/chip-design/best-edge-ai-proc...