5 ms·

How realistic is CPU-only inference in the near future?

• deleted 3y ago
  [deleted]
• travisjungroth 3y ago
  It’s in the near past. https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
  • abetlen 3y ago
    Also worth checking out https://github.com/saharNooby/rwkv.cpp https://github.com/saharNooby/rwkv.cpp which is based on Georgi's library and offers support for the RWKV family of models which are Apache-2.0 licensed.
    • BrutalCoding 3y ago
      I’ve got some of their smaller Raven models running locally on my M1 (only 16GB of RAM). I’m also in the middle of making it user friendly to run these models on all platforms (built with Flutter). First MacOS release will be out before this weekend: https://github.com/BrutalCoding/shady.ai https://github.com/BrutalCoding/shady.ai
• abetlen 3y ago
  You can see for yourself (assuming you have the model weights) https://github.com/abetlen/llama-cpp-python https://github.com/abetlen/llama-cpp-python I get around ~140 ms per token running a 13B parameter model on a thinkpad laptop with a 14 core Intel i7-9750 processor. Because it's CPU inference the initial prompt processing takes longer than on GPU so total latency is still higher than I'd like. I'm working on some caching solutions that should make this bareable for things like chat.