7 ms·
This is great! CPU only for now or can it leverage GPU for sped up inference?
by asselinpaul 3y ago
This is great! CPU only for now or can it leverage GPU for sped up inference?
- ybu 3y agoFocus is CPU for now - but it'd be very useful to have GPU support. As a baseline, this can support anything that llama.cpp. Relevant link: https://github.com/ggerganov/llama.cpp#blas-build https://github.com/ggerganov/llama.cpp#blas-build