6 ms·
Getting ~150 tok/s on an empty context with a 24 GB 7900XTX via llama.cpp's Vukan backend.
by genpfault 9mo ago
Getting ~150 tok/s on an empty context with a 24 GB 7900XTX via llama.cpp's Vukan backend.
- Tepix 9mo agoAgain, you're using some 3rd party quantisations, not the weights supplied by Nvidia (which don't fit in 24GB).