24 ms·
What type of hardware do I need to run a small model like this? I don't do Apple.
by vichle 8mo ago
What type of hardware do I need to run a small model like this? I don't do Apple.
- jychang 8mo ago1.54GB model? You can run this on a raspberry pi.
- BoredomIsFun 8mo agoPerformance of LLM inference consists of two independent metrics - prompt processing (compute intensive) and token generation (bandwidth intensive). For autocomplete with 1.5B you can get away with abysmal 10 t/s token generation performance, but you'd want as fast as possible prompt processing, pi in incapable of.
- gunalx 8mo agoif you mean on the new ai hat with npu and integrated 8gb memory, maybe.
- bodegajed 8mo ago1.5B models can run on CPU inference at around 12 tokens per second if I remember correctly.
- moffkalast 8mo agoIngesting multiple code files will take forever in prompt processing without a GPU though, tg will be the least of your worries. Especially when you don't append but change it in random places so caching doesn't work.
- bradfa 8mo agoA FIM or completion model like this won't have a large prompt and caching doesn't work anyways (per their notes). It'll get maybe a few thousand tokens in a prompt, maximum. For a 1.5B model, you should expect usable CPU-only inference on a modern CPU, like at least hundreds of tokens per second of prefill and tens of tokens per second of generation, which is decently usable in terms of responsiveness.
- moffkalast 8mo agoA thousand tokens (which would be on the low side) at 10-100 t/s in ingestion speed is 10-100 seconds. I don't seriously expect anyone to wait a solid minute after pressing tab for autocomplete, regular autocomplete gets unusably annoying if it takes more than a split second tbh.
- kevinlu1248 8mo agoUnfortunately, the main optimization (3x speedup) is using n-gram spec dec which doesn't run on CPUs. But I believe it works on Metal at least.