6 ms·
I have the same question. Noticed that Ollama got a lot of publicity and seems to be well received, but what exactly is the advantage over using llama.cpp (whic
by jpdus 3y ago
I have the same question. Noticed that Ollama got a lot of publicity and seems to be well received, but what exactly is the advantage over using llama.cpp (which also has a built-in server with OpenAI compatibility nowadays?) Directly?
- visarga 3y agoollama swaps models from the local library on the fly, based on the request args, so you can test against a bunch of models quickly
- eclectic29 3y agoOnce you've tested to your heart's content, you'll deploy your model in production. So, looks like this is really just a dev use case, not a production use case.
- silverliver 3y agoIn production, I'd be more concerned about the possibly of it going off on it's own and autoupdating and causing regressions. FLOSS LLMs are interesting to me because I can precisely control the entire stack. If Ollama doesn't have a cli flag that disables auto updating and networking altogether, I'm not letting it anywhere near my production environments. Period.
- eclectic29 3y agoIf you’re serious about production deployments vLLM is the best open source product out there. (I’m not affiliated with it)