4 ms·
Anyone know of a docker image that provides an HTTP API interface to Llama? I'm looking for a super simple sort of 'drop-in' solution like that which I can add
by TheRealClay 3y ago
Anyone know of a docker image that provides an HTTP API interface to Llama? I'm looking for a super simple sort of 'drop-in' solution like that which I can add to my web stack, to enable LLM in my web app.
- nodja 3y agohttps://github.com/abetlen/llama-cpp-python https://github.com/abetlen/llama-cpp-python has a web server mode that replicates openai's API iirc and the readme shows it has docker builds already.
- TheRealClay 3y agoThanks! As someone just getting started, I really appreciate the tip!