6 ms·
What's the cheapest way to run e.g. LLaMa2-13B and have it served as an API? I've tried Inference Endpoints and Replicate, but both would cost more than just
by ktaube 3y ago
What's the cheapest way to run e.g. LLaMa2-13B and have it served as an API?
I've tried Inference Endpoints and Replicate, but both would cost more than just using the OpenAI offering.
- lgrammel 3y agollama.cpp has a server with a REST API that you can use: https://github.com/ggerganov/llama.cpp/tree/master/examples/server https://github.com/ggerganov/llama.cpp/tree/master/examples/...
- fy20 3y agoYou can probably run it locally with llama.cpp using CPU only, but it will be slow. I have a couple year old laptop with a RTX 3060 and it runs pretty well split across the CPU and GPU.
- l5870uoo9y 3y agoI am interested in that as well. Can LLaMa2 models be deployed to VPS? (Preferable the 70B model).
- garciasn 3y agoI mean, hosting your own outside of OpenAI is mainly to avoid OpenAI accessing the data and using it for X, Y, and Z. I wouldn't roll my own if there weren't concerns about safety due to the cost and quality of the results.