6 ms·
Llama.cpp guide – Running LLMs locally on any hardware, from scratch
- wing-_-nuts 2y agoLlama.cpp is one of those projects that I want to install, but I always just wind up installing kobold.cpp because it's simply miles better with UX.
- syntaxing 2y agoLlama cpp is more of a backend software. Most front end software like kobold/open webui uses it
- tempest_ 2y agoI found it only took me ~20 minutes to get Open-WebUI and Ollama going on my machine locally. I don't really know what is happening under the hood but from 0 to chat interface was definitely not too hard.
- cwillu 2y ago“koboldcpp forked from ggerganov/llama.cpp”
- phillipcarter 2y agoI just use ollama. It works on my mac and windows machine and it's super simple to install + run most open models. And you can have another tool just shell out to it if you want more than than the CLI.
- lolinder 2y agoLlama.cpp forms the base for both Ollama and Kobold.cpp and probably a bunch of others I'm not familiar with. It's less a question of whether you want to use llama.cpp or one of the others and more of a question of whether you benefit from using one of the wrappers. I can imagine some use cases where you'd really want to use llama.cpp directly, and there are of course always people who will argue that all wrappers are bad wrappers, but for myself I like the combination of ease of use and flexibility offered by Ollama. I wrap it in Open WebUI for a GUI, but I also have some apps that reach out to Ollama directly.
- freehorse 2y agoWhat is the advantage of ollama/open webui let's say, vs llama-server? I have been using llama.cpp since when it came out, I am used to the syntax etc and I do not have problems building it (probably because I use macos which it supports better), so am I missing something from not using ollama?
- lolinder 2y agoLlama.cpp may have gotten better, but when I first started using them: * Ollama provides a very convenient way to download and manage models, which llama.cpp didn't at the time (maybe it does now?). * Last I checked, with llama.cpp server you pick a model on server startup. Ollama instead allows specifying the model in the API request, and it handles switching out which one is loaded into vram automatically. * The Modelfile abstraction is a more helpful way to keep track of different settings. When I used llama.cpp directly I had to figure out a way to track a bunch of model parameters as bash flags. Modelfiles + being able to specify the model in the request is a great synergy, allowing clients to not have to think about parameters at all, just which Modelfile to use. I'm leaving off some other reasons why I switched which I know have gotten better (like Ollama having great Docker support, which wasn't always true for llama.cpp), but some of these may also have improved with time. A glance over the docs suggests that you still can't specify a model at runtime in the request, though, which if true is probably the single biggest improvement that Ollama offers. I'm constantly switching between tiny models that fit on my GPU and large models that use RAM, and being able to download a new model with ollama pull and immediately start using it in Open WebUI is a huge plus.
- xyc 2y agoIf anyone on macOS wants to use llama.cpp with ease, check out https://recurse.chat/ https://recurse.chat/. Supports importing ChatGPT history & continue chats offline using llama.cpp. Built this so I can use local AI as a daily driver.
- nisten 2y ago[flagged]
- Havoc 2y ago>Get off the f*king adderall, Jikes
- youngNed 2y agoIf the original author (steelph0enix) is reading this, i just want to counter, the post of nisten and say: I liked your blogpost, i thought it was as concise as it needed to be - i have made notes from it and linked back to it from my notes, Thank you.
- SteelPh0enix 2y agooh, hey, thanks! i wonder what was there, i like spicy comments...
- marcodiego 2y agoFirst time I heard about Llama.cpp I got it to run on my computer. Now, my computer: a Dell laptop from 2013 with 8Gb RAM and an i5 processor, no dedicated graphic card. Since I wasn't using a MGLRU enabled kernel, It took a looong time to start but wasn't OOM-killed. Considering my amount of RAM was just the minimum required, I tried one of the smallest available models. Impressively, it worked. It was slow to spit out tokens, at a rate around a word each 1 to 5 seconds and it was able to correctly answer "What was the biggest planet in the solar system", but it quickly hallucinated talking about moons that it called "Jupterians", while I expected it to talk about Galilean Moons. Nevertheless, LLM's really impressed me and as soon as I get my hands on better hardware I'll try to run other bigger models locally in the hope that I'll finally have a personal "oracle" able to quickly answers most questions I throw at it and help me writing code and other fun things. Of course, I'll have to check its answers before using them, but current state seems impressive enough for me, specially QwQ. Is Any one running smaller experiments and can talk about your results? Is it already possible to have something like an open source co-pilot running locally?
- sorenjan 2y agoYou can use Ollama for serving a model locally, and Continue to use it in VSCode. https://ollama.com/blog/continue-code-assistant https://ollama.com/blog/continue-code-assistant
- 2y ago
- niek_pas 2y agoCan someone tell me what the advantages are of doing this over using, e.g., the ChatGPT web interface? Is it just a privacy thing?
- explorigin 2y agoPrivacy, available offline, software that lasts as long as the hardware can run it.
- JKCalhoun 2y agoYeah, that's me. nCapture a snapshot of it, from time to time — so if it ever goes offline (or off the rails: requires a subscription, begins to serve up ads), you have the last "good" one locally. I have a snapshot of Wikipedia as well (well, not the whole of Wikipedia, but 90GB worth).
- 3eb7988a1663 2y agoWhich Wikipedia snapshot do you grab? I keep meaning to do this, but whenever I skim the Wikipedia downloads pages, they offer hundreds of different flavors without any immediate documentation as to what differentiates the products.
- severine 2y agoYou can use Kiwix: https://kiwix.org/en/ https://kiwix.org/en/
- JKCalhoun 2y agowikipedia_en_all_maxi I guess that means English ... and maxi? As I say, was something around 90GB or so.
- 3eb7988a1663 2y agoWas hoping you had more insight than "maxi sounds good" which is the also the best I have.
- superkuh 2y agoI'd say avoid pulling in all the python and containers required and just download the gguf from huggingface website directly in a browser rather than doing is programmatically. That sidesteps a lot of this project's complexity since nothing about llama.cpp requires those heavy deps or abstractions.
- SteelPh0enix 2y agoYeah, I should have mentioned that in the post because I know that not everyone likes to tinker as much as I do. I'll add a note about this site in a revision soon. However, I still find it useful to know how to do that manually.
- notadoc 2y agoOllama is so easy, what's the benefit to Llama.cpp?
- SteelPh0enix 2y agoIf you're satisfied with ollama, I don't think there's a point to go "lower" and use llama.cpp directly. Unless you want to tinker.
- arendtio 2y agoI tried building and using llama.cpp multiple times, and after a while, I got so frustrated with the frequently broken build process that I switched to ollama with the following script: #!/bin/sh export OLLAMA_MODELS="/mnt/ai-models/ollama/" printf 'Starting the server now.\n' ollama serve >/dev/null 2>&1 & serverPid="$!" printf 'Starting the client (might take a moment (~3min) after a fresh boot).\n' ollama run llama3.2 2>/dev/null printf 'Stopping the server now.\n' kill "$serverPid" And it just works :-)
- boneitis 2y agothis was pretty much spot-on to my experience and track. the ridicule of people choosing to use ollama over llamacpp is so tired. i had already burned an evening trying to debug and fix issues getting nowhere fast, until i pulled ollama and had it working with just two commands. it was a shock. (granted, there is/was a crippling performance problem with sky/kabylake chips but mitigated if you had any kind of mid-tier GPU and tweaked a couple settings) anyone who tries to contribute to the general knowledge base of deploying llamacpp (like TFA) is doing heaven's work.
- SteelPh0enix 2y agoI have spent unreasonable amounts of time building llama.cpp for my hardware setup (AMD GPU) on both Windows and Linux. That was one of the main reasons of writing that blog post for me. Lmao.
- smcleod 2y agoNeat to see more folks writing blogs on their experiences. This however does seem like it's an over-complicated method of building llama.cpp. Assuming you want to do this iteratively (at least for the first time) should only need to run: ccmake . And toggle the parameters your hardware supports or that you want (e.g. if CUDA if you're using Nvidia, Metal if you're using Apple etc..), and press 'c' (configure) then 'g' (generate), then: cmake --build . -j $(expr $(nproc) / 2) Done. If you want to move the binaries into your PATH, you could then optionally run cmake install.
- SteelPh0enix 2y agoWow, i did not know about ccmake. I'll check it out and edit the post if it's really that easy to use, thanks.
- moffkalast 2y agoYeah the mingw method on windows is a ludicrous thing to even think about, and llama.cpp still has that as the suggested option in the readme for some weird reason. Endless sourcing of paths that never works quite right. I literally couldn't get it to work when I first tried it last year. Meanwhile Cmake is like two lines and somehow it's the backup fallback option? I don't get it. And well on linux it's literally just one line with make.
- blharr 2y agoBuilding anything on windows without cmake is just... I don't know why anyone would use anything else. I used to spend hours wrestling with build failures, but after setting up cmake, it just works with everything.
- SteelPh0enix 2y ago>Yeah the mingw method on windows is a ludicrous thing to even think about Building most stuff on Windows is ludicrous, that's not something uncommon. I've chosen MSYS there as it's the easiest and least paninful way of installing deps for Vulkan build.
- lukev 2y ago
- smcleod 2y agoSomewhat related - on several occasions I've come across the claim that _"Ollama is just a llama.cpp wrapper"_, which is inaccurate and completely misses the point. I am sharing my response here to avoid repeating myself repeatedly. With llama.cpp running on a machine, how do you connect your LLM clients to it and request a model gets loaded with a given set of parameters and templates? ... you can't, because llama.cpp is the inference engine - and it's bundled llama-cpp-server binary only provides relatively basic server functionality - it's really more of demo/example or MVP. Llama.cpp is all configured at the time you run the binary and manually provide it command line args for the one specific model and configuration you start it with. Ollama provides a server and client for interfacing and packaging models, such as: - Hot loading models (e.g. when you request a model from your client Ollama will load it on demand). - Automatic model parallelisation. - Automatic model concurrency. - Automatic memory calculations for layer and GPU/CPU placement. - Layered model configuration (basically docker images for models). - Templating and distribution of model parameters, templates in a container image. - Near feature complete OpenAI compatible API as well as it's native native API that supports more advanced features such as model hot loading, context management, etc... - Native libraries for common languages. - Official container images for hosting. - Provides a client/server model for running remote or local inference servers with either Ollama or openai compatible clients. - Support for both an official and self hosted model and template repositories. - Support for multi-modal / Vision LLMs - something that llama.cpp is not focusing on providing currently. - Support for serving safetensors models, as well as running and creating models directly from their Huggingface model ID. In addition to the llama.cpp engine, Ollama are working on adding additional model backends (e.g. things like exl2, awq, etc...). Ollama is not "better" or "worse" than llama.cpp because it's an entirely different tool.
- spencerchubb 2y agoI think what you just said actually reinforces the point that ollama is a llama.cpp wrapper. I don't say that to disparage ollama, in fact I love ollama. It is an impressive piece of software. If x uses y under the hood, then we say "x is a y wrapper"
- 2y ago
- HarHarVeryFunny 2y agoWhat are the limitations on which LLMs (specific transformer variants etc) llama.cpp can run? Does it require the input mode/weights to be in some self-describing format like ONNX that support different model architectures as long as they are built out of specific module/layer types, or does it more narrowly only support transformer models parameterized by depth, width, etc?
- dmezzetti 2y agoSeeing a lot of Ollama vs running llama.cpp direct talk here. I agree that setting up llama.cpp with CUDA isn't always the easiest. But there is a cost to running all inference over HTTPS. Local in-program inference will be faster. Perhaps that doesn't matter in some cases but it's worth noting. I find that running PyTorch is easier to get up and running. For quantization, AWQ models work and it's just a "pip install" away.
- varispeed 2y agoI use ChatGPT and Claude daily, but I can't see a use case for why would I use LLM outside of these services. What do you use Llama.cpp for? I get you can ask it a question in natural language and it will spit out sort of an answer, but what would you do with it, what do you ask it?
- anon373839 2y agoYou can run a model with substantially similar capabilities to Claude or ChatGPT locally, with absolute data privacy guaranteed. Whereas with Claude or ChatGPT, all you can do is trust and hope they won’t use your data against you at some point in the future. If you’re more technically minded, you can hack on the model itself, the sampling method, etc., and have a level of fine-grained control over the technology that isn’t possible with a cloud model.
- ukuina 2y ago> You can run a model with substantially similar capabilities to Claude or ChatGPT locally I am all for local models, but this is massively overselling what they are capable of on common consumer hardware (32GB RAM). If you are interested in what your hardware can pull off, find the top-ranking ~30b models on lmarena.ai and initiate a direct chat with them on the same site. Pose your common questions and see if they are answered to your satisfaction.
- anon373839 2y agoTwo points: 1) I actually think that smaller models are substantially similar to frontier models. Of course the latter are more capable, but they’re more similar than different (which I think the ELO scores on lmarena.ai suggests). 2) You can run much larger models on Apple Silicon with surprisingly decent speed.
- SteelPh0enix 2y agoI use llama.cpp mostly for working with code that i can't share with any online provider. Simple NDA stuff. Some refactors are easier to do via LLM than manually. It's a decent debugging duck too.
- nobodyandproud 2y agoThis was nice. I took the road less traveled and tried building on Windows and AMD. Spoiler: Vulkan with MSYS2 was indeed the easiest to get up and running. I actually tried w64devkit first and it worked properly for llama-server, but there were inexplicable plug-in problems with llama-bench. Edit: I tried w64devkit before I read this write-up and I was left wondering what to try next, so the timing was perfect.
- secondcoming 2y agoI just gave this a shot on my laptop and it works reasonably well considering it has no discrete GPU. One thing I’m unsure of is how to pick a model. I downloaded the 7B one from Huggingface, but how is anyone supposed to know what these models are for, or if they’re any good?
- SteelPh0enix 2y agoRead the README of the model. You'll probably find some benchmark metrics there that can tell you more-or-less how "good" the model is, but keep in mind that it's not that hard to artifically boost those scores, so don't reject every model that isn't at the top of the benchmark. I've listed some good starting models at the end of the post. Usually, most LLM models like Qwen or Llama are general-purpose, some are fine-tuned for specific stuff, like CodeQwen for programming.
- ClownsAbound 2y ago[dead]
- nothrowaways 2y agoThere are many open source alternatives to LMstudio that work just as good.
- deleted 2y ago[deleted]
- slavik81 2y agoFYI, if you're on Ubuntu 24.04, it's easy to build llama.cpp with AMD ROCm GPU acceleration. Debian enabled support for a wider variety of hardware than is available in the official AMD packages, so this should work for nearly all discrete AMD GPUs from Vega onward (with the exception of MI300, because Ubuntu 24.04 shipped with ROCm 5): sudo apt -y install git wget hipcc libhipblas-dev librocblas-dev cmake build-essential # add yourself to the video and render groups sudo usermod -aG video,render $USER # reboot to apply the group changes # download a model wget --continue -O dolphin-2.2.1-mistral-7b.Q5_K_M.gguf \ https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF/resolve/main/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf?download=true # build llama.cpp git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp git checkout b3267 HIPCXX=clang++-17 cmake -S. -Bbuild \ -DGGML_HIPBLAS=ON \ -DCMAKE_HIP_ARCHITECTURES="gfx803;gfx900;gfx906;gfx908;gfx90a;gfx1010;gfx1030;gfx1100;gfx1101;gfx1102" \ -DCMAKE_BUILD_TYPE=Release make -j8 -C build # run llama.cpp build/bin/llama-cli -ngl 32 --color -c 2048 \ --temp 0.7 --repeat_penalty 1.1 -n -1 \ -m ../dolphin-2.2.1-mistral-7b.Q5_K_M.gguf \ --prompt "Once upon a time" I think this will also work on Rembrandt, Renoir, and Cezanne integrated GPUs with Linux 6.10 or newer, so you might be able to install the HWE kernel to get it working on that hardware. With that said, users with CDNA 2 or RDNA 3 GPUs should probably use the official AMD ROCm packages instead of the built-in Ubuntu packages, as there are performance improvements for those architectures in newer versions of rocBLAS.
- inLewOf 2y agore Temperature config option: I've found it useful for trying to generate something akin to a sampling-based confidence score for chat completions (e.g., set the temperature a bit high, run the model a few times and calculate the distribution of responses). Otherwise haven't figured out a good way to get confidence scores in llama.cpp (Been tracking this git request to get log_probs https://github.com/ggerganov/llama.cpp/issues/6423 https://github.com/ggerganov/llama.cpp/issues/6423)
- marcantonio 2y agoI set up llama.cop last week on my M3. Was fairly simple via homebrew. However, I get tags like <|imstart|> in the output constantly. Is there a way to filter them out with llama-server? Seems like a major usability issue if you want to use llama.cpp by itself (with the web interface). ollama didn’t have the issue, but it’s less configurable.
- NoZZz 2y agoYou can also just download LMStudio for free, works out of the box.