6 ms·

I recently downloaded ollama on my Linux machine and even with 3060 12gb gpu and 24 GB Ram I'm unable to run mistral or dolphin and always get an out of memory

I recently downloaded ollama on my Linux machine and even with 3060 12gb gpu and 24 GB Ram I'm unable to run mistral or dolphin and always get an out of memory error. So it's amazing that these companies are able to scale these so well handling thousands of requests per minute. I wish they would do a behind the scenes on how much money, time, optimisation is done to make this all work. Also big fan of anyscale. Their pricing is just phenomenal for running models like mixtral. Not sure how they are so affordable.

mgreg 3y ago
Some details that might interest you from SemiAnalysis [1] just published yesterday. There's quite a bit that goes into optimizing inference with lots of dials to turn. One thing that does seem to have a large impact is batch size which is a benefit of scale. 1. https://www.semianalysis.com/p/inference-race-to-the-bottom-make https://www.semianalysis.com/p/inference-race-to-the-bottom-...
ilaksh 3y ago
Try https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Builds very quickly with make. But if it's slow when you try it then make sure to enable any flags related to CUDA and then try the build again. A key parameter is the one that tells it how many layers to offload to the GPU. ngl I think. Also, download the 4 bit GGUF from HuggingFace and try that. Uses much less memory.
- avereveard 3y ago
  with llama.cpp and a 12gb 3060 they can get the an entire mistral model at Q5_K_M n ram with the full 32k context. I recommend openhermes-2.5-mistral-7b-16k with USER: ASSISTANT: instructions, it's working surprisingly well for content production (let's say everything except logic and math, but that's not the strong suite of 7b models in general)
TheMatten 3y ago
I can reasonably run (quantized) Mistral-7B on a 16GB machine without GPU, using ollama. Are you sure it isn't a configuration error or bug?
- ilaksh 3y ago
  How many tokens per second and what are the specs of the machine? My attempts at CPU only have been really slow.
  - berkut 3y ago
    In my experience with llama.cpp using the CPU (on Linux) is very slow compared to GPU or NPU with the same models as my M1 MacBook Pro using Metal (or maybe it's the shared memory allowing the speedup?). Even with 12 threads of my 5900X (I've tried using the full 24 SMT - that doesn't really seem to help) with the dolphin-2.5-mixtral-8x7b.Q5_K_M model, my MacBook Pro is around 5-6x faster in terms of tokens per second...
    - ilaksh 3y ago
      I think that Metal or something is actually a built in graphics/matrix accelerator that those Macs have now. It's not really using a CPU although it seems like Apple may be trying to market it a little bit as though it's just a powerful CPU. But more like accelerator integrated with CPU. But whatever it is, it's great, and I hope that Intel and AMD will catch up. AMD has had the APUs for awhile but I think they aren't at the same level at all as the new Mac acceleration.
    - stavros 3y ago
      There must be something wrong, my 3060 does double the tokens per second as my M2 Mac (with Metal).
  - TheMatten 3y ago
    Seems to be around 3 tokens/s on my laptop, which is faster than average human, but not too fast of course. On a desktop with mid-range GPU used for offloading, I can get around 12 tokens/s, which is plenty fast for chatting.
M4v3R 3y ago
You need to pick the correct model size and quantization for the amount of GPU RAM you have. For any given model don’t download the default file, instead go to Tags section on Ollama’s page and pick a quantization whose size in GB is at most 2/3rd of your available RAM, and it should work. For example in your case Mistral-7B q4_0 and even q8_0 should work perfectly.
- swyx 3y ago
  whats the intuition for 2/3 of RAM?
  - avereveard 3y ago
    you need some leftover for holding the context
  - M4v3R 3y ago
    Because there’s always some overhead during inference plus you don’t want to fill all your available RAM because you risk swapping to disk which will make everything slow to a crawl.
    - swyx 3y ago
      so why is the overhead a 1/3 ratio instead of a constant amount? just testing the scaling assumption
deleted 3y ago
[deleted]
ignoramous 3y ago
> optimisation is done to make this all work Obviously still a nascent area but https://lmsys.org/blog https://lmsys.org/blog do a good job of diving into engineering challenges behind running these LLMs. (I'm sure there are others)
idonotknowwhy 3y ago
You can run a 7b Q4 model in your 12gb vram no problem.

mgreg 3y ago

Some details that might interest you from SemiAnalysis [1] just published yesterday. There's quite a bit that goes into optimizing inference with lots of dials to turn. One thing that does seem to have a large impact is batch size which is a benefit of scale. 1. https://www.semianalysis.com/p/inference-race-to-the-bottom-make https://www.semianalysis.com/p/inference-race-to-the-bottom-...

ilaksh 3y ago

Try https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Builds very quickly with make. But if it's slow when you try it then make sure to enable any flags related to CUDA and then try the build again. A key parameter is the one that tells it how many layers to offload to the GPU. ngl I think. Also, download the 4 bit GGUF from HuggingFace and try that. Uses much less memory.

avereveard 3y ago
with llama.cpp and a 12gb 3060 they can get the an entire mistral model at Q5_K_M n ram with the full 32k context. I recommend openhermes-2.5-mistral-7b-16k with USER: ASSISTANT: instructions, it's working surprisingly well for content production (let's say everything except logic and math, but that's not the strong suite of 7b models in general)

TheMatten 3y ago

I can reasonably run (quantized) Mistral-7B on a 16GB machine without GPU, using ollama. Are you sure it isn't a configuration error or bug?

ilaksh 3y ago
How many tokens per second and what are the specs of the machine? My attempts at CPU only have been really slow.
- berkut 3y ago
  In my experience with llama.cpp using the CPU (on Linux) is very slow compared to GPU or NPU with the same models as my M1 MacBook Pro using Metal (or maybe it's the shared memory allowing the speedup?). Even with 12 threads of my 5900X (I've tried using the full 24 SMT - that doesn't really seem to help) with the dolphin-2.5-mixtral-8x7b.Q5_K_M model, my MacBook Pro is around 5-6x faster in terms of tokens per second...
  - ilaksh 3y ago
    I think that Metal or something is actually a built in graphics/matrix accelerator that those Macs have now. It's not really using a CPU although it seems like Apple may be trying to market it a little bit as though it's just a powerful CPU. But more like accelerator integrated with CPU. But whatever it is, it's great, and I hope that Intel and AMD will catch up. AMD has had the APUs for awhile but I think they aren't at the same level at all as the new Mac acceleration.
  - stavros 3y ago
    There must be something wrong, my 3060 does double the tokens per second as my M2 Mac (with Metal).
- TheMatten 3y ago
  Seems to be around 3 tokens/s on my laptop, which is faster than average human, but not too fast of course. On a desktop with mid-range GPU used for offloading, I can get around 12 tokens/s, which is plenty fast for chatting.

M4v3R 3y ago

You need to pick the correct model size and quantization for the amount of GPU RAM you have. For any given model don’t download the default file, instead go to Tags section on Ollama’s page and pick a quantization whose size in GB is at most 2/3rd of your available RAM, and it should work. For example in your case Mistral-7B q4_0 and even q8_0 should work perfectly.

swyx 3y ago
whats the intuition for 2/3 of RAM?
- avereveard 3y ago
  you need some leftover for holding the context
- M4v3R 3y ago
  Because there’s always some overhead during inference plus you don’t want to fill all your available RAM because you risk swapping to disk which will make everything slow to a crawl.
  - swyx 3y ago
    so why is the overhead a 1/3 ratio instead of a constant amount? just testing the scaling assumption

deleted 3y ago

[deleted]

ignoramous 3y ago

> optimisation is done to make this all work Obviously still a nascent area but https://lmsys.org/blog https://lmsys.org/blog do a good job of diving into engineering challenges behind running these LLMs. (I'm sure there are others)

idonotknowwhy 3y ago

You can run a 7b Q4 model in your 12gb vram no problem.