8 ms·

This is an open weights 22B model. The download on Huggingface is 44GB. Is there a rule-of-thumb estimate for how much RAM this would need to be used locally?

This is an open weights 22B model. The download on Huggingface is 44GB. Is there a rule-of-thumb estimate for how much RAM this would need to be used locally? Is the RAM requirement the same for a GPU and "unified" RAM like Apple silicon?

fnbr 2y ago
The rule of thumb is roughly 44gb, as most models are trained in bf16, and require 16 bits per parameter, so 2 bytes. You need a bit more for activations, so maybe 50GB? you need enough RAM and HBM (GPU RAM) so it’s a constraint on both.
- sharbloop 2y ago
  Which GPU card can I buy to run this model? Can it run on commercial RTX3090 or does it need a custom GPU?
  - Havoc 2y ago
    3090 or 4090 will be able to run quantized 22B models. Though realistically for code completion smaller models will be better due to speed
  - TechDebtDevin 2y ago
    Easy..
- Novosell 2y ago
  Most GPUs still use GDDR I'm pretty sure, not HBM. Do you mean VRAM?
deleted 2y ago
[deleted]
mauricio 2y ago
22B params * 2 bytes (FP16) = 44GB just for the weights. Doesn't include KV cache and other things. When the model gets quantized to say 4bit ints, it'll be 22B params * 0.5 bytes = 11GB for example.
tosh 2y ago
B × Q / 8 B: number of parameters Q: quantization (16 = no quantization) via https://news.ycombinator.com/item?id=40090566 https://news.ycombinator.com/item?id=40090566
resource_waste 2y ago
[flagged]
- kblissett 2y ago
  Rather than teasing it might be more productive to explain why it is you think unified memory isn't relevant in this case.
  - resource_waste 2y ago
    Just call it RAM. No need to use Apple marketing.
TechDebtDevin 2y ago
I'm honestly not sure on how to measure the amount of vRAM required for these models but I suspect this would run relatively fast, depending on your use case, on a mid to high end 20 or 30 series card. No idea about Apple unified RAM. I get a lot out of performance out of even older cards such as a 1080ti but haven't tested this model.
wing-_-nuts 2y ago
Wait for a gguf release of this and it will fit neatly into a 3090 with a decent quant. I'm excited for this model and I'll be adding it to my collection.
Terretta 2y ago
Yes, RAM requirement is BnL same for GPU and using the metal/GPU in Apple Silicon. Running LLM models on a MacBook Pro with Apple Silicon vs. a PC with an Nvidia 4090 GPU has trade-offs. My 128GB MacBook Pro handles models using up to 96GB of unified memory, running at a little under half the speed of a 4090. If you use a quantized version of full floating point model, you can run the largest open models available. While the 4090 has 24GB of dedicated memory and higher bandwidth (1000 GB/s vs. 400 GB/s on M3 Max), the Mac’s unified memory system (up to 128GB) is flexible and holds smarter models (8 bit and 6 bit models act still mostly all there, 4 bit is so so, 2 bit is brain damaged). The M2 Ultra in Mac Studio offers even more (800 GB/s bandwidth and 192GB memory). So, ok, 6 or 8 of 4090 cards or 4 x A6000 cards excels in raw performance, but Apple’s unified memory in a laptop fits in your backback. It's not clear to me why Macbooks and Mac Studio Ultras with maxed out RAM aren't selling better if you look at the convenience and price relative to model size. Models that fit in one 4090 or even a pair of 4090s are toys compared to what fits on these, so for the big models you're comparing a laptop to a minifridge.
- tyfon 2y ago
  I have a 5940x with 128 gb ram. It's a bit slower perhaps than the mac, but i get the best of both worlds. That is I get a lot of RAM to hold the model and I can offload as much of it as possible to the GPU. This works especially well with models like mixtral 8x22, but also models like llama3 and the old large bloom model. I also get the utility of running Linux instead of the closed up mac os. But running large models locally is not exclusive to mac studio, you can do the same on PC for a much lower cost.
  - Terretta 2y ago
    I get the utility of a laptop that runs 20 hours on battery and slips in the side pocket of my carry-on or shoulder bag. (The Mac can also split between RAM and GPU.) Mixtral 8x22 and Llama 3 70b stream at roughly the same speed as last year's GPT-4. > closed up MacOS https://github.com/apple-oss-distributions/distribution-macOS https://github.com/apple-oss-distributions/distribution-macO... curl https://alx.sh | sh https://asahilinux.org/ https://asahilinux.org/ I prefer the "utility" of BSDs, but that's just a preference.

fnbr 2y ago

The rule of thumb is roughly 44gb, as most models are trained in bf16, and require 16 bits per parameter, so 2 bytes. You need a bit more for activations, so maybe 50GB? you need enough RAM and HBM (GPU RAM) so it’s a constraint on both.

sharbloop 2y ago
Which GPU card can I buy to run this model? Can it run on commercial RTX3090 or does it need a custom GPU?
- Havoc 2y ago
  3090 or 4090 will be able to run quantized 22B models. Though realistically for code completion smaller models will be better due to speed
- TechDebtDevin 2y ago
  Easy..
Novosell 2y ago
Most GPUs still use GDDR I'm pretty sure, not HBM. Do you mean VRAM?

deleted 2y ago

[deleted]

mauricio 2y ago

22B params * 2 bytes (FP16) = 44GB just for the weights. Doesn't include KV cache and other things. When the model gets quantized to say 4bit ints, it'll be 22B params * 0.5 bytes = 11GB for example.

tosh 2y ago

B × Q / 8 B: number of parameters Q: quantization (16 = no quantization) via https://news.ycombinator.com/item?id=40090566 https://news.ycombinator.com/item?id=40090566

resource_waste 2y ago

[flagged]

kblissett 2y ago
Rather than teasing it might be more productive to explain why it is you think unified memory isn't relevant in this case.
- resource_waste 2y ago
  Just call it RAM. No need to use Apple marketing.

TechDebtDevin 2y ago

I'm honestly not sure on how to measure the amount of vRAM required for these models but I suspect this would run relatively fast, depending on your use case, on a mid to high end 20 or 30 series card. No idea about Apple unified RAM. I get a lot out of performance out of even older cards such as a 1080ti but haven't tested this model.

wing-_-nuts 2y ago

Wait for a gguf release of this and it will fit neatly into a 3090 with a decent quant. I'm excited for this model and I'll be adding it to my collection.

Terretta 2y ago

Yes, RAM requirement is BnL same for GPU and using the metal/GPU in Apple Silicon. Running LLM models on a MacBook Pro with Apple Silicon vs. a PC with an Nvidia 4090 GPU has trade-offs. My 128GB MacBook Pro handles models using up to 96GB of unified memory, running at a little under half the speed of a 4090. If you use a quantized version of full floating point model, you can run the largest open models available. While the 4090 has 24GB of dedicated memory and higher bandwidth (1000 GB/s vs. 400 GB/s on M3 Max), the Mac’s unified memory system (up to 128GB) is flexible and holds smarter models (8 bit and 6 bit models act still mostly all there, 4 bit is so so, 2 bit is brain damaged). The M2 Ultra in Mac Studio offers even more (800 GB/s bandwidth and 192GB memory). So, ok, 6 or 8 of 4090 cards or 4 x A6000 cards excels in raw performance, but Apple’s unified memory in a laptop fits in your backback. It's not clear to me why Macbooks and Mac Studio Ultras with maxed out RAM aren't selling better if you look at the convenience and price relative to model size. Models that fit in one 4090 or even a pair of 4090s are toys compared to what fits on these, so for the big models you're comparing a laptop to a minifridge.

tyfon 2y ago
I have a 5940x with 128 gb ram. It's a bit slower perhaps than the mac, but i get the best of both worlds. That is I get a lot of RAM to hold the model and I can offload as much of it as possible to the GPU. This works especially well with models like mixtral 8x22, but also models like llama3 and the old large bloom model. I also get the utility of running Linux instead of the closed up mac os. But running large models locally is not exclusive to mac studio, you can do the same on PC for a much lower cost.
- Terretta 2y ago
  I get the utility of a laptop that runs 20 hours on battery and slips in the side pocket of my carry-on or shoulder bag. (The Mac can also split between RAM and GPU.) Mixtral 8x22 and Llama 3 70b stream at roughly the same speed as last year's GPT-4. > closed up MacOS https://github.com/apple-oss-distributions/distribution-macOS https://github.com/apple-oss-distributions/distribution-macO... curl https://alx.sh | sh https://asahilinux.org/ https://asahilinux.org/ I prefer the "utility" of BSDs, but that's just a preference.