5 ms·

An 80% size reduction is no joke, and the fact that the 1.58-bit version runs on dual H100s at 140 tokens/s is kind of mind-blowing. That said, I’m still skepti

An 80% size reduction is no joke, and the fact that the 1.58-bit version runs on dual H100s at 140 tokens/s is kind of mind-blowing. That said, I’m still skeptical about how practical this really is for most people. Like, yeah, you can run it on 24GB VRAM or even with just 20GB RAM, but "slow" is an understatement—those speeds would make even the most patient person throw their hands up. And then there’s the whole repetition issue. Infinite loops with "Pygame’s Pygame’s Pygame’s" kind of defeats the point of quantization if you ask me. Sure, the authors have fixes like adjusting the KV cache or using min_p, but doesn’t that just patch a symptom rather than solve the actual problem? A fried model is still fried, even if it stops repeating itself. On the flip side, I love that they’re making this accessible on Hugging Face... and the dynamic quantization approach is pretty brilliant. Using 1.58-bit for MoEs and leaving sensitive layers like down_proj at higher precision—super clever. Feels like they’re squeezing every last drop of juice out of the architecture, which is awesome for smaller teams who can’t afford OpenAI-scale hardware. "accessible" still comes with an asterisk. Like, I get that shared memory architectures like a 192GB Mac Ultra are a big deal, but who’s dropping $6,000+ on that setup? For that price, I’d rather build a rig with used 3090s and get way more bang for my buck (though, yeah, it’d be a power hog). Cool tech—no doubt—but the practicality is still up for debate. Guess we'll see if the next-gen models can address some of these trade-offs.

danielhanchen 2y ago
Oh the repetition issue is only on the non dynamic quants :) If you do dynamic quantization and use the 1.58bit dynamic quantized model the repetition issue fully disappears! Min_p = 0.05 was a way I found to counteract the 1.58bit model generating singular incorrect tokens which happen around 1 token per 8000!
- smcleod 2y ago
  min_p is great, do you apply a small amount of temperate as well?
  - danielhanchen 2y ago
    The recommended temperature from DeepSeek is 0.6 so I leave it at that!
    - smcleod 2y ago
      I think most of the model creators share their model usage examples so high at 0.6-0.7 simply because it's what a lot of the client apps use. IMO this is WAY too high unless you're doing creative writing. Generally I set temp to 0-0.4 at absolute most. min_p actually needs a little temperature to work effectively so with min_p I almost always use 0.2
      - danielhanchen 2y ago
        Ye lower temp is also good :) Tbh its all trial and error - I found temp=1.5, min_p=0.1 to be very useful for pass@k type workloads - ie calling the LLM multiple times and aggregating. temp=0 is also good for singular outputs. For classification tasks, it's better to actually inspect the logits. But my goto setting is always setting min_p at least 0.01 or 0.05! It vastly suppresses incorrect rare random tokens from being created, and it helps massively!
  - Der_Einzige 2y ago
    Btw, min_p (the paper about the sampler) got accepted to ICLR! As 4th author it warms my heart to so it used so much in the wild.
    - danielhanchen 2y ago
      Oh hi!! Congratulations on ICLR!!! min_p = 0.1 and temp = 1.5 is my default goto settings!!
elorant 2y ago
Not everyone needs the largest model. There are variations or R1 with fewer parameters that can easily run on consumer hardware. With 80% size reduction you could run 70B on 8-bit on an RTX 3090. Other than that, if you really need the big one you can get six 3090s and you're good to go. It's not cheap, but you're running a ChatGPT equivalent model from your basement. A year ago this was a wetdream for most enthusiasts.
- Kye 2y ago
  I ran whatever version Ollama downloaded on a 3070ti (laptop version). It's reasonably fast. Generative stuff can get weird if you do prompts like "in the style of" or "a new episode of" because it doesn't seem to have much pop culture in its training data. It knows the Stargate movie, for example, and seems to have the IMDB info for the series, but goes absolutely ham trying to summarize the series. This line in the stuff inside the <think> section suggests it's also been trained on YouTube clips: >> "I'm not entirely sure if I got all the details right, but this is what I remember from watching clips and summaries online." An excerpt from the generated summary: >> "Set in the 23rd century during a Z-Corp invasion, the series features action sequences, strategic thinking, and humor. It explores themes of international espionage, space warfare, and humanity's role in the cosmos. The show incorporates musical numbers and catchy theme songs for an engaging viewing experience. The plot involves investigating alien warships and their secret base on Kessari planet while addressing personal conflicts and philosophical questions about space." "It explores themes of international espionage, space warfare, and humanity's role in the cosmos" is the closest to correct line in the whole output.
  - Mashimo 2y ago
    > ran whatever version Ollama downloaded on a 3070ti (laptop version). It's reasonably fast. Probably was not r1, but one of the other models that got trained on r1, which apparently might still be quite good.
    - Kye 2y ago
      I'm not too hip to all the LLM terminology, so maybe someone can make sense of this and see if it's r1 or something based on r1: >>> /show info Model architecture qwen2 parameters 7.6B context length 131072 embedding length 3584 quantization Q4_K_M
smcleod 2y ago
I just ran it up on 48gb (2x 3090) + overflow into CPU RAM and it runs at around 4tk/s (only a little 8k context size though) which while absolutely not something I'd personally use daily - it is actually usable.
- eurekin 2y ago
  I have similar set-up - can you help out with running it? Was it in ollama? EDIT: It seems that original authors provided a nice write-up: https://unsloth.ai/blog/deepseekr1-dynamic#:~:text=%F0%9F%96%A5%EF%B8%8F%20Running%20Dynamic%20Quants https://unsloth.ai/blog/deepseekr1-dynamic#:~:text=%F0%9F%96...
  - smcleod 2y ago
    Yep that's pretty much what I did, their calculation for the layers was slightly off though, I found I could offload an extra 1-2 layers to the GPUs
    - danielhanchen 2y ago
      Oh yes I reduced it by 4 for just in case :) I found sometimes the formula doesn't work, so in the worst case -4 was used - glad at least it ran!
- ryao 2y ago
  Which one did you run?
huijzer 2y ago
> That said, I’m still skeptical about how practical this really is for most people. I'm running Open WebUI for months now for me and some friends as a front-end to one of the API providers (deepinfra in my case, but there are many others, see https://artificialanalysis.ai/ https://artificialanalysis.ai/). Having 1.58-bit is very practical for me. I'm looking much forward to the API provider adding this model to their system. They also added a Llama turbo (also quantized) a few months back so I have good hopes.
- danielhanchen 2y ago
  Oh I love Open WebUI as well!! But glad to hear the 1.58bit version could be helpful to you!
ricardobeat 2y ago
The repetition issue happens on simple quantization, what they are releasing is an approach that fixes that.
- danielhanchen 2y ago
  Yes exactly! I edited the blog post to make the wording a bit better!
jairuhme 2y ago
At my work, we self-host some models and have found that for anything remotely similar to RAG or use cases that are very specific, the quantized models have proven to be more than sufficient. This helps us keep them running on smaller infra and generally lower costs
- michaelt 2y ago
  Personally I've noticed major changes in performance between different quantisations of the same model. Mistral's large 123B model works well (but slowly) at 4-bit quantisation, but if I knock it down to 2.5-bit quantisation for speed, performance drops to the point where I'm better off with a 70B 4-bit model. This makes me reluctant to evaluate new models in heavily quantised forms, as you're measuring the quantisation more than the actual model.
  - sitkack 2y ago
    How are you doing your evals? Being able to do semantic diffs of the output of the two models should tell you what you need to do.
  - danielhanchen 2y ago
    That's a fair point - the trick with dynamic quants is we selectively choose not to quantize many components - ie attention is left at 4 or 6bit, just the MoE parts are 1.5bit (-1, 0, 1) There are distilled versions like Qwen 1.5, 3, 14, 32, Llama 8, 70, but those are distilled - if you want to run the original R1, then the quants are currently the only way. But I agree quants do affect perf - hence the trick for MoEs is to not quantize specific areas!
JKCalhoun 2y ago
Layman here — but I am hopeful for 1.58 bit plus custom silicon to be the Holy Grail. I suppose I am setting high expectations on Apple to integrate said in their next "A" chip. Wishful thinking.
- danielhanchen 2y ago
  Ye a custom chip would be insane! 1.5 bit with a scaling factor seems to be actually usable for MoEs with shared experts!
rafaelmn 2y ago
>Like, I get that shared memory architectures like a 192GB Mac Ultra are a big deal, but who’s dropping $6,000+ on that setup? AMD strix halo APU will have quad channel memory and will launch soon so expect these kinds of setups available for much less. Apple is charging an arm and a leg for memory upgrades, hopefully we get competition soon. From what I saw at CES OEMs are paying attention to this use case as well - hopefully not following suite on RAM markups.
- dagmx 2y ago
  Unfortunately, Apple’s RAM and Storage upgrade prices are very in line with other class comparable OEMs. I’m sure there’ll be some amount of undercutting but I don’t think it’ll be a huge difference on the RAM side itself.
  - rafaelmn 2y ago
    Highly depends on how accessible AMD makes these boards, if a lot of OEMs get it there will be good deals for sure. DDR5 prices are nowhere near Apple markups.
  - nyrikki 2y ago
    The upgrade from 8GB to 16GB cost $200...in 2012!!!! The newest Lenovo workstations that use LPDDR5x with 7467MT/s gets you a 16GB jump for the price apple charges for an 8GB. 2:1 isn't "class comparable" IMHO.
- sliken 2y ago
  Keep in mind the strix halo APU has a 256 bit wide memory bus and the Mac Ultra has a 1024 bit wide memory bus. Here's hoping the Nvidia Digit (GB10 chip) has a 512 bit or 1024 bit wide interface, otherwise the Strix Halo will be the best you can do if you don't get the Mac Ultra.
  - rafaelmn 2y ago
    I mean it remains to be seen if it will be compute or bandwidth bound, I am sure mac ultra will also have double or triple compute as well. But in either case its going to do much better than currently available CPUs with easily upgradeable ram. I would not be surprised to see 128gb configurations for around 3k (going of the ASUS g13 announced pricing of arround 2k for 32gb version and them saying it will go up to 128gb). At that point sure it might not compete with max but its at a much more acceptable price point, it will not be a device you get just for the AI, but a mobile workstation that you can also run some local models on for normal money. Will need to wait and see. I know I am not buying anything from ASUS either way.
F7F7F7 2y ago
People would only be 'throwing their hands up' because commercial LLMs have set unreasonable expectations for folks. Anyone who has a/the need for or understands the value of a local LLM would be OK with this kind of output.
- bnchrch 2y ago
  Everyone has the need for on device LLM, if the response rate was fast!
  - ddingus 2y ago
    I have MLCCHAT on my old Note 9 phone. It is actually still a great phone, but has 5GB RAM. Running an on device model is the first and only use case the RAM actually matters. And it has a headphone jack, OK? I just hate Bluetooth earbuds. And yeah, it isna problem, but I digress. When I run a 2.5B model, I get respectable output. Takes a minute or two to process the context, then output begins at somewhere on the order of 4 to 10 tokens per sec. So, I just make a query and give it a few and I have my response. Here is how I see it: That little model, which is Gemma 2.2b sorry, knows a lot of stuff. It has knowledge I don't and it gives it to me in a reasonable, though predictable way. Answers are always of a certain teacher reminding student how it all goes way. I don't care. Better is nice, but if I were stuck somewhere with no network, being able to query that model is amazing! First aid, how to make fires, materials and uses. Fixing stuff, theories of operation, what things mean and more are in that thing ready for me to take advantage of. I consider what I have fast. And it will get one or two orders faster over the next few years too. I did it on a lark (ask the model what that means) and was surprised to see I gained a nice tool.
    - jpc0 2y ago
      > First aid, how to make fires, materials and uses This scares me more than it should... Please do not trust an AI in actual life and death situations... Sure if it is literally your only option, but this implies you have a device on you that could make a phone call to an emergency number where a real human with real training and actually correct knowledge can assist you. Even as an avid hiker the amount of times I've been out off cell service is miniscule and I absolutely refresh my knowledge on first aid regularly and any potential threats before a hike somewhere new.
sliken 2y ago
I do want a 192GB Mac Ultra, I'm hoping the Nvidia Digit achieves similar at $3,000. Sadly no specifications or benchmarks, so tokens/sec is just a guess at this point.
yodsanklai 2y ago
> I’d rather build a rig with used 3090s and get way more bang for my buck I'm curious, what would you use that rig for?

danielhanchen 2y ago

Oh the repetition issue is only on the non dynamic quants :) If you do dynamic quantization and use the 1.58bit dynamic quantized model the repetition issue fully disappears! Min_p = 0.05 was a way I found to counteract the 1.58bit model generating singular incorrect tokens which happen around 1 token per 8000!

smcleod 2y ago
min_p is great, do you apply a small amount of temperate as well?
- danielhanchen 2y ago
  The recommended temperature from DeepSeek is 0.6 so I leave it at that!
  - smcleod 2y ago
    I think most of the model creators share their model usage examples so high at 0.6-0.7 simply because it's what a lot of the client apps use. IMO this is WAY too high unless you're doing creative writing. Generally I set temp to 0-0.4 at absolute most. min_p actually needs a little temperature to work effectively so with min_p I almost always use 0.2
    - danielhanchen 2y ago
      Ye lower temp is also good :) Tbh its all trial and error - I found temp=1.5, min_p=0.1 to be very useful for pass@k type workloads - ie calling the LLM multiple times and aggregating. temp=0 is also good for singular outputs. For classification tasks, it's better to actually inspect the logits. But my goto setting is always setting min_p at least 0.01 or 0.05! It vastly suppresses incorrect rare random tokens from being created, and it helps massively!
- Der_Einzige 2y ago
  Btw, min_p (the paper about the sampler) got accepted to ICLR! As 4th author it warms my heart to so it used so much in the wild.
  - danielhanchen 2y ago
    Oh hi!! Congratulations on ICLR!!! min_p = 0.1 and temp = 1.5 is my default goto settings!!

elorant 2y ago

Not everyone needs the largest model. There are variations or R1 with fewer parameters that can easily run on consumer hardware. With 80% size reduction you could run 70B on 8-bit on an RTX 3090. Other than that, if you really need the big one you can get six 3090s and you're good to go. It's not cheap, but you're running a ChatGPT equivalent model from your basement. A year ago this was a wetdream for most enthusiasts.

Kye 2y ago
I ran whatever version Ollama downloaded on a 3070ti (laptop version). It's reasonably fast. Generative stuff can get weird if you do prompts like "in the style of" or "a new episode of" because it doesn't seem to have much pop culture in its training data. It knows the Stargate movie, for example, and seems to have the IMDB info for the series, but goes absolutely ham trying to summarize the series. This line in the stuff inside the <think> section suggests it's also been trained on YouTube clips: >> "I'm not entirely sure if I got all the details right, but this is what I remember from watching clips and summaries online." An excerpt from the generated summary: >> "Set in the 23rd century during a Z-Corp invasion, the series features action sequences, strategic thinking, and humor. It explores themes of international espionage, space warfare, and humanity's role in the cosmos. The show incorporates musical numbers and catchy theme songs for an engaging viewing experience. The plot involves investigating alien warships and their secret base on Kessari planet while addressing personal conflicts and philosophical questions about space." "It explores themes of international espionage, space warfare, and humanity's role in the cosmos" is the closest to correct line in the whole output.
- Mashimo 2y ago
  > ran whatever version Ollama downloaded on a 3070ti (laptop version). It's reasonably fast. Probably was not r1, but one of the other models that got trained on r1, which apparently might still be quite good.
  - Kye 2y ago
    I'm not too hip to all the LLM terminology, so maybe someone can make sense of this and see if it's r1 or something based on r1: >>> /show info Model architecture qwen2 parameters 7.6B context length 131072 embedding length 3584 quantization Q4_K_M

smcleod 2y ago

I just ran it up on 48gb (2x 3090) + overflow into CPU RAM and it runs at around 4tk/s (only a little 8k context size though) which while absolutely not something I'd personally use daily - it is actually usable.

eurekin 2y ago
I have similar set-up - can you help out with running it? Was it in ollama? EDIT: It seems that original authors provided a nice write-up: https://unsloth.ai/blog/deepseekr1-dynamic#:~:text=%F0%9F%96%A5%EF%B8%8F%20Running%20Dynamic%20Quants https://unsloth.ai/blog/deepseekr1-dynamic#:~:text=%F0%9F%96...
- smcleod 2y ago
  Yep that's pretty much what I did, their calculation for the layers was slightly off though, I found I could offload an extra 1-2 layers to the GPUs
  - danielhanchen 2y ago
    Oh yes I reduced it by 4 for just in case :) I found sometimes the formula doesn't work, so in the worst case -4 was used - glad at least it ran!
ryao 2y ago
Which one did you run?

huijzer 2y ago

> That said, I’m still skeptical about how practical this really is for most people. I'm running Open WebUI for months now for me and some friends as a front-end to one of the API providers (deepinfra in my case, but there are many others, see https://artificialanalysis.ai/ https://artificialanalysis.ai/). Having 1.58-bit is very practical for me. I'm looking much forward to the API provider adding this model to their system. They also added a Llama turbo (also quantized) a few months back so I have good hopes.

danielhanchen 2y ago
Oh I love Open WebUI as well!! But glad to hear the 1.58bit version could be helpful to you!

ricardobeat 2y ago

The repetition issue happens on simple quantization, what they are releasing is an approach that fixes that.

danielhanchen 2y ago
Yes exactly! I edited the blog post to make the wording a bit better!

jairuhme 2y ago

At my work, we self-host some models and have found that for anything remotely similar to RAG or use cases that are very specific, the quantized models have proven to be more than sufficient. This helps us keep them running on smaller infra and generally lower costs

michaelt 2y ago
Personally I've noticed major changes in performance between different quantisations of the same model. Mistral's large 123B model works well (but slowly) at 4-bit quantisation, but if I knock it down to 2.5-bit quantisation for speed, performance drops to the point where I'm better off with a 70B 4-bit model. This makes me reluctant to evaluate new models in heavily quantised forms, as you're measuring the quantisation more than the actual model.
- sitkack 2y ago
  How are you doing your evals? Being able to do semantic diffs of the output of the two models should tell you what you need to do.
- danielhanchen 2y ago
  That's a fair point - the trick with dynamic quants is we selectively choose not to quantize many components - ie attention is left at 4 or 6bit, just the MoE parts are 1.5bit (-1, 0, 1) There are distilled versions like Qwen 1.5, 3, 14, 32, Llama 8, 70, but those are distilled - if you want to run the original R1, then the quants are currently the only way. But I agree quants do affect perf - hence the trick for MoEs is to not quantize specific areas!

JKCalhoun 2y ago

Layman here — but I am hopeful for 1.58 bit plus custom silicon to be the Holy Grail. I suppose I am setting high expectations on Apple to integrate said in their next "A" chip. Wishful thinking.

danielhanchen 2y ago
Ye a custom chip would be insane! 1.5 bit with a scaling factor seems to be actually usable for MoEs with shared experts!

rafaelmn 2y ago

>Like, I get that shared memory architectures like a 192GB Mac Ultra are a big deal, but who’s dropping $6,000+ on that setup? AMD strix halo APU will have quad channel memory and will launch soon so expect these kinds of setups available for much less. Apple is charging an arm and a leg for memory upgrades, hopefully we get competition soon. From what I saw at CES OEMs are paying attention to this use case as well - hopefully not following suite on RAM markups.

dagmx 2y ago
Unfortunately, Apple’s RAM and Storage upgrade prices are very in line with other class comparable OEMs. I’m sure there’ll be some amount of undercutting but I don’t think it’ll be a huge difference on the RAM side itself.
- rafaelmn 2y ago
  Highly depends on how accessible AMD makes these boards, if a lot of OEMs get it there will be good deals for sure. DDR5 prices are nowhere near Apple markups.
- nyrikki 2y ago
  The upgrade from 8GB to 16GB cost $200...in 2012!!!! The newest Lenovo workstations that use LPDDR5x with 7467MT/s gets you a 16GB jump for the price apple charges for an 8GB. 2:1 isn't "class comparable" IMHO.
sliken 2y ago
Keep in mind the strix halo APU has a 256 bit wide memory bus and the Mac Ultra has a 1024 bit wide memory bus. Here's hoping the Nvidia Digit (GB10 chip) has a 512 bit or 1024 bit wide interface, otherwise the Strix Halo will be the best you can do if you don't get the Mac Ultra.
- rafaelmn 2y ago
  I mean it remains to be seen if it will be compute or bandwidth bound, I am sure mac ultra will also have double or triple compute as well. But in either case its going to do much better than currently available CPUs with easily upgradeable ram. I would not be surprised to see 128gb configurations for around 3k (going of the ASUS g13 announced pricing of arround 2k for 32gb version and them saying it will go up to 128gb). At that point sure it might not compete with max but its at a much more acceptable price point, it will not be a device you get just for the AI, but a mobile workstation that you can also run some local models on for normal money. Will need to wait and see. I know I am not buying anything from ASUS either way.

F7F7F7 2y ago

People would only be 'throwing their hands up' because commercial LLMs have set unreasonable expectations for folks. Anyone who has a/the need for or understands the value of a local LLM would be OK with this kind of output.

bnchrch 2y ago
Everyone has the need for on device LLM, if the response rate was fast!
- ddingus 2y ago
  I have MLCCHAT on my old Note 9 phone. It is actually still a great phone, but has 5GB RAM. Running an on device model is the first and only use case the RAM actually matters. And it has a headphone jack, OK? I just hate Bluetooth earbuds. And yeah, it isna problem, but I digress. When I run a 2.5B model, I get respectable output. Takes a minute or two to process the context, then output begins at somewhere on the order of 4 to 10 tokens per sec. So, I just make a query and give it a few and I have my response. Here is how I see it: That little model, which is Gemma 2.2b sorry, knows a lot of stuff. It has knowledge I don't and it gives it to me in a reasonable, though predictable way. Answers are always of a certain teacher reminding student how it all goes way. I don't care. Better is nice, but if I were stuck somewhere with no network, being able to query that model is amazing! First aid, how to make fires, materials and uses. Fixing stuff, theories of operation, what things mean and more are in that thing ready for me to take advantage of. I consider what I have fast. And it will get one or two orders faster over the next few years too. I did it on a lark (ask the model what that means) and was surprised to see I gained a nice tool.
  - jpc0 2y ago
    > First aid, how to make fires, materials and uses This scares me more than it should... Please do not trust an AI in actual life and death situations... Sure if it is literally your only option, but this implies you have a device on you that could make a phone call to an emergency number where a real human with real training and actually correct knowledge can assist you. Even as an avid hiker the amount of times I've been out off cell service is miniscule and I absolutely refresh my knowledge on first aid regularly and any potential threats before a hike somewhere new.

sliken 2y ago

I do want a 192GB Mac Ultra, I'm hoping the Nvidia Digit achieves similar at $3,000. Sadly no specifications or benchmarks, so tokens/sec is just a guess at this point.

yodsanklai 2y ago

> I’d rather build a rig with used 3090s and get way more bang for my buck I'm curious, what would you use that rig for?