6 ms·

how do you low cost run a 1T param model?

by martin_ 1y ago

maven29 1y ago
32B active parameters with a single shared expert.
- JustFinishedBSG 1y ago
  This doesn’t change the VRAM usage, only the compute requirements.
  - maven29 1y ago
    You can probably run this on CPU if you have a 4090D for prompt processing, since 1TB of DDR4 only comes out to around $600. For GPU inference at scale, I think token-level batching is used.
    - t1amat 1y ago
      With 32B active parameters it would be ridiculously slow at generation.
      - selfhoster11 1y ago
        DDR3 workstation here - R1 generates at 1 token per second. In practice, this means that for complex queries, the speed of replying is closer to an email response than a chat message, but this is acceptable to me for confidential queries or queries where I need the model to be steerable. I can always hit the R1 API from a provider instead, if I want to. Given that R1 uses 37B active parameters (compared to 32B for K2), K2 should be slightly faster than that - around 1.15 tokens/second.
        CamperBob2 1y ago
        That's pretty good. Are you running the real 600B+ parameter R1, or a distill, though?
        deleted 1y ago
        [deleted]
        selfhoster11 1y ago
        The full thing, 671B. It loses some intelligence at 1.5 bit quantisation, but it's acceptable. I could actually go for around 3 bits if I max out my RAM, but I haven't done that yet.
        apitman 1y ago
        I've seen people say the models get more erratic at higher (lower?) quantization levels. What's your experience been?
        selfhoster11 1y ago
        If you mean clearly, noticeably erratic or incoherent behaviour, then that hasn't been my experience for >=4-bit inference of 32B models, or in my R1 setup. I think the others might have been referring to this happening with smaller models (sub-24B), which suffer much more after being quantised below 4 or 5 bits. My R1 most likely isn't as smart as the output coming from an int8 or FP16 API, but that's just a given. It still holds up pretty well for what I did try.
    - zackangelo 1y ago
      Typically a combination of expert level parallelism and tensor level parallelism is used. For the big MLP tensors they would be split across GPUs in a cluster. Then for the MoE parts you would spread the experts across the GPUs and route to them based on which experts are active (there would likely be more than one if the batch size is > 1).
  - selfhoster11 1y ago
    It does not have to be VRAM, it could be system RAM, or weights streamed from SSD storage. Reportedly, the latter method achieves around 1 token per second on computers with 64 GB of system RAM. R1 (and K2) is MoE, whereas Llama 3 is a dense model family. MoE actually makes these models practical to run on cheaper hardware. DeepSeek R1 is more comfortable for me than Llama 3 70B for exactly that reason - if it spills out of the GPU, you take a large performance hit. If you need to spill into CPU inference, you really want to be multiplying a different set of 32B weights for every token compared to the same 70B (or more) instead, simply because the computation takes so long.
    - refulgentis 1y ago
      The amount of people who will be using it at 1 token/sec because there's no better option, and have 64 GB of RAM, is vanishingly small. IMHO it sets the local LLM community back when we lean on extreme quantization & streaming weights from disk to say something is possible*, because when people try it out, it turns out it's an awful experience. * the implication being, anything is possible in that scenario
      - homarp 1y ago
        agentic loop can run all night long. It's just a different way to work: prepare your prompt queue, set it up, check result in the morning, adjust. 'local vibe' in 10h instead of 10mn is still better than 10 days of manual side coding.
        hereme888 1y ago
        Right on! Especially if its coding abilities are better than Claude 4 Opus. I spent thousands on my PC in anticipation of this rather than to play fancy video games. Now, where's that spare SSD...
      - selfhoster11 1y ago
        Good. Vanishingly small is still more than zero. Over time, running such models will become easier too, as people slowly upgrade to better hardware. It's not like there aren't options for the compute-constrained either. There are lots of Chinese models in the 3-32B range, and Gemma 3 is particularly good too. I will also point out that having three API-based providers deploying an impractically-large open-weights model beats the pants of having just one. Back in the day, this was called second-sourcing IIRC. With proprietary models, you're at the mercy of one corporation and their Kafkaesque ToS enforcement.

how do you low cost run a 1T param model?

maven29 1y ago

32B active parameters with a single shared expert.

JustFinishedBSG 1y ago
This doesn’t change the VRAM usage, only the compute requirements.
- maven29 1y ago
  You can probably run this on CPU if you have a 4090D for prompt processing, since 1TB of DDR4 only comes out to around $600. For GPU inference at scale, I think token-level batching is used.
  - t1amat 1y ago
    With 32B active parameters it would be ridiculously slow at generation.
    - selfhoster11 1y ago
      DDR3 workstation here - R1 generates at 1 token per second. In practice, this means that for complex queries, the speed of replying is closer to an email response than a chat message, but this is acceptable to me for confidential queries or queries where I need the model to be steerable. I can always hit the R1 API from a provider instead, if I want to. Given that R1 uses 37B active parameters (compared to 32B for K2), K2 should be slightly faster than that - around 1.15 tokens/second.
      - CamperBob2 1y ago
        That's pretty good. Are you running the real 600B+ parameter R1, or a distill, though?
        deleted 1y ago
        [deleted]
        selfhoster11 1y ago
        The full thing, 671B. It loses some intelligence at 1.5 bit quantisation, but it's acceptable. I could actually go for around 3 bits if I max out my RAM, but I haven't done that yet.
        apitman 1y ago
        I've seen people say the models get more erratic at higher (lower?) quantization levels. What's your experience been?
        selfhoster11 1y ago
        If you mean clearly, noticeably erratic or incoherent behaviour, then that hasn't been my experience for >=4-bit inference of 32B models, or in my R1 setup. I think the others might have been referring to this happening with smaller models (sub-24B), which suffer much more after being quantised below 4 or 5 bits. My R1 most likely isn't as smart as the output coming from an int8 or FP16 API, but that's just a given. It still holds up pretty well for what I did try.
  - zackangelo 1y ago
    Typically a combination of expert level parallelism and tensor level parallelism is used. For the big MLP tensors they would be split across GPUs in a cluster. Then for the MoE parts you would spread the experts across the GPUs and route to them based on which experts are active (there would likely be more than one if the batch size is > 1).
- selfhoster11 1y ago
  It does not have to be VRAM, it could be system RAM, or weights streamed from SSD storage. Reportedly, the latter method achieves around 1 token per second on computers with 64 GB of system RAM. R1 (and K2) is MoE, whereas Llama 3 is a dense model family. MoE actually makes these models practical to run on cheaper hardware. DeepSeek R1 is more comfortable for me than Llama 3 70B for exactly that reason - if it spills out of the GPU, you take a large performance hit. If you need to spill into CPU inference, you really want to be multiplying a different set of 32B weights for every token compared to the same 70B (or more) instead, simply because the computation takes so long.
  - refulgentis 1y ago
    The amount of people who will be using it at 1 token/sec because there's no better option, and have 64 GB of RAM, is vanishingly small. IMHO it sets the local LLM community back when we lean on extreme quantization & streaming weights from disk to say something is possible*, because when people try it out, it turns out it's an awful experience. * the implication being, anything is possible in that scenario
    - homarp 1y ago
      agentic loop can run all night long. It's just a different way to work: prepare your prompt queue, set it up, check result in the morning, adjust. 'local vibe' in 10h instead of 10mn is still better than 10 days of manual side coding.
      - hereme888 1y ago
        Right on! Especially if its coding abilities are better than Claude 4 Opus. I spent thousands on my PC in anticipation of this rather than to play fancy video games. Now, where's that spare SSD...
    - selfhoster11 1y ago
      Good. Vanishingly small is still more than zero. Over time, running such models will become easier too, as people slowly upgrade to better hardware. It's not like there aren't options for the compute-constrained either. There are lots of Chinese models in the 3-32B range, and Gemma 3 is particularly good too. I will also point out that having three API-based providers deploying an impractically-large open-weights model beats the pants of having just one. Back in the day, this was called second-sourcing IIRC. With proprietary models, you're at the mercy of one corporation and their Kafkaesque ToS enforcement.