5 ms·

Been running this on a few Asus GX10 machines with k3s on top, it’s been great. I’m running the new deepseek. Thank you for your work!

by redrove 2mo ago

pixelesque 2mo ago
What quant are you using, and what tps are you getting with K3?
- redrove 2mo ago
  The FP8 version from DeepSeek themselves [0], around 1800 tps prefill and 45 tokens per second decode. I’ve been running a custom VLLM image with b12x as well as nvfp4_ds_mla. I would say it’s quite fantastic in day to day, I use it mostly in Hermes and sometimes for coding. I have qwen 3.6 27b on an rtx 6000 pro as well so I use that as a workhorse in pi with DS as a reviewer/planner. [0] https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 Edit: I think you may have misread my post. k3s is NOT kimi k3, and I did mention I was running deepseek.
  - pixelesque 2mo ago
    Thanks - yeah, sorry, I mis-read that as you using both DS and K3...
    - reinitctxoffset 2mo ago
      I'm doing K3 with SoL kernels, zero hassle, myelin to sweep up the crap, on shot deploy to vast or runpods. Free to the community. I have to do like, low paying web dev to fund it, so I can't promise timelines, but it's coming and it will be free to anyone and fast as fuck. I estimate about 20 bucks an hour at current spot rates in the hundreds if not thousands of tokens per second.
      - reinitctxoffset 2mo ago
        @pg @dang you're asking me how a watch works. let's just try to keep an eye on the time. federal felony prosecution.
    - redrove 2mo ago
      No worries! Would love to run K3 but lack the hardware for a 2.8T model, like most people.
reinitctxoffset 2mo ago
i also used nixos as the base, but i went a little overboard with it, this is 7.1.2 supporting all devices and a custom driver that fixes the UMA page leak issues, and the wallpapers are all kernels at or above parity with what you can get out of the box. i'm not quite ready to OSS the whole thing, it's got a few rough edges, but if anyone wants to alpha test, caveat emptor and it's yours. https://www.youtube.com/shorts/mCUZ9XHIogw https://www.youtube.com/shorts/mCUZ9XHIogw
- ianlevesque 2mo ago
  A fix for the leaks would be great. I’m rebooting them way too often when iterating.
  - reinitctxoffset 2mo ago
    https://cdn.s4.gl/0001-Avoid-allocations-in-system-memory-page-free-path.patch https://cdn.s4.gl/0001-Avoid-allocations-in-system-memory-pa...
- deleted 2mo ago
  [deleted]
- reinitctxoffset 2mo ago
  That short got like, way more views than I thought. This is a little more of the complex plane math. https://youtu.be/WXcElxvSevM?si=HHbnzSNddIcPGCll https://youtu.be/WXcElxvSevM?si=HHbnzSNddIcPGCll
- redrove 2mo ago
  I’d be happy to test this out if you want to publish an alpha branch or something.
neobrain 2mo ago
Just curious, has suspend (to RAM) been working for you? For me the nvidia driver just keeps waking up the system instantly - but my setup is deviating from the upstream flake in a few ways, so I'm just wondering if it's worth setting up the system from scratch if it's working for other people. Other than that, can fully second that the flake is working great. Only gotcha is that CUDA-enabled packages (including Firefox) require using the flox binary cache unless you want to compile them from source, but then the package versions can lag behind a bit (and debugging nix cache issues is surprisingly difficult).

Been running this on a few Asus GX10 machines with k3s on top, it’s been great. I’m running the new deepseek. Thank you for your work!

by redrove 2mo ago

Been running this on a few Asus GX10 machines with k3s on top, it’s been great. I’m running the new deepseek. Thank you for your work!

pixelesque 2mo ago

What quant are you using, and what tps are you getting with K3?

redrove 2mo ago
The FP8 version from DeepSeek themselves [0], around 1800 tps prefill and 45 tokens per second decode. I’ve been running a custom VLLM image with b12x as well as nvfp4_ds_mla. I would say it’s quite fantastic in day to day, I use it mostly in Hermes and sometimes for coding. I have qwen 3.6 27b on an rtx 6000 pro as well so I use that as a workhorse in pi with DS as a reviewer/planner. [0] https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 Edit: I think you may have misread my post. k3s is NOT kimi k3, and I did mention I was running deepseek.
- pixelesque 2mo ago
  Thanks - yeah, sorry, I mis-read that as you using both DS and K3...
  - reinitctxoffset 2mo ago
    I'm doing K3 with SoL kernels, zero hassle, myelin to sweep up the crap, on shot deploy to vast or runpods. Free to the community. I have to do like, low paying web dev to fund it, so I can't promise timelines, but it's coming and it will be free to anyone and fast as fuck. I estimate about 20 bucks an hour at current spot rates in the hundreds if not thousands of tokens per second.
    - reinitctxoffset 2mo ago
      @pg @dang you're asking me how a watch works. let's just try to keep an eye on the time. federal felony prosecution.
  - redrove 2mo ago
    No worries! Would love to run K3 but lack the hardware for a 2.8T model, like most people.

reinitctxoffset 2mo ago

i also used nixos as the base, but i went a little overboard with it, this is 7.1.2 supporting all devices and a custom driver that fixes the UMA page leak issues, and the wallpapers are all kernels at or above parity with what you can get out of the box. i'm not quite ready to OSS the whole thing, it's got a few rough edges, but if anyone wants to alpha test, caveat emptor and it's yours. https://www.youtube.com/shorts/mCUZ9XHIogw https://www.youtube.com/shorts/mCUZ9XHIogw

ianlevesque 2mo ago
A fix for the leaks would be great. I’m rebooting them way too often when iterating.
- reinitctxoffset 2mo ago
  https://cdn.s4.gl/0001-Avoid-allocations-in-system-memory-page-free-path.patch https://cdn.s4.gl/0001-Avoid-allocations-in-system-memory-pa...
deleted 2mo ago
[deleted]
reinitctxoffset 2mo ago
That short got like, way more views than I thought. This is a little more of the complex plane math. https://youtu.be/WXcElxvSevM?si=HHbnzSNddIcPGCll https://youtu.be/WXcElxvSevM?si=HHbnzSNddIcPGCll
redrove 2mo ago
I’d be happy to test this out if you want to publish an alpha branch or something.

neobrain 2mo ago

Just curious, has suspend (to RAM) been working for you? For me the nvidia driver just keeps waking up the system instantly - but my setup is deviating from the upstream flake in a few ways, so I'm just wondering if it's worth setting up the system from scratch if it's working for other people. Other than that, can fully second that the flake is working great. Only gotcha is that CUDA-enabled packages (including Firefox) require using the flox binary cache unless you want to compile them from source, but then the package versions can lag behind a bit (and debugging nix cache issues is surprisingly difficult).