7 ms·

AMD Strix Halo RDMA Cluster Setup Guide

232 points by jakogut 3mo ago

jcastro 3mo ago
This is amazing! I'm working on a three node strix halo agentic OS factory designed to be maintained by local agents: https://github.com/projectbluefin/testing-lab https://github.com/projectbluefin/testing-lab This memory bandwidth combo is amazing for homelabbers. kyuz0's work on these containers has made the investment in this kit so valuable I hope Framework is sending you hardware! https://projectbluefin.io/server/ https://projectbluefin.io/server/ is what I'm hoping to ship, designed to just ship setups like this ootb and things like this would be so much harder without kyuz0! (Note: The 64GB ones are going for $1700-ish empty, the prices on the 128's are outrageous we can just keep making the labs more deterministic over time!)
- mestadler 3mo ago
  Yep, nice write up, seems we are all doing this. Its as close as you can get to Provider level for essentially prosumer hardware. I'll share what I've got with this running under k0s and the npu work.
pixelpoet 3mo ago
I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and model improvements, local AI becomes a reasonable practical thing! The biggest problem is all the tech companies making consumer hardware completely unaffordable, and I don't think this is accidental. Look at Micron's profits and share price lately... I got my Strix machines for ~2k eur each, best computers this 90s kid has ever owned, but those days are gone :(
- rnewme 3mo ago
  What's the advantage of ds4 over llama.cpp, esp if down the line they upstream his forked kernels?
  - pixelpoet 3mo ago
    IIRC llama.cpp doesn't implement DSv4's compressed attention mechanism, and while it does use (credited) parts of llama.cpp, it's focused on this great model for now. Much of this is covered better in the repo's readme.
    - rnewme 3mo ago
      In repo Readme and antirez reddit comments there was also expressed willingness to upstream.
  - francisduvivier 3mo ago
    I think mainly that he can move much faster with specific improvements targeting Deepseek on Systems with unified memory (Mac or Strix). It's a lot easier to optimize if you don't need to worry about all the other architectures. So optimize he did and it's just a lot faster than llama cpp for deepseek v4 pro and flash. Also interesting features are more doable, like SSD streaming, which makes it possible to load MOE weights for a model larger than your VRAM, I don't see that landing in llama cpp anytime soon.
  - mkesper 3mo ago
    Currently, llama.cpp clusters don't support tensor parallelism, have a look at Donato Capitella's detailed report: https://m.youtube.com/watch?v=PkKXm_mKCCM https://m.youtube.com/watch?v=PkKXm_mKCCM He also provides rocm toolboxes for Strix Halo: https://strix-halo-toolboxes.com/#about https://strix-halo-toolboxes.com/#about
- gruez 3mo ago
  >The biggest problem is all the tech companies making consumer hardware completely unaffordable, and I don't think this is accidental. Look at Micron's profits and share price lately... You realize "tech companies" isn't a monolith? Micron charging inflated prices doesn't magically benefit OpenAI. The "high prices keep out competitors" theory doesn't make much sense either. It's like saying Dennys benefits from higher egg prices because it makes cooking eggs at home more expensive.
mestadler 3mo ago
This is exactly the type of technical depth that makes a difference. I've been following all the work you have been doing.
jmyeet 3mo ago
So this is kind of fascinating. The main hardware costs here seem to be: - 2x Framework Desktop AI Mainboards with 128GB of RAM for $3150 each - 2x 100G Ethernet controllers for ~$500 each So the Framework board has a single PCI-e 4.0 x4 slot, which amounts to 8GB/s or 64Gbps theoretical so you're not getting 100G. Also, the 100G cards all seem to be PCI-e x16 slots for obvious reasons so you need a riser or an adapter or something to even get them to work. I don't know how hot a 100GbE copper NIC runs but, from experience, 10GbE NICs have been basically giant heatsinks, basically. So fiber might be advisable and I expect short fiber cables here probably aren't cost-prohibitive given everything else. As an aside, if you are using Ethernet for clustering and you're clustering 2 devices, in an ideal world you'd be using simplex Ethernet but that's not an option here. I wonder if the author considered USB 4.0 for clustering? I ask because I know people who have clustered Mac Studios over TB5 and that bandwidth is up to 120Gbps. The version of USB4 on the Ryzen AI 395 seems to be 40Gbps, which isn't that far off 8GB/s over PCI-e 4.0 x4. But the limiting factor with Strix Halo (and DGX Spark for that matter) is memory bandwidth, both under 300GB/s. The obvious comparison is to the Mac Studio. Unfortunately the largest spec they currently sell is 96GB. It had been as high as 512GB. And 96GB is $6700+ but you're also getting way better performance AFAICT eg [1]. The M3 Ultra has ~900GB/s memory bandwidth. You can alternatively buy a Macbook Pro with M5 Max and 128GB of RAM (now $8000, was $5500-6000 a few days ago) but that tops out at ~600GB/s, which is still double these mini AI boxes. Oh and if you don't want to go the way of these Framework motherboards, you can buy a whole 128GB Strix Halo PC for $3k or less. I think the main point here though is we're only a few years away from running 300B+ (or even 1T+) param models at useful speeds on enthusiast hardware. [1]: https://www.reddit.com/r/LocalLLaMA/comments/1u5mfaq/you_can_run_deepseek_4_flash_on_mac_m3_max_96gb/ https://www.reddit.com/r/LocalLLaMA/comments/1u5mfaq/you_can...
- kcb 3mo ago
  No reason to use fiber on short runs like that. DAC cables are cheap and better in pretty much every way over short distances. You're probably thinking of RJ-45 NICs and SFP modules which are known to run pretty hot.
  - layla5alive 3mo ago
sdlkj- 3mo ago
This is amazing work - RDMA on these smaller unified memory boxes (somewhat) bridges the gap for consumers from the ~24GB 3090/4090/7900 card that are around to 128GB/256GB! Still not cheap, especially now, but... obtainable? I do hope that apple opens up RDMA for their TB4 machines... ds4 using TB5 macs works great - but there are a lot of capable tb4 (M2/1) machines out there and afaik there's no hardware limitation preventing RDMA from working (at lower bandwidth, but with the latency gains!) on the older stuff.
Tepix 3mo ago
What‘s the advantage of using ConnectX-5 Ex VPI NICs instead of much cheaper ConnectX-3 VPI NICs to connect two machines directly, other than PCIe 4.0 instead of PCIe 3.0? Can they offload more tasks when doing RDMA? Solid information is hard to come by.
- justincormack 3mo ago
  The machine only has pcie4x4 so 50Gb bandwidth, pcie3 would halve that to 25Gb Thats the problem with these AMD laptop class cores, they have very little IO. They have been saying they will release in a desktop form factor, but then it probably wont have such good memory bandwidth... The Nvidia boxes have 200Gb ethernet thats much more useful for clustering.
- olavgg 3mo ago
  Yes CX5 can offload more. I believe CX4 has similar offloading capabilities as CX3, except that it supports 100G. Another note: In my experience, RoCE works much better on CX4+ generation. CX3 is best with Infiniband. I think some firmwares on the CX3 generation, has a messed up config for RoCE. But running Infiniband is not a complex task, is way easier than people think, like 10x easier and faster to setup than Ethernet.
  - Tepix 3mo ago
    Thanks. Yes, I'm using Infiniband with CX3. Not seeing real speedups so far. I was wondering if upgrading to CX5 could help.
kamranjon 3mo ago
Benchmarks are here: https://kyuz0.github.io/amd-strix-halo-vllm-toolboxes/ https://kyuz0.github.io/amd-strix-halo-vllm-toolboxes/ Would love to see DeepSeek V4 flash/pro and MiniMax M3 benchmarks but already these are pretty impressive, first strix Halo setup I've seen with some serious performance. EDIT: Apologies - I think I misunderstood these benchmarks - it seems this is actually very slow when compared to a M4 or M5 chip with a good amount of memory. Looking at the creators video here: https://youtu.be/Cfl3TS7ME5s?t=734 https://youtu.be/Cfl3TS7ME5s?t=734 -- it seems the performance of strix halo is much much slower than I get on my M4 MBP - which gets ~400 prefill and ~20 tok/s generation
- Tepix 3mo ago
  The pp speeds are really slow (50), I think there‘s room for improvement still.
  - kamranjon 3mo ago
    Ah yea after watching one of the creators youtube videos I realize these benchmarks are combining prefill and decode which isn't super helpful - it seems this struggles with the exact same bottlenecks as all strix halo setups, memory bandwidth. It seems this is still significantly slower than equivalent memory sizing on Mac hardware.
    - fulafel 3mo ago
      How are the memory bandwidths specs of Macbooks vs this?
      - Lwerewolf 3mo ago
        The apple silicon chips basically beat everything in bandwidth. Highest amount of memory controllers (i.e. channels) for a given capacity. That's the main party trick.
      - fulafel 3mo ago
        I looked it up: 512 GB/s for the two node AMD cluster, Macbook Pro with M5 CPU has 153 GB/s. But you can get faster Macs with M5 Pro or M5 Max.
MayeulC 3mo ago
Hmm, coing PCIe -> NIC -> NIC -> PCIe seems a bit silly, couldn't both devices communicate directly over PCIe?
- wmf 3mo ago
  You need Non-Transparent Bridging (NTB) for that; I don't know if AMD has it.
- Tepix 3mo ago
  Ideally Linux would add support for RDMA via Thunderbolt.
kristianp 3mo ago
The 2 Intel E810 100Gb NICs together cost about $800 and they won't fit in a mini PC that the AI Max often comes as. Docks or custom racks like the photo would be required. Great project though. As a sibling comment says, the author has a video of testing GLM 5.2 q2 on the pair of machines at https://www.youtube.com/watch?v=PkKXm_mKCCM https://www.youtube.com/watch?v=PkKXm_mKCCM . He also has a video about getting Deepseek 4 Flash on 1 or 2 AI Max machines using antirez's ds4 at https://www.youtube.com/watch?v=Cfl3TS7ME5s https://www.youtube.com/watch?v=Cfl3TS7ME5s . Edit: I actually have a website tracking the price of the AI Max 395+ mini pcs (on amazon, anyway). They can be had for about $3300 in the US and about 3400 euro in Germany. ( https://gpuquicklist.com/apus?models=AI%20Max+%20395%20128GB https://gpuquicklist.com/apus?models=AI%20Max+%20395%20128GB and germany: https://gpuquicklist.com/apus_de?models=AI%20Max+%20395%20128GB https://gpuquicklist.com/apus_de?models=AI%20Max+%20395%2012... )

232 points by jakogut 3mo ago

This is amazing! I'm working on a three node strix halo agentic OS factory designed to be maintained by local agents: https://github.com/projectbluefin/testing-lab https://github.com/projectbluefin/testing-lab This memory bandwidth combo is amazing for homelabbers. kyuz0's work on these containers has made the investment in this kit so valuable I hope Framework is sending you hardware! https://projectbluefin.io/server/ https://projectbluefin.io/server/ is what I'm hoping to ship, designed to just ship setups like this ootb and things like this would be so much harder without kyuz0! (Note: The 64GB ones are going for $1700-ish empty, the prices on the 128's are outrageous we can just keep making the labs more deterministic over time!)

mestadler 3mo ago
Yep, nice write up, seems we are all doing this. Its as close as you can get to Provider level for essentially prosumer hardware. I'll share what I've got with this running under k0s and the npu work.

pixelpoet 3mo ago

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and model improvements, local AI becomes a reasonable practical thing! The biggest problem is all the tech companies making consumer hardware completely unaffordable, and I don't think this is accidental. Look at Micron's profits and share price lately... I got my Strix machines for ~2k eur each, best computers this 90s kid has ever owned, but those days are gone :(

rnewme 3mo ago
What's the advantage of ds4 over llama.cpp, esp if down the line they upstream his forked kernels?
- pixelpoet 3mo ago
  IIRC llama.cpp doesn't implement DSv4's compressed attention mechanism, and while it does use (credited) parts of llama.cpp, it's focused on this great model for now. Much of this is covered better in the repo's readme.
  - rnewme 3mo ago
    In repo Readme and antirez reddit comments there was also expressed willingness to upstream.
- francisduvivier 3mo ago
  I think mainly that he can move much faster with specific improvements targeting Deepseek on Systems with unified memory (Mac or Strix). It's a lot easier to optimize if you don't need to worry about all the other architectures. So optimize he did and it's just a lot faster than llama cpp for deepseek v4 pro and flash. Also interesting features are more doable, like SSD streaming, which makes it possible to load MOE weights for a model larger than your VRAM, I don't see that landing in llama cpp anytime soon.
- mkesper 3mo ago
  Currently, llama.cpp clusters don't support tensor parallelism, have a look at Donato Capitella's detailed report: https://m.youtube.com/watch?v=PkKXm_mKCCM https://m.youtube.com/watch?v=PkKXm_mKCCM He also provides rocm toolboxes for Strix Halo: https://strix-halo-toolboxes.com/#about https://strix-halo-toolboxes.com/#about
gruez 3mo ago
>The biggest problem is all the tech companies making consumer hardware completely unaffordable, and I don't think this is accidental. Look at Micron's profits and share price lately... You realize "tech companies" isn't a monolith? Micron charging inflated prices doesn't magically benefit OpenAI. The "high prices keep out competitors" theory doesn't make much sense either. It's like saying Dennys benefits from higher egg prices because it makes cooking eggs at home more expensive.

mestadler 3mo ago

This is exactly the type of technical depth that makes a difference. I've been following all the work you have been doing.

jmyeet 3mo ago

So this is kind of fascinating. The main hardware costs here seem to be: - 2x Framework Desktop AI Mainboards with 128GB of RAM for $3150 each - 2x 100G Ethernet controllers for ~$500 each So the Framework board has a single PCI-e 4.0 x4 slot, which amounts to 8GB/s or 64Gbps theoretical so you're not getting 100G. Also, the 100G cards all seem to be PCI-e x16 slots for obvious reasons so you need a riser or an adapter or something to even get them to work. I don't know how hot a 100GbE copper NIC runs but, from experience, 10GbE NICs have been basically giant heatsinks, basically. So fiber might be advisable and I expect short fiber cables here probably aren't cost-prohibitive given everything else. As an aside, if you are using Ethernet for clustering and you're clustering 2 devices, in an ideal world you'd be using simplex Ethernet but that's not an option here. I wonder if the author considered USB 4.0 for clustering? I ask because I know people who have clustered Mac Studios over TB5 and that bandwidth is up to 120Gbps. The version of USB4 on the Ryzen AI 395 seems to be 40Gbps, which isn't that far off 8GB/s over PCI-e 4.0 x4. But the limiting factor with Strix Halo (and DGX Spark for that matter) is memory bandwidth, both under 300GB/s. The obvious comparison is to the Mac Studio. Unfortunately the largest spec they currently sell is 96GB. It had been as high as 512GB. And 96GB is $6700+ but you're also getting way better performance AFAICT eg [1]. The M3 Ultra has ~900GB/s memory bandwidth. You can alternatively buy a Macbook Pro with M5 Max and 128GB of RAM (now $8000, was $5500-6000 a few days ago) but that tops out at ~600GB/s, which is still double these mini AI boxes. Oh and if you don't want to go the way of these Framework motherboards, you can buy a whole 128GB Strix Halo PC for $3k or less. I think the main point here though is we're only a few years away from running 300B+ (or even 1T+) param models at useful speeds on enthusiast hardware. [1]: https://www.reddit.com/r/LocalLLaMA/comments/1u5mfaq/you_can_run_deepseek_4_flash_on_mac_m3_max_96gb/ https://www.reddit.com/r/LocalLLaMA/comments/1u5mfaq/you_can...

kcb 3mo ago
No reason to use fiber on short runs like that. DAC cables are cheap and better in pretty much every way over short distances. You're probably thinking of RJ-45 NICs and SFP modules which are known to run pretty hot.
- layla5alive 3mo ago

sdlkj- 3mo ago

This is amazing work - RDMA on these smaller unified memory boxes (somewhat) bridges the gap for consumers from the ~24GB 3090/4090/7900 card that are around to 128GB/256GB! Still not cheap, especially now, but... obtainable? I do hope that apple opens up RDMA for their TB4 machines... ds4 using TB5 macs works great - but there are a lot of capable tb4 (M2/1) machines out there and afaik there's no hardware limitation preventing RDMA from working (at lower bandwidth, but with the latency gains!) on the older stuff.

Tepix 3mo ago

What‘s the advantage of using ConnectX-5 Ex VPI NICs instead of much cheaper ConnectX-3 VPI NICs to connect two machines directly, other than PCIe 4.0 instead of PCIe 3.0? Can they offload more tasks when doing RDMA? Solid information is hard to come by.

justincormack 3mo ago
The machine only has pcie4x4 so 50Gb bandwidth, pcie3 would halve that to 25Gb Thats the problem with these AMD laptop class cores, they have very little IO. They have been saying they will release in a desktop form factor, but then it probably wont have such good memory bandwidth... The Nvidia boxes have 200Gb ethernet thats much more useful for clustering.
olavgg 3mo ago
Yes CX5 can offload more. I believe CX4 has similar offloading capabilities as CX3, except that it supports 100G. Another note: In my experience, RoCE works much better on CX4+ generation. CX3 is best with Infiniband. I think some firmwares on the CX3 generation, has a messed up config for RoCE. But running Infiniband is not a complex task, is way easier than people think, like 10x easier and faster to setup than Ethernet.
- Tepix 3mo ago
  Thanks. Yes, I'm using Infiniband with CX3. Not seeing real speedups so far. I was wondering if upgrading to CX5 could help.

kamranjon 3mo ago

Benchmarks are here: https://kyuz0.github.io/amd-strix-halo-vllm-toolboxes/ https://kyuz0.github.io/amd-strix-halo-vllm-toolboxes/ Would love to see DeepSeek V4 flash/pro and MiniMax M3 benchmarks but already these are pretty impressive, first strix Halo setup I've seen with some serious performance. EDIT: Apologies - I think I misunderstood these benchmarks - it seems this is actually very slow when compared to a M4 or M5 chip with a good amount of memory. Looking at the creators video here: https://youtu.be/Cfl3TS7ME5s?t=734 https://youtu.be/Cfl3TS7ME5s?t=734 -- it seems the performance of strix halo is much much slower than I get on my M4 MBP - which gets ~400 prefill and ~20 tok/s generation

Tepix 3mo ago
The pp speeds are really slow (50), I think there‘s room for improvement still.
- kamranjon 3mo ago
  Ah yea after watching one of the creators youtube videos I realize these benchmarks are combining prefill and decode which isn't super helpful - it seems this struggles with the exact same bottlenecks as all strix halo setups, memory bandwidth. It seems this is still significantly slower than equivalent memory sizing on Mac hardware.
  - fulafel 3mo ago
    How are the memory bandwidths specs of Macbooks vs this?
    - Lwerewolf 3mo ago
      The apple silicon chips basically beat everything in bandwidth. Highest amount of memory controllers (i.e. channels) for a given capacity. That's the main party trick.
    - fulafel 3mo ago
      I looked it up: 512 GB/s for the two node AMD cluster, Macbook Pro with M5 CPU has 153 GB/s. But you can get faster Macs with M5 Pro or M5 Max.

MayeulC 3mo ago

Hmm, coing PCIe -> NIC -> NIC -> PCIe seems a bit silly, couldn't both devices communicate directly over PCIe?

wmf 3mo ago
You need Non-Transparent Bridging (NTB) for that; I don't know if AMD has it.
Tepix 3mo ago
Ideally Linux would add support for RDMA via Thunderbolt.

kristianp 3mo ago

The 2 Intel E810 100Gb NICs together cost about $800 and they won't fit in a mini PC that the AI Max often comes as. Docks or custom racks like the photo would be required. Great project though. As a sibling comment says, the author has a video of testing GLM 5.2 q2 on the pair of machines at https://www.youtube.com/watch?v=PkKXm_mKCCM https://www.youtube.com/watch?v=PkKXm_mKCCM . He also has a video about getting Deepseek 4 Flash on 1 or 2 AI Max machines using antirez's ds4 at https://www.youtube.com/watch?v=Cfl3TS7ME5s https://www.youtube.com/watch?v=Cfl3TS7ME5s . Edit: I actually have a website tracking the price of the AI Max 395+ mini pcs (on amazon, anyway). They can be had for about $3300 in the US and about 3400 euro in Germany. ( https://gpuquicklist.com/apus?models=AI%20Max+%20395%20128GB https://gpuquicklist.com/apus?models=AI%20Max+%20395%20128GB and germany: https://gpuquicklist.com/apus_de?models=AI%20Max+%20395%20128GB https://gpuquicklist.com/apus_de?models=AI%20Max+%20395%2012... )