8 ms·

For anyone interested in playing with distributed systems, I'd really recommend getting a single machine with latest 16-core CPU from AMD and just running 8 vir

by densh 1y ago

For anyone interested in playing with distributed systems, I'd really recommend getting a single machine with latest 16-core CPU from AMD and just running 8 virtual machines on it. 8 virtual machines, with 4 hyper threads pinned per machine, and 1/8 of total RAM per machine. Create a network between them virtually within your virtualization software of choice (such as Proxmox). And suddenly you can start playing with distributed software, even though it's running on a single machine. For resiliency tests you can unplug one machine at a time with a single click. It will annihilate a Pi cluster in Perf/W as well, and you don't have to assemble a complex web of components to make it work. Just a single CPU, motherboard, m.2 SSD, and two sticks of RAM. Naturally, using a high core count machine without virtualization will get you best overall Perf/W in most benchmarks. What's also important but often not highlighted in benchmarks in Idle W if you'd like to keep your cluster running, and only use it occasionally.

bee_rider 1y ago
Tangentially related: I really expected running old MPI programs on stuff like the AMD multi-chip workstation packages to become a bigger thing.
- le-mark 1y ago
  I actually worked with some MPI code way back. What MPI programs are you referring to?
  - MathMonkeyMan 1y ago
    I don't know, but when I was playing with finite difference code as an undergrad in Physics, all of the docs I could find (it was a while ago, though) assumed that I was going to use MPI to run a distributed workload across the university's supercomputer. My needs were less, so I just ran my Boost.Thread code on the four cores of one node. What if you had a single server with a zillion cores in it? Maybe you could take some 15 year old MPI code and run it locally -- it'd be like a mini supercomputer with an impossibly fast network.
  - bee_rider 1y ago
    I’m not thinking of one code in particular. Just, observing that in the multi-chiplet, even inside a CPU package we’re already talking over a sort of little internal network anyway. Might as well use code that was designed to run on a network, right?
cyberpunk 1y ago
Honestly why do you need so much cpu power? You can play with distributed systems just by installing Erlang and running a couple of nodes on whatever potato-level linux box you have laying around, including a single raspberry pi.
globular-toast 1y ago
I've been saying this for years. When the last Raspberry Pi shortage happened people were scrambling to get them for building these toy clusters and it's such a shame. The Pi was made for paedogogy but I feel like most of them are wasted. I run a K8s "cluster" on a single xcp-ng instance, but you don't even really have to go that far. Docker Machine could easily spin up docker hosts with a single command, but I see that project is dead now. Docker Swarm I think still lets you scale up/down services, no hypervisor required.
- motorest 1y ago
  > I've been saying this for years. When the last Raspberry Pi shortage happened people were scrambling to get them for building these toy clusters and it's such a shame. The Pi was made for paedogogy but I feel like most of them are wasted. You're describing people using RPis to learn distributed systems, and you conclude that these RPis are wasted because RPis were made for paedogogy? > I run a K8s "cluster" on a single xcp-ng instance, but you don't even really have to go that far. That's perfectly fine. You do what works for you, just like everyone else. How would you handle someone else accusing your computer resourcss of being wasted?
  - 0xDE7EC71V3 1y ago
    I’ve learned so much setting up a pi cluster. There is something so cool about seeing code run across different pieces of hardware.
  - globular-toast 1y ago
    The point was you don't need to wait for 8 Pis to become available when most people can get going straight away with what they already have. If you want to learn physical networking or really need to "see" things happening on physically separate machines just get a free old PC from gumtree or something.
    - motorest 1y ago
      > The point was you don't need to wait for 8 Pis to become available when most people can get going straight away with what they already have. You also don't need RPis to learn anything about programming, networking, electronics, etc. But people do it anyways. I really don't see what point anyone thinks they are making regarding pedogogy. RPis are synonymous with tinkering, regardless of how you cut it. Distributed systems too.
qmr 1y ago
No need for so much CPU power, any old quad core would work.
- subscribed 1y ago
  Old quad core won't have all the virtualisation extensions.
  - afzalive 1y ago
    An old Xeon then.
  - qmr 1y ago
    Virtualization existed long before virtualization instructions. Not strictly necessary.
  - AnthonyMouse 1y ago
    > Old quad core won't have all the virtualisation extensions. Intel's first quad core was Kentsfield in 2006. It supports VT-x. AMD's first quad core likewise supports AMD-V. The newer virtualization extensions mostly just improve performance a little or do things you probably won't use anyway like SR-IOV.
  - justsomehnguy 1y ago
    Ivy Bridge is 13 years old today. You need to do the the things to buy something older than that in 2025.
- anaganisk 1y ago
  Aren’t newer CPUs especially AMDs more energy efficient?
  - AnthonyMouse 1y ago
    Newer CPUs have significantly better performance per watt under load, essentially by being a lot faster while using a similar amount of power. Idle CPU power consumption hasn't changed much in 10+ years simply because by that point it was already a single digit number of watts. The thing that matters more than the CPU for idle power consumption is how efficient the system's power supply is under light loads. The variance between them is large and newer power supplies aren't all inherently better at it.
malux85 1y ago
Yeah this is how I practiced Postgres hot standby and read replicas, It was also how I learned to setup a Hadoop cluster, and a Cassandra cluster (this was 10 years ago when these technologies were hot) Having knowledge of these systems and being able to talk about how I set them up and simulated recovery directly got me jobs that 2x and then 3x my salary, I would highly recommend all medium skilled developers setup systems like this and get practicing if you want to get up into the next level
user432678 1y ago
Yes, but this is boring. Saying this as an owner of home server with ProxMox.

by densh 1y ago

bee_rider 1y ago

Tangentially related: I really expected running old MPI programs on stuff like the AMD multi-chip workstation packages to become a bigger thing.

le-mark 1y ago
I actually worked with some MPI code way back. What MPI programs are you referring to?
- MathMonkeyMan 1y ago
  I don't know, but when I was playing with finite difference code as an undergrad in Physics, all of the docs I could find (it was a while ago, though) assumed that I was going to use MPI to run a distributed workload across the university's supercomputer. My needs were less, so I just ran my Boost.Thread code on the four cores of one node. What if you had a single server with a zillion cores in it? Maybe you could take some 15 year old MPI code and run it locally -- it'd be like a mini supercomputer with an impossibly fast network.
- bee_rider 1y ago
  I’m not thinking of one code in particular. Just, observing that in the multi-chiplet, even inside a CPU package we’re already talking over a sort of little internal network anyway. Might as well use code that was designed to run on a network, right?

cyberpunk 1y ago

Honestly why do you need so much cpu power? You can play with distributed systems just by installing Erlang and running a couple of nodes on whatever potato-level linux box you have laying around, including a single raspberry pi.

globular-toast 1y ago

I've been saying this for years. When the last Raspberry Pi shortage happened people were scrambling to get them for building these toy clusters and it's such a shame. The Pi was made for paedogogy but I feel like most of them are wasted. I run a K8s "cluster" on a single xcp-ng instance, but you don't even really have to go that far. Docker Machine could easily spin up docker hosts with a single command, but I see that project is dead now. Docker Swarm I think still lets you scale up/down services, no hypervisor required.

motorest 1y ago
> I've been saying this for years. When the last Raspberry Pi shortage happened people were scrambling to get them for building these toy clusters and it's such a shame. The Pi was made for paedogogy but I feel like most of them are wasted. You're describing people using RPis to learn distributed systems, and you conclude that these RPis are wasted because RPis were made for paedogogy? > I run a K8s "cluster" on a single xcp-ng instance, but you don't even really have to go that far. That's perfectly fine. You do what works for you, just like everyone else. How would you handle someone else accusing your computer resourcss of being wasted?
- 0xDE7EC71V3 1y ago
  I’ve learned so much setting up a pi cluster. There is something so cool about seeing code run across different pieces of hardware.
- globular-toast 1y ago
  The point was you don't need to wait for 8 Pis to become available when most people can get going straight away with what they already have. If you want to learn physical networking or really need to "see" things happening on physically separate machines just get a free old PC from gumtree or something.
  - motorest 1y ago
    > The point was you don't need to wait for 8 Pis to become available when most people can get going straight away with what they already have. You also don't need RPis to learn anything about programming, networking, electronics, etc. But people do it anyways. I really don't see what point anyone thinks they are making regarding pedogogy. RPis are synonymous with tinkering, regardless of how you cut it. Distributed systems too.

qmr 1y ago

No need for so much CPU power, any old quad core would work.

subscribed 1y ago
Old quad core won't have all the virtualisation extensions.
- afzalive 1y ago
  An old Xeon then.
- qmr 1y ago
  Virtualization existed long before virtualization instructions. Not strictly necessary.
- AnthonyMouse 1y ago
  > Old quad core won't have all the virtualisation extensions. Intel's first quad core was Kentsfield in 2006. It supports VT-x. AMD's first quad core likewise supports AMD-V. The newer virtualization extensions mostly just improve performance a little or do things you probably won't use anyway like SR-IOV.
- justsomehnguy 1y ago
  Ivy Bridge is 13 years old today. You need to do the the things to buy something older than that in 2025.
anaganisk 1y ago
Aren’t newer CPUs especially AMDs more energy efficient?
- AnthonyMouse 1y ago
  Newer CPUs have significantly better performance per watt under load, essentially by being a lot faster while using a similar amount of power. Idle CPU power consumption hasn't changed much in 10+ years simply because by that point it was already a single digit number of watts. The thing that matters more than the CPU for idle power consumption is how efficient the system's power supply is under light loads. The variance between them is large and newer power supplies aren't all inherently better at it.

malux85 1y ago

Yeah this is how I practiced Postgres hot standby and read replicas, It was also how I learned to setup a Hadoop cluster, and a Cassandra cluster (this was 10 years ago when these technologies were hot) Having knowledge of these systems and being able to talk about how I set them up and simulated recovery directly got me jobs that 2x and then 3x my salary, I would highly recommend all medium skilled developers setup systems like this and get practicing if you want to get up into the next level

user432678 1y ago

Yes, but this is boring. Saying this as an owner of home server with ProxMox.