5 ms·
Turing Pi: Kubernetes Cluster on Your Desk
- moondev 6y agoLooking at the specs, it seems almost dishonest to promote this for kubernetes. > The nodes interconnected with the onboard 1 Gbps switch. However, each node is limited with 100 Mbps USB speed. Not only that but Compute Module 3+ are limited to 1GB RAM, is it really expected someone could run a realistic workload? How stable is the control plane node with such limited resources? It seems like picking up 3 raspi4s (4GB RAM each) and powering via PoE would be a must better result.
- wedn3sday 6y agoI agree, using the USB bus for inter node communication seems like a poor design choice. Anyone got any insight on why they wouldnt use the much faster ethernet connection?
- cconstantine 6y agoMy understanding is that the ethernet module on the raspberry pi is a usb device, not a pci device.
- oakwhiz 6y agoThe SoC used in the Pi 3 does not have an Ethernet MAC. It's always provided via USB. The CM3 does not have a USB Ethernet device onboard, it's provided by the carrier board.
- afterwalk 6y agoI randomly came across a similar offering and thought about buying for fun: https://www.mininodes.com/product/5-node-raspberry-pi-3-com-carrier-board/ https://www.mininodes.com/product/5-node-raspberry-pi-3-com-... Any idea how the specs compare?
- mbreese 6y agoThis is still based on the compute modules, so you're going to have the same tradeoffs. These are interesting ideas, but until the compute modules start having more on board RAM, you'd be much better off working with a few RPi 4's. They will be faster (1Gbps ethernet, more RAM), and cheaper since you only need a gigabit switch to connect them together, not a custom carrier board. If the compute modules start to get more powerful, then having to deal with only one power supply and ethernet uplink would be nice. It's a very appealing idea. But, you're going to almost always have a better experience with a bunch of standard RPis.
- jaxn 6y agoThis is what I did. Throw PoE hats on the pis and with a PoE switch it's a real clean setup. 3D printed trays for rockmount fiber cassette blanks and set it in a network rack.
- detaro 6y agoIf you insist on using Pis (IMHO you're quickly at the point where a (potentially used) NUC or small office PC is the better choice), why would you go for extra expense and effort for PoE? Just connect a 5V PSU to the power pins? I guess it gives you individual power switching without any DIY, but other than that?
- moondev 6y agoSimply for not having to deal with all the usb-c adaptors and cables. It's also pretty cool being able to turn the pis on or off by cycling the switch port from your network control plane (Unifi in my case) You could even do it dynamically via the api for a "bare-metal autoscaling" type workflow.
- detaro 6y ago> Simply for not having to deal with all the usb-c adaptors and cables. You can get one large 5V PSU and go directly to the pins on the Pi. No need to bother with individual USB adaptors...
- deleted 6y ago[deleted]
- Floegipoky 6y agoI built a cluster of 6 pi 3b+ duct taped to a USB hub for power and a switch and router a few years ago for the explicit purpose of experimenting with clustering technologies, including kubernetes. I wasn't running it for long periods of time but it was surprisingly stable. But to your point, 1GB RAM doesn't get you very far with many of the popular distributed systems these days.
- kylek 6y agoI'd hope people wouldn't be using a setup like this for any other reason than learning k8s or tinkering/fun. edit-before-actually-posting (sorry I'm a bad person for typing a reply before clicking the link): Wait, they're selling these things? Ok, then I'm stumped. What would you do with 8 rpi's that you couldn't do with one?
- gclawes 6y agoI'm holding out to see if a CM 4 with 4GB ram comes out before I commit to something like this.
- closeparen 6y agoThere's something amusingly cyclical about a blade server architecture for Kubernetes. The tech comes out of a whole movement towards combining commodity machines using clever software instead of buying specialist hardware, but then adds the specialist hardware back in. Some deeper integration between Kubernetes and the hardware (acceleration/offload ASICs maybe), branding of k8s + this hardware as a unified product, and this would literally just be a mainframe. Which is not a terrible idea! Maybe Kubernetes is the mainframe operating system of the future.
- inetknght 6y agoThink of it as a way of using mixed hardware: mainframe hardware in conjunction with commodity hardware. It's extremely useful in that context.
- hinkley 6y agoSome mumblings have been heard accusing us all of trying to reimplement mainframes, badly. For a long time I have been watching the ebb and flow between peer to peer and client server and it’s gotten quite a bit fuzzy lately. I suppose if you treat cloud providers as a large amorphous server, it sort of still fits the mold.
- tlrobinson 6y agoNote the $189 price tag doesn't include the Raspberry Pi compute modules, which are about $30-40 each. It's a neat form factor but you could just buy some regular Raspberry Pis and an Ethernet switch.
- apple4ever 6y agoI thought it was $189 for the whole kit and caboodle, but as you said its not. $189 is a little expensive for what you get.
- sgt 6y agoThat's without eMMC though. Having a bunch if normal Pi's running SD cards would end up in tears at some point.
- a012 6y agoIsn't eMMC on each RPi board? No?
- liamdiprose 6y agoThe compute modules come with onboard eMMC
- aivarsk 6y agoIf you're looking for a cheaper alternative then https://clusterhat.com/ https://clusterhat.com/ is worth taking a look. I have one sitting on my desk (4 Pi Zero nodes and Raspberry Pi 2 controller).
- cwiggs 6y agoVery cool, Thanks for the link. Are you also running k8s on it?
- rubyn00bie 6y agoWhile Raspberry Pi's are awesome, and the power consumption is nothing to scoff at (when considering a cluster), you can accomplish this same thing for a lot less, and have quite a lot more compute power, by purchasing a used server or even something like an AMD 3600X... A single 3600X will grossly outperform this cluster (and cost less) with less headaches (you don't have N physical machines) by using KVM to deploy a few virtual machines and using Kubernetes to orchestrate and allocate within those VMs. You'll also have a lot less latency between nodes running in VMs on the same physical host. Another thing that unfortunately sucks about Raspberry Pis (less with Pi 4, but still mostly applies) is really shitty I/O performance... I spent a large amount of time over the past summer and fall trying out various ideas to have a "cluster" at home that was both practical and useful. While, the PIs were nice, they never really amounted to much more than a demo. Latency and I/O become real problems for a lot of useful interconnected services and applications. Honestly, if Ryzen 3000 hadn't come out, for cheaper cluster builds (~300-400) I still think Pis would be a solid choice but... Ryzen 3000 is just so fucking fast with a lot of cores, it's truly hard to beat. Addendum: to touch on used servers, yes your power bill will go way up, no joke, but for some applications like large storage arrays-- it's hands down the cheapest/easiest route. Search by case, not by processor, it sounds weird but the case is likely the most valuable part of the old server (like ones with 20+ SAS2 slots for $500) or PCI-E slots that GPUs can fit into.
- bproven 6y agoyou mean using a ryzen with VMs (cluster on one ryzen) I assume? bc cost of ryzan as a node would be pretty pricey compared to Pi 4s.
- rubyn00bie 6y agoYes, a single ryzen node running multiple VMs.
- nsky-world 6y agoSo, you don't see a difference between clustering on physically separated nodes and VM?
- kube-system 6y agoThis is neat, but I'm really more interested to hear about potential use cases. I'm guessing this is mostly useful for ARM workloads? Maybe some situations with low power requirements? Personally, for my multi-node test clusters, I just run VMs on cheap x86 hardware.
- hexman 6y agosome ideas https://blog.alexellis.io/ https://blog.alexellis.io/
- nsky-world 6y agoWhat's the practical point of running containers on VM?
- jonathanoliver 6y agoThis sounds like every talk given by Bryan Cantrill regarding public clouds running containers.
- kube-system 6y agoTo expand on my personal use-case, I don't want or need a whole stack of physical servers sitting around just for a test k8s environment. There are some things that you can only really test properly in a real multi-node environment rather than single-node solutions like minikube: failovers, shared storage, some networking particularities, etc. But there are reasons to run containers on VMs in production too. Hypervisors and container orchestration tools solve very different problems. Depending on what problems you are trying to solve, it might be useful to leverage both.
- nsky-world 6y agoCan you please elaborate a little bit more on the reason to run containers on VMs in production?
- kube-system 6y ago
- LargoLasskhyfv 6y agoThere is also the https://www.pine64.org/clusterboard/ https://www.pine64.org/clusterboard/ for $99 which takes these modules https://www.pine64.org/sopine/ https://www.pine64.org/sopine/ at $29 a piece, which are quad-core ARM Cortex A53 with 2GB LPDDR3. This is their wiki: https://wiki.pine64.org/index.php/PINE_A64-LTS/SOPine https://wiki.pine64.org/index.php/PINE_A64-LTS/SOPine
- hinkley 6y agoDo you happen to know how long these have been around? I build a little nanopi cluster using a combination of standoffs, cases, and a little light tinkering (aka filing), about 18 months ago and I couldn't seem to find anything like this.
- justinclift 6y agoThat looks really interesting :), but it seems like they have an pretty-much-blocker bug with the individual modules not turning on again when rebooted: https://forum.pine64.org/showthread.php?tid=5849 https://forum.pine64.org/showthread.php?tid=5849 :(
- Already__Taken 6y ago75W 5v bricks can't be that easy to come by were as 12 or even 19v we're usually tripping over.
- LargoLasskhyfv 6y agoI'd use either this https://www.mini-box.com/picoPSU-80 https://www.mini-box.com/picoPSU-80 and repurpose a spare 12V brick for it, or get something for powering LED strips. There are gazillions of them rated 5V 15A at about 30 bucks.
- hinkley 6y agoSomeone had one of these 'backplane' style boards a few years ago, but they only used it for flashing Pis for distribution. I don't think they ever made it commercially available. This looks fairly similar. And can we all just pause for a moment and look at that heat sink on the ethernet controller? Holy cats, what's goin' on there?
- Hippocrates 6y agoFinding and running docker containers on ARM is unfortunately a pain.
- tuananh 6y agofor this use case, i would rather use something like SimpleNUC[0] . It's basically just NUC machines with rack mount. It's a lot more powerful than Rpi, quite power-efficient (when compare to actual server rack) and dead silent. More importantly, this setup can handle some actual workload [0]: https://simplynuc.com/server-shelf-solution/ https://simplynuc.com/server-shelf-solution/
- barnyfried 6y agoits called a mac
- nsky-world 6y agoHey guys, I am a co-founder of Turing Pi. I see a lot of comments around Raspberry Pi, performance and VM. I just want to shed a light on some things here. Turing Pi is not about performance, it's about cluster architecture. If you look at Turing Pi as a homelab project, then yes, you can get more performance with some used cheap servers. Of course, if you get approval from your homies to occupy a closet. You can even run apps in containers with Kubernetes orchestration using VM and so on. The main idea behind Turing Pi is to deliver compute to the edge. If we look at cases where some compute will run low latency, highly available and internet independent apps to automate processes, and often in a hard to reach places, then the classic servers, not a solution. Turing Pi is an early version of edge computers with cloud-native architecture. Why it's important? Because if you are a business with some services running in the cloud and you want your edge computing organically to coexist with your cloud stack, then edge clusters could be a great choice. The speed to innovate and deploy your code into production to both cloud and the edge environment could be a critical component. The existing Turing Pi model more oriented at forward-thinking developers who want to learn and push cloud-native to the edge. Why Raspberry Pi computers? They are not the most powerful computers, but they definitely can lower the entry point for developers by offering a huge and well-documented software ecosystem.