9 ms·
Kubernetes on Oxide: How customer needs shaped our integrations
- stevehipwell 1mo agoI'm interested to see how the `oxide-cloud-controller-manager` is being built for "modern" Kubernetes and if it leads to any signify difference compared to CCMs that originated in-tree. Given the way Oxide engineer their solutions this could be really interesting. FYI I've got `karpenter-provider-oxide` on my bingo card...
- thegagne 1mo agoYes this would be great addition! Also it would be awesome to see proper load balancer, ingress controller with `gateway api` and a `api gateway` (two separate things). I’m not an oxide customer, just a fan. I have lots of ideas around how something like this should look, being disappointed with the complexity and also shortcomings of products on the market for this stuff, both in the cloud and on prem.
- sudomateo 1mo agoI'll need to record a video on what's possible today. My personal Kubernetes cluster running on Oxide uses the CCM for LoadBalancer services and NGINX Gateway Fabric for Gateway API things. I'm mostly using HTTPRoute resources today. There's an opportunity to more tightly integrate at the network layer but we'll want to get our load balancer released first.
- esseph 1mo ago> My personal Kubernetes cluster running on Oxide What a flex. It's gotten to the point I'm starting to actively look for Oxide customers so I can apply with them.
- sudomateo 1mo agoThere are few paths we can take here. The CCM's primary responsibility is to implement the cloudprovider.Interface[0] which has node, route, and service controllers. However, the CCM can run arbitrary named controllers as well, giving us a fun extension point for the future. The in-tree CCMs are being phased out in favor of out-of-tree CCMs. The AWS Cloud Provider[1] is a good example of what that's starting to look like. My colleague demo'd Karpeneter internally. We haven't committed releasing it yet but we're discussing it. [0] https://github.com/kubernetes/cloud-provider/blob/master/cloud.go https://github.com/kubernetes/cloud-provider/blob/master/clo... [1] https://github.com/kubernetes/cloud-provider-aws https://github.com/kubernetes/cloud-provider-aws
- stevehipwell 1mo agoMy point was that the major cloud CCMs are rooted (at least spiritually) in in-tree implementations. With Oxide having an API first strategy (similar in principle to a major cloud provider) the direction you take on your greenfield CCM could be interesting. RE Karpenter, it always seemed like a natural fit for Oxide, even more so than some of the currently implementors. And now you have the expertise in the team, I'd be really interested in the reason if you don't go down that path.
- sudomateo 1mo agoTotally, the out-of-tree CCM gives us a good opportunity to differentiate. Autoscaling is a requested feature, and Karpenter fits that shape naturally. The nuance is to decide where something like the Cluster API provider ends and Karpenter begins since there's a bit of overlap in concerns. Specifically, both want to manage Kubernetes nodes but for different reasons. We're discussing it though. We have a Kubernetes watercooler meeting today where we'll likely discuss the comments in this post!
- bmwagner10 1mo agoKarpenter is definitely a natural fit for Oxide. There's some interesting boundaries that we're discussing internally, like @sudomateo mentioned. One of which is CAPI. There's a CAPI Karpenter provider that elmiko built (notably back in the very early days of Karpenter). I think there's room for both use-cases. Some may want to use CAPI for everything and others may want a platform that is based on CAPI but guest clusters are not. One implementation specific detail which makes Karpenter interesting on Oxide is the tunable CPU and memory parameters rather than strict instance type shapes. The prototype I built for Karpenter on Oxide generates all possible "instance type" combinations, so you can create some really specific nodes to bin-pack pods. Another interesting area is multiple providers. This is becoming pretty common across public clouds too. A multi-provider Karpenter is something I'm interested in and I know some folks have already been gluing together, but the Karpenter story isn't great on maintaining those since you basically need to compile them together today. CAPIs multi-provider story is a bit cleaner since it only relies on CRDs. If you have ideas, let's chat in the Kubernetes slack #karpenter-dev.
- overflowy 1mo agoI would absolutely kill for them to open source their documentation system.
- bcantrill 1mo agoAre you referring to our RFD system?[0][1] Because if so, good news![2] [0] https://rfd.shared.oxide.computer/rfd/0001 https://rfd.shared.oxide.computer/rfd/0001 [1] https://oxide-and-friends.transistor.fm/episodes/rfds-the-backbone-of-oxide https://oxide-and-friends.transistor.fm/episodes/rfds-the-ba... [2] https://github.com/oxidecomputer/rfd-site https://github.com/oxidecomputer/rfd-site
- overflowy 1mo agoOMG, you just made my day!
- mixmastamyk 1mo agoWhy would a “doc system” use React?
- dcre 1mo agoBecause it’s a web site.
- pianoben 1mo agoI have seldom wanted anything as much as I want an Oxide rack at home. Maybe in 40 years we'll start to see the first ones show up in surplus auctions...
- sudomateo 1mo agoYou can run the control plane at home. I have a video on how to do it. Until we make a smaller footprint that's the only way you'd get Oxide at home.
- bakies 1mo agofwiw, I am very satisfied with talos and k8s at home.
- sudomateo 1mo agoI run this on a TuringPi board at home. I wouldn't get the TuringPi again but Talos for home use has been great. The only issue for me is I have access to Oxide so I moved most of my home lab there.
- lkasjdas 1mo agoEvery single Oxide article has comments about people wanting one at home. A great example of completely misdirected marketing and/or engineering. Oxide should have developed home microcomputers, and they would have sold like cupcakes, with their ASCII art marketing. Instead they do million dollar "mainframes" that no one (except VCs) wants to buy.
- moondev 1mo agoLove to see the CAPOx provider and buy in to Cluster API
- preisschild 1mo agoYeah me too, unfortunately I cant justify huge 100K USD servers so ill continue with cluster api provider hetzner and metal3. The whole CAPI ecosystem is quite cool.
- whazor 1mo agoKubernetes is where Oxide, from my armchair, doesn't yet quite match the public cloud. On AWS EKS Fargate, each pod runs in its own dedicated VM. With Oxide each k8s node is a VM, so you still need something like Talos. On networking it looks like it is getting closer. Where you can have external subnet give pod routed IPs without overlay. But the gap, as marked by the article, is also the load balancing. It would be nice if Kubernetes were a native feature out-of-the-box. Also integrated within the existing user/access control.
- sudomateo 1mo agoWe're discussing what "native Kubernetes" looks like on Oxide in the limit. There's a bunch to build, some of which is blocked on product gaps. We'll get there though! I don't necessarily want to match the public cloud experience if there's an opportunity for Oxide to exceed the public cloud experience. Eliminating the overlay is a good example of this. We have customers using external subnets to eliminate the overlay but we haven't integrated that into our controllers yet.
- wolttam 1mo agoSo you’d be attaching a new volume to the running worker VM for each PVC? This seems a little odd to me. Could you attach a single large volume, and do path-based provisioning on that?
- itintheory 1mo agoDoes seem like that could be an issue for RWX PVCs, unless those volumes can be mounted to multiple nodes. Depending on the workload, you might be better off using NFS.
- wsng 1mo agoThere is no one-size-fits-all, but I would say NFS is nowadays not a preferred option for most modern distributed datastores. They are fine with local storage only, and achieve coordination with higher-level protocols. Running them on top of NFS kills their performance.
- sudomateo 1mo ago> So you’d be attaching a new volume to the running worker VM for each PVC? That's what we prototyped before local disk was released and before we started disk hot-plug work. > Could you attach a single large volume, and do path-based provisioning on that? Possibly. We'd still want disk hot-plug first. Otherwise, customers would have to create their cluster in a certain shape before using PVCs.
- bitlad 1mo agoI am wondering when would you use kubernetes on oxide vs running kubernetes with kubevirt on baremetal? At first look, it feels like oxide is equivalent to proxmox or some virtualization tool, may be it uses qemu stuff underneath. Just curious. The reason I am asking this is that, we have lot onprem scenarios in our business. We are tightly coupled with k8s, to solve this we started building an internal project that is kubernetes API compatible [1] but runs containerd or WASM or our platform natively. Just curious how oxides work in this scenario [1] https://github.com/debarshibasak/superkube https://github.com/debarshibasak/superkube
- e12e 1mo agoWhen oxide provide your metal?
- bakies 1mo agoYeah me too. Like I kinda compare oxide to coreweave and coreweave is built on top of k8s afaik. I'm wondering what the underlying tech oxide leans on. If I was doing this I would be using talos and k8s on baremetal and building on top of that. Container first rather than VM first seems like a big advantage imo. Most workloads will be containers.
- redwood 1mo agoHow are folks managing stateful workloads like databases on Oxide?
- sudomateo 1mo agoI cover this a bit in the post. Today they are running something like Longhorn backed by Oxide storage. When we release our CSI plugin they can use that instead.
- dilyevsky 1mo agowhy longhorn and not mayastor? longhorn doesn't even have nvmeof that is not experimental
- sudomateo 1mo agoCould be whichever. Customers have expressed a desired for Longhorn given they were already using it so we started our testing there. Ultimately our CSI plugin will be the official Oxide path.
- kirici 1mo agoWhy Mayastor and not (Rook-)Ceph?
- dilyevsky 1mo agoSlow, buggy and resource hungry. Faster spdk-based engine rewrite has been on the way for like 7 years now
- preisschild 1mo agoOne cool feature of ceph is reed solomon erasure coding, myastor only does replication if storage efficiency is important 4+2 Erasure Coding has only half the storage overhead of 3x replication but the same fault tolerance Also re spdk https://muratkarslioglu.com/blog/io-uring-spdk-kernel-bypass/ https://muratkarslioglu.com/blog/io-uring-spdk-kernel-bypass...
- lars_francke 1mo agoDisclaimer: I'm totally biased here. I talked with a colleague from Oxide in 2024 about your Kubernetes story and he said back then "not yet but soon-ish". Seems like soon-ish is now :) We said we'd talk again when that happens but he's since left Oxide. If you (or well...your customers) are interested in a Kubernetes native data platform 100% open source we'd be very happy to talk about how that could work easily. As it's "just" Kubernetes it should be trivial but we'd be happy to test and add you to our list: https://docs.stackable.tech/home/stable/kubernetes/#supported-production-distributions https://docs.stackable.tech/home/stable/kubernetes/#supporte... The offer stands. If you're interested, my mail is in my HN profile. https://stackable.tech https://stackable.tech
- donatj 1mo agoBug report I guess, but the pages top navigation seems completely inoperable to me on my iPad. Clicking with my finger does not work. Clicking with my Apple Pencil does not work. Any sort of hover state that might be there does not work (with the pencil)
- sudomateo 1mo agoReported internally. Thank you!
- preisschild 1mo agoI think it'd be cooler if you could just run kubernetes directly on their bare metal hardware without their hypervisor in between
- ninkendo 1mo agoI’ve tried so hard to figure out what the hell oxide even is, but the website is utterly impenetrable. I guess it’s an “integrated platform” and does “AI” and it’s “purpose built for frontier workloads”, but… I have absolutely no idea what it is, still. I guess it’s a bunch of computers. It says “AMD” in a bunch of places so, I guess they’re x86. Where are the GPU’s? If it’s for “frontier workloads”, wouldn’t those be sorta important? Is it its own OS? No idea. I see people adjacent to Oxide mention IllumOS sometimes[0], so does that mean it’s a solaris-like OS? Why would I want to run k8s (presumably with linux containers) on a not-linux OS? Or does it virtualize linux instances? Do actual CTO’s go for this type of marketing, with zero details and nothing but pure fluff about “solutions”? [0] This of course is from HN comments, I see no mention of any operating system whatsoever on the website, so if it is IllumOS, it’s not like you can easily find that info anywhere.
- zdkaster 1mo agoIt's a private cloud. I guess you might not be a target customer.
- ninkendo 1mo agoI develop cloud infrastructure controller software for a living but sure. I’m certainly not a target customer, but as someone casually interested in this space it’s certainly very light on details. Re-assessing my criticism, I think I’m mostly complaining that I can’t figure out what software it’s running. But I guess the target customers don’t care, since it’s an implementation detail. I’ve only ever heard “oxide” in the past in reference to the OS they were (are?) developing, which a web search says is called “hubris”, so when I see an article like OP I’m wondering “are they running k8s on top of hubris? Wow!” But I’m imagining that’s very much not the case. In the comments in this very section an oxide employee mentions they’re using Illumos, but I don’t see that anywhere on their website either. But you’re probably right in that the target customer doesn’t give a shit what OS it runs, so long as they can get instances deployed (although I would question why, if you’re going to run k8s anyway, you don’t just run it on bare metal and skip the hypervisor, but that’s my bias showing up as someone who writes bare metal controller software.)
- nunez 1mo agoGreat post. ClusterAPI never got the love that it should. I spent a good amount of time on it at VMware, as Tanzu heavily leverages it for cluster deployment (mostly CAP-A and CAP-V). It's basically kubeadm + the spirit of Terraform, Kubernetes controller edition. There's lots of great enterprise-ready options for centralized k8s cluster fleet management these days, but it works really well for folks that are all-in on GitOps and such.
- sudomateo 1mo agoThank you so much! There are certainly a plethora of options for Kubernetes cluster management. We see Cluster API as a vendor-agnostic way for us and our customers. Perhaps one day we'll more tightly integrate Kubernetes and the Oxide control plane, but gotta start somewhere.
- zop_10 1mo ago[flagged]