7 ms·

The #1 problem with Kubernetes is it's not something that "Just Works." There's a very small subset of engineers who can stand up services on Kubernetes without

The #1 problem with Kubernetes is it's not something that "Just Works." There's a very small subset of engineers who can stand up services on Kubernetes without having it fall over in production – not to mention actually running & maintaining a Kubernetes cluster on your own VMs. In response, there's been a wave of "serverless" startups because the idea of running anything yourself has become understood as (a) a time sink, (b) incredibly error prone, and (c) very likely to fail in production. I think a Kubernetes 2.0 should consider what it would look like to have a deployment platform that engineers can easily adopt and feel confident running themselves – while still maintaining itself as a small-ish core orchestrator with strong primitives. I've been spending a lot of time building Rivet to itch my own itch of an orchestrator & deployment platform that I can self-host and scale trivially: https://github.com/rivet-gg/rivet https://github.com/rivet-gg/rivet We currently advertise as the "open-source serverless platform," but I often think of the problem as "what does Kubernetes 2.0 look like." People are already adopting it to push the limits into things that Kubernetes would traditionally be good at. We've found the biggest strong point is that you're able to build roughly the equivalent of a Kubernetes controller trivially. This unlocks features more complex workload orchestration (game servers, per-tenant deploys), multitenancy (vibe coding per-tenant backends, LLM code interpreters), metered billing per-tenant, more powerful operators, etc.

jekwoooooe 1y ago
This is just so not true. You can literally launch talos and have an N node cluster in about 10 minutes 9 of which is waiting for downloads, configs, etc. 1 min of configuration. It’s just that easy.
- motorest 1y ago
  > This is just so not true. You can literally launch talos and have an N node cluster in about 10 minutes 9 of which is waiting for downloads, configs, etc. 1 min of configuration. It’s just that easy. I agree, OP's points contrast with reality. Anyone can whip out a Kubernetes cluster with COTS hardware or cheap VMs within minutes. Take Canonical's microK8s distribution. Do a snap install to get a Kubernetes node up and running, and run a command to register it in a cluster. Does this pass nowadays as rocket science? With established cloud providers its even easier.
  - _w1tm 1y ago
    > Take Canonical's microK8s distribution. Do a snap install to get a Kubernetes node up and running, and run a command to register it in a cluster. Does this pass nowadays as rocket science? That’s only after you compile your Ubuntu kernel and all the software. Don’t even get me started on bad sectors on the floppy discs they give out at conferences!
    - motorest 1y ago
      > That’s only after you compile your Ubuntu kernel and all the software. No, that's simply wrong at so many levels.
      - p_l 1y ago
        I think you missed the sarcasm inherent in talking about bad sectors on floppy disks given away at a conference :D
- darqis 1y ago
  and how many cloud or VPS providers allow you to bring your own OS? And also split vda into multiple disks? I'll tell you how many, zero, unless you're willing to pay premium and I mean vastly overpriced
  - jekwoooooe 1y ago
    Vultr gcp aws.. what are you talking about
  - vbezhenar 1y ago
    Every cloud provider allows you to bring your own OS and configure disks in any way you want. May be you're thinking about cheapest VPS possible, driven by something like cpanel. Those are not cloud providers. But usually you wouldn't choose them for reliable service, because their whole model is overselling.
stuff4ben 1y ago
I really dislike this take and I see it all the time. Also I'm old and I'm jaded, so it is what it is... Someone decides X technology is too heavy-weight and wants to just run things simply on their laptop because "I don't need all that cruft". They spend time and resources inventing technology Y to suit their needs. Technology Y gets popular and people add to it so it can scale, because no one runs shit in production off their laptops. Someone else comes along and says, "damn, technology Y is too heavyweight, I don't need all this cruft..." "There are neither beginnings nor endings to the Wheel of Time. But it was a beginning.”
- adrianmsmith 1y ago
  It’s also possible for things to just be too complex. Just because something’s complex doesn’t necessarily mean it has to be that complex.
  - mdaniel 1y ago
    IMHO, the rest of that sentence is "be too complex for some metric within some audience" I can assure you that trying to reproduce kubernetes with a shitload of shell scripts, autoscaling groups, cloudwatch metrics, and hopes-and-prayers is too complex for my metric within the audience of people who know Kubernetes
  - wongarsu 1y ago
    Or too generic. A lot of the complexity if from trying to support all use cases. For each new feature there is a clear case of "we have X happy users, and Y people who would start using it if we just added Z". But repeat that often enough and the whole things becomes so complex and abstract that you lose those happy users. The tools I've most enjoyed (including deployment tools) are those with a clear target group and vision, along with leadership that rejects anything that falls too far outside of it. Yes, it usually doesn't have all the features I want, but it also doesn't have a myriad of features I don't need
    - deleted 1y ago
      [deleted]
themgt 1y ago
The problem Kubernetes solves is "how do I deploy this" ... so I go to Rivet (which does look cool) docs, and the options are: * single container * docker compose * manual deployment (with docker run commands) But erm, realistically how is this a viable way to deploy a "serverless infrastructure platform" at any real scale? My gut response would be ... how can I deploy Rivet on Kubernetes, either in containers or something like kube-virt to run this serverless platform across a bunch of physical/virtual machines? How is docker compose a better more reliable/scalable alternative to Kubernetes? So alternately then you sell a cloud service, but ... that's not a Kubernetes 2.0. If I was going to self-host Rivet I'd convert your docs so I could run it on Kubernetes.
- NathanFlurry 1y ago
  Our self-hosting docs are very rough right now – I'm fully aware of the irony given my comment. It's on our roadmap to get them up to snuff within the next few weeks. If you're curious on the details, we've put a lot of work to make sure that there's as few moving parts as possible: We have our own cloud VM-level autoscaler that's integrated with the core Rivet platofrm – no k8s or other orchestrators in between. You can see the meat of it here: https://github.com/rivet-gg/rivet/blob/335088d0e7b38be5d029d52556aa8ad8e101b344/packages/core/services/cluster/src/lib.rs https://github.com/rivet-gg/rivet/blob/335088d0e7b38be5d029d... For example, Rivet has an API to dynamically spin up a cluster on demand: https://github.com/rivet-gg/rivet/blob/335088d0e7b38be5d029d52556aa8ad8e101b344/packages/core/services/cluster/src/workflows/datacenter/mod.rs#L13 https://github.com/rivet-gg/rivet/blob/335088d0e7b38be5d029d... Once you start the Rivet "seed" process with your API key, everything from there is automatic. Therefore, self-hosted deployments usually look like one of: - Plugging in your cloud API token in to Rivet for autoscaling (recommended) - Fixed # of servers (hobbyist deployments that were manually set up, simple Terraform deployments, or bare metal) - Running within Kubernetes (usually because it depends on existing services)
hosh 1y ago
It's been my experience that nothing in infra and ops will ever "just work". Even something like Heroku will run into scaling issues, and how much you are willing to pay for it. If people's concerns is that they want a deployment platform that can be easily adopted and used, it's better to understand Kubernetes as the primitives on which the PaaS that people want can be built on top of it. Having said all that, Rivet looks interesting. I recognize some of the ideas from the BEAM ecosystem. Some of the appeal to me has less to do with deploying at scale, and more to do with resiliency and local-first.
teaearlgraycold 1y ago
For my needs as an engineer working on your standard simple monolithic-ish app I want a system that can’t do everything. Start with the ability to host a few services as simple as possible. I want a Heroku/Render level of complexity here. Spin up a db, web workers with simple scaling rules, background workers etc. Once you’ve perfected the design see if you can add anything else in to expand the capabilities. If you can’t do that without making it high maintenance and incomprehensible then don’t. It will forever be loved as the best way to host simple apps.
jeswin 1y ago
If you're using k8s as a service where you've outsourced all of the cluster maintenance (from any of the cloud vendors), which part due you see as super complex? The configuration and specs are vast, but you might only need to know a very small subset of it to put services into production. You can deploy to k8s with configuration that's just a little more complex than docker-compose. Having said that, perhaps for a majority of apps even that "a little more" is unnecessary - and docker-compose was what the long tail actually wanted. That docker-compose didn't get sustained traction is unfortunate.
- motorest 1y ago
  > If you're using k8s as a service where you've outsourced all of the cluster maintenance (from any of the cloud vendors), which part due you see as super complex? This point bears repeating. OP's opinion and baseless assertions contrast with reality. Even with COTS hardware nowadays we have Kubernetes distributions which are trivial to setup and run, to the point that they only require a package being installed to get a node running, and running a single command to register the newly provisioned node in a cluster. I wonder what's OP's first-hand experience with Kubernetes.
  - 15155 1y ago
    When Kubernetes was in its infancy, it was quite difficult to do the PKI and etcd setup required to run a cluster - and managed services didn't exist. Now, in every thread, people replay arguments from close to a decade ago that reflect conditions 90% of companies wouldn't face today.
    - eddythompson80 1y ago
      Yeah, it's as if someone is saying they can't run linux because compiling takes forever, dealing with header files and dependencies is a pain, plus I use bluetooth headphones.
    - motorest 1y ago
      > When Kubernetes was in its infancy, it was quite difficult to do the PKI and etcd setup required to run a cluster - and managed services didn't exist. I agree, but it's important to stress the fact that things like etcd are only a pivotal component if you're putting together an ad-hoc cluster of unreliable nodes. Let that sink in. Kubernetes is a distributed system designed for high reliability, and depends on consensus algorithms to self-heal. That's a plausible scenario if you're running a cluster of cheap unreliable COTS hardware. Is that what most people use? Absolutely not.
firesteelrain 1y ago
We run a few AKS clusters and they do just work. We rarely do any maintenance on them other than the automatic updates. The containers run 24/7 with no issues. It’s pretty amazing
kachapopopow 1y ago
This is simply not true, maintaining a k3s(k8s has a few gotchas) cluster is very easy especially with k3s auto upgrade as long as you have proper eviction rules (maybe pod distruption). Ceph can be tricky, but you can always opt for lemon or longhorn which are nearly zero maintenance. There's thousands of helm charts available that allow you to deploy even the most complicated databases within a minute. Deploying your own service is also very easy as long as you use one of the popular helm templates. Helm is by no means perfect, but it's great when you set it up the way you want. For example I have full code completion for values.yaml by simply having "deployment" charts which bundle the application database(s) and application itself into a single helm chart. You can't just "jump into" kubernetes like you can with many serverless platforms, but spending a week banging your head to possibly save hundreds of thousands in a real production environment is a no-brainer to me.
- darqis 1y ago
  But this also isn't true. Everything k8s has caveats. You can't deploy a Percona MySQL DB on ARM for instance. Various operators have various issues which require manual intervention. It's all pretty much a clusterfuck. Then debugging reasons why this service works locally with a systemd service but not on k8s is also time intensive and difficult. And the steadily changing features and frequent version bumps. It's a full time job. And many helm charts aren't that great to begin with. And what when someone deployment hangs and can't be deleted but still allocates resources? This is a common issue.
  - kachapopopow 1y ago
    I use bitnami helm charts exclusively and they have never failed me. ARM is a caviant in itself, but I had a really good experience with k3s.
- nabeards 1y ago
  Per the K3s web site and their docs, they don’t call out that it’s good for bare metal production. That tells me it’s not built for heavy workloads, and is instead for dev environments and edge compute.
arkh 1y ago
What I'd like to see is something for which adding capacity is plug&play: connect your machine to some network, boot it and automagically it will appear in your farm's UI to become a node. Like how pluging a USB stick in your PC gives you some GB of storage.
- p_l 1y ago
  Quite doable with kubernetes - honestly, the main issue there is storage, because distributed storage is inherently complex topic that you need to tackle separately, even if you run on top of k8s. However, if you can simplify your setup by allowing for example a performant NFS filer as storage provider (can be random machine with FreeNAS on top, if you have small needs even a random NAS home box), one server as "control plane", the rest can be plugged in automatically (with nfs-provisioner handling simple storage management for you)
wvh 1y ago
My take is that Kubernetes is too simple. It leaves too much open and tries to keep everything pluggable, in an attempt to be all things to all people and, more importantly, to all companies so it can be the de facto standard. So typically when you set up Kubernetes, you are left with figuring out how to get metrics, logging, alerting and whatever else on top of the core you have. There are a lot of extra decisions to make, and none of this stuff is as interoperable and fool-proof as it should be. Maybe the problem (and strength) of Kubernetes is that it's design by committee or at least common-denominator agreement so everybody stays on board. Any more clearly defined project would likely not become a de facto industry standard.
- davewritescode 1y ago
  I agree with you 100% as someone who's developed on and run many Kubernetes clusters over the years. Picking a choosing what to use can be daunting. That said, the path to ease of use usually involves making sound decisions ahead of time for users and assuming 99% will stay on that paved path. This is how frameworks like Spring and Rails because ubiquitous.
justnonono 1y ago
[flagged]

jekwoooooe 1y ago

This is just so not true. You can literally launch talos and have an N node cluster in about 10 minutes 9 of which is waiting for downloads, configs, etc. 1 min of configuration. It’s just that easy.

motorest 1y ago
> This is just so not true. You can literally launch talos and have an N node cluster in about 10 minutes 9 of which is waiting for downloads, configs, etc. 1 min of configuration. It’s just that easy. I agree, OP's points contrast with reality. Anyone can whip out a Kubernetes cluster with COTS hardware or cheap VMs within minutes. Take Canonical's microK8s distribution. Do a snap install to get a Kubernetes node up and running, and run a command to register it in a cluster. Does this pass nowadays as rocket science? With established cloud providers its even easier.
- _w1tm 1y ago
  > Take Canonical's microK8s distribution. Do a snap install to get a Kubernetes node up and running, and run a command to register it in a cluster. Does this pass nowadays as rocket science? That’s only after you compile your Ubuntu kernel and all the software. Don’t even get me started on bad sectors on the floppy discs they give out at conferences!
  - motorest 1y ago
    > That’s only after you compile your Ubuntu kernel and all the software. No, that's simply wrong at so many levels.
    - p_l 1y ago
      I think you missed the sarcasm inherent in talking about bad sectors on floppy disks given away at a conference :D
darqis 1y ago
and how many cloud or VPS providers allow you to bring your own OS? And also split vda into multiple disks? I'll tell you how many, zero, unless you're willing to pay premium and I mean vastly overpriced
- jekwoooooe 1y ago
  Vultr gcp aws.. what are you talking about
- vbezhenar 1y ago
  Every cloud provider allows you to bring your own OS and configure disks in any way you want. May be you're thinking about cheapest VPS possible, driven by something like cpanel. Those are not cloud providers. But usually you wouldn't choose them for reliable service, because their whole model is overselling.

stuff4ben 1y ago

I really dislike this take and I see it all the time. Also I'm old and I'm jaded, so it is what it is... Someone decides X technology is too heavy-weight and wants to just run things simply on their laptop because "I don't need all that cruft". They spend time and resources inventing technology Y to suit their needs. Technology Y gets popular and people add to it so it can scale, because no one runs shit in production off their laptops. Someone else comes along and says, "damn, technology Y is too heavyweight, I don't need all this cruft..." "There are neither beginnings nor endings to the Wheel of Time. But it was a beginning.”

adrianmsmith 1y ago
It’s also possible for things to just be too complex. Just because something’s complex doesn’t necessarily mean it has to be that complex.
- mdaniel 1y ago
  IMHO, the rest of that sentence is "be too complex for some metric within some audience" I can assure you that trying to reproduce kubernetes with a shitload of shell scripts, autoscaling groups, cloudwatch metrics, and hopes-and-prayers is too complex for my metric within the audience of people who know Kubernetes
- wongarsu 1y ago
  Or too generic. A lot of the complexity if from trying to support all use cases. For each new feature there is a clear case of "we have X happy users, and Y people who would start using it if we just added Z". But repeat that often enough and the whole things becomes so complex and abstract that you lose those happy users. The tools I've most enjoyed (including deployment tools) are those with a clear target group and vision, along with leadership that rejects anything that falls too far outside of it. Yes, it usually doesn't have all the features I want, but it also doesn't have a myriad of features I don't need
  - deleted 1y ago
    [deleted]

themgt 1y ago

The problem Kubernetes solves is "how do I deploy this" ... so I go to Rivet (which does look cool) docs, and the options are: * single container * docker compose * manual deployment (with docker run commands) But erm, realistically how is this a viable way to deploy a "serverless infrastructure platform" at any real scale? My gut response would be ... how can I deploy Rivet on Kubernetes, either in containers or something like kube-virt to run this serverless platform across a bunch of physical/virtual machines? How is docker compose a better more reliable/scalable alternative to Kubernetes? So alternately then you sell a cloud service, but ... that's not a Kubernetes 2.0. If I was going to self-host Rivet I'd convert your docs so I could run it on Kubernetes.

NathanFlurry 1y ago
Our self-hosting docs are very rough right now – I'm fully aware of the irony given my comment. It's on our roadmap to get them up to snuff within the next few weeks. If you're curious on the details, we've put a lot of work to make sure that there's as few moving parts as possible: We have our own cloud VM-level autoscaler that's integrated with the core Rivet platofrm – no k8s or other orchestrators in between. You can see the meat of it here: https://github.com/rivet-gg/rivet/blob/335088d0e7b38be5d029d52556aa8ad8e101b344/packages/core/services/cluster/src/lib.rs https://github.com/rivet-gg/rivet/blob/335088d0e7b38be5d029d... For example, Rivet has an API to dynamically spin up a cluster on demand: https://github.com/rivet-gg/rivet/blob/335088d0e7b38be5d029d52556aa8ad8e101b344/packages/core/services/cluster/src/workflows/datacenter/mod.rs#L13 https://github.com/rivet-gg/rivet/blob/335088d0e7b38be5d029d... Once you start the Rivet "seed" process with your API key, everything from there is automatic. Therefore, self-hosted deployments usually look like one of: - Plugging in your cloud API token in to Rivet for autoscaling (recommended) - Fixed # of servers (hobbyist deployments that were manually set up, simple Terraform deployments, or bare metal) - Running within Kubernetes (usually because it depends on existing services)

hosh 1y ago

It's been my experience that nothing in infra and ops will ever "just work". Even something like Heroku will run into scaling issues, and how much you are willing to pay for it. If people's concerns is that they want a deployment platform that can be easily adopted and used, it's better to understand Kubernetes as the primitives on which the PaaS that people want can be built on top of it. Having said all that, Rivet looks interesting. I recognize some of the ideas from the BEAM ecosystem. Some of the appeal to me has less to do with deploying at scale, and more to do with resiliency and local-first.

teaearlgraycold 1y ago

For my needs as an engineer working on your standard simple monolithic-ish app I want a system that can’t do everything. Start with the ability to host a few services as simple as possible. I want a Heroku/Render level of complexity here. Spin up a db, web workers with simple scaling rules, background workers etc. Once you’ve perfected the design see if you can add anything else in to expand the capabilities. If you can’t do that without making it high maintenance and incomprehensible then don’t. It will forever be loved as the best way to host simple apps.

jeswin 1y ago

If you're using k8s as a service where you've outsourced all of the cluster maintenance (from any of the cloud vendors), which part due you see as super complex? The configuration and specs are vast, but you might only need to know a very small subset of it to put services into production. You can deploy to k8s with configuration that's just a little more complex than docker-compose. Having said that, perhaps for a majority of apps even that "a little more" is unnecessary - and docker-compose was what the long tail actually wanted. That docker-compose didn't get sustained traction is unfortunate.

motorest 1y ago
> If you're using k8s as a service where you've outsourced all of the cluster maintenance (from any of the cloud vendors), which part due you see as super complex? This point bears repeating. OP's opinion and baseless assertions contrast with reality. Even with COTS hardware nowadays we have Kubernetes distributions which are trivial to setup and run, to the point that they only require a package being installed to get a node running, and running a single command to register the newly provisioned node in a cluster. I wonder what's OP's first-hand experience with Kubernetes.
- 15155 1y ago
  When Kubernetes was in its infancy, it was quite difficult to do the PKI and etcd setup required to run a cluster - and managed services didn't exist. Now, in every thread, people replay arguments from close to a decade ago that reflect conditions 90% of companies wouldn't face today.
  - eddythompson80 1y ago
    Yeah, it's as if someone is saying they can't run linux because compiling takes forever, dealing with header files and dependencies is a pain, plus I use bluetooth headphones.
  - motorest 1y ago
    > When Kubernetes was in its infancy, it was quite difficult to do the PKI and etcd setup required to run a cluster - and managed services didn't exist. I agree, but it's important to stress the fact that things like etcd are only a pivotal component if you're putting together an ad-hoc cluster of unreliable nodes. Let that sink in. Kubernetes is a distributed system designed for high reliability, and depends on consensus algorithms to self-heal. That's a plausible scenario if you're running a cluster of cheap unreliable COTS hardware. Is that what most people use? Absolutely not.

firesteelrain 1y ago

We run a few AKS clusters and they do just work. We rarely do any maintenance on them other than the automatic updates. The containers run 24/7 with no issues. It’s pretty amazing

kachapopopow 1y ago

This is simply not true, maintaining a k3s(k8s has a few gotchas) cluster is very easy especially with k3s auto upgrade as long as you have proper eviction rules (maybe pod distruption). Ceph can be tricky, but you can always opt for lemon or longhorn which are nearly zero maintenance. There's thousands of helm charts available that allow you to deploy even the most complicated databases within a minute. Deploying your own service is also very easy as long as you use one of the popular helm templates. Helm is by no means perfect, but it's great when you set it up the way you want. For example I have full code completion for values.yaml by simply having "deployment" charts which bundle the application database(s) and application itself into a single helm chart. You can't just "jump into" kubernetes like you can with many serverless platforms, but spending a week banging your head to possibly save hundreds of thousands in a real production environment is a no-brainer to me.

darqis 1y ago
But this also isn't true. Everything k8s has caveats. You can't deploy a Percona MySQL DB on ARM for instance. Various operators have various issues which require manual intervention. It's all pretty much a clusterfuck. Then debugging reasons why this service works locally with a systemd service but not on k8s is also time intensive and difficult. And the steadily changing features and frequent version bumps. It's a full time job. And many helm charts aren't that great to begin with. And what when someone deployment hangs and can't be deleted but still allocates resources? This is a common issue.
- kachapopopow 1y ago
  I use bitnami helm charts exclusively and they have never failed me. ARM is a caviant in itself, but I had a really good experience with k3s.
nabeards 1y ago
Per the K3s web site and their docs, they don’t call out that it’s good for bare metal production. That tells me it’s not built for heavy workloads, and is instead for dev environments and edge compute.

arkh 1y ago

What I'd like to see is something for which adding capacity is plug&play: connect your machine to some network, boot it and automagically it will appear in your farm's UI to become a node. Like how pluging a USB stick in your PC gives you some GB of storage.

p_l 1y ago
Quite doable with kubernetes - honestly, the main issue there is storage, because distributed storage is inherently complex topic that you need to tackle separately, even if you run on top of k8s. However, if you can simplify your setup by allowing for example a performant NFS filer as storage provider (can be random machine with FreeNAS on top, if you have small needs even a random NAS home box), one server as "control plane", the rest can be plugged in automatically (with nfs-provisioner handling simple storage management for you)

wvh 1y ago

My take is that Kubernetes is too simple. It leaves too much open and tries to keep everything pluggable, in an attempt to be all things to all people and, more importantly, to all companies so it can be the de facto standard. So typically when you set up Kubernetes, you are left with figuring out how to get metrics, logging, alerting and whatever else on top of the core you have. There are a lot of extra decisions to make, and none of this stuff is as interoperable and fool-proof as it should be. Maybe the problem (and strength) of Kubernetes is that it's design by committee or at least common-denominator agreement so everybody stays on board. Any more clearly defined project would likely not become a de facto industry standard.

davewritescode 1y ago
I agree with you 100% as someone who's developed on and run many Kubernetes clusters over the years. Picking a choosing what to use can be daunting. That said, the path to ease of use usually involves making sound decisions ahead of time for users and assuming 99% will stay on that paved path. This is how frameworks like Spring and Rails because ubiquitous.

justnonono 1y ago

[flagged]