6 ms·

This feels like a post ranting against SystemD written from someone who likes init. I understand that K8 does many things but its also how you look at the prob

This feels like a post ranting against SystemD written from someone who likes init. I understand that K8 does many things but its also how you look at the problem. K8 does one thing well, manage complex distributed systems such as knowing when to scale up and down if you so choose and when to start up new pods when they fail. Arguably, this is one problem that is made up of smaller problems that are solved by smaller services just like SystemD works. Sometimes I wonder if the Perlis-Thompson Principle and the Unix Philosophy have become a way to force a legalistic view of software development or are just out-dated.

dolni 5y ago
I don't find the comparison to systemd to be convincing here. The end-result of systemd for the average administrator is that you no longer need to write finicky, tens or hundreds of line init scripts. They're reduced to unit files which are often just 10-15 lines. systemd is designed to replace old stuff. The result of Kubernetes for the average administrator is a massively complex system with its own unique concepts. It needs to be well understood if you want to be able to administrate it effectively. Updates come fast and loose, and updates are going to impact an entire cluster. Kubernetes, unlike systemd, is designed to be built _on top of_ existing technologies you'd be using anyway (cloud provider autoscaling, load balancing, storage). So rather than being like systemd, which adds some complexity and also takes some away, Kubernetes only adds.
- 0xEFF 5y ago
  Kubernetes removes the complexity of keeping a process (service) available. There’s a lot to unpack in that sentence, which is to say there’s a lot of complexity it removes. Agree it does add as well. I’m not convinced k8s is a net increase in complexity after everything is accounted for. Authentication, authorization, availability, monitoring, logging, deployment tooling, auto scaling, abstracting the underlying infrastructure, etc…
  - dolni 5y ago
    > Kubernetes removes the complexity of keeping a process (service) available. Does it really do that if it you just use it to provision an AWS load balancer, which can do health checks and terminate unhealthy instances for you? No. Sure, you could run some other ingress controller but now you have _yet another_ thing to manage.
    - cjalmeida 5y ago
      If that’s all you use k8s for, you don’t need it. Myself I need a to setup a bunch of other cloud services for day 2 operations. And I need to do it consistently across clouds. The kind of clients I serve won’t use my product as a SaaS due to regulatory/security reasons.
      - dolni 5y ago
        Multi-cloud is one of the few compelling use cases I can think of for Kubernetes. That said, there are relatively few organizations that actually require it.
    - randomswede 5y ago
      Do AWS load balancers distinguish between "do not send traffic" and "needs termination"? Kubernetes has readiness checks and health checks for a reason. The readiness check is a gate for "should receive traffic" and the health check is a gate for "should be restarted".
- throwaway894345 5y ago
  > So rather than being like systemd, which adds some complexity and also takes some away, Kubernetes only adds. Here are some bits of complexity that managed Kubernetes takes away: * SSH configuration * Key management * Certificate management (via cert-manager) * DNS management (via external-dns) * Auto-scaling * Process management * Logging * Host monitoring * Infra as code * Instance profiles * Reverse proxy * TLS * HTTP -> HTTPS redirection So maybe your point was "the VMs still exist" which is true, but I generally don't care because the work required of me goes away. Alternatively, you have to have most/all of these things anyway, so if you're not using Kubernetes you're cobbling together solutions for these things which has the following implications: 1. You will not be able to find candidates who know your bespoke solution, whereas you can find people who know Kubernetes. 2. Training people on your bespoke solution will be harder. You will have to write a lot more documentation whereas there is an abundance of high quality documentation and training material available for Kubernetes. 3. When something inevitably breaks with your bespoke solution, you're unlikely to get much help Googling around, whereas it's very likely that you'll find what you need to diagnose / fix / work around your Kubernetes problem. 4. Kubernetes improves at a rapid pace, and you can get those improvements for nearly free. To improve your bespoke solution, you have to take the time to do it all yourself. 5. You're probably not going to have the financial backing to build your bespoke solution to the same quality caliber that the Kubernetes folks are able to devote (yes, Kubernetes has its problems, but unless you're at a FAANG then your homegrown solution is almost certainly going to be poorer quality if only because management won't give you the resources you need to build it properly).
  - dolni 5y ago
    Respectfully, I think you have a lot of ignorance about what a typical cloud provider offers. Let's go through these each step-by-step. > SSH configuration Do you mean the configuration for sshd? What special requirements would have that Kubernetes would help fulfill? > Key management Assuming you mean SSH authorized keys since you left this unspecified. AWS does this with EC2 instance connect. > Certificate management (via cert-manager) AWS has ACM. > DNS management (via external-dns) This is not even a problem if you use AWS cloud primatives. You point Route 53 at a load balancer, which automatically discovers instances from a target group. > Auto-scaling AWS already does this via autoscaling. > Process management systemd and/or docker do this for you. > Logging AWS can send instance logs to CloudWatch. See https://docs.aws.amazon.com/systems-manager/latest/userguide/monitoring-cloudwatch-agent.html https://docs.aws.amazon.com/systems-manager/latest/userguide.... > Host monitoring In what sense? Amazon target groups can monitor the health of a service and automatically replace instances that report unhealthy, time out, or otherwise. > Infra as code I mean, you have to have a description somewhere of your pods. It's still "infra as code", just in the form prescribed by Kubernetes. > Instance profiles Instance profiles are replaced by secrets, which I'm not sure is better, just different. In either case, if you're following best practices, you need to configure security policies and apply them appropriately. > Reverse proxy AWS load balancers and target groups do this for you. > HTTPS AWS load balancers, CloudFront, do this for you. ACM issues the certificates. I won't address the remainder of your post because it seems contingent on the incorrect assumption that all of these are "bespoke solutions" that just have to be completely reinvented if you choose not to use Kubernetes.
- mst 5y ago
  Right, I really dislike systemd in many ways ... but I love what it enables people to do and accept that for all my grumpyness about it, it is overall a net win in many scenarios. k8s ... I think is often overkill in a way that simply doesn't apply to systemd.
  - parasubvert 5y ago
    If you have to manage a large distributed software code base or set of datacenters, Kubernetes is a win in that it provides a consistent, elegant solution to a nearly universal set of problems. Systemd comparatively feels like a complete waste of time given the heat it has generated for the benefit.
- thethethethe 5y ago
  > The end-result of systemd for the average administrator is that you no longer need to write finicky, tens or hundreds of line init scripts. Wouldn't the hundreds of lines of finicky, bespoke Ansible/Chef/Puppet configs required to manage non-k8s infra be the equivalent to this?
  - mathw 5y ago
    In my work, absolutely yes. Using Kubernetes has saved us sooo much nonsense. Yes we have a mix of Terraform and k8s manifests to deploy to Azure Kubernetes Service, but it works out pretty well in the end. Honestly most of the annoyance is Azure stuff. Kubernetes stuff is pretty joyful and, unlike Azure, the documentation sometimes even explains how it works.
    - dolni 5y ago
      I can't say I have had the same experience. Kubernetes cluster changes potentially create issues for all services operating in that cluster. Provisioning logic that is baked into an image means changes to one service have no chance of affecting other services (app updates that create poor netizen behavior, notwithstanding). Rolling back an AMI is as trivial as setting the AMI back in the launch template and respinning instances. There is a lot to be said for being able to make changes that you are confident will have a limited scope.
  - dolni 5y ago
    Does Kubernetes infrastructure also not require some form of configuration? Yes, there is a trade off here. You are trading a staggeringly complex external dependency for a little bit of configuration you write yourself. The Kubernetes master branch weighs in at ~4.6 million lines of code right now. Ansible sits at ~286k on their devel branch (this includes the core functionality of Ansible but not every single module). You could choose not to even use Ansible and just write a small shell script that builds out an image which does something useful in less than 500 lines of your own code, easily. Kubernetes does useful stuff and may take some work off your plate. It's also a risk. If it breaks, you get to keep both of the pieces. Kubernetes occupies the highly unenviable space of having to do highly available network clustering. As a piece of software, it is complex because it has to be. Most people don't need the functionality provided by Kubernetes. There are some niceties. But if I have to choose between "this ~500 line homebrew shell script broke" and "a Kubernetes upgrade went wrong" I know which one I am choosing, and it's not the Kubernetes problem. Managed Kubernetes, like managed cloud services, mitigate some of those issues. But you can still end up with issues like mismatched node sizes and pod resource requirements, so there is a bunch of unused compute. TL;DR of course there are trade-offs, no solution is magic.
throwaway894345 5y ago
> I understand that K8 does many things but its also how you look at the problem. K8 does one thing well, manage complex distributed systems such as knowing when to scale up and down if you so choose and when to start up new pods when they fail. Also, in the sense of "many small components that each do one thing well", k8s is even more Unix-like than Unix in that almost everything in k8s is just a controller for a specific resource type.
deleted 5y ago
[deleted]
rossmohax 5y ago
> K8 does one thing well, manage complex distributed systems such as knowing when to scale up and down if you so choose and when to start up new pods when they fail. K8S does very simple stateless case well, but anything more complicated and you are on your own. Statefull services is still a major pain especially thus with leader elections. There is not feedback to K8S about application state of the cluster, so it can't know which instancess are less disruptive to shut down or which shard needs more capacity.

dolni 5y ago

I don't find the comparison to systemd to be convincing here. The end-result of systemd for the average administrator is that you no longer need to write finicky, tens or hundreds of line init scripts. They're reduced to unit files which are often just 10-15 lines. systemd is designed to replace old stuff. The result of Kubernetes for the average administrator is a massively complex system with its own unique concepts. It needs to be well understood if you want to be able to administrate it effectively. Updates come fast and loose, and updates are going to impact an entire cluster. Kubernetes, unlike systemd, is designed to be built _on top of_ existing technologies you'd be using anyway (cloud provider autoscaling, load balancing, storage). So rather than being like systemd, which adds some complexity and also takes some away, Kubernetes only adds.

0xEFF 5y ago
Kubernetes removes the complexity of keeping a process (service) available. There’s a lot to unpack in that sentence, which is to say there’s a lot of complexity it removes. Agree it does add as well. I’m not convinced k8s is a net increase in complexity after everything is accounted for. Authentication, authorization, availability, monitoring, logging, deployment tooling, auto scaling, abstracting the underlying infrastructure, etc…
- dolni 5y ago
  > Kubernetes removes the complexity of keeping a process (service) available. Does it really do that if it you just use it to provision an AWS load balancer, which can do health checks and terminate unhealthy instances for you? No. Sure, you could run some other ingress controller but now you have _yet another_ thing to manage.
  - cjalmeida 5y ago
    If that’s all you use k8s for, you don’t need it. Myself I need a to setup a bunch of other cloud services for day 2 operations. And I need to do it consistently across clouds. The kind of clients I serve won’t use my product as a SaaS due to regulatory/security reasons.
    - dolni 5y ago
      Multi-cloud is one of the few compelling use cases I can think of for Kubernetes. That said, there are relatively few organizations that actually require it.
  - randomswede 5y ago
    Do AWS load balancers distinguish between "do not send traffic" and "needs termination"? Kubernetes has readiness checks and health checks for a reason. The readiness check is a gate for "should receive traffic" and the health check is a gate for "should be restarted".
throwaway894345 5y ago
> So rather than being like systemd, which adds some complexity and also takes some away, Kubernetes only adds. Here are some bits of complexity that managed Kubernetes takes away: * SSH configuration * Key management * Certificate management (via cert-manager) * DNS management (via external-dns) * Auto-scaling * Process management * Logging * Host monitoring * Infra as code * Instance profiles * Reverse proxy * TLS * HTTP -> HTTPS redirection So maybe your point was "the VMs still exist" which is true, but I generally don't care because the work required of me goes away. Alternatively, you have to have most/all of these things anyway, so if you're not using Kubernetes you're cobbling together solutions for these things which has the following implications: 1. You will not be able to find candidates who know your bespoke solution, whereas you can find people who know Kubernetes. 2. Training people on your bespoke solution will be harder. You will have to write a lot more documentation whereas there is an abundance of high quality documentation and training material available for Kubernetes. 3. When something inevitably breaks with your bespoke solution, you're unlikely to get much help Googling around, whereas it's very likely that you'll find what you need to diagnose / fix / work around your Kubernetes problem. 4. Kubernetes improves at a rapid pace, and you can get those improvements for nearly free. To improve your bespoke solution, you have to take the time to do it all yourself. 5. You're probably not going to have the financial backing to build your bespoke solution to the same quality caliber that the Kubernetes folks are able to devote (yes, Kubernetes has its problems, but unless you're at a FAANG then your homegrown solution is almost certainly going to be poorer quality if only because management won't give you the resources you need to build it properly).
- dolni 5y ago
  Respectfully, I think you have a lot of ignorance about what a typical cloud provider offers. Let's go through these each step-by-step. > SSH configuration Do you mean the configuration for sshd? What special requirements would have that Kubernetes would help fulfill? > Key management Assuming you mean SSH authorized keys since you left this unspecified. AWS does this with EC2 instance connect. > Certificate management (via cert-manager) AWS has ACM. > DNS management (via external-dns) This is not even a problem if you use AWS cloud primatives. You point Route 53 at a load balancer, which automatically discovers instances from a target group. > Auto-scaling AWS already does this via autoscaling. > Process management systemd and/or docker do this for you. > Logging AWS can send instance logs to CloudWatch. See https://docs.aws.amazon.com/systems-manager/latest/userguide/monitoring-cloudwatch-agent.html https://docs.aws.amazon.com/systems-manager/latest/userguide.... > Host monitoring In what sense? Amazon target groups can monitor the health of a service and automatically replace instances that report unhealthy, time out, or otherwise. > Infra as code I mean, you have to have a description somewhere of your pods. It's still "infra as code", just in the form prescribed by Kubernetes. > Instance profiles Instance profiles are replaced by secrets, which I'm not sure is better, just different. In either case, if you're following best practices, you need to configure security policies and apply them appropriately. > Reverse proxy AWS load balancers and target groups do this for you. > HTTPS AWS load balancers, CloudFront, do this for you. ACM issues the certificates. I won't address the remainder of your post because it seems contingent on the incorrect assumption that all of these are "bespoke solutions" that just have to be completely reinvented if you choose not to use Kubernetes.
mst 5y ago
Right, I really dislike systemd in many ways ... but I love what it enables people to do and accept that for all my grumpyness about it, it is overall a net win in many scenarios. k8s ... I think is often overkill in a way that simply doesn't apply to systemd.
- parasubvert 5y ago
  If you have to manage a large distributed software code base or set of datacenters, Kubernetes is a win in that it provides a consistent, elegant solution to a nearly universal set of problems. Systemd comparatively feels like a complete waste of time given the heat it has generated for the benefit.
thethethethe 5y ago
> The end-result of systemd for the average administrator is that you no longer need to write finicky, tens or hundreds of line init scripts. Wouldn't the hundreds of lines of finicky, bespoke Ansible/Chef/Puppet configs required to manage non-k8s infra be the equivalent to this?
- mathw 5y ago
  In my work, absolutely yes. Using Kubernetes has saved us sooo much nonsense. Yes we have a mix of Terraform and k8s manifests to deploy to Azure Kubernetes Service, but it works out pretty well in the end. Honestly most of the annoyance is Azure stuff. Kubernetes stuff is pretty joyful and, unlike Azure, the documentation sometimes even explains how it works.
  - dolni 5y ago
    I can't say I have had the same experience. Kubernetes cluster changes potentially create issues for all services operating in that cluster. Provisioning logic that is baked into an image means changes to one service have no chance of affecting other services (app updates that create poor netizen behavior, notwithstanding). Rolling back an AMI is as trivial as setting the AMI back in the launch template and respinning instances. There is a lot to be said for being able to make changes that you are confident will have a limited scope.
- dolni 5y ago
  Does Kubernetes infrastructure also not require some form of configuration? Yes, there is a trade off here. You are trading a staggeringly complex external dependency for a little bit of configuration you write yourself. The Kubernetes master branch weighs in at ~4.6 million lines of code right now. Ansible sits at ~286k on their devel branch (this includes the core functionality of Ansible but not every single module). You could choose not to even use Ansible and just write a small shell script that builds out an image which does something useful in less than 500 lines of your own code, easily. Kubernetes does useful stuff and may take some work off your plate. It's also a risk. If it breaks, you get to keep both of the pieces. Kubernetes occupies the highly unenviable space of having to do highly available network clustering. As a piece of software, it is complex because it has to be. Most people don't need the functionality provided by Kubernetes. There are some niceties. But if I have to choose between "this ~500 line homebrew shell script broke" and "a Kubernetes upgrade went wrong" I know which one I am choosing, and it's not the Kubernetes problem. Managed Kubernetes, like managed cloud services, mitigate some of those issues. But you can still end up with issues like mismatched node sizes and pod resource requirements, so there is a bunch of unused compute. TL;DR of course there are trade-offs, no solution is magic.

throwaway894345 5y ago

> I understand that K8 does many things but its also how you look at the problem. K8 does one thing well, manage complex distributed systems such as knowing when to scale up and down if you so choose and when to start up new pods when they fail. Also, in the sense of "many small components that each do one thing well", k8s is even more Unix-like than Unix in that almost everything in k8s is just a controller for a specific resource type.

deleted 5y ago

[deleted]

rossmohax 5y ago

> K8 does one thing well, manage complex distributed systems such as knowing when to scale up and down if you so choose and when to start up new pods when they fail. K8S does very simple stateless case well, but anything more complicated and you are on your own. Statefull services is still a major pain especially thus with leader elections. There is not feedback to K8S about application state of the cluster, so it can't know which instancess are less disruptive to shut down or which shard needs more capacity.