7 ms·

> To move to Kubernetes, an organization needs a full engineering team just to keep the Kubernetes clusters running Perhaps my team runs a simpler cluster, but

> To move to Kubernetes, an organization needs a full engineering team just to keep the Kubernetes clusters running Perhaps my team runs a simpler cluster, but we have been running a Kubernetes cluster for 2+ years as a team of 2 and it has been nothing less than worth it The way the author describes the costs of moving to Kubernetes makes me think that they don't have the experience with Kubernetes to actually realize the major benefits over the initial costs

cassianoleal 5y ago
I've been on a team of 6-8 running relatively large-scale clusters (2 big ones with lower hundreds of workloads through tens of namespaces/tenants, plus a couple smaller ones). To "keep the Kubernetes clusters running" is an afterthought at most. I mostly agree with your comment.
- RamRodification 5y ago
  Do/Did you use some specific method or orchestration product to set it up?
  - Sahbak 5y ago
    Not OP, but similar workloads, team of 5 We use kops to deplot and manage the clusters. The hardest part is updating, due to the workloads running on it. Other than that, little to no problems with kube itself.
  - cassianoleal 5y ago
    We use GKE but there's nothing stopping most organisations to do the same. It doesn't bear much weight though, and I've had experience with other toolings largely to the same effect. I agree with the sibling comment about upgrades - this is IMO where GKE really shines as a cluster upgrade is mostly a button-pressing exercise.
theptip 5y ago
This comes up a lot around here; many people look at “Kubernetes the hard way” and think they need to run their own cluster. Just use GKE or whatever the equivalent managed offering is for your cloud provider. A couple of clicks in the ui and you have a cluster. Its really easy to run, I rarely have to look at the cluster at all.
heydonovan 5y ago
Yes! Was going to say the same. Kubernetes is far easier to learn than some random bespoke setup. After reading the article, it just sounds like they reinvented the wheel but made it AWS specific for some reason. Was brought on as a consultant years ago and found their bespoke setup of random ec2 instances and lambda scripts to be far more difficult to understand than just spinning up a managed Kubernetes cluster and having a generic interface to deploy the application, as well as monitoring, logging, metrics, etc.
- cortesoft 5y ago
  > Kubernetes is far easier to learn than some random bespoke setup This, to me, is the biggest advantage of kubernetes. Yes, you can do all the things yourself, with your own custom solution that does everything you need to do just as well as kubernetes. What kubernetes gives you is a shared way of doing things. By using a common method, you can easily integrate different services together, as well as onboard new hires easily. Using something ubiquitous like kubernetes helps both your code onboarding and your people onboarding.
  - throwaway894345 5y ago
    Also not having to debug EC2 lifecycle issues is really nice. And you don't have to deal with all of the garbage of setting up ec2 instances (configuring SSH, process management, configuration management, log exfiltration, monitoring, infra as code, nginx, albs, etc).
  - zinclozenge 5y ago
    Well, that and also a "nice" programmatic interface.
- ldiracdelta 5y ago
  In addition to ease, why would I, as a s/w engr, want to invest in learning your weird stack instead of learning or using an equivalent tech stack that is actually marketable? Learning a technology can be a huge investment. Why would I want to invest in a technology with not much future and not much ability to translate into another position should this gig go south.
Fiahil 5y ago
Exactly, plus all major cloud provider will happily host a kubernetes cluster for you if you ask them with your money. In a previous project, we managed more than 20 clusters on Azure (AKS) and AWS (EC2) with 4 people. The only tool that made this possible was kubernetes. I've been running my own kubernetes cluster on a Raspberry pi. Does my cat count as an engineering team?
- sandebert 5y ago
  Sounds like a useless use of cat.
- throwawayboise 5y ago
  I feel like I must be stupid because I've tried several times to set up k8s on in-house hardware and it has never worked. I've tried several different "recipes" and the networking never works right. IDK if it's because things are changing so fast that the popular "recipes" are already outdated, or if they are being published without actually testing them, but it's left a bad taste in my mouth. I'm sure the big cloud providers make it easy for end users to use, but that doesn't help me.
  - Okkef 5y ago
    Don't run your own cluster, unless you do it to learn. If you're in the cloud, just use a managed instance and focus on building software.
    - foobiekr 5y ago
      How do people who do this debug issues in the k8s layer? Or is this just the Jenga Model of Modern Software Development in action?
      - GauntletWizard 5y ago
        There's definitely a lot of Jenga going on here, but on the other hand: Kubernetes, when set up, has some very simple constraints that work. You don't often need to touch things at that layer; They become invisible, and when you're dealing with a cloud provider's hosted K8s, you don't get to touch them directly anyway. K8s was a lot more simple earlier on. It's actually dying from adoption: There's a ton of dumb ideas bolted on top, that have become "standard" and "supported" because of demands from bad customers. The core is very clean, though, and you rarely need to interact with that.
marcinzm 5y ago
Same. We've been running EKS at my last two companies using Terraform without issues and very small teams. There's a learning curve for sure but the ongoing maintenance is pretty minimal. I suspect going with GKE would have a smaller learning curve.
arwhatever 5y ago
Genuine question because I honestly have no idea: how much additional learning curve is added by the average cloud provider to create a Kubernetes hosting context/setup? Knowing nothing about K, I’m constantly wondering how it could be simpler than dumping a binary into a PaaS (web) hosting context and letting it do its thing. I’m interested to learn.
maxbreaker 5y ago
We used to run a very well run Kubernetes clusters with a single devops, then two. If you have senior devops engineers who know what they are doing and have the opportunity to greenfield it's really not that bad. We now have 4 devops engineers. At the end of the day it's not necessarily Kubernetes that's difficult/time consuming, it's all the context around it. How do you monitor/observability? Security? What's your CI/CD to deploy everything? How do you fully automate everything in reproducible ways within the k8s clusters, and outside? I've been doing infrastructure management / devops for well over 23 years. Built & managed datacenters running tens of thousands of servers. Kubernetes and AWS have made my life easier. What's really important is having the right senior people fully aware of the latest practices to get started. Greenfield now is easier than ever.
miraculixx 5y ago
I am running 5 k8s clusters on behalf of customers on 3 different cloud providers. By myself.

cassianoleal 5y ago

I've been on a team of 6-8 running relatively large-scale clusters (2 big ones with lower hundreds of workloads through tens of namespaces/tenants, plus a couple smaller ones). To "keep the Kubernetes clusters running" is an afterthought at most. I mostly agree with your comment.

RamRodification 5y ago
Do/Did you use some specific method or orchestration product to set it up?
- Sahbak 5y ago
  Not OP, but similar workloads, team of 5 We use kops to deplot and manage the clusters. The hardest part is updating, due to the workloads running on it. Other than that, little to no problems with kube itself.
- cassianoleal 5y ago
  We use GKE but there's nothing stopping most organisations to do the same. It doesn't bear much weight though, and I've had experience with other toolings largely to the same effect. I agree with the sibling comment about upgrades - this is IMO where GKE really shines as a cluster upgrade is mostly a button-pressing exercise.

theptip 5y ago

This comes up a lot around here; many people look at “Kubernetes the hard way” and think they need to run their own cluster. Just use GKE or whatever the equivalent managed offering is for your cloud provider. A couple of clicks in the ui and you have a cluster. Its really easy to run, I rarely have to look at the cluster at all.

heydonovan 5y ago

Yes! Was going to say the same. Kubernetes is far easier to learn than some random bespoke setup. After reading the article, it just sounds like they reinvented the wheel but made it AWS specific for some reason. Was brought on as a consultant years ago and found their bespoke setup of random ec2 instances and lambda scripts to be far more difficult to understand than just spinning up a managed Kubernetes cluster and having a generic interface to deploy the application, as well as monitoring, logging, metrics, etc.

cortesoft 5y ago
> Kubernetes is far easier to learn than some random bespoke setup This, to me, is the biggest advantage of kubernetes. Yes, you can do all the things yourself, with your own custom solution that does everything you need to do just as well as kubernetes. What kubernetes gives you is a shared way of doing things. By using a common method, you can easily integrate different services together, as well as onboard new hires easily. Using something ubiquitous like kubernetes helps both your code onboarding and your people onboarding.
- throwaway894345 5y ago
  Also not having to debug EC2 lifecycle issues is really nice. And you don't have to deal with all of the garbage of setting up ec2 instances (configuring SSH, process management, configuration management, log exfiltration, monitoring, infra as code, nginx, albs, etc).
- zinclozenge 5y ago
  Well, that and also a "nice" programmatic interface.
ldiracdelta 5y ago
In addition to ease, why would I, as a s/w engr, want to invest in learning your weird stack instead of learning or using an equivalent tech stack that is actually marketable? Learning a technology can be a huge investment. Why would I want to invest in a technology with not much future and not much ability to translate into another position should this gig go south.

Fiahil 5y ago

Exactly, plus all major cloud provider will happily host a kubernetes cluster for you if you ask them with your money. In a previous project, we managed more than 20 clusters on Azure (AKS) and AWS (EC2) with 4 people. The only tool that made this possible was kubernetes. I've been running my own kubernetes cluster on a Raspberry pi. Does my cat count as an engineering team?

sandebert 5y ago
Sounds like a useless use of cat.
throwawayboise 5y ago
I feel like I must be stupid because I've tried several times to set up k8s on in-house hardware and it has never worked. I've tried several different "recipes" and the networking never works right. IDK if it's because things are changing so fast that the popular "recipes" are already outdated, or if they are being published without actually testing them, but it's left a bad taste in my mouth. I'm sure the big cloud providers make it easy for end users to use, but that doesn't help me.
- Okkef 5y ago
  Don't run your own cluster, unless you do it to learn. If you're in the cloud, just use a managed instance and focus on building software.
  - foobiekr 5y ago
    How do people who do this debug issues in the k8s layer? Or is this just the Jenga Model of Modern Software Development in action?
    - GauntletWizard 5y ago
      There's definitely a lot of Jenga going on here, but on the other hand: Kubernetes, when set up, has some very simple constraints that work. You don't often need to touch things at that layer; They become invisible, and when you're dealing with a cloud provider's hosted K8s, you don't get to touch them directly anyway. K8s was a lot more simple earlier on. It's actually dying from adoption: There's a ton of dumb ideas bolted on top, that have become "standard" and "supported" because of demands from bad customers. The core is very clean, though, and you rarely need to interact with that.

marcinzm 5y ago

Same. We've been running EKS at my last two companies using Terraform without issues and very small teams. There's a learning curve for sure but the ongoing maintenance is pretty minimal. I suspect going with GKE would have a smaller learning curve.

arwhatever 5y ago

Genuine question because I honestly have no idea: how much additional learning curve is added by the average cloud provider to create a Kubernetes hosting context/setup? Knowing nothing about K, I’m constantly wondering how it could be simpler than dumping a binary into a PaaS (web) hosting context and letting it do its thing. I’m interested to learn.

maxbreaker 5y ago

We used to run a very well run Kubernetes clusters with a single devops, then two. If you have senior devops engineers who know what they are doing and have the opportunity to greenfield it's really not that bad. We now have 4 devops engineers. At the end of the day it's not necessarily Kubernetes that's difficult/time consuming, it's all the context around it. How do you monitor/observability? Security? What's your CI/CD to deploy everything? How do you fully automate everything in reproducible ways within the k8s clusters, and outside? I've been doing infrastructure management / devops for well over 23 years. Built & managed datacenters running tens of thousands of servers. Kubernetes and AWS have made my life easier. What's really important is having the right senior people fully aware of the latest practices to get started. Greenfield now is easier than ever.

miraculixx 5y ago

I am running 5 k8s clusters on behalf of customers on 3 different cloud providers. By myself.