7 ms·

Learning to operate Kubernetes reliably

357 points by mglukhovsky 9y ago

perfmode 9y ago
Why do you need a 99.99% from job completion rate? Why not just design for failure and inevitable retries? Almost seems like you grant platform users a false sense of security by making it very reliable but not perfect.
- hinkley 9y ago
  I’ll hazard a guess that this has to do with the fact that the work load is a set of scheduled tasks. Their customers expect the cron jobs to run when they expected and how they expected. With that constraint restarts look a lot less acceptable.
- sisk 9y ago
  My guess: because financial systems. A lot of traditional financial instruments 1) are not resilient to failure and 2) run at fixed times in batches. I’m confident it’s not their own systems that set the requirement of rigidity.
- YesThatTom2 9y ago
  How are those two things different?
KaiserPro 9y ago
Much as it burns me to admit this, for this usecase, jenkins is king. <60 nodes and its perfect. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user environment meant that we could execute as user and inherit their environment. It has sensible retry logic, and lots of hooks for all your hooking needs. pipelines are useful for chaining jobs together. We _could_ have written them as normal jobs to be run somewhere in the 36k node farm, but that was more hassle than its worth. Sure its fun, but having to contend with sharing a box that's doing a fluid sim or similar, so we'd have to carve off a section anyway. However kuberenetes to _just_ run cron is a massive waste. It smacks of shiny new tool syndrome. seriously jenkins is a single day deployment. transplanting the cron jobs is again less than a day (assuming your slaves have got a decent environment.) So, with the greatest of respect, talking about building a business case is pretty moot when you are effectively wasting what appears to be > two man months on what should be a week long migration. Think gaffer tape, not carbon fibre bonded to aluminium. If however, the rest of the platform lives on kuberenetes, then I could see the logic, having all your stuff running on one platform is very appealing, especially if you have invested time in translating comprehensive monitoring into business relevant alerts.
- hinkley 9y ago
  My current company keeps trying to cook up elaborate systems to keep certain deployments from happening while others are going on and I couldn’t recall ever having to solve this previously which is odd because of course this has been a problem before. Yeah I was using my CI system to handle the CD constraints and it was so straightforward it hardly registered as work. I was setting up one build agent with a custom property and all the builds that couldn’t run simultaneously would all require an agent with that property. So they just queued in chronological order of arrival. Done. Next problem.
- ironjunkie 9y ago
  Agree with the shiny new tool syndrome. Also remember this is Stripe, and they like to advertise through Engineering blogs (and they do that quite well to be honest). I'm getting cynical here, but I'm sometimes wondering if they didn't specifically chose a cool shiny tool, so that they can speak about it (and advertise through blogging)
- hinkley 9y ago
  I wonder if it's feasible or worthwhile for someone to try to extract the task and batch processing code from Jenkins into a separate project. Perhaps the analytics too. With a little work you could expand that out to make a travis equivalent using the same code base.
- lima 9y ago
  Red Hat's OpenShift has a very nice Jenkins <-> Kubernetes integration, too.
- jvns 9y ago
  Hi! Post author here! I agree that it's really important to be careful of "shiny new tool" syndrome -- one of my primary goals in writing this post was to show that operating Kubernetes in production is complicated and to encourage people to think carefully before introducing a Kubernetes cluster into their infrastructure. As you say -- I think by itself "we want to run some cron jobs" isn't a good enough reason by itself to use Kubernetes (though it might be a good enough reason if you’re using a managed Kubernetes cluster where someone else handles the cluster operations). A goal for this project was to prove to ourselves that we actually could run production code in Kubernetes, to learn about how much work operating Kubernetes actually is, and to lay the groundwork for moving more things to Kubernetes in the future. In my mind, a huge advantage of Kubernetes is that Kubernetes' code is very readable and they're great at accepting contributions. In the past when we've run into performance problems with Jenkins (we also use jenkins-job-builder to manage our 1k node Jenkins cluster), they've been extremely difficult to debug and it's hard to get visibility into what's going on inside Jenkins. I find Kubernetes’ code a lot easier to read, it's fairly easy to monitor the internals, and the core components have pprof included by default if you want to get profiling information out. Being able to easily fix bugs in Kubernetes and get the patches merged upstream has been a big deal for us.
minimaxir 9y ago
Kubernetes very recently added native Cronjob support: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ https://kubernetes.io/docs/concepts/workloads/controllers/cr... How does Stripe's approach differ?
- tarmstrong 9y ago
  No difference — we are using Kubernetes's native cronjob support. This post is about how we migrated to that system.
mephitix 9y ago
Setting aside the k8s content itself, I love the way this article is written. It's not a typical tutorial or tips/tricks but takes you time-traveling through the experience of a big company adopting nascent tech. Lot of great things to take away even outside of the kubernetes tips.
- unmole 9y ago
  Julia Evans is something of a celebrity. Her personal blog is an absolute gold mine: https://jvns.ca https://jvns.ca
robszumski 9y ago
> “Sometimes when we do an etcd failover, the API server starts timing out requests until we restart it.” This is likely related a set of Kubernetes bugs [1][2] (and grpc[3]) that CoreOS is working diligently to get fixed. The first set of these, the endpoint reconciler[4], has landed in 1.9. More work is pending on the etcd client in Kubernetes. The good news is that the client is used everywhere, so one fix and all components will benefit. [1]: https://github.com/kubernetes/community/pull/939 https://github.com/kubernetes/community/pull/939 [2]: https://github.com/kubernetes/kubernetes/issues/22609 https://github.com/kubernetes/kubernetes/issues/22609 [3]: https://github.com/kubernetes/kubernetes/issues/47131 https://github.com/kubernetes/kubernetes/issues/47131 [4]: https://github.com/kubernetes/kubernetes/pull/51698 https://github.com/kubernetes/kubernetes/pull/51698
- pishpash 9y ago
  I don't get this. Didn't Kubernetes come out of Google Borg that had been in use forever? The second write should be more elegant and impressive -- why so many basic bugs?
  - ecnahc515 9y ago
    The concepts are based on similar ideas in Borg, but the things it shares with Borg ends there. The implementation is effectively entirely from scratch, so bugs will exist.
  - rsanders 9y ago
    Kubernetes didn't come with all of the other Google infrastructure.
  - alpb 9y ago
    Kubernetes takes some concepts from Borg. A system like Borg would be very closely coupled to Google‘s infrastructure that there’s probably very little to open source from there without open sourcing the entire machinery. Also, any large scale system like Borg developed at a large company like Facebook or Google will have completely opinionated one-way-of-doing-things for a lot of aspects. This doesn’t work for the world outside where lots of developers from different backgrounds, lots of projects with different requirements exist.
asimpletune 9y ago
What is the benefit of using Kubernetes over Mesos (or in conjunction with Mesos)?
- gtaylor 9y ago
  The answer to that depends on your particular usage case and requirements. There is no simple, always-true response.
  - asimpletune 9y ago
    What are some examples of each?
- whalesalad 9y ago
  I cannot think of a single reason to choose Mesos over Kubernetes unless you are literally Twitter.
- vicaya 9y ago
  FTFA: "We’d previously been using Chronos (with Mesos) as a cron job scheduling system, but it was no longer meeting our reliability requirements and it’s mostly unmaintained (1 commit in the last 9 months, and the last time a pull request was merged was March 2016) Because Chronos is unmaintained, we decided it wasn’t worth continuing to invest in improving our existing cluster." Though Chronos has a release recently with a bunch of fixes, Mesos is inevitably fading as a legacy platform.
  - asimpletune 9y ago
    > Mesos is inevitably fading as a legacy platform. Because of Chronos? This is a bizarre thing to say. Mesos actually works extremely well. Whenever I ask the why kube over Mesos question, I never get a good answer. I think because people just don’t know Mesos. Also it wasn’t made by google.
    - vicaya 9y ago
YesThatTom2 9y ago
Such good writing style AND useful technical content. Why can't all blog posts be this good?
- nindalf 9y ago
  The author writes regularly and her posts almost always reach the top of HN. Like most skills, improvement comes with practice. If a person is willing to put in the same time and effort as jvns has, I'm sure they would be rewarded with similar results.
ad_hominem 9y ago
How do you deal with sidecar containers in CronJobs (and regular batch Jobs) not terminating correctly? https://github.com/kubernetes/kubernetes/issues/25908 https://github.com/kubernetes/kubernetes/issues/25908
- jvns 9y ago
  We don't run sidecar containers in cron jobs yet. That said, here's a workaround (from that issue): https://github.com/kubernetes/kubernetes/issues/25908#issuecomment-308569672 https://github.com/kubernetes/kubernetes/issues/25908#issuec...
  - ad_hominem 9y ago
    I'm aware of the workarounds in that thread. Just wondering if Stripe had a different workaround but I guess not.
    - jmillikin 9y ago
      That GitHub comment is Stripe's workaround! I copied it nearly as-is from our internal job setup boilerplate.
scarface74 9y ago
I'm curious about what people think about HashiCorp's Nad bs Kubernetes. I chose Nomad because I'm already using Consul and I wanted to run raw .Net executables. Would it have been worth it to use Docker with .Net Core? Not trying to change my infrastructure now, but just curious about whether it is worth the time to play with it on the side.
- wmf 9y ago
  Nomad appears to be better designed, more scalable, and easier to operate than k8s, but it will fall behind pretty rapidly since k8s has 100x more developers.
  - pm90 9y ago
    That isn't necessarily true (playing devils advocate): OpenStack had gajillion developers and still failed (mostly). Although k8s does seem to be designed much better. I use it personally too and hope for its success.
djsumdog 9y ago
I haven't been at a k8s shop yet, but at my last job we used Marathon (on DC/OS). I know you can run Kubernetes on DC/OS, but the default scheduler it comes with is Marathon. Is there an advantage to one over the other? It looks like in both cases, you need a platform team (at least 2, maybe 3 people; we had a large complex setup and had like 10) to setup things like K8s, DC/OS or Nomad, because they are complex systems with a lot of different components .. components like Flanel vs Weavenet vs some other container networks, handling storage volumes, labels and automatic configuration of HAProxy from them (marathon-lb on DC/OS). All schedulers (k8s, swarm, marathon) seems to use a json format for job information that's pretty specific, not only to the scheduler, but to the way other tooling is setup at your specific shop.
alexebird 9y ago
I always search for mentions of Hashicorp Nomad in the comments section of front-page Kubernetes articles like this. There are often few or no mentions, so I’d like to add a plug for the Hashistack. For some reason Nomad seems to get noticeably less publicity than some of the other Hashicorp offerings like Consul, Vault, and Terraform. In my opinion Nomad is right up there with them. The documentation is excellent. I haven’t had to fix any upstream issues in about a year of development on two separate Nomad clusters. Upgrading versions live is straightforward, and I rarely find myself in a situation where I can’t accomplish something I envisioned because Nomad is missing a feature. It schedules batch jobs, cron jobs, long running services, and system services that run on every node. It has a variety of job drivers outside of Docker. Nomad, Consul, Vault, and the Consul-aware Fabio load balancer run together to form most of what one might need for a cluster scheduler based deployment, somewhat reminiscent of the “do one thing well” Unix philosophy of composability. Certainly it isn’t perfect, but I’d recommend it to anyone who is considering using a cluster scheduler but is apprehensive about the operational complexity of the more widely discussed options such as Kubernetes.
- SahAssar 9y ago
  What's your take on Fabio vs. Traefik? I had not heard of Fabio before, but they seem to support a similar featureset.
- toong 9y ago
  Catch 22: the lack of traction/adoption is the main point that stops me from exploring it more. I would have to put so much effort in convincing customers and management to not go the (now almost default?) Kubernetes-route, that it's risky trying something else. A small hiccup in Nomad, would be enough for the pitchforks to come out.
- akvadrako 9y ago
  I never heard of Nomad, but I can't see why I would choose it over the much more popular and standardised k8s. The biggest benefits seem to be (1) simplicity, but GCE and minikube are easy enough to learn in a day and (2) ability to run non-containers, but docker containers are generic - they can run java apps just fine.

357 points by mglukhovsky 9y ago

Why do you need a 99.99% from job completion rate? Why not just design for failure and inevitable retries? Almost seems like you grant platform users a false sense of security by making it very reliable but not perfect.

hinkley 9y ago
I’ll hazard a guess that this has to do with the fact that the work load is a set of scheduled tasks. Their customers expect the cron jobs to run when they expected and how they expected. With that constraint restarts look a lot less acceptable.
sisk 9y ago
My guess: because financial systems. A lot of traditional financial instruments 1) are not resilient to failure and 2) run at fixed times in batches. I’m confident it’s not their own systems that set the requirement of rigidity.
YesThatTom2 9y ago
How are those two things different?

KaiserPro 9y ago

Much as it burns me to admit this, for this usecase, jenkins is king. <60 nodes and its perfect. At previous job, we had migrated from a nasty cron orchestration system to jenkins. It did a number of things including building software, batch generating thumbnails and moving data about on around 30 nodes, of which about 25 were fungible. Jenkins job builder meant that everything was defined in yaml, stored in git and was repeatable. A sane user environment meant that we could execute as user and inherit their environment. It has sensible retry logic, and lots of hooks for all your hooking needs. pipelines are useful for chaining jobs together. We _could_ have written them as normal jobs to be run somewhere in the 36k node farm, but that was more hassle than its worth. Sure its fun, but having to contend with sharing a box that's doing a fluid sim or similar, so we'd have to carve off a section anyway. However kuberenetes to _just_ run cron is a massive waste. It smacks of shiny new tool syndrome. seriously jenkins is a single day deployment. transplanting the cron jobs is again less than a day (assuming your slaves have got a decent environment.) So, with the greatest of respect, talking about building a business case is pretty moot when you are effectively wasting what appears to be > two man months on what should be a week long migration. Think gaffer tape, not carbon fibre bonded to aluminium. If however, the rest of the platform lives on kuberenetes, then I could see the logic, having all your stuff running on one platform is very appealing, especially if you have invested time in translating comprehensive monitoring into business relevant alerts.

hinkley 9y ago
My current company keeps trying to cook up elaborate systems to keep certain deployments from happening while others are going on and I couldn’t recall ever having to solve this previously which is odd because of course this has been a problem before. Yeah I was using my CI system to handle the CD constraints and it was so straightforward it hardly registered as work. I was setting up one build agent with a custom property and all the builds that couldn’t run simultaneously would all require an agent with that property. So they just queued in chronological order of arrival. Done. Next problem.
ironjunkie 9y ago
Agree with the shiny new tool syndrome. Also remember this is Stripe, and they like to advertise through Engineering blogs (and they do that quite well to be honest). I'm getting cynical here, but I'm sometimes wondering if they didn't specifically chose a cool shiny tool, so that they can speak about it (and advertise through blogging)
hinkley 9y ago
I wonder if it's feasible or worthwhile for someone to try to extract the task and batch processing code from Jenkins into a separate project. Perhaps the analytics too. With a little work you could expand that out to make a travis equivalent using the same code base.
lima 9y ago
Red Hat's OpenShift has a very nice Jenkins <-> Kubernetes integration, too.
jvns 9y ago
Hi! Post author here! I agree that it's really important to be careful of "shiny new tool" syndrome -- one of my primary goals in writing this post was to show that operating Kubernetes in production is complicated and to encourage people to think carefully before introducing a Kubernetes cluster into their infrastructure. As you say -- I think by itself "we want to run some cron jobs" isn't a good enough reason by itself to use Kubernetes (though it might be a good enough reason if you’re using a managed Kubernetes cluster where someone else handles the cluster operations). A goal for this project was to prove to ourselves that we actually could run production code in Kubernetes, to learn about how much work operating Kubernetes actually is, and to lay the groundwork for moving more things to Kubernetes in the future. In my mind, a huge advantage of Kubernetes is that Kubernetes' code is very readable and they're great at accepting contributions. In the past when we've run into performance problems with Jenkins (we also use jenkins-job-builder to manage our 1k node Jenkins cluster), they've been extremely difficult to debug and it's hard to get visibility into what's going on inside Jenkins. I find Kubernetes’ code a lot easier to read, it's fairly easy to monitor the internals, and the core components have pprof included by default if you want to get profiling information out. Being able to easily fix bugs in Kubernetes and get the patches merged upstream has been a big deal for us.

minimaxir 9y ago

Kubernetes very recently added native Cronjob support: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ https://kubernetes.io/docs/concepts/workloads/controllers/cr... How does Stripe's approach differ?

tarmstrong 9y ago
No difference — we are using Kubernetes's native cronjob support. This post is about how we migrated to that system.

mephitix 9y ago

Setting aside the k8s content itself, I love the way this article is written. It's not a typical tutorial or tips/tricks but takes you time-traveling through the experience of a big company adopting nascent tech. Lot of great things to take away even outside of the kubernetes tips.

unmole 9y ago
Julia Evans is something of a celebrity. Her personal blog is an absolute gold mine: https://jvns.ca https://jvns.ca

robszumski 9y ago

> “Sometimes when we do an etcd failover, the API server starts timing out requests until we restart it.” This is likely related a set of Kubernetes bugs [1][2] (and grpc[3]) that CoreOS is working diligently to get fixed. The first set of these, the endpoint reconciler[4], has landed in 1.9. More work is pending on the etcd client in Kubernetes. The good news is that the client is used everywhere, so one fix and all components will benefit. [1]: https://github.com/kubernetes/community/pull/939 https://github.com/kubernetes/community/pull/939 [2]: https://github.com/kubernetes/kubernetes/issues/22609 https://github.com/kubernetes/kubernetes/issues/22609 [3]: https://github.com/kubernetes/kubernetes/issues/47131 https://github.com/kubernetes/kubernetes/issues/47131 [4]: https://github.com/kubernetes/kubernetes/pull/51698 https://github.com/kubernetes/kubernetes/pull/51698

pishpash 9y ago
I don't get this. Didn't Kubernetes come out of Google Borg that had been in use forever? The second write should be more elegant and impressive -- why so many basic bugs?
- ecnahc515 9y ago
  The concepts are based on similar ideas in Borg, but the things it shares with Borg ends there. The implementation is effectively entirely from scratch, so bugs will exist.
- rsanders 9y ago
  Kubernetes didn't come with all of the other Google infrastructure.
- alpb 9y ago
  Kubernetes takes some concepts from Borg. A system like Borg would be very closely coupled to Google‘s infrastructure that there’s probably very little to open source from there without open sourcing the entire machinery. Also, any large scale system like Borg developed at a large company like Facebook or Google will have completely opinionated one-way-of-doing-things for a lot of aspects. This doesn’t work for the world outside where lots of developers from different backgrounds, lots of projects with different requirements exist.

asimpletune 9y ago

What is the benefit of using Kubernetes over Mesos (or in conjunction with Mesos)?

gtaylor 9y ago
The answer to that depends on your particular usage case and requirements. There is no simple, always-true response.
- asimpletune 9y ago
  What are some examples of each?
whalesalad 9y ago
I cannot think of a single reason to choose Mesos over Kubernetes unless you are literally Twitter.
vicaya 9y ago
FTFA: "We’d previously been using Chronos (with Mesos) as a cron job scheduling system, but it was no longer meeting our reliability requirements and it’s mostly unmaintained (1 commit in the last 9 months, and the last time a pull request was merged was March 2016) Because Chronos is unmaintained, we decided it wasn’t worth continuing to invest in improving our existing cluster." Though Chronos has a release recently with a bunch of fixes, Mesos is inevitably fading as a legacy platform.
- asimpletune 9y ago
  > Mesos is inevitably fading as a legacy platform. Because of Chronos? This is a bizarre thing to say. Mesos actually works extremely well. Whenever I ask the why kube over Mesos question, I never get a good answer. I think because people just don’t know Mesos. Also it wasn’t made by google.
  - vicaya 9y ago

YesThatTom2 9y ago

Such good writing style AND useful technical content. Why can't all blog posts be this good?

nindalf 9y ago
The author writes regularly and her posts almost always reach the top of HN. Like most skills, improvement comes with practice. If a person is willing to put in the same time and effort as jvns has, I'm sure they would be rewarded with similar results.

ad_hominem 9y ago

How do you deal with sidecar containers in CronJobs (and regular batch Jobs) not terminating correctly? https://github.com/kubernetes/kubernetes/issues/25908 https://github.com/kubernetes/kubernetes/issues/25908

jvns 9y ago
We don't run sidecar containers in cron jobs yet. That said, here's a workaround (from that issue): https://github.com/kubernetes/kubernetes/issues/25908#issuecomment-308569672 https://github.com/kubernetes/kubernetes/issues/25908#issuec...
- ad_hominem 9y ago
  I'm aware of the workarounds in that thread. Just wondering if Stripe had a different workaround but I guess not.
  - jmillikin 9y ago
    That GitHub comment is Stripe's workaround! I copied it nearly as-is from our internal job setup boilerplate.

scarface74 9y ago

I'm curious about what people think about HashiCorp's Nad bs Kubernetes. I chose Nomad because I'm already using Consul and I wanted to run raw .Net executables. Would it have been worth it to use Docker with .Net Core? Not trying to change my infrastructure now, but just curious about whether it is worth the time to play with it on the side.

wmf 9y ago
Nomad appears to be better designed, more scalable, and easier to operate than k8s, but it will fall behind pretty rapidly since k8s has 100x more developers.
- pm90 9y ago
  That isn't necessarily true (playing devils advocate): OpenStack had gajillion developers and still failed (mostly). Although k8s does seem to be designed much better. I use it personally too and hope for its success.

djsumdog 9y ago

I haven't been at a k8s shop yet, but at my last job we used Marathon (on DC/OS). I know you can run Kubernetes on DC/OS, but the default scheduler it comes with is Marathon. Is there an advantage to one over the other? It looks like in both cases, you need a platform team (at least 2, maybe 3 people; we had a large complex setup and had like 10) to setup things like K8s, DC/OS or Nomad, because they are complex systems with a lot of different components .. components like Flanel vs Weavenet vs some other container networks, handling storage volumes, labels and automatic configuration of HAProxy from them (marathon-lb on DC/OS). All schedulers (k8s, swarm, marathon) seems to use a json format for job information that's pretty specific, not only to the scheduler, but to the way other tooling is setup at your specific shop.

alexebird 9y ago

I always search for mentions of Hashicorp Nomad in the comments section of front-page Kubernetes articles like this. There are often few or no mentions, so I’d like to add a plug for the Hashistack. For some reason Nomad seems to get noticeably less publicity than some of the other Hashicorp offerings like Consul, Vault, and Terraform. In my opinion Nomad is right up there with them. The documentation is excellent. I haven’t had to fix any upstream issues in about a year of development on two separate Nomad clusters. Upgrading versions live is straightforward, and I rarely find myself in a situation where I can’t accomplish something I envisioned because Nomad is missing a feature. It schedules batch jobs, cron jobs, long running services, and system services that run on every node. It has a variety of job drivers outside of Docker. Nomad, Consul, Vault, and the Consul-aware Fabio load balancer run together to form most of what one might need for a cluster scheduler based deployment, somewhat reminiscent of the “do one thing well” Unix philosophy of composability. Certainly it isn’t perfect, but I’d recommend it to anyone who is considering using a cluster scheduler but is apprehensive about the operational complexity of the more widely discussed options such as Kubernetes.

SahAssar 9y ago
What's your take on Fabio vs. Traefik? I had not heard of Fabio before, but they seem to support a similar featureset.
toong 9y ago
Catch 22: the lack of traction/adoption is the main point that stops me from exploring it more. I would have to put so much effort in convincing customers and management to not go the (now almost default?) Kubernetes-route, that it's risky trying something else. A small hiccup in Nomad, would be enough for the pitchforks to come out.
akvadrako 9y ago
I never heard of Nomad, but I can't see why I would choose it over the much more popular and standardised k8s. The biggest benefits seem to be (1) simplicity, but GCE and minikube are easy enough to learn in a day and (2) ability to run non-containers, but docker containers are generic - they can run java apps just fine.