7 ms·

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm

by DDub 9y ago

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

MattRogish 9y ago
We (ReactiveOps) use a combination of CircleCI and some scripts wrapping kubectl https://github.com/reactiveops/rok8s-scripts https://github.com/reactiveops/rok8s-scripts
jameskilton 9y ago
I've got a simple setup that makes use of YAML files, Rake tasks, and raw kubectl. I've yet to take a look at helm or spinnaker but it's on my list. You really can go a long way just with K8s' own tooling.
- jomakasi 9y ago
  I've started with creating Rake tasks. Then I found https://github.com/CommercialTribe/psykube https://github.com/CommercialTribe/psykube which is opinionated. Then I've ended up creating my own simple ruby tool to manage kubernetes with my own directory structure and configuration.
- danielodio 9y ago
  We've written a bunch about Spinnaker & K8s at http://go.Armory.io/kubernetes http://go.Armory.io/kubernetes ; hope that's helpful!
foxylion 9y ago
We also did some evaluation and then decided to stick to KISS an chose kubectl commands combined with cat and kexpand. Really simple approach to allow dynamic kubernetes deployments. Example command can be cat service.yml | kexpand expand -v image-tag=git-135afed4 | kubectl apply -f - The service.yml contains the full deployment configuration, service definition and ingress rules. So this works without preconfiguring anything in kubernetes when deploying an new service. An engineer only has to create the service.yml and Jenkins does deploy it automatically on every master build. *kexpand is a small tool which does something similar to sed, but in a simpler and less powerful way (keep it simple): https://github.com/kopeio/kexpand https://github.com/kopeio/kexpand
- nikon 9y ago
  I do something similar, but envsubst does the job.
  - ff_ 9y ago
    +1 on envsubst, it's the minimal solution to the problem of templating kubernetes (of course YMMV, we are a small team and don't need more complex stuff)
  - brianwawok 9y ago
    Ditto me too to get my git sha into my image names and deploy with deployments.
- howinator 9y ago
  We basically did the same thing except we used Ansible for our templating. This allows us to store all our shared "environment configuration data", e.g., name of RDS for services in prod environment, name of backup S3 bucket for services in dev environment, in an Ansible role then just pull that information into our templated deployment manifest file. So far, it's worked out pretty well for us.
deadmik3 9y ago
I'd recommend looking into openshift. it's basically kubectl + cool deployment features. there's also free, paid, and dedicated online hosted options. disclaimer: I work on openshift
- awill 9y ago
  You had a big announcement about openshift.io. Everyone on HN signed up, but it's been months, but I'm still 'awaiting approval' Have you let anyone in? What's the value in all the marketing hype if you then don't let people in.
  - yebyen 9y ago
    Seconded. The last e-mail I got from OpenShift.io indicated they haven't let anyone in. I've already practically lost interest. It was evidently only "coming soon" but that announcement really looked like "coming tomorrow." The original OpenShift Developer Preview made you sign up, but you would be allowed into the platform within hours or days.
    - deadmik3 9y ago
      OpenShift.io is a different product than hosted OpenShift on its own. If you were to sign up for the Pro tier at openshift.com your account would be immediately provisioned. We are working on provisioning enqueued users for our Starter tier as well but again, these are separate from OpenShift.io
      - yebyen 9y ago
        Yeah, OpenShift Pro is also quite expensive... starting at $10k for 5 nodes right? I just checked out your pricing page and I'm 100% wrong about this. You have a $50/mo tier now! That's fantastic, thanks for pointing me at that. You should definitely spam everyone on the OpenShift.io waiting list and let us know about your new pricing /s No seriously, I would have liked to get some spam about this. Maybe you sent it and I missed it. I am a lot more interested at $50/mo than I would have been at $10k/yr!
bryanlarsen 9y ago
We're using helm, but that was chosen mostly based on gut feel. It's an official project, and has momentum. We didn't want to spend too much time choosing a tool until we knew our requirements, and we don't really have a firm grasp of requirements until we've used something for a while. It seems to be working well for us so far, but it's still early. There are lots of answers here that aren't helm, so I'm curious if there are any particular reasons that people ruled out helm?
- nikon 9y ago
  Are you doing what Gitlab claims below "We (GitLab) use it mostly as a templating system as well" or really using it to manage complex apps? I think it doesn't really offer much for normal microservices. I do use it for things like nginx ingress but for stuff I've built a service.yml / deployment.yml are fine.
- henridf 9y ago
  We ran into a number of issues with Helm when deploying - failures leading us to have to rollback, with rollbacks then failing, requiring manual changes to unblock. I think that for third-party packages and related templating (which seems like the original use-case) it works well, but I would be wary of using it for high-res deploys of our own stuff.
yebyen 9y ago
You should take a look at Deis Workflow. I say this in spite of the fact that it was announced last week[1], the next release of Deis Workflow will be the last (under the current stewardship, and probably under that name.) It's just such a solid system, I would even more strongly recommend the (already EOL'ed early last year)[2] Deis v1 PaaS, except that you've already indicated you're moving to K8S, and Deis v2 is designed for K8S. I still recommend the v1 PaaS for people learning about principles of HA clusters. (Another disclosure: I have published[3] about how to do this, a work on how to do a cheap HA cluster using Deis v1 PaaS.) I have a strong suspicion that Deis will live on after March under stewardship of new leadership from the community. In the mean time, you have roughly 6 months of support from Microsoft, maybe I am overstating to say that they have committed to keeping the lights on for that long, but they have committed to merging critical fixes for that long (and we hope that in 6 months, Kubernetes will have solidified enough that we don't have to worry too much about breaking changes from upstream release mongers anymore.) Personally I don't buy commercial support and it would not be the deal maker or breaker for me. [1]: https://deis.com/blog/2017/deis-workflow-final-release/#future https://deis.com/blog/2017/deis-workflow-final-release/#futu... [2]: https://deis.com/blog/2016/deis-1-13-lts/ https://deis.com/blog/2016/deis-1-13-lts/ [3]: https://deis.com/blog/2016/cheapest-fault-tolerant-cluster-deis/ https://deis.com/blog/2016/cheapest-fault-tolerant-cluster-d...
- gtirloni 9y ago
  I'd advise against choosing core infrastructure components (that have a clear EOL deadline) based on strong suspicion. Even more so in a landscape that's constantly changing like Kubernetes. You have zero guarantees that it'll be maintained and will keep up with new breaking changes.
  - yebyen 9y ago
    You know it's open source, right? I have zero guarantees that any of my open source projects that I use for business critical infrastructure aren't going to pack up shop and quit maintaining their stuff tomorrow. You should know how your infrastructure works well enough to maintain it for yourself. I (personally) will be maintaining this one in the future, if necessary! We're working it out now. What do you mean by "strong suspicion?" Please don't downvote because you read a few words you didn't like, I was upfront about this EOL date because I don't want it coming back later that I was dishonest about it, but my perception is not that "EOL" means it's dead, it is that "EOL" means it's done. Stability is a good thing. Microsoft also EOL'ed MSPaint.exe, and I remember how the community reacted. I think the quote was about "works for 99% of users and has been stable for over a decade? sounds like a good candidate for deletion!" The project is cancelled because it's not strategically important to Microsoft, not because it's not viable or having technical issues. The core devs have chosen to work on more kubernetes-native tooling. They aren't abandoning Kubernetes, and I'll bet you don't have a competing product you can show me that has guaranteed to keep the lights on for the next 6 months.
dguaraglia 9y ago
I feel you. About a month ago I was fighting with the same feeling. In the end, decided to use Kubernetes only for a single piece of infrastructure so it's all pretty manageable through scripts. Managing secrets in particular is a pain in the ass. One route I started checking but didn't commit to was using Ansible. They have a relatively good Kubernetes playbook and a facility to store secrets. That said, every damn task needs to be pointed to the K8S API endpoint, which is not the greatest.
- smarterclayton 9y ago
  Agree - we've been talking about how we can more natively tie the inventory into clusters, contexts, and apps. The host focus of Ansible doesn't always map to other domains, but I think it has a real chance with Kube.
lobster_johnson 9y ago
We wrote an internal tool that wraps Helm and GPG. But we're really using Helm as a glorified templating system; since we deploy from git, Helm's release management is useless to us, and is even somewhat in the way. We might decide to drop Helm at some point, I think.
- xur17 9y ago
  We went down a similar path and ended up using helm-template [0] to render our helm charts without tiller. We also use an internal tool that: - maps applications to namespaces within clusters for different environments (since we have 1 cluster per environment) - does some magic around secrets to make them easier to interface with [0] https://github.com/technosophos/helm-template https://github.com/technosophos/helm-template
- Snappy 9y ago
  That's a good point. We (GitLab) use it mostly as a templating system as well. It's a step up from piping `sed` output to `kubectl`. But we have our own tools for managing redeploys and rollbacks.
sytse 9y ago
If you're using our GitLab consider using Auto Deploy. Our CTO recently made a quick start guide for it https://docs.gitlab.com/ee/ci/autodeploy/quick_start_guide.html https://docs.gitlab.com/ee/ci/autodeploy/quick_start_guide.h...
- Snappy 9y ago
  FWIW, the next version of GitLab's Auto Deploy will use Helm under the hood (and let you bring-your-own-chart).
majewsky 9y ago
At SAP, we're using Helm [1] to deploy OpenStack (plus a bunch of extra services like a custom Rails-based dashboard [2]) on baremetal Kubernetes. For image building, testing and deployment, we use Concourse CI [3], and OpenStack assets (like service users, service projects and roles) are deployed with a custom Kubernetes operator [4]. Our charts are at [5] if you want to take a look. [1] https://github.com/kubernetes/helm https://github.com/kubernetes/helm [2] https://github.com/sapcc/elektra https://github.com/sapcc/elektra [3] https://concourse.ci https://concourse.ci [4] https://github.com/sapcc/kubernetes-operators https://github.com/sapcc/kubernetes-operators in the "openstack-operator" directory [5] https://github.com/sapcc/openstack-helm https://github.com/sapcc/openstack-helm and https://github.com/sapcc/helm-charts https://github.com/sapcc/helm-charts (two different repos since we're in the middle of modularizing the original monolithic OpenStack chart in the first repo into per-service charts in the second one)
invisible 9y ago
We are using ecfg (from Shopify) and Jenkins+kubectl. We use ansible for a couple of things but it's largely only because of some parameters of our architecture for a legacy monolith.
theptip 9y ago
A lot of folks are using Helm, but I find it very opaque to debug when templates go wrong (and I feel quite strongly that we shouldn't be writing untyped templates for our models). Also I found writing reusable spec components to be very difficult, e.g. a reverse proxy that I add to a number of pods. I use pykube (also worth looking at the incubator project client-python) to write deploy scripts in Python; client-python is particularly nice as it uses OpenAPI specs to give you type hinting on the API objects/fields that you're writing. Much more civilized than typing bare yaml into a text editor. If Python isn't your thing you can generate your own client from the OpenAPI specs, though I've found the client generation process to be a bit buggy.
rubenv 9y ago
I recently spoke about the approach we use at Ticketmatic: https://rocketeer.be/articles/coreos-fest-2017/ https://rocketeer.be/articles/coreos-fest-2017/
- kalev 9y ago
  That must have been an awesome talk. Thanks for the write-up!

by DDub 9y ago

MattRogish 9y ago

We (ReactiveOps) use a combination of CircleCI and some scripts wrapping kubectl https://github.com/reactiveops/rok8s-scripts https://github.com/reactiveops/rok8s-scripts

jameskilton 9y ago

I've got a simple setup that makes use of YAML files, Rake tasks, and raw kubectl. I've yet to take a look at helm or spinnaker but it's on my list. You really can go a long way just with K8s' own tooling.

jomakasi 9y ago
I've started with creating Rake tasks. Then I found https://github.com/CommercialTribe/psykube https://github.com/CommercialTribe/psykube which is opinionated. Then I've ended up creating my own simple ruby tool to manage kubernetes with my own directory structure and configuration.
danielodio 9y ago
We've written a bunch about Spinnaker & K8s at http://go.Armory.io/kubernetes http://go.Armory.io/kubernetes ; hope that's helpful!

foxylion 9y ago

We also did some evaluation and then decided to stick to KISS an chose kubectl commands combined with cat and kexpand. Really simple approach to allow dynamic kubernetes deployments. Example command can be cat service.yml | kexpand expand -v image-tag=git-135afed4 | kubectl apply -f - The service.yml contains the full deployment configuration, service definition and ingress rules. So this works without preconfiguring anything in kubernetes when deploying an new service. An engineer only has to create the service.yml and Jenkins does deploy it automatically on every master build. *kexpand is a small tool which does something similar to sed, but in a simpler and less powerful way (keep it simple): https://github.com/kopeio/kexpand https://github.com/kopeio/kexpand

nikon 9y ago
I do something similar, but envsubst does the job.
- ff_ 9y ago
  +1 on envsubst, it's the minimal solution to the problem of templating kubernetes (of course YMMV, we are a small team and don't need more complex stuff)
- brianwawok 9y ago
  Ditto me too to get my git sha into my image names and deploy with deployments.
howinator 9y ago
We basically did the same thing except we used Ansible for our templating. This allows us to store all our shared "environment configuration data", e.g., name of RDS for services in prod environment, name of backup S3 bucket for services in dev environment, in an Ansible role then just pull that information into our templated deployment manifest file. So far, it's worked out pretty well for us.

deadmik3 9y ago

I'd recommend looking into openshift. it's basically kubectl + cool deployment features. there's also free, paid, and dedicated online hosted options. disclaimer: I work on openshift

awill 9y ago
You had a big announcement about openshift.io. Everyone on HN signed up, but it's been months, but I'm still 'awaiting approval' Have you let anyone in? What's the value in all the marketing hype if you then don't let people in.
- yebyen 9y ago
  Seconded. The last e-mail I got from OpenShift.io indicated they haven't let anyone in. I've already practically lost interest. It was evidently only "coming soon" but that announcement really looked like "coming tomorrow." The original OpenShift Developer Preview made you sign up, but you would be allowed into the platform within hours or days.
  - deadmik3 9y ago
    OpenShift.io is a different product than hosted OpenShift on its own. If you were to sign up for the Pro tier at openshift.com your account would be immediately provisioned. We are working on provisioning enqueued users for our Starter tier as well but again, these are separate from OpenShift.io
    - yebyen 9y ago
      Yeah, OpenShift Pro is also quite expensive... starting at $10k for 5 nodes right? I just checked out your pricing page and I'm 100% wrong about this. You have a $50/mo tier now! That's fantastic, thanks for pointing me at that. You should definitely spam everyone on the OpenShift.io waiting list and let us know about your new pricing /s No seriously, I would have liked to get some spam about this. Maybe you sent it and I missed it. I am a lot more interested at $50/mo than I would have been at $10k/yr!

bryanlarsen 9y ago

We're using helm, but that was chosen mostly based on gut feel. It's an official project, and has momentum. We didn't want to spend too much time choosing a tool until we knew our requirements, and we don't really have a firm grasp of requirements until we've used something for a while. It seems to be working well for us so far, but it's still early. There are lots of answers here that aren't helm, so I'm curious if there are any particular reasons that people ruled out helm?

nikon 9y ago
Are you doing what Gitlab claims below "We (GitLab) use it mostly as a templating system as well" or really using it to manage complex apps? I think it doesn't really offer much for normal microservices. I do use it for things like nginx ingress but for stuff I've built a service.yml / deployment.yml are fine.
henridf 9y ago
We ran into a number of issues with Helm when deploying - failures leading us to have to rollback, with rollbacks then failing, requiring manual changes to unblock. I think that for third-party packages and related templating (which seems like the original use-case) it works well, but I would be wary of using it for high-res deploys of our own stuff.

yebyen 9y ago

You should take a look at Deis Workflow. I say this in spite of the fact that it was announced last week[1], the next release of Deis Workflow will be the last (under the current stewardship, and probably under that name.) It's just such a solid system, I would even more strongly recommend the (already EOL'ed early last year)[2] Deis v1 PaaS, except that you've already indicated you're moving to K8S, and Deis v2 is designed for K8S. I still recommend the v1 PaaS for people learning about principles of HA clusters. (Another disclosure: I have published[3] about how to do this, a work on how to do a cheap HA cluster using Deis v1 PaaS.) I have a strong suspicion that Deis will live on after March under stewardship of new leadership from the community. In the mean time, you have roughly 6 months of support from Microsoft, maybe I am overstating to say that they have committed to keeping the lights on for that long, but they have committed to merging critical fixes for that long (and we hope that in 6 months, Kubernetes will have solidified enough that we don't have to worry too much about breaking changes from upstream release mongers anymore.) Personally I don't buy commercial support and it would not be the deal maker or breaker for me. [1]: https://deis.com/blog/2017/deis-workflow-final-release/#future https://deis.com/blog/2017/deis-workflow-final-release/#futu... [2]: https://deis.com/blog/2016/deis-1-13-lts/ https://deis.com/blog/2016/deis-1-13-lts/ [3]: https://deis.com/blog/2016/cheapest-fault-tolerant-cluster-deis/ https://deis.com/blog/2016/cheapest-fault-tolerant-cluster-d...

gtirloni 9y ago
I'd advise against choosing core infrastructure components (that have a clear EOL deadline) based on strong suspicion. Even more so in a landscape that's constantly changing like Kubernetes. You have zero guarantees that it'll be maintained and will keep up with new breaking changes.
- yebyen 9y ago
  You know it's open source, right? I have zero guarantees that any of my open source projects that I use for business critical infrastructure aren't going to pack up shop and quit maintaining their stuff tomorrow. You should know how your infrastructure works well enough to maintain it for yourself. I (personally) will be maintaining this one in the future, if necessary! We're working it out now. What do you mean by "strong suspicion?" Please don't downvote because you read a few words you didn't like, I was upfront about this EOL date because I don't want it coming back later that I was dishonest about it, but my perception is not that "EOL" means it's dead, it is that "EOL" means it's done. Stability is a good thing. Microsoft also EOL'ed MSPaint.exe, and I remember how the community reacted. I think the quote was about "works for 99% of users and has been stable for over a decade? sounds like a good candidate for deletion!" The project is cancelled because it's not strategically important to Microsoft, not because it's not viable or having technical issues. The core devs have chosen to work on more kubernetes-native tooling. They aren't abandoning Kubernetes, and I'll bet you don't have a competing product you can show me that has guaranteed to keep the lights on for the next 6 months.

dguaraglia 9y ago

I feel you. About a month ago I was fighting with the same feeling. In the end, decided to use Kubernetes only for a single piece of infrastructure so it's all pretty manageable through scripts. Managing secrets in particular is a pain in the ass. One route I started checking but didn't commit to was using Ansible. They have a relatively good Kubernetes playbook and a facility to store secrets. That said, every damn task needs to be pointed to the K8S API endpoint, which is not the greatest.

smarterclayton 9y ago
Agree - we've been talking about how we can more natively tie the inventory into clusters, contexts, and apps. The host focus of Ansible doesn't always map to other domains, but I think it has a real chance with Kube.

lobster_johnson 9y ago

We wrote an internal tool that wraps Helm and GPG. But we're really using Helm as a glorified templating system; since we deploy from git, Helm's release management is useless to us, and is even somewhat in the way. We might decide to drop Helm at some point, I think.

xur17 9y ago
We went down a similar path and ended up using helm-template [0] to render our helm charts without tiller. We also use an internal tool that: - maps applications to namespaces within clusters for different environments (since we have 1 cluster per environment) - does some magic around secrets to make them easier to interface with [0] https://github.com/technosophos/helm-template https://github.com/technosophos/helm-template
Snappy 9y ago
That's a good point. We (GitLab) use it mostly as a templating system as well. It's a step up from piping `sed` output to `kubectl`. But we have our own tools for managing redeploys and rollbacks.

sytse 9y ago

If you're using our GitLab consider using Auto Deploy. Our CTO recently made a quick start guide for it https://docs.gitlab.com/ee/ci/autodeploy/quick_start_guide.html https://docs.gitlab.com/ee/ci/autodeploy/quick_start_guide.h...

Snappy 9y ago
FWIW, the next version of GitLab's Auto Deploy will use Helm under the hood (and let you bring-your-own-chart).

majewsky 9y ago

At SAP, we're using Helm [1] to deploy OpenStack (plus a bunch of extra services like a custom Rails-based dashboard [2]) on baremetal Kubernetes. For image building, testing and deployment, we use Concourse CI [3], and OpenStack assets (like service users, service projects and roles) are deployed with a custom Kubernetes operator [4]. Our charts are at [5] if you want to take a look. [1] https://github.com/kubernetes/helm https://github.com/kubernetes/helm [2] https://github.com/sapcc/elektra https://github.com/sapcc/elektra [3] https://concourse.ci https://concourse.ci [4] https://github.com/sapcc/kubernetes-operators https://github.com/sapcc/kubernetes-operators in the "openstack-operator" directory [5] https://github.com/sapcc/openstack-helm https://github.com/sapcc/openstack-helm and https://github.com/sapcc/helm-charts https://github.com/sapcc/helm-charts (two different repos since we're in the middle of modularizing the original monolithic OpenStack chart in the first repo into per-service charts in the second one)

invisible 9y ago

We are using ecfg (from Shopify) and Jenkins+kubectl. We use ansible for a couple of things but it's largely only because of some parameters of our architecture for a legacy monolith.

theptip 9y ago

A lot of folks are using Helm, but I find it very opaque to debug when templates go wrong (and I feel quite strongly that we shouldn't be writing untyped templates for our models). Also I found writing reusable spec components to be very difficult, e.g. a reverse proxy that I add to a number of pods. I use pykube (also worth looking at the incubator project client-python) to write deploy scripts in Python; client-python is particularly nice as it uses OpenAPI specs to give you type hinting on the API objects/fields that you're writing. Much more civilized than typing bare yaml into a text editor. If Python isn't your thing you can generate your own client from the OpenAPI specs, though I've found the client generation process to be a bit buggy.

rubenv 9y ago

I recently spoke about the approach we use at Ticketmatic: https://rocketeer.be/articles/coreos-fest-2017/ https://rocketeer.be/articles/coreos-fest-2017/

kalev 9y ago
That must have been an awesome talk. Thanks for the write-up!