8 ms·

Operator Framework: Building Apps on Kubernetes

133 points by polvi 8y ago

pgalgali 8y ago
Can we use Operator Framework for building operators that also require new api server using api aggregator?
pm90 8y ago
Can anyone talk about the positives/negatives of Operators v/s Helm Charts? From what I see, it seems like Operators are a better tool for defining and deploying Custom k8s resources whereas helm charts are good way to organize applications (deployment, service etc. templates packaged into one tar).
- elsonrodriguez 8y ago
  You can think of K8s Controllers/Operators as a way to extend the K8s API at runtime with a new type of object that is managed by a controller you've written yourself (as opposed to the built in k8s objects that are handled by the default controller-manager). A Helm chart by comparison is a way to template out K8s objects to make them configurable for different environments. Some shops will end up combining one with the other.
  - smarterclayton 8y ago
    And that’s why the team created a Helm-operator based on Lostromos (a tool for expanding Helm charts without running a tillers erver) that makes it easy to use a Helm chart for templating but still add additional lifecycle on top: https://github.com/operator-framework/helm-app-operator-kit https://github.com/operator-framework/helm-app-operator-kit The templating aspect of Helm and the set of quality content is complementary to being able to add higher level lifecycle.
- alpb 8y ago
  The main difference of Operators is explained here https://coreos.com/blog/introducing-operators.html https://coreos.com/blog/introducing-operators.html Operators basically encapsulate human operational knowledge on a particular system, like how do you run a Memcached Ring on Kubernetes capsulated by https://github.com/ianlewis/memcached-operator https://github.com/ianlewis/memcached-operator.
- mfer 8y ago
  Helm and Community Charts maintainer here... Helm is a package manager. Think of it like apt for Kubernetes. Operators enable you to manage the operation of applications within Kubernetes. They are complementary. You can deploy an operator as part of a Helm Chart. I recently wrote a blog post that explains how the different tools relate. https://codeengineered.com/blog/2018/kubernetes-helm-related-tools/ https://codeengineered.com/blog/2018/kubernetes-helm-related...
  - nikolay 8y ago
    Any plans to switch from templatized YAML to Jsonnet/Ksonnet?
    - bryanlarsen 8y ago
      The helm 3 proposal uses Lua. Not sure if they've actually started working on it: https://github.com/kubernetes-helm/community/blob/master/helm-v3/000-helm-v3.md https://github.com/kubernetes-helm/community/blob/master/hel...
      - nikolay 8y ago
        Honestly, this is a terrible choice! I like Lua, but it's not mainstream, and it's not going to be! There are tons of embeddable JavaScript engines - they could've used TypeScript, too!
        hobofan 8y ago
        > but it's not mainstream, and it's not going to be! Why are you suggesting jsonnet then?
        nikolay 8y ago
        Because it's popular in Kubernetes and makes sense, unlike Lua in the context.
        mfer 8y ago
- streetcat1 8y ago
  To understand operators you should think differently about k8s. I.e. as it core, k8s is a distributed messaging queue, where the messages are in the form of a declarative desired state (defined by yaml file) of a thing. The operator (or to be more precise one or more controllers) listen to those messages and try to reconcile the desired state with the current state. So an operator is a combination of the message types and the controllers that take of the reconciliation process. Taking this point of view, k8s is much more than container orchestration framework. I.e. it can orchestrate anything in the real world, as long as there is a way to define the desired state of a thing and a controller that can affect the real world. Back to the original question. Helm was created in order to raise the abstraction of resource definition (I.e. the desired state) from plain yaml to property file which is much more readable and smaller. Along the way, it also became a packaging tool.
cookiecaper 8y ago
> You may be familiar with Operators from the concept’s introduction in 2016. An Operator is a method of packaging, deploying and managing a Kubernetes application. "Operators", as introduced in 2016, were just bespoke Go programs that communicated with Kubernetes internals in a pretty low-level way. You were writing special-case plugins for Kubernetes, but they didn't want to make it sound that way, because I guess that just doesn't sound hip or devopsy. This branding exercise worked out for CoreOS -- Red Hat just bought them. This whole space is massively infused with bullshit. It's because all of these companies want to make money selling you cloud stuff, because it's profitable to rent computers at 3-5x the TCO. Google especially is hungry to claw back the lead in the cloud space from Amazon, and it's not hard to conceive why Kubernetes doesn't seem to work without fuss anywhere except GKE, or to understand the massive marketing dollars that Google is pumping into this whole Kubernetes farce (and for the record, Google seems to consider HN an important platform for k8s PR; I've been censured after too many Googlers found my k8s-skeptical posts "tedious"). Anyway, I guess that's neither here nor there. Just annoyed at what is by now the totally conventional status quo of overhyped empty promises made by people who seem more like ignorant promoters and fanboys than serious engineers. This "Operator Framework" seems to be the same concept of Operators, just with additional library support for the plugins -- err, "Operators". It may be a good improvement, will have to research more.
- gtirloni 8y ago
  Your comment would be more useful if it shared any hands-on experience with Operators instead of the usual ranting. I think you've made your general opinion on containers and Kubernetes pretty clear already.
  - cookiecaper 8y ago
    First, I don't think anyone notices or cares when I post specifically, so it's hard to really feel like my "general opinion" is well-known enough to not talk about it anymore. I'm no Joel Spolsky over here! Second, I feel like it's valid to point out that Operators are not really just a method of "packaging", in a post that tries to make it sound like Operators are just a small bit of YAML or metadata. You're writing real, non-trivial Go code that tells Kubernetes explicitly how to deploy and manage the lifecycles of specific types of applications. At least until now with the "Operator Framework", there wasn't really even anything that firmly defined an Operator as an Operator; it's just what some people called their Go code that manipulated k8s's object handling and lifecycle internals. But, if you insist, here's one operator I've worked with: https://github.com/coreos/prometheus-operator https://github.com/coreos/prometheus-operator . This is from CoreOS itself. Here's a patch I submitted about a year ago: https://github.com/coreos/prometheus-operator/pull/289 https://github.com/coreos/prometheus-operator/pull/289 . This required updating the way the software handled HTTP response codes in one of its "watcher" daemons (because all packaging methods need those, right?), and fixing the order of operations in the bootstrap scripts. Some more general info about this repo: $ du -sh prometheus-operator/.git 51M prometheus-operator/.git The repo size is 51M. $ git rev-list --count master 1716 There have been almost 2000 commits. $ cloc --vcs=git --exclude-dir vendor,example,contrib . 290 text files. 278 unique files. 121 files ignored. github.com/AlDanial/cloc v 1.74 T=0.82 s (295.4 files/s, 48117.2 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Go 50 1581 1392 20622 JSON 9 0 0 6276 YAML 132 260 885 4164 Markdown 30 792 0 2957 Bourne Shell 17 65 58 257 make 1 34 2 91 Python 1 10 5 40 TOML 1 11 20 30 Dockerfile 2 8 0 27 ------------------------------------------------------------------------------- SUM: 243 2761 2362 34464 ------------------------------------------------------------------------------- It appears there are over 20k lines of Go code after excluding vendor libraries and the example and contrib directories (arguably, contrib should've been included). I dunno, it just feels a little disingenuous, to me, to say that something that involves this much code is just a "packaging method" for a normal application. "Sure, just write an operator to package that up" like it's comparable to a package.json manifest or something. It's not! You need custom daemons that watch files to make sure that your k8s deployment stays in sync, and then you need to exert very meticulous and specific control over Kubernetes' behavior to make things work well. I think it's demonstrative that it takes north of 20k lines of Go code to package an application for deployment on Kubernetes. What do you think? ------------- EDIT: And one clarification: my opinion on containers as such is probably not well-known, since you're conflating it with my opinion on Kubernetes. I like containers conceptually (who wouldn't?) and I run several of them through LXC: NAME STATE AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED axxxx-dev STOPPED 0 - - - false gentoo-encoder STOPPED 0 - - - false jeff-arch-base-lxc STOPPED 0 - - - false jeff-crypto RUNNING 0 - xxx.xxx.xx.xxx - false jeff-ffmpeg STOPPED 0 - - - false jeff-netsec STOPPED 0 - - - false jeff-ocr STOPPED 0 - - - false localtorrents-lxc RUNNING 0 - xxx.xxx.xx.xxx - false nim-dev STOPPED 0 - - - false plex-2018 RUNNING 1 - xxx.xxx.xxx.xxx - true unifi STOPPED 0 - - - true I believe this is the kind of thing people actually want. Highly efficient, thin "VMs" that are easy to manage and run as independent systems without requiring the resource commitment. There is a good place for Kubernetes in probably about 1% of deployments where it's used. Most other people are just trying to run something like LXC, but they're confused because everyone who is critical of k8s drops to -4 and gets HN's mods after them. :)
thedevopsguy 8y ago
I've been an early adopter of docker. Used Compose when it was still called Fig, used and deployed kubernetes beta up to version 1 for in-house PAAS/heroku like environment. Must say I do miss those days when K8s was an idea that could fit in your head. The primitives were just enough back then. It was powerful developer tool for teams and we used it aggressively to accelerate our development process. K8s has now moved beyond this and seems to me to be focussing strongly on its operational patterns. You can see these operational patterns being used together to create a fairly advanced on-prem cloud infrastructure. At times, to me, it looks like over-engineering. Looking at the borg papers, I don't remember seeing operational primitives this advanced. The develop interface was fairly simple i.e this is my app, give me these resources, go! I know you don't have to use this new construct but it sure does make the landscape a lot more complicated.
- GauntletWizard 8y ago
  There's some truth and some wistful hope in your post; In my time at Google, the only thing that was anything like these "Operators" was what was developed by the MySQL SRE team, which was great but they also admitted it was a bit "round peg, square hole". There's a shared persistence layer that hasn't quite shown up yet; you need a low-latency POSIX filesystem and a throughput-heavy non-POSIX system (Chubby and GFS in the Borg world/Etcd and ??? in k8s). Not having the ability to work with persistent, shared objects is the biggest detriment to the ecosystem. S3 sorta works if you're in AWS, GCE supports Bigtable and etc
- merb 8y ago
  > You can see these operational patterns being used together to create a fairly advanced on-prem cloud infrastructure. At times, to me, it looks like over-engineering. well consider you wanted to have a High Available solutions that supports Blue Green / Rolling Deploys without downtime. You either built it yourself or you rely on something like k8s. It's not that much over engineering. K8s is a lot of code, yes. But the constructs is still pretty simple. I think deploying k8s is still way easier than most other solutions out there, like all these PaaS's and Cloud solutions. Spinning up K8s is basically just using ignition/cloud-config, coreos and PXE or better iPXE. Yeah sometimes it's troublesome to upgrade a k8s version or a etcd cluster. However everything on top of k8s or even coreos itself is extremly simple to upgrade. inb4 or our current system is using consul, haproxy, ansible and some custom built stuff to actually run our stuff. System upgrades are still done manually or trough ansible and my company plans to replace that with k8s. it's just way simpler to keep everything up-to date and run for high availability without disruption on deployments. it's also way simpler to actually get new services/tools into production, i.e. redis/elasticsearch without needing to keep them up to date/running.
hardwaresofton 8y ago
tl;dr - this is a rant. the operator pattern should stay a pattern. "framework" and "application" are meaningless these days. Stop trying to make things so easy a "monkey could do it". No no no no no. The words "framework" and "application" are so meaningless now that even reading this post is draining. CoreOS pioneered the Operator pattern, but I think building up that pattern into a framework to get people developing it away from the knowing the basics of k8s is such a mistake. The operator pattern falls out of the primitives that k8s offers (there's literally a concept called a controller) -- this makes it seem like another app platform. I think the level of abstraction isn't even right, this is like trying to enable people to write daemons without knowing anything about linux or signals or processes. Then again, I also dislike tools like Helm because they do the same thing. Why is everyone so in a rush to make inevitably leaky abstractions to make everything so easy a monkey could do it? All you're doing is encouraging cookie cutter programmers to write cookie cutter poorly understood code that will break on someone inevitably. All essential complexity introduced by features in the layer below an abstraction cannot be simplified, it can only be hidden or removed. It is OK for things to be hard, as long as they are simple (in the rich hickey easy vs simple analogy).

133 points by polvi 8y ago

Can we use Operator Framework for building operators that also require new api server using api aggregator?

pm90 8y ago

Can anyone talk about the positives/negatives of Operators v/s Helm Charts? From what I see, it seems like Operators are a better tool for defining and deploying Custom k8s resources whereas helm charts are good way to organize applications (deployment, service etc. templates packaged into one tar).

elsonrodriguez 8y ago
You can think of K8s Controllers/Operators as a way to extend the K8s API at runtime with a new type of object that is managed by a controller you've written yourself (as opposed to the built in k8s objects that are handled by the default controller-manager). A Helm chart by comparison is a way to template out K8s objects to make them configurable for different environments. Some shops will end up combining one with the other.
- smarterclayton 8y ago
  And that’s why the team created a Helm-operator based on Lostromos (a tool for expanding Helm charts without running a tillers erver) that makes it easy to use a Helm chart for templating but still add additional lifecycle on top: https://github.com/operator-framework/helm-app-operator-kit https://github.com/operator-framework/helm-app-operator-kit The templating aspect of Helm and the set of quality content is complementary to being able to add higher level lifecycle.
alpb 8y ago
The main difference of Operators is explained here https://coreos.com/blog/introducing-operators.html https://coreos.com/blog/introducing-operators.html Operators basically encapsulate human operational knowledge on a particular system, like how do you run a Memcached Ring on Kubernetes capsulated by https://github.com/ianlewis/memcached-operator https://github.com/ianlewis/memcached-operator.
mfer 8y ago
Helm and Community Charts maintainer here... Helm is a package manager. Think of it like apt for Kubernetes. Operators enable you to manage the operation of applications within Kubernetes. They are complementary. You can deploy an operator as part of a Helm Chart. I recently wrote a blog post that explains how the different tools relate. https://codeengineered.com/blog/2018/kubernetes-helm-related-tools/ https://codeengineered.com/blog/2018/kubernetes-helm-related...
- nikolay 8y ago
  Any plans to switch from templatized YAML to Jsonnet/Ksonnet?
  - bryanlarsen 8y ago
    The helm 3 proposal uses Lua. Not sure if they've actually started working on it: https://github.com/kubernetes-helm/community/blob/master/helm-v3/000-helm-v3.md https://github.com/kubernetes-helm/community/blob/master/hel...
    - nikolay 8y ago
      Honestly, this is a terrible choice! I like Lua, but it's not mainstream, and it's not going to be! There are tons of embeddable JavaScript engines - they could've used TypeScript, too!
      - hobofan 8y ago
        > but it's not mainstream, and it's not going to be! Why are you suggesting jsonnet then?
        nikolay 8y ago
        Because it's popular in Kubernetes and makes sense, unlike Lua in the context.
      - mfer 8y ago
streetcat1 8y ago
To understand operators you should think differently about k8s. I.e. as it core, k8s is a distributed messaging queue, where the messages are in the form of a declarative desired state (defined by yaml file) of a thing. The operator (or to be more precise one or more controllers) listen to those messages and try to reconcile the desired state with the current state. So an operator is a combination of the message types and the controllers that take of the reconciliation process. Taking this point of view, k8s is much more than container orchestration framework. I.e. it can orchestrate anything in the real world, as long as there is a way to define the desired state of a thing and a controller that can affect the real world. Back to the original question. Helm was created in order to raise the abstraction of resource definition (I.e. the desired state) from plain yaml to property file which is much more readable and smaller. Along the way, it also became a packaging tool.

cookiecaper 8y ago

> You may be familiar with Operators from the concept’s introduction in 2016. An Operator is a method of packaging, deploying and managing a Kubernetes application. "Operators", as introduced in 2016, were just bespoke Go programs that communicated with Kubernetes internals in a pretty low-level way. You were writing special-case plugins for Kubernetes, but they didn't want to make it sound that way, because I guess that just doesn't sound hip or devopsy. This branding exercise worked out for CoreOS -- Red Hat just bought them. This whole space is massively infused with bullshit. It's because all of these companies want to make money selling you cloud stuff, because it's profitable to rent computers at 3-5x the TCO. Google especially is hungry to claw back the lead in the cloud space from Amazon, and it's not hard to conceive why Kubernetes doesn't seem to work without fuss anywhere except GKE, or to understand the massive marketing dollars that Google is pumping into this whole Kubernetes farce (and for the record, Google seems to consider HN an important platform for k8s PR; I've been censured after too many Googlers found my k8s-skeptical posts "tedious"). Anyway, I guess that's neither here nor there. Just annoyed at what is by now the totally conventional status quo of overhyped empty promises made by people who seem more like ignorant promoters and fanboys than serious engineers. This "Operator Framework" seems to be the same concept of Operators, just with additional library support for the plugins -- err, "Operators". It may be a good improvement, will have to research more.

gtirloni 8y ago
Your comment would be more useful if it shared any hands-on experience with Operators instead of the usual ranting. I think you've made your general opinion on containers and Kubernetes pretty clear already.
- cookiecaper 8y ago
  First, I don't think anyone notices or cares when I post specifically, so it's hard to really feel like my "general opinion" is well-known enough to not talk about it anymore. I'm no Joel Spolsky over here! Second, I feel like it's valid to point out that Operators are not really just a method of "packaging", in a post that tries to make it sound like Operators are just a small bit of YAML or metadata. You're writing real, non-trivial Go code that tells Kubernetes explicitly how to deploy and manage the lifecycles of specific types of applications. At least until now with the "Operator Framework", there wasn't really even anything that firmly defined an Operator as an Operator; it's just what some people called their Go code that manipulated k8s's object handling and lifecycle internals. But, if you insist, here's one operator I've worked with: https://github.com/coreos/prometheus-operator https://github.com/coreos/prometheus-operator . This is from CoreOS itself. Here's a patch I submitted about a year ago: https://github.com/coreos/prometheus-operator/pull/289 https://github.com/coreos/prometheus-operator/pull/289 . This required updating the way the software handled HTTP response codes in one of its "watcher" daemons (because all packaging methods need those, right?), and fixing the order of operations in the bootstrap scripts. Some more general info about this repo: $ du -sh prometheus-operator/.git 51M prometheus-operator/.git The repo size is 51M. $ git rev-list --count master 1716 There have been almost 2000 commits. $ cloc --vcs=git --exclude-dir vendor,example,contrib . 290 text files. 278 unique files. 121 files ignored. github.com/AlDanial/cloc v 1.74 T=0.82 s (295.4 files/s, 48117.2 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Go 50 1581 1392 20622 JSON 9 0 0 6276 YAML 132 260 885 4164 Markdown 30 792 0 2957 Bourne Shell 17 65 58 257 make 1 34 2 91 Python 1 10 5 40 TOML 1 11 20 30 Dockerfile 2 8 0 27 ------------------------------------------------------------------------------- SUM: 243 2761 2362 34464 ------------------------------------------------------------------------------- It appears there are over 20k lines of Go code after excluding vendor libraries and the example and contrib directories (arguably, contrib should've been included). I dunno, it just feels a little disingenuous, to me, to say that something that involves this much code is just a "packaging method" for a normal application. "Sure, just write an operator to package that up" like it's comparable to a package.json manifest or something. It's not! You need custom daemons that watch files to make sure that your k8s deployment stays in sync, and then you need to exert very meticulous and specific control over Kubernetes' behavior to make things work well. I think it's demonstrative that it takes north of 20k lines of Go code to package an application for deployment on Kubernetes. What do you think? ------------- EDIT: And one clarification: my opinion on containers as such is probably not well-known, since you're conflating it with my opinion on Kubernetes. I like containers conceptually (who wouldn't?) and I run several of them through LXC: NAME STATE AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED axxxx-dev STOPPED 0 - - - false gentoo-encoder STOPPED 0 - - - false jeff-arch-base-lxc STOPPED 0 - - - false jeff-crypto RUNNING 0 - xxx.xxx.xx.xxx - false jeff-ffmpeg STOPPED 0 - - - false jeff-netsec STOPPED 0 - - - false jeff-ocr STOPPED 0 - - - false localtorrents-lxc RUNNING 0 - xxx.xxx.xx.xxx - false nim-dev STOPPED 0 - - - false plex-2018 RUNNING 1 - xxx.xxx.xxx.xxx - true unifi STOPPED 0 - - - true I believe this is the kind of thing people actually want. Highly efficient, thin "VMs" that are easy to manage and run as independent systems without requiring the resource commitment. There is a good place for Kubernetes in probably about 1% of deployments where it's used. Most other people are just trying to run something like LXC, but they're confused because everyone who is critical of k8s drops to -4 and gets HN's mods after them. :)

thedevopsguy 8y ago

I've been an early adopter of docker. Used Compose when it was still called Fig, used and deployed kubernetes beta up to version 1 for in-house PAAS/heroku like environment. Must say I do miss those days when K8s was an idea that could fit in your head. The primitives were just enough back then. It was powerful developer tool for teams and we used it aggressively to accelerate our development process. K8s has now moved beyond this and seems to me to be focussing strongly on its operational patterns. You can see these operational patterns being used together to create a fairly advanced on-prem cloud infrastructure. At times, to me, it looks like over-engineering. Looking at the borg papers, I don't remember seeing operational primitives this advanced. The develop interface was fairly simple i.e this is my app, give me these resources, go! I know you don't have to use this new construct but it sure does make the landscape a lot more complicated.

GauntletWizard 8y ago
There's some truth and some wistful hope in your post; In my time at Google, the only thing that was anything like these "Operators" was what was developed by the MySQL SRE team, which was great but they also admitted it was a bit "round peg, square hole". There's a shared persistence layer that hasn't quite shown up yet; you need a low-latency POSIX filesystem and a throughput-heavy non-POSIX system (Chubby and GFS in the Borg world/Etcd and ??? in k8s). Not having the ability to work with persistent, shared objects is the biggest detriment to the ecosystem. S3 sorta works if you're in AWS, GCE supports Bigtable and etc
merb 8y ago
> You can see these operational patterns being used together to create a fairly advanced on-prem cloud infrastructure. At times, to me, it looks like over-engineering. well consider you wanted to have a High Available solutions that supports Blue Green / Rolling Deploys without downtime. You either built it yourself or you rely on something like k8s. It's not that much over engineering. K8s is a lot of code, yes. But the constructs is still pretty simple. I think deploying k8s is still way easier than most other solutions out there, like all these PaaS's and Cloud solutions. Spinning up K8s is basically just using ignition/cloud-config, coreos and PXE or better iPXE. Yeah sometimes it's troublesome to upgrade a k8s version or a etcd cluster. However everything on top of k8s or even coreos itself is extremly simple to upgrade. inb4 or our current system is using consul, haproxy, ansible and some custom built stuff to actually run our stuff. System upgrades are still done manually or trough ansible and my company plans to replace that with k8s. it's just way simpler to keep everything up-to date and run for high availability without disruption on deployments. it's also way simpler to actually get new services/tools into production, i.e. redis/elasticsearch without needing to keep them up to date/running.

hardwaresofton 8y ago

tl;dr - this is a rant. the operator pattern should stay a pattern. "framework" and "application" are meaningless these days. Stop trying to make things so easy a "monkey could do it". No no no no no. The words "framework" and "application" are so meaningless now that even reading this post is draining. CoreOS pioneered the Operator pattern, but I think building up that pattern into a framework to get people developing it away from the knowing the basics of k8s is such a mistake. The operator pattern falls out of the primitives that k8s offers (there's literally a concept called a controller) -- this makes it seem like another app platform. I think the level of abstraction isn't even right, this is like trying to enable people to write daemons without knowing anything about linux or signals or processes. Then again, I also dislike tools like Helm because they do the same thing. Why is everyone so in a rush to make inevitably leaky abstractions to make everything so easy a monkey could do it? All you're doing is encouraging cookie cutter programmers to write cookie cutter poorly understood code that will break on someone inevitably. All essential complexity introduced by features in the layer below an abstraction cannot be simplified, it can only be hidden or removed. It is OK for things to be hard, as long as they are simple (in the rich hickey easy vs simple analogy).