7 ms·

Kubernetes StatefulSets are Broken

107 points by mjg235 4y ago

nullwarp 4y ago
I have completely given up on anything related to stateful applications in Kubernetes. Most of the time it can work okay but the amount of headache's we've experienced with it we've completely abandoned using Kubernetes for anything isn't stateless and requiring dynamic scaling.
aeyes 4y ago
> Manually edit the StatefulSet volume claim with the new storage size and add a dummy pod annotation to force a rolling update If the PVC size changes Kubernetes automatically does an online resize of the filesystem: https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/ https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/ This has been possible for 2-3 years if you had the flag enabled.
- arianvanp 4y ago
  Problem is that you don't create PVCs yourself usually but the statefulset manages them based on a template.
- sweaver 4y ago
  How consistent have the results been? I used this in the past with some flakiness...
  - jeppesen-io 4y ago
    Not OP, but I've been doing it for a few years. Can't say I've ever seen an issue - with Prometheus Operator, which the article mentions, as few times as well
- uberduper 4y ago
  Online resizing of pv has been possible for quite a while. The complaint from the author is that you can do this for any existing pv that was created by the sts pvc definition, but scaling up the sts will create a new pod and pvc using the original spec. Altering that spec in the sts manifest is unpleasant when it really shouldn't be.
0xbadcafebee 4y ago
If you need stateful files in a microservice, just mount a network filesystem. Yes, it's slow. Yes, it's buggy. Yes, it's not very portable. Yes, there are locking issues. But if you really need stateful files in a microservice, something is already fucked up. Network filesystems are a simple and cheap way to add that functionality and more. You get 'infinite' storage, redundancy, persistence, shared volumes, intra-cluster data availability, centralized data lifecycle management, etc, and no need for more custom Kubernetes logic just to persist some files. It's a shitty solution that is actually good enough.
- mjg235 4y ago
  I actually think k8s is (or can be) a decent solve for non-critical, low cost databases if you're confident about navigating the toolchain. The one big gap is storage resizing, which is part of why we're pointing it out.
- spmurrayzzz 4y ago
  I'd double down even more on your latter comment of "just dont use k8s" for stateful needs as its a minefield. But mounting a network fs is just gonna create a whole host of headaches when they decide to use the mount with anything that doesn't have native io fencing etc.
- oceanplexian 4y ago
  NFS doesn't have to be slow and buggy, its just that people try to set it up on a buggy cloud provider, and get burned. Tons of massive, mission-critical bare metal VMWare infra is backed by NFS and something like a NetApp. The one I used to admin years ago was connected via redundant 40Gb ethernet and it could easily push many GB/s at very consistent and low latency, and would fail over instantly without interruption.
powerhour 4y ago
This is more a criticism of operators than StatefulSets. The promise of operators is easier automated management of your resources. If the operator can't do basic things like resizing disks, it's flawed and needs to be fixed. However, I've always been suspicious of operators, so maybe my bias is leaking here. Kubernetes is already a level of abstraction and adding another seems risky and unnecessary.
- mjg235 4y ago
  The flaw is definitely most acute when it is combined with operators, but it would make storage resizes vastly more accessible for less capable devs if it were able to be done declaratively via the spec instead of via a painful workaround.
- dmead 4y ago
  I agree with this. I manage a monorepo of 10s of terraform/terragrunt modules most of which are deployed to a k8s cluster. the whole point of code as infrastructure is that your git repo is an audit log of whats changed in your infra. Once you leave it up to operators or persistent volume claim templates (or whatever else makes side effectful changes with being told) you're throwing all that niceness of control out the window.
  - encryptluks2 4y ago
    A Git repo can still be a good audit log even if you use templates, especially if you're adding a step to transform files with each push.
  - raffraffraff 4y ago
    Infra as code can't actively manage things like database clusters, handling promotion of a secondary, automatic deployment of new replicas, scaling. Using it to deploy RDS Aurora doesn't count. My impression is that operators are designed to handle those "active" decisions in real-time, and that's outside the realm of terraform, GitOps etc.
    - 4y ago
Spivak 4y ago
I cannot understand why people in this thread are saying people who want state are holding it wrong. If you are saying that k8s falls down on state — literally the only hard problem — then why are you bothering take on the complexity of k8s? What are you running on k8s that is stateless and can’t be served by ec2 instances in auto scaling groups? State management is the killer feature for k8s; it manages all the complexity and presents you a fiction where stateful apps can be written like they’re actually stateless.
- mjg235 4y ago
  People are definitely sleeping on k8s' capability in managing stateful workloads. We run various datastores on it all the time, and it's quite reliable for a lot of use cases (largely because the underlying cloud block stores it's orchestrating are incredibly robust). That said there are warts that should be removed, but that's not surprising.
- oceanplexian 4y ago
  The vast majority of developers don't understand what "stateless" means, since if you want to get technical is no such thing as a stateless system. You might have drained requests off of a pod, but it still has a "state", it still has things cached in memory, it might still have connections open to some outside entity (Database, whatever), and the developer, not K8s, is responsible for catching those signals, cleaning up things in memory, gracefully terminating connections, handing off in-flight workloads to another pod, etc. Even in the upper echelons of the tech I see a very small minority of developers actually aware of all the things that can make stateless workloads stateful. Which is OK for something where the stakes are low, but if you're a DBA or a Systems oriented person you'll see people make these (very wrong) assumptions all the time and recoil in horror.
  - GauntletWizard 4y ago
    And that's why the title DBA is going the way of the dodo; state in this instance does not include cache and connections, because those are theoretically idempotent. Theoretically is not actually, and a ton of the assumptions in k8s break down when you look at them super closely, but they break down when things have gone wrong. If your cache is properly designed it doesn't matter if it's a little stale; you're not using it for things where correct and up to date is the most important thing. Connections, too; it shouldn't matter if your connection pool needs to spin up another connection, because if any of the conditions exist that the connection fails (or takes too long and stalls requests for any appreciable time), you already should have been paged and be on the way to resolving the incident.
lokar 4y ago
This totally misunderstands the motivation for statefull sets. The only thing they offer is the stable per-task names. They are a reimplantation of jobs in borg where you always have stable task numbers. It's really more for shared services then ones with persistent state.
- vorpalhex 4y ago
  Incorrect. An STS changes the deployment/rollout mechanisms away from blue-green.
- bogomipz 4y ago
  From the latest documentation[1]: StatefulSets are valuable for applications that require one or more of the following. Stable, unique network identifiers. Stable, persistent storage. Ordered, graceful deployment and scaling. Ordered, automated rolling updates. [1] https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ https://kubernetes.io/docs/concepts/workloads/controllers/st...
  - Filligree 4y ago
    Which confuses me, frankly. When do you not need automated rolling updates? Or stable network identifiers? I'm sure there are cases, but it seems like supporting them should be the default -- are statefulsets somehow expensive?
    - lokar 4y ago
      Consider a job of N identical API servers. Your request can go to any of them. To update we could just add new tasks and take away old tasks. It's a gradual update, but not "rolling" is the sens that each task name has a single love instance at a time.
    - mjg235 4y ago
      A basic stateless web app behind a load balancer can tolerate unstable network identifiers as long as you automate registering/deregistering the new names, which kubernetes services do very well. Stateful apps can be a bit more exceptional because they often have client libraries that require you to hardcode network addresses via connection strings and other tooling that makes that glue automation more tricky and worth the added guarantee around consistent naming.
linuxftw 4y ago
Sometimes operators interfere with what you're trying to do, as the article points out the Prometheus operator. Pro tip: Scale down the operator, make your changes, put everything back the way the operator expects it, scale operator back up.
- project00 4y ago
  I had to do a MongoDB migration and I was glad we had originally deployed it using the standard k8s yaml manifests instead of operators. It was a layer of complexity removed.
CodesInChaos 4y ago
Sparse block storage is one of the cloud features I miss most. What I'd expect: * You can create a volume, but pay only for in-use blocks. * You create a volume bigger than what you'll ever need => no resizing required (At least 16 TiB, but possibly even something crazy like an exabyte) * There is some kind of "trim" operation that marks a block as free * Ideally it's possible to choose the maximum block index (size of block-volume) independently from the maximum number of in-use blocks (for cost control)
- lifty 4y ago
  Great feature, but cloud providers are incentivised to make it difficult for people to save money. This would instantly vaporise a big chunk of revenue, because people would not have to pay for pre-provisioned storage.
  - advisedwang 4y ago
    Maybe one of the lower tier providers should use it as a differentiator.
    - djbusby 4y ago
      Why add a low margin feature to your product/service?
      - bo0tzz 4y ago
        Because it draws in customers that will then also use your higher margin features.
      - discodave 4y ago
        Amazon historically has gone after 'low margin' businesses. The cloud, at 25% margins (for AWS) is is 'low margin' compared to traditional software sales.
  - pgwhalen 4y ago
    > Great feature, but cloud providers are incentivised to make it difficult for people to save money. Every business is incentivized this way. Competition is what pushes prices down, of which there is plenty in the cloud industry (though maybe still not as much as we would like).
AtlasBarfed 4y ago
k8s is made for stateless servers. But "stateful sets" has always seemed like a bolt on. Also, for important "big data" stores with replication, I don't want to trust all-in-one third party operators, which is what all K8S operators seem to aspire to be. Your data is important, and you should evaluate all your data operations use cases yourself.
- MrStonedOne 4y ago
openplatypus 4y ago
Agreed, we started with Kafka, PostgreSQL and one other service in K8s with StatefulSets. It was painful. We quickly realized that rolling upgrades often got blocked. The reason being peristent volume claims sometimes get stuck. And this is just one of few problems. Since we moved databases and stateful services outside Kubernetes, everything got faster and more reliable.
- moreira 4y ago
  As someone who's been eyeing Kubernetes as a learning experience, that actually is really disheartening to hear. If you're being forced to manage your database outside of Kubernetes, what are you using Kubernetes for? Running your web/worker servers?
  - anchochilis 4y ago
    My mid-sized shop uses managed offerings for MySQL and Postgres but we run several stateful workloads, including ElasticSearch and MongoDB, in GKE. I personally haven't experienced the issues the OP is describing. It might be that StatefulSets and PVCs are more stable now then when OP tried them. Of course using managed/hosted K8s makes a big difference.
    - openplatypus 4y ago
      We use managed K8s. And we dropped StatefulSet maybe 4 months ago. There might have been a way to support our use case better. But this was managed service and we couldn't easily tweak every settings. But also, we didn't want to. Our goal of using Kubernetes is to make difficult stuff easy. Not to make easy stuff difficult. And in our skillset and capacity moving stateful services outside K8s was easier, cheaper and more reliable.
  - deleted 4y ago
    [deleted]
  - openplatypus 4y ago
    Microservices, automations, APIs. K8s is great way to easily deploy and scale services. As we leverage Kafka, a lot of applications fallback on Topics and KTables for storage. Kubernetes with jobs, cron jobs, operators, secrets etc is effectively an Operating System for the cloud. Definitely worth the investment to learn and practice.
MrStonedOne 4y ago
singron 4y ago
I just looked at the scripts for the last StatefulSet thing I did, and it has the exact workaround in the apply script. I.e. recreate the ss, edit the pvcs, and restart the rollout when there is a volume size change. K8s has really been an experience of encountering 2-4 year old bugs (closed by the stale bot of course) and missing features. We accumulate increasingly more workarounds and additional pieces of complexity to deal with this. Everything is always changing, but somehow still staying the same.
- mjg235 4y ago
  Yeah it's wild that it's been a pain point for so long without a properly supported solve.
ec109685 4y ago
This fundamentally untrue: “But, Kubernetes was originally intended to act as a container orchestration platform for stateless workloads, not stateful applications.” Even from the early days, there were constructs to allow stateful applications to be built on top of it. They have made it easier to run stateful applications, but to say it wasn’t designed to support stateful applications is incorrect: https://kubernetes.io/blog/2016/08/stateful-applications-using-kubernetes-datera/ https://kubernetes.io/blog/2016/08/stateful-applications-usi... “Stateful applications such as MySQL, Kafka, Cassandra, and Couchbase for a while, the introduction of Pet Sets has significantly improved this support.”
zoomzoom 4y ago
Generally, from teams we talk to at Coherence (withcoherence.com), it seems to be a mistake to try to run stateful workloads in Kubernetes. As much as the cost savings compared to managed services like AWS RDS or GCP Cloud SQL are attractive, the configuration time and maintenance are just not worth it. Our perspective is that container orchestration tools should be used to manage stateless workloads, and that stateful workloads should delegate to managed services where volume reliability, scalability, and backup can be handled with purpose-built tools.
koprulusector 4y ago
These are valid points. I’ve personally needed to do things like delete the StatefulSet, change volume size, update/change pvc, update StatefulSet, reapply. I just took it as a given and hadn’t thought much about the ergonomics. It’s really not that big of a deal and probably a ten minute process, tops. That said, OP makes a good point that mechanisms to handle this kind of stuff exist in CSI, so why not leverage / take advantage of that versus pissing in the wind (especially when you run into CRD that wants to undo the above work while you’re in the process)?
- smarterclayton 4y ago
  At the time we designed the StatefulSet, the resize behavior hadn’t been designed / finalized. Now that it has, it’s really a matter of someone having the time and patience to drive it though. I don’t know whether someone has sorted out all the design implications (like what happens if someone adds new resource requests, or what happens if a new resource request is rejected, and how that gets reported to the user), but it’s a small change that needs a fair amount of design thought, which tends to be hard to drive through.
de6u99er 4y ago
There's a pending pull request that solves this problem since end of June. https://github.com/kubernetes/enhancements/pull/3412 https://github.com/kubernetes/enhancements/pull/3412
lazyant 4y ago
StatefulSets are introduced as "when you need persistent storage" and not a lot more than that. A key idea that is often skipped is that k8s doesn't want to presume how a cluster of "database" nodes ("database" in the wide sense, including RMQ, Redis etc) coordinate their HA. So when a k8s node dies with a sts on it, unlike other pods, the sts pod is NOT recreated (not wanting to interfere with whatever mechanism the db cluster has to deal with it). I've found most people working with k8s don't know about this behaviour.
rdelpret 4y ago
Write a KEP not a blog post?
fathead_glacier 4y ago
Managing state is hard and always has been regardless of the underlying technology. I have been using Kubernetes for a few years now and whenever I see people putting manual steps they should instead be using an operator. What the author is describing as a problem has a solution (they even added it to their own operator). Kubernetes has a steep learning curve but if you want to do nontrivial thing like have state than you need to learn nontrivial things like operators. The default controllers that come with Kubernetes do not cover everything under the sun and they do require extensions via the operator pattern for some tasks. Just my two cents.
nunez 4y ago
The Kubernetes authors were explicit in stating that stateless workloads were their design target, and it is somewhat obvious that StatefulSets were a concession, so this isn't terribly surprising to me. I have a feeling that OpenShift handles stateful workloads more gracefully due to their enterprise-oriented customer base, but I have nothing to back that.

107 points by mjg235 4y ago

I have completely given up on anything related to stateful applications in Kubernetes. Most of the time it can work okay but the amount of headache's we've experienced with it we've completely abandoned using Kubernetes for anything isn't stateless and requiring dynamic scaling.

aeyes 4y ago

> Manually edit the StatefulSet volume claim with the new storage size and add a dummy pod annotation to force a rolling update If the PVC size changes Kubernetes automatically does an online resize of the filesystem: https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/ https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/ This has been possible for 2-3 years if you had the flag enabled.

arianvanp 4y ago
Problem is that you don't create PVCs yourself usually but the statefulset manages them based on a template.
sweaver 4y ago
How consistent have the results been? I used this in the past with some flakiness...
- jeppesen-io 4y ago
  Not OP, but I've been doing it for a few years. Can't say I've ever seen an issue - with Prometheus Operator, which the article mentions, as few times as well
uberduper 4y ago
Online resizing of pv has been possible for quite a while. The complaint from the author is that you can do this for any existing pv that was created by the sts pvc definition, but scaling up the sts will create a new pod and pvc using the original spec. Altering that spec in the sts manifest is unpleasant when it really shouldn't be.

0xbadcafebee 4y ago

If you need stateful files in a microservice, just mount a network filesystem. Yes, it's slow. Yes, it's buggy. Yes, it's not very portable. Yes, there are locking issues. But if you really need stateful files in a microservice, something is already fucked up. Network filesystems are a simple and cheap way to add that functionality and more. You get 'infinite' storage, redundancy, persistence, shared volumes, intra-cluster data availability, centralized data lifecycle management, etc, and no need for more custom Kubernetes logic just to persist some files. It's a shitty solution that is actually good enough.

mjg235 4y ago
I actually think k8s is (or can be) a decent solve for non-critical, low cost databases if you're confident about navigating the toolchain. The one big gap is storage resizing, which is part of why we're pointing it out.
spmurrayzzz 4y ago
I'd double down even more on your latter comment of "just dont use k8s" for stateful needs as its a minefield. But mounting a network fs is just gonna create a whole host of headaches when they decide to use the mount with anything that doesn't have native io fencing etc.
oceanplexian 4y ago
NFS doesn't have to be slow and buggy, its just that people try to set it up on a buggy cloud provider, and get burned. Tons of massive, mission-critical bare metal VMWare infra is backed by NFS and something like a NetApp. The one I used to admin years ago was connected via redundant 40Gb ethernet and it could easily push many GB/s at very consistent and low latency, and would fail over instantly without interruption.

powerhour 4y ago

This is more a criticism of operators than StatefulSets. The promise of operators is easier automated management of your resources. If the operator can't do basic things like resizing disks, it's flawed and needs to be fixed. However, I've always been suspicious of operators, so maybe my bias is leaking here. Kubernetes is already a level of abstraction and adding another seems risky and unnecessary.

mjg235 4y ago
The flaw is definitely most acute when it is combined with operators, but it would make storage resizes vastly more accessible for less capable devs if it were able to be done declaratively via the spec instead of via a painful workaround.
dmead 4y ago
I agree with this. I manage a monorepo of 10s of terraform/terragrunt modules most of which are deployed to a k8s cluster. the whole point of code as infrastructure is that your git repo is an audit log of whats changed in your infra. Once you leave it up to operators or persistent volume claim templates (or whatever else makes side effectful changes with being told) you're throwing all that niceness of control out the window.
- encryptluks2 4y ago
  A Git repo can still be a good audit log even if you use templates, especially if you're adding a step to transform files with each push.
- raffraffraff 4y ago
  Infra as code can't actively manage things like database clusters, handling promotion of a secondary, automatic deployment of new replicas, scaling. Using it to deploy RDS Aurora doesn't count. My impression is that operators are designed to handle those "active" decisions in real-time, and that's outside the realm of terraform, GitOps etc.
  - 4y ago

Spivak 4y ago

I cannot understand why people in this thread are saying people who want state are holding it wrong. If you are saying that k8s falls down on state — literally the only hard problem — then why are you bothering take on the complexity of k8s? What are you running on k8s that is stateless and can’t be served by ec2 instances in auto scaling groups? State management is the killer feature for k8s; it manages all the complexity and presents you a fiction where stateful apps can be written like they’re actually stateless.

mjg235 4y ago
People are definitely sleeping on k8s' capability in managing stateful workloads. We run various datastores on it all the time, and it's quite reliable for a lot of use cases (largely because the underlying cloud block stores it's orchestrating are incredibly robust). That said there are warts that should be removed, but that's not surprising.
oceanplexian 4y ago
The vast majority of developers don't understand what "stateless" means, since if you want to get technical is no such thing as a stateless system. You might have drained requests off of a pod, but it still has a "state", it still has things cached in memory, it might still have connections open to some outside entity (Database, whatever), and the developer, not K8s, is responsible for catching those signals, cleaning up things in memory, gracefully terminating connections, handing off in-flight workloads to another pod, etc. Even in the upper echelons of the tech I see a very small minority of developers actually aware of all the things that can make stateless workloads stateful. Which is OK for something where the stakes are low, but if you're a DBA or a Systems oriented person you'll see people make these (very wrong) assumptions all the time and recoil in horror.
- GauntletWizard 4y ago
  And that's why the title DBA is going the way of the dodo; state in this instance does not include cache and connections, because those are theoretically idempotent. Theoretically is not actually, and a ton of the assumptions in k8s break down when you look at them super closely, but they break down when things have gone wrong. If your cache is properly designed it doesn't matter if it's a little stale; you're not using it for things where correct and up to date is the most important thing. Connections, too; it shouldn't matter if your connection pool needs to spin up another connection, because if any of the conditions exist that the connection fails (or takes too long and stalls requests for any appreciable time), you already should have been paged and be on the way to resolving the incident.

lokar 4y ago

This totally misunderstands the motivation for statefull sets. The only thing they offer is the stable per-task names. They are a reimplantation of jobs in borg where you always have stable task numbers. It's really more for shared services then ones with persistent state.

vorpalhex 4y ago
Incorrect. An STS changes the deployment/rollout mechanisms away from blue-green.
bogomipz 4y ago
From the latest documentation[1]: StatefulSets are valuable for applications that require one or more of the following. Stable, unique network identifiers. Stable, persistent storage. Ordered, graceful deployment and scaling. Ordered, automated rolling updates. [1] https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ https://kubernetes.io/docs/concepts/workloads/controllers/st...
- Filligree 4y ago
  Which confuses me, frankly. When do you not need automated rolling updates? Or stable network identifiers? I'm sure there are cases, but it seems like supporting them should be the default -- are statefulsets somehow expensive?
  - lokar 4y ago
    Consider a job of N identical API servers. Your request can go to any of them. To update we could just add new tasks and take away old tasks. It's a gradual update, but not "rolling" is the sens that each task name has a single love instance at a time.
  - mjg235 4y ago
    A basic stateless web app behind a load balancer can tolerate unstable network identifiers as long as you automate registering/deregistering the new names, which kubernetes services do very well. Stateful apps can be a bit more exceptional because they often have client libraries that require you to hardcode network addresses via connection strings and other tooling that makes that glue automation more tricky and worth the added guarantee around consistent naming.

linuxftw 4y ago

Sometimes operators interfere with what you're trying to do, as the article points out the Prometheus operator. Pro tip: Scale down the operator, make your changes, put everything back the way the operator expects it, scale operator back up.

project00 4y ago
I had to do a MongoDB migration and I was glad we had originally deployed it using the standard k8s yaml manifests instead of operators. It was a layer of complexity removed.

CodesInChaos 4y ago

Sparse block storage is one of the cloud features I miss most. What I'd expect: * You can create a volume, but pay only for in-use blocks. * You create a volume bigger than what you'll ever need => no resizing required (At least 16 TiB, but possibly even something crazy like an exabyte) * There is some kind of "trim" operation that marks a block as free * Ideally it's possible to choose the maximum block index (size of block-volume) independently from the maximum number of in-use blocks (for cost control)

lifty 4y ago
Great feature, but cloud providers are incentivised to make it difficult for people to save money. This would instantly vaporise a big chunk of revenue, because people would not have to pay for pre-provisioned storage.
- advisedwang 4y ago
  Maybe one of the lower tier providers should use it as a differentiator.
  - djbusby 4y ago
    Why add a low margin feature to your product/service?
    - bo0tzz 4y ago
      Because it draws in customers that will then also use your higher margin features.
    - discodave 4y ago
      Amazon historically has gone after 'low margin' businesses. The cloud, at 25% margins (for AWS) is is 'low margin' compared to traditional software sales.
- pgwhalen 4y ago
  > Great feature, but cloud providers are incentivised to make it difficult for people to save money. Every business is incentivized this way. Competition is what pushes prices down, of which there is plenty in the cloud industry (though maybe still not as much as we would like).

AtlasBarfed 4y ago

k8s is made for stateless servers. But "stateful sets" has always seemed like a bolt on. Also, for important "big data" stores with replication, I don't want to trust all-in-one third party operators, which is what all K8S operators seem to aspire to be. Your data is important, and you should evaluate all your data operations use cases yourself.

MrStonedOne 4y ago

openplatypus 4y ago

Agreed, we started with Kafka, PostgreSQL and one other service in K8s with StatefulSets. It was painful. We quickly realized that rolling upgrades often got blocked. The reason being peristent volume claims sometimes get stuck. And this is just one of few problems. Since we moved databases and stateful services outside Kubernetes, everything got faster and more reliable.

moreira 4y ago
As someone who's been eyeing Kubernetes as a learning experience, that actually is really disheartening to hear. If you're being forced to manage your database outside of Kubernetes, what are you using Kubernetes for? Running your web/worker servers?
- anchochilis 4y ago
  My mid-sized shop uses managed offerings for MySQL and Postgres but we run several stateful workloads, including ElasticSearch and MongoDB, in GKE. I personally haven't experienced the issues the OP is describing. It might be that StatefulSets and PVCs are more stable now then when OP tried them. Of course using managed/hosted K8s makes a big difference.
  - openplatypus 4y ago
    We use managed K8s. And we dropped StatefulSet maybe 4 months ago. There might have been a way to support our use case better. But this was managed service and we couldn't easily tweak every settings. But also, we didn't want to. Our goal of using Kubernetes is to make difficult stuff easy. Not to make easy stuff difficult. And in our skillset and capacity moving stateful services outside K8s was easier, cheaper and more reliable.
- deleted 4y ago
  [deleted]
- openplatypus 4y ago
  Microservices, automations, APIs. K8s is great way to easily deploy and scale services. As we leverage Kafka, a lot of applications fallback on Topics and KTables for storage. Kubernetes with jobs, cron jobs, operators, secrets etc is effectively an Operating System for the cloud. Definitely worth the investment to learn and practice.

MrStonedOne 4y ago

singron 4y ago

I just looked at the scripts for the last StatefulSet thing I did, and it has the exact workaround in the apply script. I.e. recreate the ss, edit the pvcs, and restart the rollout when there is a volume size change. K8s has really been an experience of encountering 2-4 year old bugs (closed by the stale bot of course) and missing features. We accumulate increasingly more workarounds and additional pieces of complexity to deal with this. Everything is always changing, but somehow still staying the same.

mjg235 4y ago
Yeah it's wild that it's been a pain point for so long without a properly supported solve.

ec109685 4y ago

This fundamentally untrue: “But, Kubernetes was originally intended to act as a container orchestration platform for stateless workloads, not stateful applications.” Even from the early days, there were constructs to allow stateful applications to be built on top of it. They have made it easier to run stateful applications, but to say it wasn’t designed to support stateful applications is incorrect: https://kubernetes.io/blog/2016/08/stateful-applications-using-kubernetes-datera/ https://kubernetes.io/blog/2016/08/stateful-applications-usi... “Stateful applications such as MySQL, Kafka, Cassandra, and Couchbase for a while, the introduction of Pet Sets has significantly improved this support.”

zoomzoom 4y ago

Generally, from teams we talk to at Coherence (withcoherence.com), it seems to be a mistake to try to run stateful workloads in Kubernetes. As much as the cost savings compared to managed services like AWS RDS or GCP Cloud SQL are attractive, the configuration time and maintenance are just not worth it. Our perspective is that container orchestration tools should be used to manage stateless workloads, and that stateful workloads should delegate to managed services where volume reliability, scalability, and backup can be handled with purpose-built tools.

koprulusector 4y ago

These are valid points. I’ve personally needed to do things like delete the StatefulSet, change volume size, update/change pvc, update StatefulSet, reapply. I just took it as a given and hadn’t thought much about the ergonomics. It’s really not that big of a deal and probably a ten minute process, tops. That said, OP makes a good point that mechanisms to handle this kind of stuff exist in CSI, so why not leverage / take advantage of that versus pissing in the wind (especially when you run into CRD that wants to undo the above work while you’re in the process)?

smarterclayton 4y ago
At the time we designed the StatefulSet, the resize behavior hadn’t been designed / finalized. Now that it has, it’s really a matter of someone having the time and patience to drive it though. I don’t know whether someone has sorted out all the design implications (like what happens if someone adds new resource requests, or what happens if a new resource request is rejected, and how that gets reported to the user), but it’s a small change that needs a fair amount of design thought, which tends to be hard to drive through.

de6u99er 4y ago

There's a pending pull request that solves this problem since end of June. https://github.com/kubernetes/enhancements/pull/3412 https://github.com/kubernetes/enhancements/pull/3412

lazyant 4y ago

StatefulSets are introduced as "when you need persistent storage" and not a lot more than that. A key idea that is often skipped is that k8s doesn't want to presume how a cluster of "database" nodes ("database" in the wide sense, including RMQ, Redis etc) coordinate their HA. So when a k8s node dies with a sts on it, unlike other pods, the sts pod is NOT recreated (not wanting to interfere with whatever mechanism the db cluster has to deal with it). I've found most people working with k8s don't know about this behaviour.

rdelpret 4y ago

Write a KEP not a blog post?

fathead_glacier 4y ago

Managing state is hard and always has been regardless of the underlying technology. I have been using Kubernetes for a few years now and whenever I see people putting manual steps they should instead be using an operator. What the author is describing as a problem has a solution (they even added it to their own operator). Kubernetes has a steep learning curve but if you want to do nontrivial thing like have state than you need to learn nontrivial things like operators. The default controllers that come with Kubernetes do not cover everything under the sun and they do require extensions via the operator pattern for some tasks. Just my two cents.

nunez 4y ago

The Kubernetes authors were explicit in stating that stateless workloads were their design target, and it is somewhat obvious that StatefulSets were a concession, so this isn't terribly surprising to me. I have a feeling that OpenShift handles stateful workloads more gracefully due to their enterprise-oriented customer base, but I have nothing to back that.