6 ms·

Heh. I feel your pain. EKS is a nice platform, but I have seen the light and try not to run anything that requires a persistent file system on it. I feel like I

Heh. I feel your pain. EKS is a nice platform, but I have seen the light and try not to run anything that requires a persistent file system on it. I feel like I'm shouting in the wilderness though- just adout every day is a "can't we deploy mysql here?"

m0xte 6y ago
Yes I’m a big proponent of stateless stuff. In fact 20 years ago I was crowing this to a deaf audience. Instead I find queues in databases that didn’t even need to exist. And our solutions, kubernetes for example, are complicated Rube Goldberg machines rather than things that make us sleep easy. However really the software industry is all about state and where to keep it because that’s where the trolls collect their gold coins. So what can we do?
- codeduck 6y ago
  I find gin helps to numb the pain...
- simiones 6y ago
  So where would a queue go in a distributed system if not in some kind of database/file system? How do you handle resilience if the queue is not stored on disk somewhere? Paxos?
  - m0xte 6y ago
    Well the point is more that a lot of stuff doesn't even need to be in a queue. It's just cookie cutter engineering. "I did this before and it worked so I'll do it again" without any consideration on what it does and why it does it.
chrisbolt 6y ago
Is this an EKS problem or a Kubernetes problem?
- user5994461 6y ago
  Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his...
  - cbsmith 6y ago
    Unless they are sharded and replicated distributed databases...
  - rad_gruchalski 6y ago
    > Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Really? Can you share details? I'm asking because I know people who run databases in Docker for years and have no issues. Just store data on a volume. > Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much Maybe not, haven't had a storage issue with Docker myself since maybe 2017. That article feels like a big rant but yes, it's ... old.
    - Townley 6y ago
      Docker volumes are great. I've also run databases in docker without issue. But they're different from kubernetes "persistent volumes" and "persistent volume claims" The issues come when you move into high availability (specifically in a Kubernetes context). K8s may decide to move your database to a different node (server) on a whim, and suddenly your data is gone. You have to solve this by having mounted, block-level storage available to your cluster (eg AWS's EBS; the product being discussed) so you can use persistent volumes. That comes with added complexity, and usually the need to incorporate db operators into your stack. As a result, a lot of k8s admins (myself included) choose to pay crazy amounts for RDS and S3 to avoid the complexities of persistent data.

m0xte 6y ago

Yes I’m a big proponent of stateless stuff. In fact 20 years ago I was crowing this to a deaf audience. Instead I find queues in databases that didn’t even need to exist. And our solutions, kubernetes for example, are complicated Rube Goldberg machines rather than things that make us sleep easy. However really the software industry is all about state and where to keep it because that’s where the trolls collect their gold coins. So what can we do?

codeduck 6y ago
I find gin helps to numb the pain...
simiones 6y ago
So where would a queue go in a distributed system if not in some kind of database/file system? How do you handle resilience if the queue is not stored on disk somewhere? Paxos?
- m0xte 6y ago
  Well the point is more that a lot of stuff doesn't even need to be in a queue. It's just cookie cutter engineering. "I did this before and it worked so I'll do it again" without any consideration on what it does and why it does it.

chrisbolt 6y ago

Is this an EKS problem or a Kubernetes problem?

user5994461 6y ago
Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his...
- cbsmith 6y ago
  Unless they are sharded and replicated distributed databases...
- rad_gruchalski 6y ago
  > Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Really? Can you share details? I'm asking because I know people who run databases in Docker for years and have no issues. Just store data on a volume. > Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much Maybe not, haven't had a storage issue with Docker myself since maybe 2017. That article feels like a big rant but yes, it's ... old.
  - Townley 6y ago
    Docker volumes are great. I've also run databases in docker without issue. But they're different from kubernetes "persistent volumes" and "persistent volume claims" The issues come when you move into high availability (specifically in a Kubernetes context). K8s may decide to move your database to a different node (server) on a whim, and suddenly your data is gone. You have to solve this by having mounted, block-level storage available to your cluster (eg AWS's EBS; the product being discussed) so you can use persistent volumes. That comes with added complexity, and usually the need to incorporate db operators into your stack. As a result, a lot of k8s admins (myself included) choose to pay crazy amounts for RDS and S3 to avoid the complexities of persistent data.