7 ms·

If you find yourself using something like this, you seriously fucked up as DevOps / cloud admin / whatever.

by devops99 2y ago

cameronh90 2y ago
Or you're budget constrained, and haven't yet been able to allocate resources to things like GitOps.
devops99 2y ago
[flagged]
- deleted 2y ago
  [deleted]
llama052 2y ago
Agreed, this feels like the kubernetes descheduler as a service or something. Wild.
paolop 2y ago
Possible, but if someone needs it and it works well, why not?
pescerosso 2y ago
I understand where you’re coming from, and ideally, we strive for well-managed Kubernetes environments. However, as DevOps practitioners, we often face complexities that lead to stale or orphaned resources due to fast deployment cycles, changing application needs or teams. Even the public clouds make lots of money from services that are left running and not used for which some companies make a living helping clean things up. K8s-cleaner serves as a helpful safety net to automate the identification and cleanup of these resources, reducing manual overhead and minimizing human error. It allows teams to focus on strategic tasks instead of day-to-day resource management.
- devops99 2y ago
  > However, as DevOps practitioners, we often face complexities that lead to stale or orphaned resources due to fast deployment cycles So, as a DevOps practitioner myself, I had enough say within the organizations I worked at, who are now clients, and also my other clients, that anything not in a dev environment goes through our GitOps pipeline. Other than the GitOps pipeline, there is zero write access to anything not dev. If we stop using a resource, we remove a line or two (usually just one) in a manifest file, the GitOps pipeline takes care of the rest. Not a single thing is unaccounted for, even if indirectly. That said, the DevOps-in-name-only clowns far outnumber actual DevOps people, and there is no doubt a large market for your product. edited: added clarity
  - nine_k 2y ago
    How do you realize that you have stopped using a resource? Can there be cases when you're hesitant to remove a resource just yet, because you want a possible rollback to be fast, and then it lingers, forgotten?
    - devops99 2y ago
      With our GitOps patterns, anything you could call an "environment" has a Git branch that reflects "intended state". So a K8s namespace named test02, another named test-featurebranch, and another named staging-featurebranch, all have their own Git branch with whatever Helm, OpenTOFU, etc. With this pattern, and other patterns, we have a principle "if it's in the Git branch, we intend for it to there, if it's not in the Git branch it can't be there". We use Helm to apply and remove things -- and we loved version 3 when it came out -- so there's not really any way for anything to linger.
  - photonthug 2y ago
    > I had enough say within the organizations I worked at, who are now clients This sounds like experience that’s mainly at small/medium sized orgs. At large orgs the devops/cloud people are constantly under pressure to install random stuff from random vendors. That pressure comes from every direction because every department head (infosec/engineering/data science) is trying to spend huge budgets to justify their own salary/headcount and maintain job security, because it’s harder to fire someone if you’re in the middle of a migrate-to-vendor process they championed, and you’re locked into the vendor contract, etc etc. People also will seek to undermine every reasonable standard about isolation and break down the walls you design between environments so that even QA or QC type vendors want their claws in prod. Best practice or not, You can’t really say no to all of it all the time or it’s perceived as obstructionist. Thus there’s constant churn of junk you don’t want and don’t need that’s “supposed to be available” everywhere and the list is always changing. Of course in the limit there is crusty unused junk and we barely know what’s running anywhere in clouds or clusters. Regardless of the state of the art with Devops, most orgs are going to have clutter because those orgs are operating in a changing world and without a decisive or even consistent vision of what they want/need.
thephyber 2y ago
Or the person working that role before you did and you are trying to manage the situation as best as possible.
- devops99 2y ago
  As a cloud/DevOps consultant, I don't believe in letting management drag on the life support of failed deployments. We (the team I built out since a couple years ago) carve out a new AWS subaccount / GCP Project or bare-metal K8s or whatever the environment is, instantiate our GitOps pipeline, and services get cut-over in order to get supported. When I was working in individual contributor roles, I managed to "manage upward" enough that I could establish boundaries on what could be supported or not. Yes this did involve a lot of "soft skills" (something I'm capable of doing even though my posts on this board are rather curt). "Every DevOps job is a political job" rings true. Do not support incumbent K8s clusters, as a famous -- more like infamous -- radio host used to say on 97.1 FM in the 90s: dump that bitch.
  - remram 2y ago
    It's really hard to parse what you are trying to say in the middle of this colorful imagery and quotations, but if you are spinning up a new Kubernetes cluster (and AWS/GCP account) per deployment, it's obvious you don't need this tool.
    - devops99 2y ago
      Specifically, leaving the corporate political dynamics aside, we move the workloads to the deployments that are up to standard and then archive+drop the old ones. Very simple.
      - remram 2y ago
        A more cynical man than I would say that if you need to recreate all your workloads on a new cluster to bring them up to standard, "you seriously fucked up as DevOps / cloud admin / whatever"
benreesman 2y ago
No one likes leaking resources, but it happens to the very best teams. It seems like a tool that can propose a good guess about where you landed is strictly useful and good?
- devops99 2y ago
  > but it happens to the very best teams. I am very convinced it does not. I think where the apex between our viewpoints lies is what we recognize (or not) as "best" teams.
  - benreesman 2y ago
    If you know how to do serious software without spilling any memory then you should write it up and collect the most significant Turing Award ever. Hackers would be immediately bifurcated into those who followed that practice and those who are helpless.
    - devops99 2y ago
      > If you know how to do serious software without spilling any memory This thread is about K8s Pods (and other K8s resources) that have been sitting idle, not memory leaks in software. As far as "spilling" memory, the problem has already been solved by Rust which does not do garbage collection because it has static memory mapping. Does this mean egregious amounts of memory won't be used by some Rust programs? No. But unlike languages with garbage collection, where Rust is using that memory it is actually doing something with that memory.
      - benreesman 2y ago
        I can assure you that memory leaks specifically and resource leaks generally are possible in any language more expressive than a stack machine with an arena. Rust makes an interesting and demonstrably pragmatic set of tradeoffs here as opposed to e.g. C/C++: treating std::move as a default (aka linear typing) prevents a lot of leaks. But it still has pointers (Arc, etc.) and it still has tables: it’s still easy to leak in a long-lived process doing interesting things. For a lot of use cases it’s the better default and it’s popular as a result. But neither Rust nor k8s have solved computer science.
jrsdav 2y ago
Not all third party Kubernetes controllers are created equal, unfortunately. And it is also the reality that not every infra team gets final say on what operators we have to deploy to our clusters (yet we're stuck cleaning up the messes they leave behind).
remram 2y ago
I don't agree, you can see it as just another linting tool. Just like your IDE will warn of unused variables and functions, and offer a quick-fix to remove them. You wouldn't call someone a fucked up programmer for using an IDE.
- devops99 2y ago
  [flagged]
  - remram 2y ago
    I am assuming that the tool provides a list and asks for confirmation before deleting? Does it just go and automatically delete stuff in the background at any time? edit: yes I see a "dry run" option, which is how I would use it. I also see a "scheduled" option which is probably what you're criticizing. Hard to tell, you're quicker with the insulting than the arguing.

If you find yourself using something like this, you seriously fucked up as DevOps / cloud admin / whatever.

cameronh90 2y ago

Or you're budget constrained, and haven't yet been able to allocate resources to things like GitOps.

devops99 2y ago

[flagged]

deleted 2y ago
[deleted]

llama052 2y ago

Agreed, this feels like the kubernetes descheduler as a service or something. Wild.

paolop 2y ago

Possible, but if someone needs it and it works well, why not?

pescerosso 2y ago

I understand where you’re coming from, and ideally, we strive for well-managed Kubernetes environments. However, as DevOps practitioners, we often face complexities that lead to stale or orphaned resources due to fast deployment cycles, changing application needs or teams. Even the public clouds make lots of money from services that are left running and not used for which some companies make a living helping clean things up. K8s-cleaner serves as a helpful safety net to automate the identification and cleanup of these resources, reducing manual overhead and minimizing human error. It allows teams to focus on strategic tasks instead of day-to-day resource management.

devops99 2y ago
> However, as DevOps practitioners, we often face complexities that lead to stale or orphaned resources due to fast deployment cycles So, as a DevOps practitioner myself, I had enough say within the organizations I worked at, who are now clients, and also my other clients, that anything not in a dev environment goes through our GitOps pipeline. Other than the GitOps pipeline, there is zero write access to anything not dev. If we stop using a resource, we remove a line or two (usually just one) in a manifest file, the GitOps pipeline takes care of the rest. Not a single thing is unaccounted for, even if indirectly. That said, the DevOps-in-name-only clowns far outnumber actual DevOps people, and there is no doubt a large market for your product. edited: added clarity
- nine_k 2y ago
  How do you realize that you have stopped using a resource? Can there be cases when you're hesitant to remove a resource just yet, because you want a possible rollback to be fast, and then it lingers, forgotten?
  - devops99 2y ago
    With our GitOps patterns, anything you could call an "environment" has a Git branch that reflects "intended state". So a K8s namespace named test02, another named test-featurebranch, and another named staging-featurebranch, all have their own Git branch with whatever Helm, OpenTOFU, etc. With this pattern, and other patterns, we have a principle "if it's in the Git branch, we intend for it to there, if it's not in the Git branch it can't be there". We use Helm to apply and remove things -- and we loved version 3 when it came out -- so there's not really any way for anything to linger.
- photonthug 2y ago
  > I had enough say within the organizations I worked at, who are now clients This sounds like experience that’s mainly at small/medium sized orgs. At large orgs the devops/cloud people are constantly under pressure to install random stuff from random vendors. That pressure comes from every direction because every department head (infosec/engineering/data science) is trying to spend huge budgets to justify their own salary/headcount and maintain job security, because it’s harder to fire someone if you’re in the middle of a migrate-to-vendor process they championed, and you’re locked into the vendor contract, etc etc. People also will seek to undermine every reasonable standard about isolation and break down the walls you design between environments so that even QA or QC type vendors want their claws in prod. Best practice or not, You can’t really say no to all of it all the time or it’s perceived as obstructionist. Thus there’s constant churn of junk you don’t want and don’t need that’s “supposed to be available” everywhere and the list is always changing. Of course in the limit there is crusty unused junk and we barely know what’s running anywhere in clouds or clusters. Regardless of the state of the art with Devops, most orgs are going to have clutter because those orgs are operating in a changing world and without a decisive or even consistent vision of what they want/need.

thephyber 2y ago

Or the person working that role before you did and you are trying to manage the situation as best as possible.

devops99 2y ago
As a cloud/DevOps consultant, I don't believe in letting management drag on the life support of failed deployments. We (the team I built out since a couple years ago) carve out a new AWS subaccount / GCP Project or bare-metal K8s or whatever the environment is, instantiate our GitOps pipeline, and services get cut-over in order to get supported. When I was working in individual contributor roles, I managed to "manage upward" enough that I could establish boundaries on what could be supported or not. Yes this did involve a lot of "soft skills" (something I'm capable of doing even though my posts on this board are rather curt). "Every DevOps job is a political job" rings true. Do not support incumbent K8s clusters, as a famous -- more like infamous -- radio host used to say on 97.1 FM in the 90s: dump that bitch.
- remram 2y ago
  It's really hard to parse what you are trying to say in the middle of this colorful imagery and quotations, but if you are spinning up a new Kubernetes cluster (and AWS/GCP account) per deployment, it's obvious you don't need this tool.
  - devops99 2y ago
    Specifically, leaving the corporate political dynamics aside, we move the workloads to the deployments that are up to standard and then archive+drop the old ones. Very simple.
    - remram 2y ago
      A more cynical man than I would say that if you need to recreate all your workloads on a new cluster to bring them up to standard, "you seriously fucked up as DevOps / cloud admin / whatever"

benreesman 2y ago

No one likes leaking resources, but it happens to the very best teams. It seems like a tool that can propose a good guess about where you landed is strictly useful and good?

devops99 2y ago
> but it happens to the very best teams. I am very convinced it does not. I think where the apex between our viewpoints lies is what we recognize (or not) as "best" teams.
- benreesman 2y ago
  If you know how to do serious software without spilling any memory then you should write it up and collect the most significant Turing Award ever. Hackers would be immediately bifurcated into those who followed that practice and those who are helpless.
  - devops99 2y ago
    > If you know how to do serious software without spilling any memory This thread is about K8s Pods (and other K8s resources) that have been sitting idle, not memory leaks in software. As far as "spilling" memory, the problem has already been solved by Rust which does not do garbage collection because it has static memory mapping. Does this mean egregious amounts of memory won't be used by some Rust programs? No. But unlike languages with garbage collection, where Rust is using that memory it is actually doing something with that memory.
    - benreesman 2y ago
      I can assure you that memory leaks specifically and resource leaks generally are possible in any language more expressive than a stack machine with an arena. Rust makes an interesting and demonstrably pragmatic set of tradeoffs here as opposed to e.g. C/C++: treating std::move as a default (aka linear typing) prevents a lot of leaks. But it still has pointers (Arc, etc.) and it still has tables: it’s still easy to leak in a long-lived process doing interesting things. For a lot of use cases it’s the better default and it’s popular as a result. But neither Rust nor k8s have solved computer science.

jrsdav 2y ago

Not all third party Kubernetes controllers are created equal, unfortunately. And it is also the reality that not every infra team gets final say on what operators we have to deploy to our clusters (yet we're stuck cleaning up the messes they leave behind).

remram 2y ago

I don't agree, you can see it as just another linting tool. Just like your IDE will warn of unused variables and functions, and offer a quick-fix to remove them. You wouldn't call someone a fucked up programmer for using an IDE.

devops99 2y ago
[flagged]
- remram 2y ago
  I am assuming that the tool provides a list and asks for confirmation before deleting? Does it just go and automatically delete stuff in the background at any time? edit: yes I see a "dry run" option, which is how I would use it. I also see a "scheduled" option which is probably what you're criticizing. Hard to tell, you're quicker with the insulting than the arguing.