8 ms·

If you spin up Kubernetes for "a couple of containers to run your web app", I think you're doing something wrong in the first place, also coupled with your comm

If you spin up Kubernetes for "a couple of containers to run your web app", I think you're doing something wrong in the first place, also coupled with your comment about adding SDN to Kubernetes. People use Kubernetes for way too small things, and it sounds like you don't have the scale for actually running Kubernetes.

dajonker 5mo ago
I totally agree, but that's not what happens in reality: the average devops knows k8s and will slap it onto anything they see (if only so they can put in on their resume). The average manager hears about k8s, gets convinced they need and hires beforementioned devops to build it.
- goombaskoop 5mo ago
  > the average devops knows k8s and will slap it onto anything they see This is certainly the case from all the third person accounts I hear. Online. I never actually met a single one that is like that, if anything, those same people are the ones that are first to tell me about their Hetzner setups.
  - hkt 5mo ago
    DevOps here. The trouble is that we are literally expected to do this everywhere we go. I've personally advocated for approaches which use say, a pair of dedicated servers, or VMs as in GPs example. If you want it outside of AWS/GCP/Azure, you're regarded as a crazy person. If you don't adopt "best practices" (as defined by vendors) then management are scared. Management very often trust the sales and marketing departments of big vendors more than their own staff. Many of us have given up fighting this, because what it comes down to is a massive asymmetry of information and trust.
    - regularfry 5mo ago
      There is a kernel of validity lurking in the heart of all this, which is that immutable images you have the ability to throw away and refresh regularly are genuinely better than long-running VMs with an OS you've got to maintain, with the scope for vulnerabilities unrelated to the app you actually want to run. Management has absorbed this one good thing and slapped layer after layer of pointless rubbish on it, like a sort of inverse pearl. Being able to say "we've minimised our attack surface with a scratch image" (or alpine, or something from one of the secure image vendors) is a genuinely valuable thing. It's just the all of the everything that goes along with it...
      - dijit 5mo ago
        Sure. The challenge is convincing people that "golden images" and containers share a history, and that kubernetes didn't invent containers: they just solved load balancing and storage abstraction for stateless message architectures in a nice way. If you're doing something highly stateful, or that requires a heavy deployment (game servers are typically 10's of GB and have rich dynamic configuration in my experience) then kubernetes starts to become round-peg-square-hole. But people buy into it because the surrounding tooling is just so nice; and like GP says: those cloud sales guys are really good at their jobs, and kubernetes is so difficult to run reliably yourself that it gets you hooked on cloud. There's a literal army of highly charismatic, charming people who are economically incentivised to push this technology and it can be made to work so- the odds, as they say, are against you.
    - nz 5mo ago
      The following happened to a friend. Around the time of the pandemic, a company wanted to make some Javascript code do a kind of transformation over large number of web-pages (a billion or so, fetched as WARC files from the web archive). Their engineers suggested setting up SmartOS VMs and deploying Manta (which would have allowed the use of the Javascript code in a totally unmodified way -- map-reduce from the command-line, that scales with the number storage/processing nodes) which should have taken a few weeks at most. After a bit of googling and meeting, the higher ups decided to use AWS Lambdas and Google Cloud Functions, because that's what everyone else was doing, and they figured that this was a sensible business move because the job-market must be full of people who know how to modify/maintain Lambda/GCF code. Needless to say, Lambda/GCF were not built for this kind of workload, and they could not scale. In fact, the workload was so out-of-distribution, that the GCP folks moved the instances (if you can call them that) to a completely different data-center, because the workload was causing performance problems, for _other_ customers in the original data-center. Once it became clear that this approach cannot scale to a billion or so web-pages, it was decided to -- no, not to deploy Manta or an equivalent -- but to build a custom "pipeline" from scratch, that would do this. This system was in development for 6 months or so, and never really worked correctly/reliably. This is the kind of thing that happens when non-engineers can override or veto engineering decisions -- and the only reason they can do that, is because the non-engineers sign the paychecks (it does not matter how big the paycheck is, because market will find a way to extract all of it). One of the fallacies of the tech-industry (I do not mean to paint with too broad a brush, there are obviously companies out there that know what they are doing) is that there are trade-offs to be made between business-decisions and engineering-decisions. I think this is more a kind of psychological distortion or a false-choice (forcing an engineering decision on the basis of what the job market will be like some day in the future -- during a pandemic no less -- is practically delusional). Also, if such trade-offs are true trade-offs, then maybe the company is not really an engineering company (which is fine, but that is kind of like a shoe-store having a few podiatrists on staff -- it is wasteful, but they can now walk around in white lab-coats, and pretend to be a healthcare institution instead of a shoe-store). Personally, I believe that the tech industry sustains itself via technical debt, much like the real economy sustains itself on real debt. In some sense, everyone is trying to gaslight everyone else into incurring as much technical debt as possible, so that a way to service the debt can be sold. Most of the technical debt is not necessary, and if people were empowered to just not incur it, I suspect it would orient tech companies towards making things that actually push the state of the art forward.
    - jcgrillo 5mo ago
      > Management very often trust the sales and marketing departments of big vendors more than their own staff. They're getting kickbacks from cloud vendors. Prove me wrong.
      - r_lee 5mo ago
        not sure if this is a thing with Cloud vendors, but e.g. in Finance, you'll definitely get the opportunity to call your rep over for free fancy dinners or whatever you want, because those are "customer meetings" better than nothing, I don't blame em.
        jcgrillo 5mo ago
        > better than nothing, I don't blame em. Money ain't got no owners, only spenders.
    - vladvasiliu 5mo ago
      > If you want it outside of AWS/GCP/Azure, you're regarded as a crazy person. If you don't adopt "best practices" (as defined by vendors) then management are scared. Management very often trust the sales and marketing departments of big vendors more than their own staff. Many of us have given up fighting this, because what it comes down to is a massive asymmetry of information and trust. I think this is the crux of the matter. Also, "everybody is doing it, so they must be right" is also a very common way of thinking amongst this population.
    - jimbokun 5mo ago
      “Nobody ever got fired for buying… …IBM …Microsoft …AWS.”
  - ownagefool 5mo ago
    To be fair, I have k8s on my hetzner :p
- darkwater 5mo ago
  And the average developer doesn't even know where to start to deploy things in prod. When the feature product asks passes QA... to the next sprint! we are done!
  - chrisweekly 5mo ago
    Whose responsibility is it to establish the prerequisite CICD pipelines, HITL workflows, and Observability infr in order for devs to shepherd changes to prod (and track their impact)? Hint: it's not the developer's.
    - liveoneggs 5mo ago
      But those things all require more pods on the cluster! We've looped back around to the beginning.
      - darkwater 5mo ago
        Exactly my point. But then developers: "I just want to go to my Heroku days again!" but then with a sufficient big company there are maaany developers doing things their slightly different way, and then other effects start compounding, and then costs go up because 15 different teams are using 27 different solutions and and and... But yeah, let's just spin-up a shadow IT VM with Debian like GP said, it's easy!
        throwup238 5mo ago
        > But yeah, let's just spin-up a shadow IT VM with Debian like GP said, it's easy! That’s literally how they sold AWS in the beginning. Cloud won not because of costs or flexibility but because it allowed teams to provision their own machines from their budget instead of going through all the red tape with their IT departments creating… a bunch of shadow IT VMs! Everything old is new again, except it works on an accelerated ten year cycle in the IT industry.
        5mo ago
- tete 5mo ago
  > the average devops knows k8s If you'd know Kubernetes, you know not to use it. I say that as someone who used to do consulting for it. The reality is that yet again "making money" completely collides with efficient, quality, sane productive work. For me one of the main reasons to leave that space is that I couldn't really deal with the fact that my work collides with a client's success. That said I have helped to get off that stuff and other things that they thought they needed, that just wasted time and money. It just feels odd going into a company that hired you to consult on a topic only to end up telling them "The best approach for you is not doing that at all". Often never. Like some people thought "Well, if we have hundreds of thousands or even millions of users" and the reality was that even in these scenarios if you went away from that abstract thought and discussed a hypothetical based on their product they realized that they'd still be better off without it. Besides the fact that this hypothetical often was in a future that made it likely that they said they'd likely have completely different setup so preparing for that didn't even make sense. I think a big thing related to that was/is the microservice craze where people end up moving to a complex architecture for not many good reasons and then they increase complexity way faster than what they actually deliver in terms of the product, because it somehow feels good. I know it does, I've been there. When in reality the outcome often is just a complex mess with what could have been a relatively simple monolith. And these monoliths do work. And in the vast majority of cases they are easy to scale, because your problem switches from "how do we best allocate that huge amount of very different services across our infrastructure" to (for the most part) "how do we spin up our monolith on one more server" which tends to be a way easier to tackle service. And nothing stops you from still using everything else if you want. Just because it's a monolith doesn't mean you need to skip on any of the cloud offerings, etc. For some reason there seems to be that idea that if you write a monolith you are somehow barred from using modern tooling, infrastructure, services, etc. Not sure where that comes from.
  - r_lee 5mo ago
    I think one big problem is that using microservice architecture doesn't mean that literally everything has to be a "microservice". if you don't truly need granual scaling (i.e. your "app" doesn't get a bunch of asymmetric loads across different paths), then you can just have more monolithic "microservices" until they need to be split up imo this should achieve a nice balance?
rvz 5mo ago
They use it for inflating their resume for career progression rather than actually evaluating if they need it in the first place. This is why you get many folks over-thinking the solution and picking the most hyped technologies and using them to solve the wrong problems without thinking about what they are selling. You don't need K8s + AWS EC2 + S3 just to host a web app. That tells me they like lighting money on fire and bankrupting the company and moving to the next one.
- p_l 5mo ago
  Often the alternatives presented as cheaper to me in discussions are actually burning money. But given how I always see "you don't need k8s because you're not going to scale so fast" I am feel like even professional k8s operators have missed the fundamental design goals of it :/ (maximizing utilization of finite compute)
Thanemate 5mo ago
I know that "resume-driven development" exists, where the tradeoffs between approaches aren't about the technical fit of the solution but the career trajectory. I've seen people making plain workstation preparation scripts using Rust, only to have something to flex about in interviews. I'm not surprised even in the slightest that DevOps workers will slap k8s on everything, to show "real industry experience" in a job market where the resume matches the tools.
- ororoo 5mo ago
  there are alsp people with devops title that do not know anything else than the hammer, and then everything is a hammer problem. I mean, I worked with people who were suprised that you can run more applications inside ec2 vm than just 1 app.
  - tete 5mo ago
    > there are alsp people with devops title that do not know anything else than the hammer, and then everything is a hammer problem. To be fair though, that's true for every profession or skill. > I mean, I worked with people who were suprised that you can run more applications inside ec2 vm than just 1 app. I've seen something similar where people were surprised that you can use an object storage (so effectively "make HTTP requests") from every server.
  - jauntywundrkind 5mo ago
    Conversely, we had millions of server huggers before, who each knew their company's stuff in a way that wasn't really applicable if they went somewhere else. Every company used to have a bespoke collection of build, deployment, monitoring, scaling, etc concerns. Everyone had their own practices, their own wikis to try to make sense of what they had. I think we critically under-appreciate that k8s is a social technology that is broadly applicable. Not just for hosting containers, but as a cloud-native form of thinking, where it becomes much easier to ask: what do we have here, and is it running well, and to have systems that are helping you keep that all on track (autonomic behavior/control loops). I see such rebellion & disdain for where we are now, but so few people who seem able to recognize and grapple with what absolute muck we so recently have crawled out of.
altmanaltman 5mo ago
yeah it's like wanting to drive to the mall in the Space Shuttle and then complaining how its too complicated
littlestymaar 5mo ago
I have nom doubt that there are legit use cases for something like k8s at Google or other multi-billion companies. But if its use was confined to this use case, pretty much nobody would be using it (unless as a customer of the organization's infra) and barely would be talking about it (like how there isn't too much talk about Borg). The reason k8s is a thing in the first place is because it's being used by way too many people for their own goods. (Most people having worked in startups have met too many architecture astronauts in our lives). If I had to bet, I'd wager that 99% of k8s users are in the “spin a few containers to run your web app” category (for the simple reason that for one billion-dollar tech business using it for legit reasons, there's many thousands early startups who do not).
- rantanplan 5mo ago
  The legit use case for companies like Google/Amazon etc is only to sell it to customers. None of these companies use K8s internally for real critical workloads.
  - bitexploder 5mo ago
    Ehm, that is simply not true. Google built it for themselves first. It is essentially the open source version of the internal architecture. It gets used.
    - zaphar 5mo ago
      I worked at google. k8s does not really look at all like what they used internally when I was there, aside from sharing some similar looking building blocks.
      - oblio 5mo ago
        Yeah, but is the internal tool simpler? I'd be surprised.
        zaphar 5mo ago
sdevonoes 5mo ago
Depends. For personal projects, yeah definitely. But at work? Typically the “Platform” team can only afford to support 1 (maybe 2) ways of deployment, and k8s is quite versatile, so even if you need 1 small service, you’ll go with the self-service-k8s approach your Platform team offers. Because the alternative is for you (or your team) to own the whole infrastructure stack for your new deloyment model (ecs? lambda? Whatever): so you need to setup service accounts, secret paths, firewalls, security, pipelines, registries, and a large etc. And most likely, no one will give you access rights for all of that , and your PM won’t accept the overhead either. So having everyone use the same deployment model (and that’s typically k8s) saves effort. I don’t like it for sure
- limaho 5mo ago
  This is where I'm at. Using Podman daily to run Python scripts and apps and it's been going great! However trying to build things like monitoring, secure secret injection, centralized inventory, remote logging, etc. has fallen on us. Has lead to some shadow IT (running our own container image registry, hashicorp vault instance, etc.) which makes me hesitant to share with others in the company how we're operating. I like to think if we had a K8s environment a lot of this would be built out within it. Having that functionality abstracted away from the developer would be a huge win in my opinion.
  - evanphx 5mo ago
    Are you doing that across a fleet of machines or just one?
    - limaho 5mo ago
      We have 4 servers we run containers on. Calling that a fleet feels too generous. Not much rhyme or reason as to what containers run on which server
ownagefool 5mo ago
It depends what you're doing it. My app is fairly simple node process with some side car worker processes. k8s enables me to deploy it 30 times for 30 PRs, trivially, in a standard way, with standard cleanup. Can I do that without k8s? Yes. To the same standard with the same amount of effort? Probably not. Here, I'd argue the k8s APIs and interfaces are better than trying to do this on AWS ( or your preferred cloud provider ). Where things get complicated is k8s itself is borderline cloud provider software. So teams who were previously good using a managed service are now owning more of the stack, and these random devops heros aren't necessarily making good decisions everywhere. So you really have three obvious use cases: a) You're doing something interesting with the k8s APIs, that aren't easy to do on a cloud provider. Essentially, you're a power user. b) You want a cloud abstraction layer because you're multi-cloud or you want a lock-in bargaining chip. c) You want cloud semantics without being on a cloud provider. However, if you're a single developer with a single machine, or a very small team and you're happy working through contended static environments, you can pretty much just put a process on a box and call it done. k8s is overkill here, though not as much as people claim until the devops heros start their work.
- shimman 5mo ago
  Call me old fashion but I prefer tools like Dokploy that make deployment across different VPS extremely easy. Dokploy allows me to utilize my home media server, using local instances of forgejo to deploy code, to great effect. k8s appears to be a corporate welfare jobs program where trillion dollar multinational monopolistic companies are the only ones who can collectively spend 100s of millions sustaining. Since most companies aren't trillion dollar monopolies, adopting such measures seems extremely poor. All it signals to me is that we have to stop letting SV + VC dictate the direction of tech in our industry, because their solutions are unsustainable and borderline useless for the vast majority of use cases. I'll never forget the insurance companies I worked at that orchestrated every single repo with a k8s deployment whose cloud spend was easily in the high six figures a month to handle a work load of 100k/MAU where the concurrent peak never went more than 5,000 users, something the company did know with 40 years of records. Literally had a 20 person team whose entire existence was managing the companies k8s setup. Only reason the company could sustain this was that it's an insurance company (insurance companies are highly profitable, don't let them convince you otherwise; so profitable that the government has to regulate how much profit they're legally allowed to make). Absolute insanity, unsustainable, and a tremendous waste of limited human resources. Glad you like it for your node app tho, happy for you.
tjarjoura 5mo ago
In some sense, Kubernetes is just a portable platform for running Linux services, even on a single node using something like K3s. I almost see it as being an extension of the Linux OS layer.
- sgt 5mo ago
  Then why can't we put a wrapper onto systemd and make that into a light weight k8s?
  - enos_feedler 5mo ago
    Remember fleet?
  - marcosdumay 5mo ago
    Systemd is on the wrong layer here. You need something that can set your machine up, like docker.
    - sgt 5mo ago
      Okay it sets the machine up, but not the underlying host machine though.
    - jasonjayr 5mo ago
      Systemd seems to be moving in that direction, the features are coming together to actually enable this. Though imagining the unholy existence of an init system who's only job is to spin up containers, that can contain other inits, OS images, or whatever ..... turtles all the way down.
      - sgt 5mo ago
        I don't see why not. Maybe it should be on the same layer going forward - for true cloud compute (including on-premise cloud)
    - necovek 5mo ago
      Docker does not set your machine up: it sets your process up. Tech like Flatpak or Snap is closer to Docker than "machines" are — except that Docker has local, virtualized networking built-in as the IPC layer.
hombre_fatal 5mo ago
k8s is useful when you have services that must spin up and down together, and you want to swap out services and deploy all/some/one. and then also package this so that you and other developers can get the infrastructure running locally or on other machines.
geodel 5mo ago
Doing Kubernetes like doing Agile is mandatory nowadays. I've been asked to package a 20 line worth of bash script as docker image so it can be delivered via CI/CD pipeline via Kubernetes pods in cloud. Value is not that I got job done at a day's notice. It is black mark that I couldn't package it as per industry best practices. Not doing would mean out of job/work. Whether it is happening correctly is not something decision makers care as long it is getting done anyhow.
- necovek 5mo ago
  In my 20+ years in the industry, I've been at one company which really did Agile, and that was the one I started with. Everyone else is communicating they are doing Agile while being very far away from it ;)
  - bdangubic 5mo ago
    if anyone knew what agile is maybe more would have a chance if making it work (it won’t). in my 30* the only “process” that worked and works is “hire the right people and get the F out of the way.”
    - necovek 5mo ago
      That's pretty much what agile is :)
- radlad 5mo ago
  There are many organizations which still ship software without Kubernetes. Perhaps even the vast majority.
  - geodel 5mo ago
    Of course. I used to think I am working for one such organization for long time. Until leadership decided "modernization" as top priority for IT teams as we are lagging far.
- jauntywundrkind 5mo ago
throwaway894345 5mo ago
Even if using just one VM, I'll probably slap k3s on it and manage my application using manifests. It's just so much easier than dealing with puppet or chef or vanilla cloud-init. Docker compose works too, but at that point it's just easier to stick with k3s and then I can have nice things like background jobs, a straightforward path to HA, access to an ecosystem of existing software, and a nicer CLI.
- tayo42 5mo ago
  Thats what I don't get when people bring up this idea k8s is complicated. All of those other tools are complicated and fragile
  - throwaway894345 5mo ago
    I think the things that trip people up are: 1. People expect k8s to be an opinionated platform and it's very happy to let you make a mess 2. People think k8s is supposed to be a cross platform portability layer and ... it maybe can be if you're very careful, but it's mostly not that 3. People compare k8s/cloud/etc to some monolithic application with admin permissions to everything and they compare that to the "difficulty" of dealing with RBAC/IAM/networking/secrets management 4. People don't realize how much more complicated vanilla Linux tooling and how much more accidental complexity is involved
moooo99 5mo ago
> People use Kubernetes for way too small things, and it sounds like you don't have the scale for actually running Kubernetes. This is a problem I've run into enterprise deployments. K8s is often the lowest common denominator semi small platform engineering teams arrive on. At my current employer, a platform managed K8s namespace is the only thing we got in terms of PaaS offering, so it is what we use. Is it overpowered? Yes. Is it overly complex for our usecase? Definitely. Could we basically get by hosting our services on a few cheap mini computers with no performance penalty? Also yes.
firesteelrain 5mo ago
We have a hobby web based app that consists of multiple containers. It runs in docker compose. Serves 1000 users right now (runs 24/7). Single VM. No Kubernetes whatsoever. I agree with you.
- giobox 5mo ago
  Docker compose is brilliant while your stack remains on a single box, and will scale quite nicely for some time this way for most applications with minimum maintenance overhead. My personal strategy has always been to start off in docker compose, and break out to a k8s configuration later if I have to start scaling beyond single box.
jen20 5mo ago
The problem with Kubernetes is that it doesn't scale down to small deployments very well, but it sure as shit doesn't scale up to large ones either. Large shared multi-tenant clusters have massive problems even when running parts of the same application with the same incentives, it falls apart completely when the tenants are diverse. Nomad has neither of these problems.
mbesto 5mo ago
> it sounds like you don't have the scale for actually running Kubernetes. You don't set up k8s because your current load can't be handled, you do for future growth. Sometimes that growth doesn't pan out and now you're left with a complex infrastructure that is expensive to maintain and not getting any of the benefit.
deleted 5mo ago
[deleted]

dajonker 5mo ago

I totally agree, but that's not what happens in reality: the average devops knows k8s and will slap it onto anything they see (if only so they can put in on their resume). The average manager hears about k8s, gets convinced they need and hires beforementioned devops to build it.

goombaskoop 5mo ago
> the average devops knows k8s and will slap it onto anything they see This is certainly the case from all the third person accounts I hear. Online. I never actually met a single one that is like that, if anything, those same people are the ones that are first to tell me about their Hetzner setups.
- hkt 5mo ago
  DevOps here. The trouble is that we are literally expected to do this everywhere we go. I've personally advocated for approaches which use say, a pair of dedicated servers, or VMs as in GPs example. If you want it outside of AWS/GCP/Azure, you're regarded as a crazy person. If you don't adopt "best practices" (as defined by vendors) then management are scared. Management very often trust the sales and marketing departments of big vendors more than their own staff. Many of us have given up fighting this, because what it comes down to is a massive asymmetry of information and trust.
  - regularfry 5mo ago
    There is a kernel of validity lurking in the heart of all this, which is that immutable images you have the ability to throw away and refresh regularly are genuinely better than long-running VMs with an OS you've got to maintain, with the scope for vulnerabilities unrelated to the app you actually want to run. Management has absorbed this one good thing and slapped layer after layer of pointless rubbish on it, like a sort of inverse pearl. Being able to say "we've minimised our attack surface with a scratch image" (or alpine, or something from one of the secure image vendors) is a genuinely valuable thing. It's just the all of the everything that goes along with it...
    - dijit 5mo ago
      Sure. The challenge is convincing people that "golden images" and containers share a history, and that kubernetes didn't invent containers: they just solved load balancing and storage abstraction for stateless message architectures in a nice way. If you're doing something highly stateful, or that requires a heavy deployment (game servers are typically 10's of GB and have rich dynamic configuration in my experience) then kubernetes starts to become round-peg-square-hole. But people buy into it because the surrounding tooling is just so nice; and like GP says: those cloud sales guys are really good at their jobs, and kubernetes is so difficult to run reliably yourself that it gets you hooked on cloud. There's a literal army of highly charismatic, charming people who are economically incentivised to push this technology and it can be made to work so- the odds, as they say, are against you.
  - nz 5mo ago
    The following happened to a friend. Around the time of the pandemic, a company wanted to make some Javascript code do a kind of transformation over large number of web-pages (a billion or so, fetched as WARC files from the web archive). Their engineers suggested setting up SmartOS VMs and deploying Manta (which would have allowed the use of the Javascript code in a totally unmodified way -- map-reduce from the command-line, that scales with the number storage/processing nodes) which should have taken a few weeks at most. After a bit of googling and meeting, the higher ups decided to use AWS Lambdas and Google Cloud Functions, because that's what everyone else was doing, and they figured that this was a sensible business move because the job-market must be full of people who know how to modify/maintain Lambda/GCF code. Needless to say, Lambda/GCF were not built for this kind of workload, and they could not scale. In fact, the workload was so out-of-distribution, that the GCP folks moved the instances (if you can call them that) to a completely different data-center, because the workload was causing performance problems, for _other_ customers in the original data-center. Once it became clear that this approach cannot scale to a billion or so web-pages, it was decided to -- no, not to deploy Manta or an equivalent -- but to build a custom "pipeline" from scratch, that would do this. This system was in development for 6 months or so, and never really worked correctly/reliably. This is the kind of thing that happens when non-engineers can override or veto engineering decisions -- and the only reason they can do that, is because the non-engineers sign the paychecks (it does not matter how big the paycheck is, because market will find a way to extract all of it). One of the fallacies of the tech-industry (I do not mean to paint with too broad a brush, there are obviously companies out there that know what they are doing) is that there are trade-offs to be made between business-decisions and engineering-decisions. I think this is more a kind of psychological distortion or a false-choice (forcing an engineering decision on the basis of what the job market will be like some day in the future -- during a pandemic no less -- is practically delusional). Also, if such trade-offs are true trade-offs, then maybe the company is not really an engineering company (which is fine, but that is kind of like a shoe-store having a few podiatrists on staff -- it is wasteful, but they can now walk around in white lab-coats, and pretend to be a healthcare institution instead of a shoe-store). Personally, I believe that the tech industry sustains itself via technical debt, much like the real economy sustains itself on real debt. In some sense, everyone is trying to gaslight everyone else into incurring as much technical debt as possible, so that a way to service the debt can be sold. Most of the technical debt is not necessary, and if people were empowered to just not incur it, I suspect it would orient tech companies towards making things that actually push the state of the art forward.
  - jcgrillo 5mo ago
    > Management very often trust the sales and marketing departments of big vendors more than their own staff. They're getting kickbacks from cloud vendors. Prove me wrong.
    - r_lee 5mo ago
      not sure if this is a thing with Cloud vendors, but e.g. in Finance, you'll definitely get the opportunity to call your rep over for free fancy dinners or whatever you want, because those are "customer meetings" better than nothing, I don't blame em.
      - jcgrillo 5mo ago
        > better than nothing, I don't blame em. Money ain't got no owners, only spenders.
  - vladvasiliu 5mo ago
    > If you want it outside of AWS/GCP/Azure, you're regarded as a crazy person. If you don't adopt "best practices" (as defined by vendors) then management are scared. Management very often trust the sales and marketing departments of big vendors more than their own staff. Many of us have given up fighting this, because what it comes down to is a massive asymmetry of information and trust. I think this is the crux of the matter. Also, "everybody is doing it, so they must be right" is also a very common way of thinking amongst this population.
  - jimbokun 5mo ago
    “Nobody ever got fired for buying… …IBM …Microsoft …AWS.”
- ownagefool 5mo ago
  To be fair, I have k8s on my hetzner :p
darkwater 5mo ago
And the average developer doesn't even know where to start to deploy things in prod. When the feature product asks passes QA... to the next sprint! we are done!
- chrisweekly 5mo ago
  Whose responsibility is it to establish the prerequisite CICD pipelines, HITL workflows, and Observability infr in order for devs to shepherd changes to prod (and track their impact)? Hint: it's not the developer's.
  - liveoneggs 5mo ago
    But those things all require more pods on the cluster! We've looped back around to the beginning.
    - darkwater 5mo ago
      Exactly my point. But then developers: "I just want to go to my Heroku days again!" but then with a sufficient big company there are maaany developers doing things their slightly different way, and then other effects start compounding, and then costs go up because 15 different teams are using 27 different solutions and and and... But yeah, let's just spin-up a shadow IT VM with Debian like GP said, it's easy!
      - throwup238 5mo ago
        > But yeah, let's just spin-up a shadow IT VM with Debian like GP said, it's easy! That’s literally how they sold AWS in the beginning. Cloud won not because of costs or flexibility but because it allowed teams to provision their own machines from their budget instead of going through all the red tape with their IT departments creating… a bunch of shadow IT VMs! Everything old is new again, except it works on an accelerated ten year cycle in the IT industry.
        5mo ago
tete 5mo ago
> the average devops knows k8s If you'd know Kubernetes, you know not to use it. I say that as someone who used to do consulting for it. The reality is that yet again "making money" completely collides with efficient, quality, sane productive work. For me one of the main reasons to leave that space is that I couldn't really deal with the fact that my work collides with a client's success. That said I have helped to get off that stuff and other things that they thought they needed, that just wasted time and money. It just feels odd going into a company that hired you to consult on a topic only to end up telling them "The best approach for you is not doing that at all". Often never. Like some people thought "Well, if we have hundreds of thousands or even millions of users" and the reality was that even in these scenarios if you went away from that abstract thought and discussed a hypothetical based on their product they realized that they'd still be better off without it. Besides the fact that this hypothetical often was in a future that made it likely that they said they'd likely have completely different setup so preparing for that didn't even make sense. I think a big thing related to that was/is the microservice craze where people end up moving to a complex architecture for not many good reasons and then they increase complexity way faster than what they actually deliver in terms of the product, because it somehow feels good. I know it does, I've been there. When in reality the outcome often is just a complex mess with what could have been a relatively simple monolith. And these monoliths do work. And in the vast majority of cases they are easy to scale, because your problem switches from "how do we best allocate that huge amount of very different services across our infrastructure" to (for the most part) "how do we spin up our monolith on one more server" which tends to be a way easier to tackle service. And nothing stops you from still using everything else if you want. Just because it's a monolith doesn't mean you need to skip on any of the cloud offerings, etc. For some reason there seems to be that idea that if you write a monolith you are somehow barred from using modern tooling, infrastructure, services, etc. Not sure where that comes from.
- r_lee 5mo ago
  I think one big problem is that using microservice architecture doesn't mean that literally everything has to be a "microservice". if you don't truly need granual scaling (i.e. your "app" doesn't get a bunch of asymmetric loads across different paths), then you can just have more monolithic "microservices" until they need to be split up imo this should achieve a nice balance?

rvz 5mo ago

They use it for inflating their resume for career progression rather than actually evaluating if they need it in the first place. This is why you get many folks over-thinking the solution and picking the most hyped technologies and using them to solve the wrong problems without thinking about what they are selling. You don't need K8s + AWS EC2 + S3 just to host a web app. That tells me they like lighting money on fire and bankrupting the company and moving to the next one.

p_l 5mo ago
Often the alternatives presented as cheaper to me in discussions are actually burning money. But given how I always see "you don't need k8s because you're not going to scale so fast" I am feel like even professional k8s operators have missed the fundamental design goals of it :/ (maximizing utilization of finite compute)

Thanemate 5mo ago

I know that "resume-driven development" exists, where the tradeoffs between approaches aren't about the technical fit of the solution but the career trajectory. I've seen people making plain workstation preparation scripts using Rust, only to have something to flex about in interviews. I'm not surprised even in the slightest that DevOps workers will slap k8s on everything, to show "real industry experience" in a job market where the resume matches the tools.

ororoo 5mo ago
there are alsp people with devops title that do not know anything else than the hammer, and then everything is a hammer problem. I mean, I worked with people who were suprised that you can run more applications inside ec2 vm than just 1 app.
- tete 5mo ago
  > there are alsp people with devops title that do not know anything else than the hammer, and then everything is a hammer problem. To be fair though, that's true for every profession or skill. > I mean, I worked with people who were suprised that you can run more applications inside ec2 vm than just 1 app. I've seen something similar where people were surprised that you can use an object storage (so effectively "make HTTP requests") from every server.
- jauntywundrkind 5mo ago
  Conversely, we had millions of server huggers before, who each knew their company's stuff in a way that wasn't really applicable if they went somewhere else. Every company used to have a bespoke collection of build, deployment, monitoring, scaling, etc concerns. Everyone had their own practices, their own wikis to try to make sense of what they had. I think we critically under-appreciate that k8s is a social technology that is broadly applicable. Not just for hosting containers, but as a cloud-native form of thinking, where it becomes much easier to ask: what do we have here, and is it running well, and to have systems that are helping you keep that all on track (autonomic behavior/control loops). I see such rebellion & disdain for where we are now, but so few people who seem able to recognize and grapple with what absolute muck we so recently have crawled out of.

altmanaltman 5mo ago

yeah it's like wanting to drive to the mall in the Space Shuttle and then complaining how its too complicated

littlestymaar 5mo ago

I have nom doubt that there are legit use cases for something like k8s at Google or other multi-billion companies. But if its use was confined to this use case, pretty much nobody would be using it (unless as a customer of the organization's infra) and barely would be talking about it (like how there isn't too much talk about Borg). The reason k8s is a thing in the first place is because it's being used by way too many people for their own goods. (Most people having worked in startups have met too many architecture astronauts in our lives). If I had to bet, I'd wager that 99% of k8s users are in the “spin a few containers to run your web app” category (for the simple reason that for one billion-dollar tech business using it for legit reasons, there's many thousands early startups who do not).

rantanplan 5mo ago
The legit use case for companies like Google/Amazon etc is only to sell it to customers. None of these companies use K8s internally for real critical workloads.
- bitexploder 5mo ago
  Ehm, that is simply not true. Google built it for themselves first. It is essentially the open source version of the internal architecture. It gets used.
  - zaphar 5mo ago
    I worked at google. k8s does not really look at all like what they used internally when I was there, aside from sharing some similar looking building blocks.
    - oblio 5mo ago
      Yeah, but is the internal tool simpler? I'd be surprised.
      - zaphar 5mo ago

sdevonoes 5mo ago

Depends. For personal projects, yeah definitely. But at work? Typically the “Platform” team can only afford to support 1 (maybe 2) ways of deployment, and k8s is quite versatile, so even if you need 1 small service, you’ll go with the self-service-k8s approach your Platform team offers. Because the alternative is for you (or your team) to own the whole infrastructure stack for your new deloyment model (ecs? lambda? Whatever): so you need to setup service accounts, secret paths, firewalls, security, pipelines, registries, and a large etc. And most likely, no one will give you access rights for all of that , and your PM won’t accept the overhead either. So having everyone use the same deployment model (and that’s typically k8s) saves effort. I don’t like it for sure

limaho 5mo ago
This is where I'm at. Using Podman daily to run Python scripts and apps and it's been going great! However trying to build things like monitoring, secure secret injection, centralized inventory, remote logging, etc. has fallen on us. Has lead to some shadow IT (running our own container image registry, hashicorp vault instance, etc.) which makes me hesitant to share with others in the company how we're operating. I like to think if we had a K8s environment a lot of this would be built out within it. Having that functionality abstracted away from the developer would be a huge win in my opinion.
- evanphx 5mo ago
  Are you doing that across a fleet of machines or just one?
  - limaho 5mo ago
    We have 4 servers we run containers on. Calling that a fleet feels too generous. Not much rhyme or reason as to what containers run on which server

ownagefool 5mo ago

It depends what you're doing it. My app is fairly simple node process with some side car worker processes. k8s enables me to deploy it 30 times for 30 PRs, trivially, in a standard way, with standard cleanup. Can I do that without k8s? Yes. To the same standard with the same amount of effort? Probably not. Here, I'd argue the k8s APIs and interfaces are better than trying to do this on AWS ( or your preferred cloud provider ). Where things get complicated is k8s itself is borderline cloud provider software. So teams who were previously good using a managed service are now owning more of the stack, and these random devops heros aren't necessarily making good decisions everywhere. So you really have three obvious use cases: a) You're doing something interesting with the k8s APIs, that aren't easy to do on a cloud provider. Essentially, you're a power user. b) You want a cloud abstraction layer because you're multi-cloud or you want a lock-in bargaining chip. c) You want cloud semantics without being on a cloud provider. However, if you're a single developer with a single machine, or a very small team and you're happy working through contended static environments, you can pretty much just put a process on a box and call it done. k8s is overkill here, though not as much as people claim until the devops heros start their work.

shimman 5mo ago
Call me old fashion but I prefer tools like Dokploy that make deployment across different VPS extremely easy. Dokploy allows me to utilize my home media server, using local instances of forgejo to deploy code, to great effect. k8s appears to be a corporate welfare jobs program where trillion dollar multinational monopolistic companies are the only ones who can collectively spend 100s of millions sustaining. Since most companies aren't trillion dollar monopolies, adopting such measures seems extremely poor. All it signals to me is that we have to stop letting SV + VC dictate the direction of tech in our industry, because their solutions are unsustainable and borderline useless for the vast majority of use cases. I'll never forget the insurance companies I worked at that orchestrated every single repo with a k8s deployment whose cloud spend was easily in the high six figures a month to handle a work load of 100k/MAU where the concurrent peak never went more than 5,000 users, something the company did know with 40 years of records. Literally had a 20 person team whose entire existence was managing the companies k8s setup. Only reason the company could sustain this was that it's an insurance company (insurance companies are highly profitable, don't let them convince you otherwise; so profitable that the government has to regulate how much profit they're legally allowed to make). Absolute insanity, unsustainable, and a tremendous waste of limited human resources. Glad you like it for your node app tho, happy for you.

tjarjoura 5mo ago

In some sense, Kubernetes is just a portable platform for running Linux services, even on a single node using something like K3s. I almost see it as being an extension of the Linux OS layer.

sgt 5mo ago
Then why can't we put a wrapper onto systemd and make that into a light weight k8s?
- enos_feedler 5mo ago
  Remember fleet?
- marcosdumay 5mo ago
  Systemd is on the wrong layer here. You need something that can set your machine up, like docker.
  - sgt 5mo ago
    Okay it sets the machine up, but not the underlying host machine though.
  - jasonjayr 5mo ago
    Systemd seems to be moving in that direction, the features are coming together to actually enable this. Though imagining the unholy existence of an init system who's only job is to spin up containers, that can contain other inits, OS images, or whatever ..... turtles all the way down.
    - sgt 5mo ago
      I don't see why not. Maybe it should be on the same layer going forward - for true cloud compute (including on-premise cloud)
  - necovek 5mo ago
    Docker does not set your machine up: it sets your process up. Tech like Flatpak or Snap is closer to Docker than "machines" are — except that Docker has local, virtualized networking built-in as the IPC layer.

hombre_fatal 5mo ago

k8s is useful when you have services that must spin up and down together, and you want to swap out services and deploy all/some/one. and then also package this so that you and other developers can get the infrastructure running locally or on other machines.

geodel 5mo ago

Doing Kubernetes like doing Agile is mandatory nowadays. I've been asked to package a 20 line worth of bash script as docker image so it can be delivered via CI/CD pipeline via Kubernetes pods in cloud. Value is not that I got job done at a day's notice. It is black mark that I couldn't package it as per industry best practices. Not doing would mean out of job/work. Whether it is happening correctly is not something decision makers care as long it is getting done anyhow.

necovek 5mo ago
In my 20+ years in the industry, I've been at one company which really did Agile, and that was the one I started with. Everyone else is communicating they are doing Agile while being very far away from it ;)
- bdangubic 5mo ago
  if anyone knew what agile is maybe more would have a chance if making it work (it won’t). in my 30* the only “process” that worked and works is “hire the right people and get the F out of the way.”
  - necovek 5mo ago
    That's pretty much what agile is :)
radlad 5mo ago
There are many organizations which still ship software without Kubernetes. Perhaps even the vast majority.
- geodel 5mo ago
  Of course. I used to think I am working for one such organization for long time. Until leadership decided "modernization" as top priority for IT teams as we are lagging far.
jauntywundrkind 5mo ago

throwaway894345 5mo ago

Even if using just one VM, I'll probably slap k3s on it and manage my application using manifests. It's just so much easier than dealing with puppet or chef or vanilla cloud-init. Docker compose works too, but at that point it's just easier to stick with k3s and then I can have nice things like background jobs, a straightforward path to HA, access to an ecosystem of existing software, and a nicer CLI.

tayo42 5mo ago
Thats what I don't get when people bring up this idea k8s is complicated. All of those other tools are complicated and fragile
- throwaway894345 5mo ago
  I think the things that trip people up are: 1. People expect k8s to be an opinionated platform and it's very happy to let you make a mess 2. People think k8s is supposed to be a cross platform portability layer and ... it maybe can be if you're very careful, but it's mostly not that 3. People compare k8s/cloud/etc to some monolithic application with admin permissions to everything and they compare that to the "difficulty" of dealing with RBAC/IAM/networking/secrets management 4. People don't realize how much more complicated vanilla Linux tooling and how much more accidental complexity is involved

moooo99 5mo ago

> People use Kubernetes for way too small things, and it sounds like you don't have the scale for actually running Kubernetes. This is a problem I've run into enterprise deployments. K8s is often the lowest common denominator semi small platform engineering teams arrive on. At my current employer, a platform managed K8s namespace is the only thing we got in terms of PaaS offering, so it is what we use. Is it overpowered? Yes. Is it overly complex for our usecase? Definitely. Could we basically get by hosting our services on a few cheap mini computers with no performance penalty? Also yes.

firesteelrain 5mo ago

We have a hobby web based app that consists of multiple containers. It runs in docker compose. Serves 1000 users right now (runs 24/7). Single VM. No Kubernetes whatsoever. I agree with you.

giobox 5mo ago
Docker compose is brilliant while your stack remains on a single box, and will scale quite nicely for some time this way for most applications with minimum maintenance overhead. My personal strategy has always been to start off in docker compose, and break out to a k8s configuration later if I have to start scaling beyond single box.

jen20 5mo ago

The problem with Kubernetes is that it doesn't scale down to small deployments very well, but it sure as shit doesn't scale up to large ones either. Large shared multi-tenant clusters have massive problems even when running parts of the same application with the same incentives, it falls apart completely when the tenants are diverse. Nomad has neither of these problems.

mbesto 5mo ago

> it sounds like you don't have the scale for actually running Kubernetes. You don't set up k8s because your current load can't be handled, you do for future growth. Sometimes that growth doesn't pan out and now you're left with a complex infrastructure that is expensive to maintain and not getting any of the benefit.

deleted 5mo ago

[deleted]