7 ms·

Or optimize a bit earlier and prevent having to scale out to a bazillion systems.

• deleted 20d ago
  [deleted]
• bcrosby95 20d ago
  The way I usually prevent having to scale out to a bazillion systems is never getting more than 10 users.
  • steve_adams_86 20d ago
    I wonder why Cloudflare didn’t think of this
    • kevin_thibedeau 20d ago
      This is Broadcom's business model
      • gerdesj 20d ago
        Quite. I was a VMware fanboi (25+ years, man and boy) I still look after a few VMware estates and a lot of Proxmox ones (that used to run VMware). Hilariously, VMware is described as "enterprise class", which I can only conclude means MVP and a bit wanky. Today I repaired a Proxmox HA + Ceph node using boring old normal Linux skills and as it turns out I have 30 years of those. Part way through a remote v8 to 9 upgrade I think I lost comms due to using OpenvSwitch for networking and despite using tmux for the upgrade session. Anyway, the Proxmox ISO was useless for rescue but the classic systemrescuecd worked nicely and I could run dpkg in a chroot. VMware "used" Linux and never really gave back. I don't miss fixing vCentres and all the other nonsense that "Enterprise" wankery has foisted on me over the years.
        • taneq 20d ago
          When I was first getting into software dev, I thought 'enterprise' meant 'industrial grade', 'powerful', 'high-performing'. Then I actually met some enterprise software, and realised that it means 'expensive', 'bespoke', 'one-off', and usually 'janky'.
          • phoghed 20d ago
            Enterprise means it has SSO and a support contract
            • esseph 20d ago
              Lol, so true
            • jacquesm 20d ago
              It means you are paying for a support contract. Whether you actually have one time will tell.
              • PeterStuer 20d ago
                Only seasoned engineers will understand you are not joking.
                • phoghed 20d ago
                  Sorry, the very simple thing you’re trying to do is too complex and non-standard for our support team to handle. We’ll gladly sell you a consultant for $400/hr. He’ll work on modifying our system, and then we’ll sell those modifications to everyone else.
            • valenterry 20d ago
              And in-transparent pricing and this annoying dude that asks you once in a while if everything is okay, usually before the renewal.
          • 20d ago
  • bch 20d ago
    The Art of Production
  • froh 20d ago
    pro move. made my evening.
  • forgot-my-pw 20d ago
    The art of premature optimizations
    • smolder 20d ago
      The art is in knowing how to write software that doesn't perform like shit without doing all the work of measuring and refining. If you can save $100k in hardware costs in a couple days by just knowing what you're doing, that optimization is not premature.
    • jodrellblank 20d ago
      And yet when Prof. Donald Knuth wrote that in 1974 paper[1] it was in this context: > "The improvement in speed from Example 2 to Example 2a is only about 12%, and many people would pronounce that insignificant. The conventional wisdom shared by many of today's software engineers calls for ignoring efficiency in the small; but I believe this is simply an overreaction to the abuses they see being practiced by pennywise-and-pound-foolish programmers, who can't debug or maintain their "optimized" programs. In established engineering disciplines a 12% improvement, easily obtained, is never considered marginal" also: > "In the late 1960's we witnessed a "software crisis", which many people thought was paradoxical because programming was supposed to be so easy. As a result of the crisis, people are now beginning to renounce every feature of programming that can be considered guilty by virtue of its association with difficulties. Not only go to statements are being questioned; we also hear complaints about floating-point calculations, global variables, semaphores, pointer variables, and even assignment statements. Soon we might be restricted to only a dozen or so programs that are sufficiently simple to be allowable" In a recent comment I mentioned a youtube interview with Rico Mariani, a performance engineer from Microsoft, and he said that he often got called into projects approaching their deadlines and not meeting their performance goals. In one anecdote he spent a couple of hours with a team and showed how their design could never meat the goal even with the fastest disks, CPUs, memory, and network. And commented how strange it is if they had spent a day at the start of the project whiteboarding out the design against hardware specs at the start of the project - and avoided months of wasted effort - that would be called "premature optimization". [1] https://dl.acm.org/doi/pdf/10.1145/356635.356640 https://dl.acm.org/doi/pdf/10.1145/356635.356640
  • bigbuppo 20d ago
    And that's why I charge $10,000,000/user/mo.
• dakolli 20d ago
  You can build foundations that aren't extermely optimal but have future optimisations in mind.
• scottlamb 20d ago
  You're never going to get promoted with that attitude! I'm joking...but not entirely. It sounds impressive on a promo packet when you say you've saved 100 TB of RAM / $$$ through whatever technique. But it sounds a lot less impressive when you say if this system grows to this size in x years, I will have saved 100 TB, especially when no one yet knows how large the system will really be in that time or what the cost of RAM will be. I dunno, maybe if you say that x years ago, I made a decision that now is saving us 100 TB, that's kinda impressive, but you're also getting credit for it x years after you did the work. It also doesn't have the implication that it must be inherently complex/hard because some other smart person chose the other way. And there is a bias to care more about recent accomplishments. So I don't really think it'd be valued the same at all. Also, in general big tech (at least Google) prefers growing the userbase over improving efficiency. Periodically efficiency is rewarded, e.g. when RAM cost suddenly balloons or some big must-have feature has suddenly used up capacity planned for something else. You get rewarded for doing efficiency work on demand, not eagerly. I once got a $100 peer bonus for finding 100,000 cores that were essentially stranded by an accounting error in another team's migration script.
• Dylan16807 20d ago
  It was already reasonably lean. If they had 10 bazillion systems, they now need somewhere between 6 and 8 bazillion systems.
• otterley 20d ago
  Remember that everything has an opportunity cost. Running a lot of servers might cost $10 million annually, but if the product team had to choose between a project that would recoup $5 million of that vs. an opportunity to earn $50 million ARR for the same amount of work, the logical answer would be obvious.
  • vanviegen 20d ago
    That's assuming the ops team has infinite capacity.
  • Retric 20d ago
    In my experience optimizations actually preformed tend to have ridiculously high ROI because they are so rarely prioritized. Better performance = saving money + better user experience.
    • mtrovo 20d ago
      Depends on how you measure your ROI and how it could be different from how your company measure their ROI. The problem with optimizations is that you are competing in prioritization with other features. Reducing the baseline cost always has a limit of zero, while the upside from new features is infinite according to your leadership and investors, so it is very hard to argue against. In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.
      • 27183 20d ago
        > The problem with optimizations is that you are competing in prioritization with other features. I think companies often over-indulge in features nobody wants, needs, or cares about. I quit my previous company because they were forcing us to build something that had single digit weekly active users. It was utterly pointless, driven entirely by some half baked navel gazing harebrained ideas about what a "nontechnical user" might want. But nobody ever asked any real users. I estimate the company probably blew the greater part of $10M on this bullshit, not counting opportunity cost. > In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn. People like that are problematic not just because they don't understand tech debt. They also don't understand products. There are shitloads of people in the industry who market themselves as some kind of mystical gurus, are able to deliver impressive monologues talking over everyone on the zoom call, but contribute nothing else than a sense of urgency and frustration. If you find yourself in their company, better to just leave.
• hinkley 19d ago
  This is perhaps the hardest nut to crack. We give perverse incentives to the marketing department to bring us the most customers they can, instead of the most appropriate customers. Someone told me at my second lead position that they had landed a top-tier customer for our demo-ware and the first words out of my mouth were, "FUCK ME". Not the response they were expecting, but then that guy never did end up understanding me the entire time we worked together. My bosses did though. When your system is new you're a loss leader for all of your customers. Every dollar they bring in costs you two, possibly more. You have to get to positive MRR before the venture capital runs out, and over a long enough time horizon, you will go for a new round and find out that a recession is about to start and the VC guys getting cagey is the first clue it's coming. Meanwhile if you spend all of your time and energy on reducing costs instead of increasing revenue, then your competitors catch up with you. Particularly if their funding rounds are half a cycle off from yours - one of you will be the last one to have gotten a cash infusion before the money dried up. Most recently I worked at a place that didn't understand my calls for sobriety until it was too late. So I got put in charge of a rear-guard action that was too little too late, and our customers all fled to much cheaper competitors who could do 80% of what we could for half the price. I learned some good stuff, but the company got bought by a competitor who scrapped those systems.