5 ms·

I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion-2f5 https://isolveproblems.subst

• eviks 6mo ago
  Though this doesn't make much sense on its surface - a bug means something is already broken, and he tells of millions of crashes per month, so it was visibly broken. 100% chance of being broken (bug) > some chance of breakage from fixing it (sure, the value of current and potential bug isn't accounted for here, but then neither is it in "afraid to break something, do nothing")
  • jiggawatts 6mo ago
    I've experienced a nearly identical scenario where a large fleet of identical servers (Citrix session hosts) were crashing at a "rate" high enough that I had to "scale up" my crash dump collection scripts with automated analysis, distribution into about a hundred buckets, and then per-bucket statistical analysis of the variables. I had to compress, archive, and then simply throw away crash dumps because I had too many. It was pure insanity, the crashes were variously caused by things like network drivers so old and vulnerable that "drive by" network scans by malware would BSOD the servers. Alternatively, successful virus infections would BSOD the servers because the viruses were written for desktop editions of Windows and couldn't handle the differences in the server edition, so they'd just crash the system. On and on. It was a shambling zombie horde, not a server farm. I was made to jump through flaming hoops backwards to prove beyond a shadow of a doubt that every single individual critical Microsoft security patch a) definitely fixed one of the crash bugs and b) didn't break any apps. I did so! I demonstrated a 3x improvement in overall performance -- which by itself is staggering -- and that BSODs dropped by a factor of hundreds. I had pages written up on each and every patch, specifically calling out how they precisely matched a bucket of BSODs exactly. I tested the apps. I showed that some of them that were broken before suddenly started working. I did extensive UAT, etc. "No." was the firm answer from management. "Too dangerous! Something could break! You don't know what these patches could do!" etc, etc. The arguments were pure insanity, totally illogical, counter to all available evidence, and motived only by animal fear. These people had been burned before, and they're never touching the stove again, or even going into the kitchen. You cannot fix an organisation like this "from below" as an IC, or even a mid-level manager. CEOs would have a hard time turning a ship like this around. Heads would have to roll, all the way up to CIO, before anything could possibly be fixed.
    • eviks 6mo ago
      Yeah, long periods of total disfunction get ingrained Though just to ref my original point > burned before, and they're never touching the stove again Except they are sitting on the stove with their asses burning, which cuts all the needed cooling off their heads!
      • jiggawatts 6mo ago
        The better analogy is that they ran out of the kitchen in a panic, and left the pots on the burners. Some time later there is smoke curling up from under the kitchen door, but they’re used to the burning smell by now so it’s “not that big a deal”.
• bombcar 6mo ago
  > Once you reach this stage, the only escape is to first cover everything with tests and then meticulously fix bugs, without shipping any new features. Isn't this where Oracle is with their DB? Wasn't HN complaining about that?
• gherkinnn 6mo ago
  Once you reach this stage, the only escape is to jump ship. Either mentally or, ideally, truly. You're in an unwinnable position. Don't take the brunt for management's mistakes. Don't try to fix what you have no agency over.
  • chii 6mo ago
    unfortunately, what you will find is that unless you get lucky, the next ship is more of the same. The system/management style is ingrained in corporate culture of large-ish companies (i would say if it has more than 2 layers of management from you to someone owning the equity of the business and calling the shots, it's "large"). It stems from the fact that when an executive is bestowed the responsibility of managing a company from the shareholders, the responsibility is diluted, and the agent-principle problem rears their ugly head. When several more layers of this starts growing in a large company, the divergence and the path of least resistance is to have zero trust in the "subordinates", lest they make a choice that is contrary to what their managers want. The only way to make good software is to have a small, nimble organization, where the craftsman (doing the work) makes the call, gets the rewards, and suffers the consequences (if any). That aligns the agent-principle together.
    • bigstrat2003 6mo ago
      > I would say if it has more than 2 layers of management from you to someone owning the equity of the business and calling the shots, it's "large" By that metric, my 50 employee company is "large".
      • chii 6mo ago
        well, does this company have more than 2 layers of management? Why do you need that much for only 50 people, instead of enpowering those people to make choices (after training and providing guidance on what makes for a good choice in various circumstances)?
    • cineticdaffodil 6mo ago
      Hierachy is the enemy of succeding projects and information flow. The more important and complex hierarchy in a culture the less likely it is to have a working software industry. Germanys and japanese endless :"old vs young, seniority vs new, internal vs external, company wide management vs project local management come to mind. Its guerilla vs army, startup vs company allover..
• idorosen 6mo ago
  Or to simplify the product and rebuild.
  • axelriet 6mo ago
    Exactly. But he’s right about management, first the problem must be acknowledged and that may make some people look bad.
  • teeray 6mo ago
    “Rebuild” is also a four-letter word though at this stage too. The customer has a panel of knob-and-tube wiring and aluminum paper-wrapped wire in the house. They want a new hot tub. They don’t want some electrician telling them they need to completely rewire their house first at huge expense, such that they cannot afford the hot tub anymore. They’ll just throw the electrician out and get some kid in a pickup truck (“You’re Absolutely Right Handyman LLC”) to run a lamp cord to their new hot tub. Once the house burns to the ground, the new owners will wire their new construction correctly.
• praptak 6mo ago
  This isn't incentivized in corporate environment. Noticed how "the talent left after the launch" is mentioned in the article? Same problem. You don't get rewarded for cleaning up mess (despite lip service from management) nor for maintaining the product after the launch. Only big launches matter. The other corporate problem is that it takes time before the cleanup produces measurable benefits and you may as well get reorged before this happens.
  • steveBK123 6mo ago
    > You don't get rewarded for cleaning up mess (despite lip service from management) nor for maintaining the product after the launch I have never worked at a shop or on a codebase where "move fast & break things, then fix it later" ever got to the "fix it later" party. I've worked at large orgs with large old codebases where the % of effort needed for BAU / KTLO slowly climbs to 100%. Usually some combination of tech debt accumulation, staffing reduction, and scale/scope increases pushing the existing system to its limits. This is related to a worry I have about AI. I hear a lot of expectations that we're just going to increase code velocity 5x from people that have never maintained a product before. So moving faster & breaking more things (accumulating more tech debt) will probably have more rapid catastrophic outcomes for products in this new phase. Then we will have some sort of butlerian jihad or agile v2.
    • asdfman123 6mo ago
      People are still trying to figure out how to use AI. Right now the meme is it's used by juniors to churn out slop, but I think people will start to recognize it's far more powerful in the hands of competent senior devs. It actually surprised me that you can use AI to write even better code: tell it to write a test to catch the suspected bug, then tell it to fix the bug, then have it write documentation. Maybe also split out related functionality into a new file while you're at it. I might have skipped all that pre-AI, but now all that takes 15 minutes. And the bonus creating more understandable code allows AI to fix even more bugs. So it could actually become a virtuous cycle of using AI to clean up debt to understand more code. In fact, right now, we're selling technical debt cleanup projects that I've been begging for for years as "we have to do this so the codebase will be more understandable by AI."
• dbdr 6mo ago
  > I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something. And that, my friends, is why you want a memory safe language with as many static guarantees as possible checked automatically by the compiler.
  • axelriet 6mo ago
    I was waiting for that comment :) Remember that everybody, eventually, calls into code written in C.
    • dbdr 6mo ago
      If 90% of the code I run is in safe rust (including the part that's new and written by me, therefore most likely to introduce bugs) and 10% is in C or unsafe rust, are you saying that has no value? Il meglio è l'inimico del bene. Le mieux est l'ennemi du bien. Perfect is the enemy of good.
      • axelriet 6mo ago
        That is an unexpected interpretation. Use the best tool for the job, also factoring what you (and your org) are comfortable with.
        • RyujiYasukochi 6mo ago
          [flagged]
    • tux3 6mo ago
      If you're sufficiently stubborn, it's certainly possible to call directly into code written in Verilog, held together with inscrutable Perl incantations. High-level languages like C certainly have their place, but the space seems competitive these days. Who knows where the future will lead.
      • axelriet 6mo ago
• doctorpangloss 6mo ago
  if the service is so shitty, why are people paying so much fucking money for it? is microsoft committing an accounting fraud?
  • llama052 6mo ago
    Yeah it’s entirely business people and executives who make these decisions in most companies. Not the ones who use it or implement on it.
  • fxtentacle 6mo ago
    It’s more of a hostage situation.
  • rawgabbit 6mo ago
    The US government’s experts called Azure “a pile of shit”; they got overruled. https://www.propublica.org/article/microsoft-cloud-fedramp-cybersecurity-government https://www.propublica.org/article/microsoft-cloud-fedramp-c...
  • hunterpayne 6mo ago
    Because Azure customers are companies that still, in 2026 only use Windows. Anyone else uses something else. Turns out, companies like that don't tend to have the best engineering teams. So moving an entire cloud infrastructure from Azure to say AWS, probably is either really expensive, really risky or too disruptive to do for the type of engineering team that Azure customers have. I would expect MS to bleed from this slowly for a long time until they actually fix it. I seriously doubt they ever will but stranger things have happened.
    • pjmlp 6mo ago
      Turns out outside companies shipping software products aspiring to be the next Google or Apple, most companies that work outside software industry also need software to run their business and they couldn't care less about HN technology cool factor. They use whatever they can to ship their products into trucks, outsourcing their IT and development costs , and that is about it.
• rk06 6mo ago
  once you reach the stage, the only escape is to give up on it. and move on. somethings are beyond your control and capabilities
• monocasa 6mo ago
  No joke, I worked at a place where in our copy of system headers we had to #define near and far to nothing. That was because (despite not having supported any systems where this was applicable for more than a decade) there was a set of files that were considered too risky to make changes in that still had dos style near and far pointers that we had to compile for a more sane linear address space. https://www.geeksforgeeks.org/c/what-are-near-far-and-huge-pointers/ https://www.geeksforgeeks.org/c/what-are-near-far-and-huge-p... Now, I'm just a simple country engineer, but a sane take on risk management probably doesn't prefer de facto editing files by hijacking keywords with template magic compared with, you know just making the actual change, reviewing it, and checking it in.
• hikarudo 6mo ago
  > Once you reach this stage, the only escape is to first cover everything with tests and then meticulously fix bugs The exact same approach is recommended in the book "Working effectively with legacy code" by Michael Feathers, with several techniques on how to do it. He describes legacy code as 'code with no tests'.
  • coredog64 6mo ago
    "Show me the incentives, and I will show you the outcomes" - Charlie Munger I once worked in a shop where we had high and inflexible test coverage requirements. Developers eventually figured out that you could run a bunch of random scenarios and then `assert true` in the finally clause of the exception handler. Eventually you'd be guaranteed to cover enough to get by that gate. Pushing back on that practice led to a management fight about feature velocity and externally publicized deadlines.
• neya 6mo ago
  Once you reach this stage, honestly the only escape is real escape. Put your papers in and start looking for a job elsewhere, because when they go down, they will go down hard and drag you with them. It's not like you didn't try.
• egorfine 6mo ago
  writing tests and then meticulously fixing bugs does not increase shareholders' value.
  • branko_d 6mo ago
    Dave Cutler and his team are a clear counter-example. They famously shipped Windows NT with zero known bugs, which clearly brought enormous shareholder value. The problem, of course, is that this sort of thing doesn’t bring value next quarter.
• deleted 6mo ago
  [deleted]
• bob1029 6mo ago
  > first cover everything with tests Beware this goal. I'm dealing with the consequences of TDD taken way too far right now. Someone apparently had this same idea. > management who do not fully understand the problem nor are incentivized to understand it They are definitely incentivized to understand the problem. However the developers often take it upon themselves to deceive management. This happens to be their incentive. The longer they can hoodwink leadership, the longer they can pad their resume and otherwise play around in corporate Narnia. It's amazing how far you can bullshit leaders under the pretense of how proper and cultured things like TDD are. There are compelling metrics and it has a very number-go-up feel to it. It's really easy to pervert all other aspects of the design such that they serve at the altar of TDD. Integration testing is the only testing that matters to the customer. No one cares if your user service works flawlessly with fake everything being plugged into it. I've never seen it not come off like someone playing sim city or factorio with the codebase in the end.
  • carols10cents 6mo ago
    If you're writing the tests after writing the code, you're not doing TDD though.
  • caoilte 6mo ago
    Unit tests are just as important as integration tests as long as they're tightly scoped to business logic and aren't written just to improve coverage. Anything can be done badly, especially if it is quantified and used as a metric of success (Goodhart's law applies). Integration tests can be just as bad in this regard. They can be flakey and take hours, give you a false sense of security and not even address the complexity of the business domain. I've seen people argue against unit tests because they force you to decompose your system into discrete pieces. I hope that's not the core concern here becuase a well decomposed system is easier to maintain and extend as well as write unit tests for.
    • bwfan123 6mo ago
      The problem with unit tests these days is that AI writes them entirely and does a great job at it. That defeats the purpose of unit tests in the first place since the human doesnt have the patience to review the reams of over-mocked test-code produced by AI. The end-result of this are things like the code leak of claude code presumably caused by ai generated ci/cd packaging code nobody bothered to review since the attitude is: who reviews test or ci/cd code ? If they break big-deal, ai will fix it.
• varispeed 6mo ago
  I was once in such a position. I persuaded management to first cover the entire project with extensive test suite before touching anything. It took us around 3 months to have "good" coverage and then we started refactor of parts that were 100% covered. 5 months in the shareholders got impatient and demanded "results". We were not ready yet and in their mind we were doing nothing. No amount of explanation helped and they thought we are just adding superficial work ("the project worked before and we were shipping new features! Maybe you are just not skilled enough?") Eventually they decided to scrap whole thing. Project was killed and entire team sacked.
  • jimbokun 6mo ago
    I’m a developer and if a team spent five months only refactoring with zero features added I would fire you too. Refactoring and quality improvements must happen incrementally and in parallel with shipping new features and fixing bugs.
    • bmurphy1976 6mo ago
      I'm a director and one of our teams just spent 8 months doing just that and it was totally justified. They're finally coming up for air and the foundation is significantly improved. There's nuance here. Every project/team/org is different.
    • eviks 6mo ago
      Welcome to Microsoft! Enjoy the ever-growing backlog of bugs to fix!
• staticassertion 6mo ago
  It is so hard to test those codebases too. A lot of the time there's IO and implicit state changes through the code. Even getting testing in place, let alone good testing, is often an incredibly difficult task. And no one will refactor the code to make testing easier because they're too afraid to break the code.