5 ms·

All of those are product that creates huge risks when deployed to mission critical environments and this is exactly the problem. The entire wintel ecosystem de

All of those are product that creates huge risks when deployed to mission critical environments and this is exactly the problem. The entire wintel ecosystem depends on people putting their heads in the sand and repeating "nobody ever got fired for buying Microsoft/crowdstrike/IBM" and neglecting to run even the most trivial simulation of what happens when the very well understood design flaws of those platforms gets triggered by a QA department you have no control over drops the ball. The problem is that as long as nobody dares recognizing that the current mono culture around the "market leading providers" this kind of event will remain really likely even if nobody is trying to break it and and extremely likely once you insert well funded malicious actors(ranging from bored teenagers to criminal gangs and geopolitical rivals). The problem is that adding fair weather product that gives the illusion of control though fancy dashboards on the days they work is not really an substitute for proper reliance testing and security hardening but far less disruptive to companies that don't really want to leave the 90ies PC metaphor behind.

zifpanachr23 2y ago
To be fair, IBM products back in the day when that saying made sense never had these kinds of problems. It's straight up insulting to compare them to somebody like Crowdstrike. Wintel won by being cheaper and shittier and getting a critical mass of fly by night OEMs and software vendors on board. IBM was more analogous to the way Apple handles things. Heavy vertical integration and premium price point with a select few software and hardware vendors working very closely with IBM when software and hardware analogous to Crowdstrike in terms of access was created.
EvanAnderson 2y ago
How should corporate IT do it? You have 100,000 devices to manage. How do you handle that efficiently without creating a monoculture? It's not a "90ies PC metaphor" problem. Swap Chromebooks for PCs and you still have the problem-- how do you handle centralized management of that "fleet"? Should every employee "bring their own device" leaving corporate IT "hands-off"? There are still monocultures within that world. Poor quality assurance on the part of software providers is the root cause. The monocultures and having software that treats the symptoms of bad computing metaphors aren't good either, but bad software quality assurance is the reason this happened today.
- Stranger43 2y ago
  If crowdsource QA department is all that stands between you and days of no operations then you chose to live with the near certainty that you will have days rather then hours of unplanned company wide downtime. And if you cannot actually abandon someone like microsoft that consistantly screws up their QA then it's basically dishonest for you to claim that reliability is even a concern for your desktop platform. And that's essentially what i say when i accuse the modern enterprise it's client device teams of being stuck in the 90ies as those risk were totally acceptable back when the stakes were low and outages only impacted non time critical back office clerical work. but what we saw today was that those high risk cost optimized systems got deployed into roles where the risk/consequence profile is entirely different. So what you do is that you keep the low impact data entry clerks and spreadsheet wranglers on the windows platform but threat the customer facing workers dealing with time sensitive task something a bit less risky. It's might not be as easy as just deploying the same old platform designed back in the 90ies to everyone but once you leave the Microsoft ecosystem dual sourcing based on open standards become totally feasible, at costs that might not be prohibitive as everything in the unix like ecosystem including web browsers have multiple independent implementations so you basically just have to standardize of 2-4 rather then one platform which again isnt unfeasible. It's telling that an Azure region failed this news cycle without anyone noticing because companies just don't tolerate the kind of risk people takes with their wintel desktop for their backends so most critical services hosted in microsofts Iowa datacenter had and second site on standby.
  - jimnotgym 2y ago
    >And if you cannot actually abandon someone like microsoft that consistantly screws up their QA The last outage I can remember due to an ms update was 7 or 8 years ago. Desktops got stuck on 'update 100% complete'. After a couple of minutes I pressed ctrl+alt+del and it cleared. Before that...I don't remember. Btw MS provides excellent tools to manage updates, and you can apply them on a rolling basis.
  - EvanAnderson 2y ago
    > If crowdsource QA department is all that stands between you and days of no operations ... For companies of a certain large size, I guess. For all but the largest companies, though, there's no choice but to outsource software risks to software manufacturers. The idea that every company is going to shoulder the burden of maintaining their own software is ridiculous. Companies use off-the-shelf software because it makes good financial sense. > And if you cannot actually abandon someone like microsoft that consistantly screws up their QA then it's basically dishonest for you to claim that reliability is even a concern for your desktop platform. When a company has significant software assets tied to a Microsoft platform there's no alternative. A company is going to use the software that best-suits their needs. Platform is a consideration, however I've never seen it be the dominant consideration. Today's issue isn't a Microsoft problem. The blame rests squarely on Crowdstrike and their inability to do QA. The culture of allowing "security software" to automatically update is bad, but Crowdstrike threw the lit match into that tinderbox by pushing out this update globally. As another comment points out, Microsoft has good tools for rolling update releases for corporate environments. They're not perfect but they're not terrible either. > It's might not be as easy as just deploying the same old platform ... When a company doesn't control their software platform they don't have this choice. Off-the-shelf software is going to dictate this. In some fantasy world where every application is web-based and legacy code is all gone maybe that's a possibility. I have yet to work in that environment. Companies aren't maintaining the "wintel desktop" because they want to.

zifpanachr23 2y ago

To be fair, IBM products back in the day when that saying made sense never had these kinds of problems. It's straight up insulting to compare them to somebody like Crowdstrike. Wintel won by being cheaper and shittier and getting a critical mass of fly by night OEMs and software vendors on board. IBM was more analogous to the way Apple handles things. Heavy vertical integration and premium price point with a select few software and hardware vendors working very closely with IBM when software and hardware analogous to Crowdstrike in terms of access was created.

EvanAnderson 2y ago

How should corporate IT do it? You have 100,000 devices to manage. How do you handle that efficiently without creating a monoculture? It's not a "90ies PC metaphor" problem. Swap Chromebooks for PCs and you still have the problem-- how do you handle centralized management of that "fleet"? Should every employee "bring their own device" leaving corporate IT "hands-off"? There are still monocultures within that world. Poor quality assurance on the part of software providers is the root cause. The monocultures and having software that treats the symptoms of bad computing metaphors aren't good either, but bad software quality assurance is the reason this happened today.

Stranger43 2y ago
If crowdsource QA department is all that stands between you and days of no operations then you chose to live with the near certainty that you will have days rather then hours of unplanned company wide downtime. And if you cannot actually abandon someone like microsoft that consistantly screws up their QA then it's basically dishonest for you to claim that reliability is even a concern for your desktop platform. And that's essentially what i say when i accuse the modern enterprise it's client device teams of being stuck in the 90ies as those risk were totally acceptable back when the stakes were low and outages only impacted non time critical back office clerical work. but what we saw today was that those high risk cost optimized systems got deployed into roles where the risk/consequence profile is entirely different. So what you do is that you keep the low impact data entry clerks and spreadsheet wranglers on the windows platform but threat the customer facing workers dealing with time sensitive task something a bit less risky. It's might not be as easy as just deploying the same old platform designed back in the 90ies to everyone but once you leave the Microsoft ecosystem dual sourcing based on open standards become totally feasible, at costs that might not be prohibitive as everything in the unix like ecosystem including web browsers have multiple independent implementations so you basically just have to standardize of 2-4 rather then one platform which again isnt unfeasible. It's telling that an Azure region failed this news cycle without anyone noticing because companies just don't tolerate the kind of risk people takes with their wintel desktop for their backends so most critical services hosted in microsofts Iowa datacenter had and second site on standby.
- jimnotgym 2y ago
  >And if you cannot actually abandon someone like microsoft that consistantly screws up their QA The last outage I can remember due to an ms update was 7 or 8 years ago. Desktops got stuck on 'update 100% complete'. After a couple of minutes I pressed ctrl+alt+del and it cleared. Before that...I don't remember. Btw MS provides excellent tools to manage updates, and you can apply them on a rolling basis.
- EvanAnderson 2y ago
  > If crowdsource QA department is all that stands between you and days of no operations ... For companies of a certain large size, I guess. For all but the largest companies, though, there's no choice but to outsource software risks to software manufacturers. The idea that every company is going to shoulder the burden of maintaining their own software is ridiculous. Companies use off-the-shelf software because it makes good financial sense. > And if you cannot actually abandon someone like microsoft that consistantly screws up their QA then it's basically dishonest for you to claim that reliability is even a concern for your desktop platform. When a company has significant software assets tied to a Microsoft platform there's no alternative. A company is going to use the software that best-suits their needs. Platform is a consideration, however I've never seen it be the dominant consideration. Today's issue isn't a Microsoft problem. The blame rests squarely on Crowdstrike and their inability to do QA. The culture of allowing "security software" to automatically update is bad, but Crowdstrike threw the lit match into that tinderbox by pushing out this update globally. As another comment points out, Microsoft has good tools for rolling update releases for corporate environments. They're not perfect but they're not terrible either. > It's might not be as easy as just deploying the same old platform ... When a company doesn't control their software platform they don't have this choice. Off-the-shelf software is going to dictate this. In some fantasy world where every application is web-based and legacy code is all gone maybe that's a possibility. I have yet to work in that environment. Companies aren't maintaining the "wintel desktop" because they want to.