4 ms·

Hello, IT, have you tried turning it on and off again 15 times? Seriously though - this entire outage is the poster child for why you NEVER have software tha

Hello, IT, have you tried turning it on and off again 15 times? Seriously though - this entire outage is the poster child for why you NEVER have software that updates without explicit permission from a sysadmin. If I were in congress, I would make it illegal, it's an obvious national security issue.

pas 2y ago
That's not the big no-no here. Lack of any real DRP is. Sure, it's cheaper to just buy CS Falcon (and who knows what other amazing vendors supplied timebombs are ticking silently) than paying sysadmins and developers ... and letting them build something that does what it needs, not much else, so there's no need to put these fantastic "single agents" from these RCE-as-a-service vendors on all the fucking servers.
- Connector2542 2y ago
  both are true
travoc 2y ago
It was a data update that triggered a software bug. It was not a software update. I don't think it's reasonable to make data updates illegal.
- CoastalCoder 2y ago
  I'm a general purpose computer, the distinction between software and data can be pretty fuzzy.
- fire_lake 2y ago
  Code is data and data is code
  - surfingdino 2y ago
    Unless you use PIC controllers with https://en.wikipedia.org/wiki/Harvard_architecture https://en.wikipedia.org/wiki/Harvard_architecture
- duped 2y ago
  This distinction is meaningless at best and harmful at worst. If a code path isn't followed until a config file updates, that is practically the same thing as the code path being introduced by the update.
rfoo 2y ago
> why you NEVER have software that updates without explicit permission from a sysadmin In general I agree, but this case is quite messy. It's more like your anti-virus had a bug since forever that if it loads a broken virus definition it bricks your system. And a broken virus definition finally happened today. Do you want every virus definition (that is updated every few hours) to require explicit permission from a sysadmin?
johnkizer 2y ago
What % of those sysadmins are then going to turn around and script something to auto-approve those updates, once they realize that they are A) requested at inconvenient times and B) are related to security? Who's going to take the risk of appearing to have sat on an important update, while the org they support is ravaged by ThreatOfTheDay, because they thought they knew better than a multi-billion dollar, tops-in-their-field company? (I'm not necessarily saying that's actually objectively correct, but I can't imagine that many folks are willing to risk the downside)
avs733 2y ago
I understand the logic of this but it is somewhat based on the assumption - which most industries have in droves - that people in THAT industry are the competent bullwhark against stupidity. I consulted for a company for a while where the 'sysadmin' was the owner's mother - who bought laptops from walmart. Not only could she NOT have approved updates like this, even if she could she would have she wouldn't have had any knowledge whatsoever with which to make a determination if it worked. In an abstraction, the problem really is with externalities. These approaches to updates exist because people who CAN'T do what you describe are likely a more dominant part of the threat model than this happening to people you do describe. The resulting fix, as we're seeing, is very reliable until it isn't...and if the isn't is enormous in scale the systems aren't setup to fail gracefully. If you want to make a rule...require graceful failure.
MBCook 2y ago
Nah. That’s not the problem. Kernel level code blindly loading arbitrary files? Panicking when the file doesn’t parse because it’s not a memory safe language? Not validating the files before loading them? Not validating the files before SHIPPING them? No CI? No safety net? No staged rollout in case of explosion? There are far FAR bigger mistakes here than “sys admin didn’t have to press button”.
- teeheelol 2y ago
  Exactly this. This is a faulty and dangerous product from conception to execution.
- jcgrillo 2y ago
  Is it normal to make outbound connections during boot? Doesn't that circumvent a firewall? That seems like something a security team evaluating whether they want this software on their network might care about during an eval period.. right?
  - sudosysgen 2y ago
    CrowdStrike is so invasive that it needs firewall exceptions. It does a lot of the actual antivirus work in the cloud. It's a security nightmare.
  - teeheelol 2y ago
    Looking at the contents of c:\windows\system32\drivers\crowdstrike suggests it does all sorts of weird shit right down to injecting itself into UEFI and futzing with firmware. It's literally in everything. Unfortunately "security" folk these days are box ticking fuckwits and this product brief ticked all the boxes. They do not understand any more traditional methodologies other than "install these magic beans and action the reports". Invest in better software and network architecture and DR strategy instead.
- dns_snek 2y ago
  > Panicking when the file doesn’t parse because it’s not a memory safe language? Whether a program panics or recovers when attempting to parse bad data is entirely orthogonal to memory safety. Do you have any in-depth technical information about the bug itself that you're basing this on?
a0123 2y ago
They still run Windows XP (og edition, not this patched rubbish) to make sure national security isn't compromised. The really important machines are still on Win 3.1.
mardifoufs 2y ago
What would the sysadmins do in this context? Read the release notes of the update? The only thing they would do is update and then be responsible for the problem, and in that case you're back to this exact problem. It's not like they'd read the source code or examine every file that's been changed or downloaded for a proprietary kernel module for every crowdstrike update (there must be a LOT of them).
- dmurray 2y ago
  They would release the update in a testing/sandbox environment first before rolling out kernel-level changes to every computer on their network. They're the same team who mandate you use a 3-year-old browser version and 5-year-old OS, because you can't be trusted to manage your own updates, so they do know the idea.
  - mardifoufs 2y ago
    Would this have changed something for this specific problem? I usually 100% agree with you fwiw, I just don't think this would've helped here because it seems like an almost "non update"? Most people claim there has been no update to the software, and no prompt or option to update it or not
    - dmurray 2y ago
      It's a file that was downloaded from Crowdstrike's servers, which have presumably been whitelisted in the firewall, and used to configure the software. Of course it's a software update, regardless of whether the file says .exe or .dll or .sys or .txt, and regardless of whether there was a prompt. Again, the same team in most enterprises wouldn't dream of letting you have an auto- updating Firefox Nightly, they know how to configure software so it doesn't phone home for updates or is blocked from phoning home.
more_corn 2y ago
You’re learning the wrong lesson here. Automatic security updates in Debian and Ubuntu actually get tested and work. The RCE in ssh a week ago is an argument for enabling automatic security updates. (And for security in depth, putting everything behind VPN for example) This example is probably an argument for not running windows on critical systems due to insufficient focus on security from the beginning which has lead to a need for things like crowdstrike. They do make a version of CS for Linux but nobody runs it unless they’re forced to by overzealous compliance drones.
- strunz 2y ago
  >They do make a version of CS for Linux but nobody runs it unless they’re forced to by overzealous compliance drones. I wish people would stop making blanket statement as if they know how every company in the world runs. Plenty of Linux machines are running CS, and it's not only because they are forced to for compliance. NG AV has been picking up speed as a "just in case" thing for Linux and Mac for years now. Your anecdote does not apply to everyone.

pas 2y ago

That's not the big no-no here. Lack of any real DRP is. Sure, it's cheaper to just buy CS Falcon (and who knows what other amazing vendors supplied timebombs are ticking silently) than paying sysadmins and developers ... and letting them build something that does what it needs, not much else, so there's no need to put these fantastic "single agents" from these RCE-as-a-service vendors on all the fucking servers.

Connector2542 2y ago
both are true

travoc 2y ago

It was a data update that triggered a software bug. It was not a software update. I don't think it's reasonable to make data updates illegal.

CoastalCoder 2y ago
I'm a general purpose computer, the distinction between software and data can be pretty fuzzy.
fire_lake 2y ago
Code is data and data is code
- surfingdino 2y ago
  Unless you use PIC controllers with https://en.wikipedia.org/wiki/Harvard_architecture https://en.wikipedia.org/wiki/Harvard_architecture
duped 2y ago
This distinction is meaningless at best and harmful at worst. If a code path isn't followed until a config file updates, that is practically the same thing as the code path being introduced by the update.

rfoo 2y ago

> why you NEVER have software that updates without explicit permission from a sysadmin In general I agree, but this case is quite messy. It's more like your anti-virus had a bug since forever that if it loads a broken virus definition it bricks your system. And a broken virus definition finally happened today. Do you want every virus definition (that is updated every few hours) to require explicit permission from a sysadmin?

johnkizer 2y ago

What % of those sysadmins are then going to turn around and script something to auto-approve those updates, once they realize that they are A) requested at inconvenient times and B) are related to security? Who's going to take the risk of appearing to have sat on an important update, while the org they support is ravaged by ThreatOfTheDay, because they thought they knew better than a multi-billion dollar, tops-in-their-field company? (I'm not necessarily saying that's actually objectively correct, but I can't imagine that many folks are willing to risk the downside)

avs733 2y ago

I understand the logic of this but it is somewhat based on the assumption - which most industries have in droves - that people in THAT industry are the competent bullwhark against stupidity. I consulted for a company for a while where the 'sysadmin' was the owner's mother - who bought laptops from walmart. Not only could she NOT have approved updates like this, even if she could she would have she wouldn't have had any knowledge whatsoever with which to make a determination if it worked. In an abstraction, the problem really is with externalities. These approaches to updates exist because people who CAN'T do what you describe are likely a more dominant part of the threat model than this happening to people you do describe. The resulting fix, as we're seeing, is very reliable until it isn't...and if the isn't is enormous in scale the systems aren't setup to fail gracefully. If you want to make a rule...require graceful failure.

MBCook 2y ago

Nah. That’s not the problem. Kernel level code blindly loading arbitrary files? Panicking when the file doesn’t parse because it’s not a memory safe language? Not validating the files before loading them? Not validating the files before SHIPPING them? No CI? No safety net? No staged rollout in case of explosion? There are far FAR bigger mistakes here than “sys admin didn’t have to press button”.

teeheelol 2y ago
Exactly this. This is a faulty and dangerous product from conception to execution.
jcgrillo 2y ago
Is it normal to make outbound connections during boot? Doesn't that circumvent a firewall? That seems like something a security team evaluating whether they want this software on their network might care about during an eval period.. right?
- sudosysgen 2y ago
  CrowdStrike is so invasive that it needs firewall exceptions. It does a lot of the actual antivirus work in the cloud. It's a security nightmare.
- teeheelol 2y ago
  Looking at the contents of c:\windows\system32\drivers\crowdstrike suggests it does all sorts of weird shit right down to injecting itself into UEFI and futzing with firmware. It's literally in everything. Unfortunately "security" folk these days are box ticking fuckwits and this product brief ticked all the boxes. They do not understand any more traditional methodologies other than "install these magic beans and action the reports". Invest in better software and network architecture and DR strategy instead.
dns_snek 2y ago
> Panicking when the file doesn’t parse because it’s not a memory safe language? Whether a program panics or recovers when attempting to parse bad data is entirely orthogonal to memory safety. Do you have any in-depth technical information about the bug itself that you're basing this on?

a0123 2y ago

They still run Windows XP (og edition, not this patched rubbish) to make sure national security isn't compromised. The really important machines are still on Win 3.1.

mardifoufs 2y ago

What would the sysadmins do in this context? Read the release notes of the update? The only thing they would do is update and then be responsible for the problem, and in that case you're back to this exact problem. It's not like they'd read the source code or examine every file that's been changed or downloaded for a proprietary kernel module for every crowdstrike update (there must be a LOT of them).

dmurray 2y ago
They would release the update in a testing/sandbox environment first before rolling out kernel-level changes to every computer on their network. They're the same team who mandate you use a 3-year-old browser version and 5-year-old OS, because you can't be trusted to manage your own updates, so they do know the idea.
- mardifoufs 2y ago
  Would this have changed something for this specific problem? I usually 100% agree with you fwiw, I just don't think this would've helped here because it seems like an almost "non update"? Most people claim there has been no update to the software, and no prompt or option to update it or not
  - dmurray 2y ago
    It's a file that was downloaded from Crowdstrike's servers, which have presumably been whitelisted in the firewall, and used to configure the software. Of course it's a software update, regardless of whether the file says .exe or .dll or .sys or .txt, and regardless of whether there was a prompt. Again, the same team in most enterprises wouldn't dream of letting you have an auto- updating Firefox Nightly, they know how to configure software so it doesn't phone home for updates or is blocked from phoning home.

more_corn 2y ago

You’re learning the wrong lesson here. Automatic security updates in Debian and Ubuntu actually get tested and work. The RCE in ssh a week ago is an argument for enabling automatic security updates. (And for security in depth, putting everything behind VPN for example) This example is probably an argument for not running windows on critical systems due to insufficient focus on security from the beginning which has lead to a need for things like crowdstrike. They do make a version of CS for Linux but nobody runs it unless they’re forced to by overzealous compliance drones.

strunz 2y ago
>They do make a version of CS for Linux but nobody runs it unless they’re forced to by overzealous compliance drones. I wish people would stop making blanket statement as if they know how every company in the world runs. Plenty of Linux machines are running CS, and it's not only because they are forced to for compliance. NG AV has been picking up speed as a "just in case" thing for Linux and Mac for years now. Your anecdote does not apply to everyone.