7 ms·

Technical Details on Today's Outage

124 points by xjia 2y ago

isthisreallife2 2y ago
So - a malformed configuration is capable of crashing a kernel process. Sounds very exploitable. Very
jchiu1106 2y ago
Where are the technical details?
- Zamiel_Snawley 2y ago
  Yeah, this PR statement is pretty much devoid of information other than it is not a cyber attack.
- deleted 2y ago
  [deleted]
- augustk 2y ago
  This is proprietary (closed-source) software as far as I understand.
hatsunearu 2y ago
I'm not a big expert but honestly this read like a bunch of garbage. > Although Channel Files end with the SYS extension, they are not kernel drivers. OK, but I'm pretty sure usermode software can't cause a BSOD. Clearly something running in kernel mode ate shit and that brought the system down. Just because a channel file not in kernel mode ate shit doesn't mean your kernel mode software isn't culpable. This just seems like a sleezy dodge.
- SoftTalker 2y ago
  The kernel driver reads the channel files. It choked on this one, and crashed.
  - hatsunearu 2y ago
    That's what I thought. So saying "it's not a kernel mode driver" is technically true, but I don't need to explain why it's a bunch of nonsense to try to damage control their incompetence.
    - epcoa 2y ago
      No idea why you’re getting downvoted. A configuration file for code that runs in kernel space is usually effectively kernel code (it certainly was in this case) - obviously there are formal methods to allow kernel code to be configured in a “safe” fashion, but it’s obvious that’s not going on here.
  - teeray 2y ago
    The kernel driver is an interpreter that executed an HCF instruction from the channel file.
  - SAI_Peregrinus 2y ago
    Which implies that any malware capable of replacing these channel files can crash their kernel driver. I wonder if there's a non-crashing way to exploit this & get kernel-space code execution.
    - numbsafari 2y ago
      I can guarantee you that you aren't the only one thinking this right now.
    - ytch 2y ago
      IIRC, there were some security software exploits, that trigger RCE (or DoS) of the scanning engine by malicious file?
    - joshka 2y ago
      If you have write access a path like C:\Windows\System32\drivers\CrowdStrike\ (and I'd assume the parent directory), then you pretty much can crash the kernel many ways. If you have the means to insert an AV config file update in between the config servers and the user's host then you probably can PWN the system pretty easily as well. What this probably does mean is that Crowdstrike will be receiving some attention from hackers of both hat colors. Here's the bug bounty page ... https://hackerone.com/crowdstrike?type=team https://hackerone.com/crowdstrike?type=team
- gjm11 2y ago
  It doesn't read to me as trying to dodge anything. They aren't saying "they're not kernel drivers, so everything is OK", they're saying "seeing the .sys on the filenames, you might think they're kernel drivers, but as it happens they're something else". (Maybe there's some subtext that I'm missing, but I don't see how saying "these aren't kernel drivers" makes them look any better, and I do see why they might say it to be informative, so it looks like to me like they're doing the latter.)
  - hatsunearu 2y ago
    > It doesn't read to me as trying to dodge anything. It absolutely reads like this. They are getting blasted online for shipping kernel mode driver updates without proper QA and release engineering. Which just from face value just seems like some insano style engineering. They are saying "it's not actually a kernel mode value" to deflect blame. I mean, I really don't understand why they would make this statement otherwise. If they are innocently just trying to say "this is just a channel file", there are other ways to say this, and it really isn't relevant enough to underline and emphasize.
    - gjm11 2y ago
      They're getting blasted for causing a massive worldwide outage due to what is clearly inadequate quality control. I don't see why this is any better if it's "pushed a kernel-mode driver update with bugs in it" than if it's "released a product with buggy kernel-mode stuff that can be made to crash by an innocuous-looking data file, and then pushed a data file that made it crash". Same result either way. Same demonstration of inadequate quality control either way. I think the story they're telling now, which so far as I know is the truth, looks worse for them, because it requires them to have screwed up their QC twice. Once when they made a product that do such bad things, and once when they pushed the data file to millions of PCs without checking what it did. So I still don't see how "this particular file happens not to be kernel-mode code" makes them look any better, and therefore I don't see why they'd be saying it "to deflect blame". It doesn't deflect blame; they look just as bad either way.
      - 2y ago
- deleted 2y ago
  [deleted]
tail_exchange 2y ago
Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.
- Zamiel_Snawley 2y ago
  Truly, how the extent the damage was so widespread is my main question at this point. Everyone has a buggy release at some point, but impacting global customers at this level is damn near unforgivable. Heads need to roll for this oversight.
- hatsunearu 2y ago
  It sounds like Channel files are just basically definition updates in normal antivirus software; it's not actually code, just some stuff on what the software should "look out for". And it sounds like they shipped some malformed channel file and the software that interprets it can't handle malformed inputs and ate shit. That software happened to be kernel mode, and also marked as boot-critical, so it if falls over, it causes a BSOD and inability to boot. and it's kind of understandable that channel files might seem safe to update constantly without oversight, but that's just assuming that the file that interprets the channel file isn't a bunch of dogshit code.
  - Murky3515 2y ago
    It's not understandable imo. At the very least they should have tests for the loader component that shows it can handle corrupted input. Amateur hour.
    - throwaway346434 2y ago
      Halting problem is undecidable. On the scale of "no one bothered to put error handling or validation in" to "a subtle problem exists for this given input"; you and I lack the information to make a judgement.
patrickthebold 2y ago
>The configuration update triggered a logic error that resulted in an operating system crash. > We understand how this issue occurred and we are doing a thorough root cause analysis to determine how this logic flaw occurred. There's always going to be flaws in the logic of the code, the trick is to not have single errors be so catastrophic.
- chris_nielsen 2y ago
  Yeah “how this logic flaw occurred” is the wrong question. How a common bug was rolled out globally with no controls, testing, or rollback strategy is the right question
  - YZF 2y ago
    They're all good questions. The thing that reads the config should have been fuzz tested with something like AFL. Likely should have a lot more tests. Maybe shouldn't run in a device driver. There's almost no doubt there are engineering process and culture issues here. And then absolutely the release process. Rollback is hard I guess once your OS can't boot.
    - cube00 2y ago
      > Rollback is hard I guess once your OS can't boot. This is why the client needs have enough error handling to realise it's latest update has now caused unsuccessful boot and roll that update back locally to the last known good configuration (or completely back to factory and pull all updates again).
ungreased0675 2y ago
This seems very unsatisfying. Not sure if I was expecting too much, but that’s a lot of words for very little information. I’d like more information on how these Channel Files are created, tested, and deployed. What’s the minimum number of people that can do it? How fast can the process go?
xyst 2y ago
“Technical” detail report reads more like a lawyer generated report. This company is awful. If I ever get a sales pitch from these shit brains, they will get immediately shut down. Also fuck MS and their awful operating system that then spawned this god awful product/company known as “CrowdStike Falcon”
- userbinator 2y ago
  If Windows wasn't as popular, then this might've happened to Linux to macOS instead. Blame CrowdStrike's incompetence, not MS.
  - acdha 2y ago
    It couldn't happen on macOS: Apple stopped letting third-parties run code in the kernel after years of failures like this. It also wouldn't happen on Linux: they use eBPF there which was designed by grownups and validates its inputs.
    - j16sdiz 2y ago
      Kext still exist on macos
      - acdha 2y ago
        Kind of: they’ve been deprecated for 4 years and you have to disable SIP to load them. https://developer.apple.com/support/kernel-extensions/ https://developer.apple.com/support/kernel-extensions/ They’ve added system extension mechanisms for the most common needs trying to balance the various things people use kexts for against the impact on security, performance, and reliability many kexts had.
    - mynameisvlad 2y ago
      eBPF exists on Windows, too: https://microsoft.github.io/ebpf-for-windows/ https://microsoft.github.io/ebpf-for-windows/ They’re just not using it. They could have not used it for Linux too. The presence of the feature is not enough to guarantee this would’ve never happened in a hypothetical.
PedroBatista 2y ago
Light on technical and light on details. Putting the actual blast radius aside, this whole thing seems a bit amateurish for a "security company" that pulls the contracts they do.
- grecy 2y ago
  You assume the most competent company got the contracts, which is simply not the world we live in. The company that lobbied the hardest and paid the most in bribes got the contracts.
  - manquer 2y ago
    Rather point I think is there technical and evaluation gates companies of this nature regularly go through while contracting, part of that is being able to talk the language of the industry properly . This seems very amateurish for companies who regularly talk professionally to win said contracts , whether the best product or not. My guess is C-suite, crisis consultants and lawyers are involved heavily so the actual engineering folks have little voice now in any communication and we get stuff like this.
  - tootie 2y ago
    Having been through enough procurement cycles as both a buyer and seller there does not need to be a whit of malfeasance for a bad decision to occur. It's aggressive sales, price wars, poorly informed decision makers, gut instinct, favoritism, familiarity, incumbency, network effects. You notice how this outage affected hospitals and airlines? There is a strong tendency in software sales for industries to align around one or two leaders. Oh, American chose Crowdstrike? Maybe we at Delta should just do what they did. Or literally Delta hires the VP from American to be their CISO and he just does what he did before. Vendor selection is hard and buyer's remorse is frequently hard to deal with once you've sunk cost into a migration.
- dev-jayson 2y ago
  Yeah, I think I'm getting more detailed analysis on Social Media from strangers, which I know I should take with a grain of salt. But I guess I'm expecting a lot more than "a filed caused this" from the company that caused this havoc.
automatoney 2y ago
To my biased ears it sounds like these configuration-like files are a borderline DSL that maybe isn't being treated as such. I feel like that's a common issue - people assume because you call it a config file, it's not a language, and so it doesn't get treated as actual code that gets interpreted.
pneumonic 2y ago
> we are doing a "root cause analysis to determine how this logic flaw occurred" That's going to find a cause: a programmer made an error. That's not the root of the problem. The root of the problem is allowing such an error to be released (especially obvious because of its widespread impact).
geuis 2y ago
Weak. Very weak and over corporate level of ass covering. And it doesn't even come close to doing that. They should just let the EM of the team involved provide a public detailed response that I'm sure is floating around internally. Just own the problem and address the questions rather than trying to play at politics, quite poorly.
canistel 2y ago
> This issue is not the result of or related to a cyberattack. Must be corrected to "the issue is not the result of or related to a cyberattack by external agents".
kyriakos 2y ago
Why is everyone blaming Microsoft? Is this something of an oversight in their side too? Can someone explain?
- sgammon 2y ago
  For letting a failure of this magnitude be possible, I suspect
- cyrnel 2y ago
  I'm no kernel expert, but people are saying Microsoft deserves some blame for not exposing necessary functionality to user space, requiring the use of a very-unsafe kernel driver. Linux provides eBPF and macOS provides system extensions. I'll also add that Windows itself heavily prioritizes backwards-compatibility over security, which leads companies to seek out third-party solutions for stopping malware instead of design-based mitigations being built into Windows.
  - mardifoufs 2y ago
    I don't agree. I'm glad Microsoft doesn't provide the functionality to do what crowdstrike does to user space. Crowdstrike acts in a similar way to deeply seated malware, except that it is usually installed voluntarily. But the behavior and capabilities that it has are basically what any malware would dream of, and exposing them to user space would imo create a mess (especially on windows). If anything, this is good as it will make people even more weary of kernel mode software. And I'm not sure epbf actually allows you to do a lot of the stuff crowdstrike-like software does. I know they use it on Linux though so maybe eBPF has evolved a lot since I last looked at it.
    - cyrnel 2y ago
      I generally agree with you. It's an either-or thing: either Microsoft secures their OS, or they provide safe ways for users to secure their OS. The first option is a million times better, but having neither option leads us to this mess.
timbelina 2y ago
Can someone aim me at some RTFM that describes the sensor release and patching process, please? I'm lost trying to understand: When a new version 'n' of the sensor is released, we upgrade a selected batch of machines and do some tests (mostly waiting around :-)) to see that all is well. Then we upgrade the rest of the fleet by OU. However, 'cause we're scaredy cats, we leave some critical kit on n-1 for longer. And some really critical kit even on n-2. (Yeah, there's a risk in not applying patches I know but there are other outage-related risks that we balance; forget that for now) Our assumption is that n-1, n-2, etc are old, stable releases, and so when fan and shit collided yesterday, we just hopped on the console and did a policy update to revert to n-2 and assumed we'd dodged the bullet. But of course, that failed... you know what they say about assumptions :-) So in a long-winded way that leads to my three questions: Why did the 'content update' take out not just n but n-whatever sensors equally as effectively? Are the n-whatever versions not actually stable? And if the n-whatever versions are not actually stable and are being patched, what's the point of the versioning? Cheers!
bryan_w 2y ago
It kinda feels like someone added a watch for c:\COM\COM like we did back in the day on AOL
dang 2y ago
Related ongoing thread: CrowdStrike Update: Windows Bluescreen and Boot Loops - https://news.ycombinator.com/item?id=41002195 https://news.ycombinator.com/item?id=41002195 - July 2024 (3590 comments)
rdtsc 2y ago
> The update that occurred at 04:09 UTC was designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks The obvious joke here is CS runs the malicious C2 framework. So the system worked as designed: it prevented further execution and quarantined the affected machines. But given they say that’s just a configuration file (then why the hell is it suffixed with .sys?), it’s actually plausible. A smart attacker could disguise themselves and use the same facilities as the CS. CS will try to block them and blocks itself in the process?
nonfamous 2y ago
>>> Systems that are not currently impacted will continue to operate as expected, continue to provide protection, and have no risk of experiencing this event in the future. Given that this incident has now happened twice in the space of months (first on Linux, then on Windows), and that as stated in this very post the root cause analysis is not yet complete, I find that statement of “NO RISK” very hard to believe.
0nate 2y ago
The lower you go in system architecture, the greater the impact when defects occur. In this instance, the Crowdstrike agent is embedded within the Windows Kernel, and registered with the Kernel Filter Engine illustrated in the diagram below. https://www.nathanhandy.blog/images/blog/OSI%20Model%20in%20Practice%20v1.1%20-%20SingleSystem%20-%20Large.png https://www.nathanhandy.blog/images/blog/OSI%20Model%20in%20... If the initial root cause analysis is correct, Crowdstrike has pushed out a bug that could have been easily stopped had software engineering best practices been followed: Unit Testing, Code Coverage, Integration Testing, Definition of Done.
bkjshki 2y ago
[flagged]

124 points by xjia 2y ago

isthisreallife2 2y ago

So - a malformed configuration is capable of crashing a kernel process. Sounds very exploitable. Very

jchiu1106 2y ago

Where are the technical details?

Zamiel_Snawley 2y ago
Yeah, this PR statement is pretty much devoid of information other than it is not a cyber attack.
deleted 2y ago
[deleted]
augustk 2y ago
This is proprietary (closed-source) software as far as I understand.

hatsunearu 2y ago

I'm not a big expert but honestly this read like a bunch of garbage. > Although Channel Files end with the SYS extension, they are not kernel drivers. OK, but I'm pretty sure usermode software can't cause a BSOD. Clearly something running in kernel mode ate shit and that brought the system down. Just because a channel file not in kernel mode ate shit doesn't mean your kernel mode software isn't culpable. This just seems like a sleezy dodge.

SoftTalker 2y ago
The kernel driver reads the channel files. It choked on this one, and crashed.
- hatsunearu 2y ago
  That's what I thought. So saying "it's not a kernel mode driver" is technically true, but I don't need to explain why it's a bunch of nonsense to try to damage control their incompetence.
  - epcoa 2y ago
    No idea why you’re getting downvoted. A configuration file for code that runs in kernel space is usually effectively kernel code (it certainly was in this case) - obviously there are formal methods to allow kernel code to be configured in a “safe” fashion, but it’s obvious that’s not going on here.
- teeray 2y ago
  The kernel driver is an interpreter that executed an HCF instruction from the channel file.
- SAI_Peregrinus 2y ago
  Which implies that any malware capable of replacing these channel files can crash their kernel driver. I wonder if there's a non-crashing way to exploit this & get kernel-space code execution.
  - numbsafari 2y ago
    I can guarantee you that you aren't the only one thinking this right now.
  - ytch 2y ago
    IIRC, there were some security software exploits, that trigger RCE (or DoS) of the scanning engine by malicious file?
  - joshka 2y ago
    If you have write access a path like C:\Windows\System32\drivers\CrowdStrike\ (and I'd assume the parent directory), then you pretty much can crash the kernel many ways. If you have the means to insert an AV config file update in between the config servers and the user's host then you probably can PWN the system pretty easily as well. What this probably does mean is that Crowdstrike will be receiving some attention from hackers of both hat colors. Here's the bug bounty page ... https://hackerone.com/crowdstrike?type=team https://hackerone.com/crowdstrike?type=team
gjm11 2y ago
It doesn't read to me as trying to dodge anything. They aren't saying "they're not kernel drivers, so everything is OK", they're saying "seeing the .sys on the filenames, you might think they're kernel drivers, but as it happens they're something else". (Maybe there's some subtext that I'm missing, but I don't see how saying "these aren't kernel drivers" makes them look any better, and I do see why they might say it to be informative, so it looks like to me like they're doing the latter.)
- hatsunearu 2y ago
  > It doesn't read to me as trying to dodge anything. It absolutely reads like this. They are getting blasted online for shipping kernel mode driver updates without proper QA and release engineering. Which just from face value just seems like some insano style engineering. They are saying "it's not actually a kernel mode value" to deflect blame. I mean, I really don't understand why they would make this statement otherwise. If they are innocently just trying to say "this is just a channel file", there are other ways to say this, and it really isn't relevant enough to underline and emphasize.
  - gjm11 2y ago
    They're getting blasted for causing a massive worldwide outage due to what is clearly inadequate quality control. I don't see why this is any better if it's "pushed a kernel-mode driver update with bugs in it" than if it's "released a product with buggy kernel-mode stuff that can be made to crash by an innocuous-looking data file, and then pushed a data file that made it crash". Same result either way. Same demonstration of inadequate quality control either way. I think the story they're telling now, which so far as I know is the truth, looks worse for them, because it requires them to have screwed up their QC twice. Once when they made a product that do such bad things, and once when they pushed the data file to millions of PCs without checking what it did. So I still don't see how "this particular file happens not to be kernel-mode code" makes them look any better, and therefore I don't see why they'd be saying it "to deflect blame". It doesn't deflect blame; they look just as bad either way.
    - 2y ago
deleted 2y ago
[deleted]

tail_exchange 2y ago

Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.

Zamiel_Snawley 2y ago
Truly, how the extent the damage was so widespread is my main question at this point. Everyone has a buggy release at some point, but impacting global customers at this level is damn near unforgivable. Heads need to roll for this oversight.
hatsunearu 2y ago
It sounds like Channel files are just basically definition updates in normal antivirus software; it's not actually code, just some stuff on what the software should "look out for". And it sounds like they shipped some malformed channel file and the software that interprets it can't handle malformed inputs and ate shit. That software happened to be kernel mode, and also marked as boot-critical, so it if falls over, it causes a BSOD and inability to boot. and it's kind of understandable that channel files might seem safe to update constantly without oversight, but that's just assuming that the file that interprets the channel file isn't a bunch of dogshit code.
- Murky3515 2y ago
  It's not understandable imo. At the very least they should have tests for the loader component that shows it can handle corrupted input. Amateur hour.
  - throwaway346434 2y ago
    Halting problem is undecidable. On the scale of "no one bothered to put error handling or validation in" to "a subtle problem exists for this given input"; you and I lack the information to make a judgement.

patrickthebold 2y ago

>The configuration update triggered a logic error that resulted in an operating system crash. > We understand how this issue occurred and we are doing a thorough root cause analysis to determine how this logic flaw occurred. There's always going to be flaws in the logic of the code, the trick is to not have single errors be so catastrophic.

chris_nielsen 2y ago
Yeah “how this logic flaw occurred” is the wrong question. How a common bug was rolled out globally with no controls, testing, or rollback strategy is the right question
- YZF 2y ago
  They're all good questions. The thing that reads the config should have been fuzz tested with something like AFL. Likely should have a lot more tests. Maybe shouldn't run in a device driver. There's almost no doubt there are engineering process and culture issues here. And then absolutely the release process. Rollback is hard I guess once your OS can't boot.
  - cube00 2y ago
    > Rollback is hard I guess once your OS can't boot. This is why the client needs have enough error handling to realise it's latest update has now caused unsuccessful boot and roll that update back locally to the last known good configuration (or completely back to factory and pull all updates again).

ungreased0675 2y ago

This seems very unsatisfying. Not sure if I was expecting too much, but that’s a lot of words for very little information. I’d like more information on how these Channel Files are created, tested, and deployed. What’s the minimum number of people that can do it? How fast can the process go?

xyst 2y ago

“Technical” detail report reads more like a lawyer generated report. This company is awful. If I ever get a sales pitch from these shit brains, they will get immediately shut down. Also fuck MS and their awful operating system that then spawned this god awful product/company known as “CrowdStike Falcon”

userbinator 2y ago
If Windows wasn't as popular, then this might've happened to Linux to macOS instead. Blame CrowdStrike's incompetence, not MS.
- acdha 2y ago
  It couldn't happen on macOS: Apple stopped letting third-parties run code in the kernel after years of failures like this. It also wouldn't happen on Linux: they use eBPF there which was designed by grownups and validates its inputs.
  - j16sdiz 2y ago
    Kext still exist on macos
    - acdha 2y ago
      Kind of: they’ve been deprecated for 4 years and you have to disable SIP to load them. https://developer.apple.com/support/kernel-extensions/ https://developer.apple.com/support/kernel-extensions/ They’ve added system extension mechanisms for the most common needs trying to balance the various things people use kexts for against the impact on security, performance, and reliability many kexts had.
  - mynameisvlad 2y ago
    eBPF exists on Windows, too: https://microsoft.github.io/ebpf-for-windows/ https://microsoft.github.io/ebpf-for-windows/ They’re just not using it. They could have not used it for Linux too. The presence of the feature is not enough to guarantee this would’ve never happened in a hypothetical.

PedroBatista 2y ago

Light on technical and light on details. Putting the actual blast radius aside, this whole thing seems a bit amateurish for a "security company" that pulls the contracts they do.

grecy 2y ago
You assume the most competent company got the contracts, which is simply not the world we live in. The company that lobbied the hardest and paid the most in bribes got the contracts.
- manquer 2y ago
  Rather point I think is there technical and evaluation gates companies of this nature regularly go through while contracting, part of that is being able to talk the language of the industry properly . This seems very amateurish for companies who regularly talk professionally to win said contracts , whether the best product or not. My guess is C-suite, crisis consultants and lawyers are involved heavily so the actual engineering folks have little voice now in any communication and we get stuff like this.
- tootie 2y ago
  Having been through enough procurement cycles as both a buyer and seller there does not need to be a whit of malfeasance for a bad decision to occur. It's aggressive sales, price wars, poorly informed decision makers, gut instinct, favoritism, familiarity, incumbency, network effects. You notice how this outage affected hospitals and airlines? There is a strong tendency in software sales for industries to align around one or two leaders. Oh, American chose Crowdstrike? Maybe we at Delta should just do what they did. Or literally Delta hires the VP from American to be their CISO and he just does what he did before. Vendor selection is hard and buyer's remorse is frequently hard to deal with once you've sunk cost into a migration.
dev-jayson 2y ago
Yeah, I think I'm getting more detailed analysis on Social Media from strangers, which I know I should take with a grain of salt. But I guess I'm expecting a lot more than "a filed caused this" from the company that caused this havoc.

automatoney 2y ago

To my biased ears it sounds like these configuration-like files are a borderline DSL that maybe isn't being treated as such. I feel like that's a common issue - people assume because you call it a config file, it's not a language, and so it doesn't get treated as actual code that gets interpreted.

pneumonic 2y ago

> we are doing a "root cause analysis to determine how this logic flaw occurred" That's going to find a cause: a programmer made an error. That's not the root of the problem. The root of the problem is allowing such an error to be released (especially obvious because of its widespread impact).

geuis 2y ago

Weak. Very weak and over corporate level of ass covering. And it doesn't even come close to doing that. They should just let the EM of the team involved provide a public detailed response that I'm sure is floating around internally. Just own the problem and address the questions rather than trying to play at politics, quite poorly.

canistel 2y ago

> This issue is not the result of or related to a cyberattack. Must be corrected to "the issue is not the result of or related to a cyberattack by external agents".

kyriakos 2y ago

Why is everyone blaming Microsoft? Is this something of an oversight in their side too? Can someone explain?

sgammon 2y ago
For letting a failure of this magnitude be possible, I suspect
cyrnel 2y ago
I'm no kernel expert, but people are saying Microsoft deserves some blame for not exposing necessary functionality to user space, requiring the use of a very-unsafe kernel driver. Linux provides eBPF and macOS provides system extensions. I'll also add that Windows itself heavily prioritizes backwards-compatibility over security, which leads companies to seek out third-party solutions for stopping malware instead of design-based mitigations being built into Windows.
- mardifoufs 2y ago
  I don't agree. I'm glad Microsoft doesn't provide the functionality to do what crowdstrike does to user space. Crowdstrike acts in a similar way to deeply seated malware, except that it is usually installed voluntarily. But the behavior and capabilities that it has are basically what any malware would dream of, and exposing them to user space would imo create a mess (especially on windows). If anything, this is good as it will make people even more weary of kernel mode software. And I'm not sure epbf actually allows you to do a lot of the stuff crowdstrike-like software does. I know they use it on Linux though so maybe eBPF has evolved a lot since I last looked at it.
  - cyrnel 2y ago
    I generally agree with you. It's an either-or thing: either Microsoft secures their OS, or they provide safe ways for users to secure their OS. The first option is a million times better, but having neither option leads us to this mess.

timbelina 2y ago

Can someone aim me at some RTFM that describes the sensor release and patching process, please? I'm lost trying to understand: When a new version 'n' of the sensor is released, we upgrade a selected batch of machines and do some tests (mostly waiting around :-)) to see that all is well. Then we upgrade the rest of the fleet by OU. However, 'cause we're scaredy cats, we leave some critical kit on n-1 for longer. And some really critical kit even on n-2. (Yeah, there's a risk in not applying patches I know but there are other outage-related risks that we balance; forget that for now) Our assumption is that n-1, n-2, etc are old, stable releases, and so when fan and shit collided yesterday, we just hopped on the console and did a policy update to revert to n-2 and assumed we'd dodged the bullet. But of course, that failed... you know what they say about assumptions :-) So in a long-winded way that leads to my three questions: Why did the 'content update' take out not just n but n-whatever sensors equally as effectively? Are the n-whatever versions not actually stable? And if the n-whatever versions are not actually stable and are being patched, what's the point of the versioning? Cheers!

bryan_w 2y ago

It kinda feels like someone added a watch for c:\COM\COM like we did back in the day on AOL

dang 2y ago

Related ongoing thread: CrowdStrike Update: Windows Bluescreen and Boot Loops - https://news.ycombinator.com/item?id=41002195 https://news.ycombinator.com/item?id=41002195 - July 2024 (3590 comments)

rdtsc 2y ago

> The update that occurred at 04:09 UTC was designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks The obvious joke here is CS runs the malicious C2 framework. So the system worked as designed: it prevented further execution and quarantined the affected machines. But given they say that’s just a configuration file (then why the hell is it suffixed with .sys?), it’s actually plausible. A smart attacker could disguise themselves and use the same facilities as the CS. CS will try to block them and blocks itself in the process?

nonfamous 2y ago

>>> Systems that are not currently impacted will continue to operate as expected, continue to provide protection, and have no risk of experiencing this event in the future. Given that this incident has now happened twice in the space of months (first on Linux, then on Windows), and that as stated in this very post the root cause analysis is not yet complete, I find that statement of “NO RISK” very hard to believe.

0nate 2y ago

The lower you go in system architecture, the greater the impact when defects occur. In this instance, the Crowdstrike agent is embedded within the Windows Kernel, and registered with the Kernel Filter Engine illustrated in the diagram below. https://www.nathanhandy.blog/images/blog/OSI%20Model%20in%20Practice%20v1.1%20-%20SingleSystem%20-%20Large.png https://www.nathanhandy.blog/images/blog/OSI%20Model%20in%20... If the initial root cause analysis is correct, Crowdstrike has pushed out a bug that could have been easily stopped had software engineering best practices been followed: Unit Testing, Code Coverage, Integration Testing, Definition of Done.

bkjshki 2y ago

[flagged]