6 ms·

TL;DR: Windows Defender had a bug that made certain system calls expensive on CPU cycles when Defender's Real-time Protection feature is enabled. After discover

TL;DR: Windows Defender had a bug that made certain system calls expensive on CPU cycles when Defender's Real-time Protection feature is enabled. After discovery, Mozilla reported this issue to Microsoft. Microsoft is releasing a patch that should result in lower CPU usage when using Firefox on sites like YouTube (a ~75% CPU usage reduction was noted when browsing YouTube in Firefox with the fixed version of Defender). It seems like the HN submission form truncated the # from the end of the URL I linked to, which linked to the relevant comment. I'll try that here: https://bugzilla.mozilla.org/show_bug.cgi?id=1441918#c82 https://bugzilla.mozilla.org/show_bug.cgi?id=1441918#c82 and https://bugzilla.mozilla.org/show_bug.cgi?id=1441918#c91 https://bugzilla.mozilla.org/show_bug.cgi?id=1441918#c91

dang 3y ago
Ok, I've put that back in the URL above. Thanks.
IronWolve 3y ago
It's not just mozilla, been working defender issues for the last few years on thousands of windows vm's. Mostly due to the enabling the more intensive heuristic real time engine and they have different code bases depending on versions installed on different windows builds, and patching does seem to trigger it. For months we had issues where we couldnt log into some vm's due to high cpu for defender, and had to bounce the vm and apply a temp defender fix. I think its a growing issue, as they mature/migrate their older code base, issues become less frequent.
- psychphysic 3y ago
  I have malwarebytes premium and defender CPU usage is nearly 100% at times bringin Firefox to a halt. Chrome works fine..I've been blaming Firefox so far.
  - rejectfinite 3y ago
    I am on Windows 10, Malwarebytes premium and using Firefox Nightly on Youtube right now and it is using miniscule CPU and has so for a long time. On a i7 4790k desktop machine. Firefox itself is at 4-5% and the whole machine is at 14% Normal Firefox was also fine last I used it.
  - Yoric 3y ago
    In my experience (as a former Firefox dev), antivirus / antimalware software are really poorly behaved. They tend to: - require admin rights (which means that if they have vulnerabilities, it can take control of the entire machine, even if Firefox itself is sanboxed); - monkey-patch the Firefox executable in memory, which works (when it does) as long as the version of the software tracks closely the version of Firefox, which may or may not be the case; - ... and also decreases the memory-safety of Firefox, which makes it easier to pwn; - ... and also makes the crash reports unreliable; - install encryption certificates that are actually less trustworthy than Mozilla's, hence decreasing the security of https; - block Firefox and add-on security updates, also decreasing security; - install privileged add-ons, many of which are easy to exploit from any webpage; - ... Part of the work on Crash Scene Investigations was attempting to determine whether the crash was in Firefox or in code or in some bogus foreign code. Depressingly often, it was the latter. In your case, it's entirely possible that malwarebytes was simply untested on Firefox.
    - genocidicbunny 3y ago
      > - monkey-patch the Firefox executable in memory, which works (when it does) as long as the version of the software tracks closely the version of Firefox, which may or may not be the case; This one was a frustratingly common cause of crashes when I worked in gamedev. So many crashes would end up being some overlay or antivirus monkeying about with memory.
    - jbritton 3y ago
      I had always assumed that one application could not touch the memory of another application. Does running as Admin allow breaking this boundary?
      - codedokode 3y ago
        This is wrong, on Windows there are system calls to access memory of other process and on Linux you can do it using debugging. Also on Windows there is a tradition to inject libraries into other processes, create threads in processes etc.
  - gcp 3y ago
    Did you report this on Bugzilla? Even if it's not Firefox's fault if you don't report it no-one knows about the issue. Recent versions of Firefox allow you to block some stuff like that: https://support.mozilla.org/en-US/kb/identify-problems-third-party-modules-firefox-windows#w_block-modules-that-cause-firefox-to-crash https://support.mozilla.org/en-US/kb/identify-problems-third... Though it's possible they use different code injection tricks to make blocking impossible. (You can't block Defender from listening to events for example)
Diggsey 3y ago
Well, also Firefox is making an excessive number of calls to that slow system call compared to other browsers (Chrome, Edge).
- sfink 3y ago
  My understanding is that until recently (January), V8 (inside Chrome & Edge) made a similar number of calls. The main use is making it so that JIT-generated code is not writable while it is executing. It's an important security measure. V8 switched to a more recent mechanism (memory protection keys) that have been gradually getting support from the various OSes. But IIUC, they switched off the mprotect/VirtualProtect calls unconditionally, and added in the protection key stuff only where supported, which suggests that they left some configurations without any protection at all. SpiderMonkey (in Firefox) has not yet switched to the cheaper mechanism. I may have some of the details wrong. https://source.chromium.org/chromium/_/chromium/v8/v8.git/+/32f82d0f4831dbdb242940d3dfb5048c986f5560 https://source.chromium.org/chromium/_/chromium/v8/v8.git/+/...
  - nagisa 3y ago
    pkeys are hardware-specific as far as I am aware, and at least last time I tried them didn't work on hardware as recent as zen 1.
- zokier 3y ago
  Well, it was fast system call until MS added AV hook to it.
  - Diggsey 3y ago
    My comment was only intended to add missing information to the TLDR (since this fact is important in the linked thread) not to say that Firefox is at fault. Now that you raised it however, even if the system call used to be fast, Firefox is making an extremely high number of calls to that sytem call, and there's always going to be some overhead to that. There are almost certainly ways that Firefox could reduce the number of calls it needs to make.
zokier 3y ago
Note that this issue is not exclusive to MS Defender, but likely all Windows AV products to varying degrees: > > I would also like to add that this high CPU usage issue while using Firefox is not exclusive to Microsoft Defender. It's an issue for Norton's AV products also and should be the same for Symantec Endpoint products too. > > So, you should also test them. > It is true that we should analyze the situation with other AV vendors, however, given the numbers shared above, and given how relevant it is to keep track of memory protection changes in order to detect malicious behavior, it is very likely that the explanation for Windows Defender also applies (at least in part) to other AV vendors. Can we get edit on the title?
jodrellblank 3y ago
> “Windows Defender had a bug that made certain system calls expensive” It also has a bug(?) which makes method calls 100x slower in PowerShell 7: https://github.com/PowerShell/PowerShell/issues/19431 https://github.com/PowerShell/PowerShell/issues/19431
cjblack 3y ago
I'm curious how much excess energy has been consumed, and won't be consumed any longer, as a result of this improvement - even just limited to reduced CPU usage on Windows machines using Firefox to watch Youtube. I love thinking about the impacts of tiny improvements at scale like this, might do some napkin math on it later and see if I can come up with something in the right order of magnitude.
- fsckboy 3y ago
  firefox browser share is teeny tiny these days
  - botanical 3y ago
    It's 200 million people, with 80% on Windows. https://data.firefox.com/dashboard/user-activity https://data.firefox.com/dashboard/user-activity https://data.firefox.com/dashboard/hardware https://data.firefox.com/dashboard/hardware
  - tomrod 3y ago
    Teeny tiny multiplied by 7 Billion by 365 days per year by 24 hours per day by a fraction of a kW does add up.
    - beAbU 3y ago
      7B people are not watching youtube on Firefox 24/7 365 days a year.
      - mulmen 3y ago
        But at any given moment someone is.
      - tomrod 3y ago
        Correct. Some teeny tiny fraction of market share is. For the conceptual calculation, I refer you to my earlier comment.
    - 3y ago
Sakos 3y ago
I've seen some really weird performance behavior from Defender and I just keep it disabled on my desktop device now. I'm not surprised to see it affecting Firefox like this. Defender's dropped all the way to the bottom of the list in effectiveness anyway, so I don't feel it's a big loss.
nikanj 3y ago
The biggest surprise for me was Microsoft actually fixing it.
- elygre 3y ago
  Is that because you don't expect programmers in general to fix their bugs? Or do you think Microsoft in particular don't care about their products?
  - nikanj 3y ago
    Because once a corporation grows larger than some singularity threshold, there seems to be a bug event horizon where all bug reports just disappear. Send a bug report to a five-person software company, their lead dev contacts you the same day and has a patched version ready to go in a week. Send a bug report to Microsoft / Citrix / Apple / etc, and you'll never hear back.
- hoffs 3y ago
  They said that Microsoft was already in progress of changing the offending code before Mozilla report.

dang 3y ago

Ok, I've put that back in the URL above. Thanks.

IronWolve 3y ago

It's not just mozilla, been working defender issues for the last few years on thousands of windows vm's. Mostly due to the enabling the more intensive heuristic real time engine and they have different code bases depending on versions installed on different windows builds, and patching does seem to trigger it. For months we had issues where we couldnt log into some vm's due to high cpu for defender, and had to bounce the vm and apply a temp defender fix. I think its a growing issue, as they mature/migrate their older code base, issues become less frequent.

psychphysic 3y ago
I have malwarebytes premium and defender CPU usage is nearly 100% at times bringin Firefox to a halt. Chrome works fine..I've been blaming Firefox so far.
- rejectfinite 3y ago
  I am on Windows 10, Malwarebytes premium and using Firefox Nightly on Youtube right now and it is using miniscule CPU and has so for a long time. On a i7 4790k desktop machine. Firefox itself is at 4-5% and the whole machine is at 14% Normal Firefox was also fine last I used it.
- Yoric 3y ago
  In my experience (as a former Firefox dev), antivirus / antimalware software are really poorly behaved. They tend to: - require admin rights (which means that if they have vulnerabilities, it can take control of the entire machine, even if Firefox itself is sanboxed); - monkey-patch the Firefox executable in memory, which works (when it does) as long as the version of the software tracks closely the version of Firefox, which may or may not be the case; - ... and also decreases the memory-safety of Firefox, which makes it easier to pwn; - ... and also makes the crash reports unreliable; - install encryption certificates that are actually less trustworthy than Mozilla's, hence decreasing the security of https; - block Firefox and add-on security updates, also decreasing security; - install privileged add-ons, many of which are easy to exploit from any webpage; - ... Part of the work on Crash Scene Investigations was attempting to determine whether the crash was in Firefox or in code or in some bogus foreign code. Depressingly often, it was the latter. In your case, it's entirely possible that malwarebytes was simply untested on Firefox.
  - genocidicbunny 3y ago
    > - monkey-patch the Firefox executable in memory, which works (when it does) as long as the version of the software tracks closely the version of Firefox, which may or may not be the case; This one was a frustratingly common cause of crashes when I worked in gamedev. So many crashes would end up being some overlay or antivirus monkeying about with memory.
  - jbritton 3y ago
    I had always assumed that one application could not touch the memory of another application. Does running as Admin allow breaking this boundary?
    - codedokode 3y ago
      This is wrong, on Windows there are system calls to access memory of other process and on Linux you can do it using debugging. Also on Windows there is a tradition to inject libraries into other processes, create threads in processes etc.
- gcp 3y ago
  Did you report this on Bugzilla? Even if it's not Firefox's fault if you don't report it no-one knows about the issue. Recent versions of Firefox allow you to block some stuff like that: https://support.mozilla.org/en-US/kb/identify-problems-third-party-modules-firefox-windows#w_block-modules-that-cause-firefox-to-crash https://support.mozilla.org/en-US/kb/identify-problems-third... Though it's possible they use different code injection tricks to make blocking impossible. (You can't block Defender from listening to events for example)

Diggsey 3y ago

Well, also Firefox is making an excessive number of calls to that slow system call compared to other browsers (Chrome, Edge).

sfink 3y ago
My understanding is that until recently (January), V8 (inside Chrome & Edge) made a similar number of calls. The main use is making it so that JIT-generated code is not writable while it is executing. It's an important security measure. V8 switched to a more recent mechanism (memory protection keys) that have been gradually getting support from the various OSes. But IIUC, they switched off the mprotect/VirtualProtect calls unconditionally, and added in the protection key stuff only where supported, which suggests that they left some configurations without any protection at all. SpiderMonkey (in Firefox) has not yet switched to the cheaper mechanism. I may have some of the details wrong. https://source.chromium.org/chromium/_/chromium/v8/v8.git/+/32f82d0f4831dbdb242940d3dfb5048c986f5560 https://source.chromium.org/chromium/_/chromium/v8/v8.git/+/...
- nagisa 3y ago
  pkeys are hardware-specific as far as I am aware, and at least last time I tried them didn't work on hardware as recent as zen 1.
zokier 3y ago
Well, it was fast system call until MS added AV hook to it.
- Diggsey 3y ago
  My comment was only intended to add missing information to the TLDR (since this fact is important in the linked thread) not to say that Firefox is at fault. Now that you raised it however, even if the system call used to be fast, Firefox is making an extremely high number of calls to that sytem call, and there's always going to be some overhead to that. There are almost certainly ways that Firefox could reduce the number of calls it needs to make.

zokier 3y ago

Note that this issue is not exclusive to MS Defender, but likely all Windows AV products to varying degrees: > > I would also like to add that this high CPU usage issue while using Firefox is not exclusive to Microsoft Defender. It's an issue for Norton's AV products also and should be the same for Symantec Endpoint products too. > > So, you should also test them. > It is true that we should analyze the situation with other AV vendors, however, given the numbers shared above, and given how relevant it is to keep track of memory protection changes in order to detect malicious behavior, it is very likely that the explanation for Windows Defender also applies (at least in part) to other AV vendors. Can we get edit on the title?

jodrellblank 3y ago

> “Windows Defender had a bug that made certain system calls expensive” It also has a bug(?) which makes method calls 100x slower in PowerShell 7: https://github.com/PowerShell/PowerShell/issues/19431 https://github.com/PowerShell/PowerShell/issues/19431

cjblack 3y ago

I'm curious how much excess energy has been consumed, and won't be consumed any longer, as a result of this improvement - even just limited to reduced CPU usage on Windows machines using Firefox to watch Youtube. I love thinking about the impacts of tiny improvements at scale like this, might do some napkin math on it later and see if I can come up with something in the right order of magnitude.

fsckboy 3y ago
firefox browser share is teeny tiny these days
- botanical 3y ago
  It's 200 million people, with 80% on Windows. https://data.firefox.com/dashboard/user-activity https://data.firefox.com/dashboard/user-activity https://data.firefox.com/dashboard/hardware https://data.firefox.com/dashboard/hardware
- tomrod 3y ago
  Teeny tiny multiplied by 7 Billion by 365 days per year by 24 hours per day by a fraction of a kW does add up.
  - beAbU 3y ago
    7B people are not watching youtube on Firefox 24/7 365 days a year.
    - mulmen 3y ago
      But at any given moment someone is.
    - tomrod 3y ago
      Correct. Some teeny tiny fraction of market share is. For the conceptual calculation, I refer you to my earlier comment.
  - 3y ago

Sakos 3y ago

I've seen some really weird performance behavior from Defender and I just keep it disabled on my desktop device now. I'm not surprised to see it affecting Firefox like this. Defender's dropped all the way to the bottom of the list in effectiveness anyway, so I don't feel it's a big loss.

nikanj 3y ago

The biggest surprise for me was Microsoft actually fixing it.

elygre 3y ago
Is that because you don't expect programmers in general to fix their bugs? Or do you think Microsoft in particular don't care about their products?
- nikanj 3y ago
  Because once a corporation grows larger than some singularity threshold, there seems to be a bug event horizon where all bug reports just disappear. Send a bug report to a five-person software company, their lead dev contacts you the same day and has a patched version ready to go in a week. Send a bug report to Microsoft / Citrix / Apple / etc, and you'll never hear back.
hoffs 3y ago
They said that Microsoft was already in progress of changing the offending code before Mozilla report.