5 ms·

You shouldn't trust someone who misleads with: "Now, 70% of the CVEs originating at Microsoft are memory safety issues, Levick said. “There is no real trend it’

You shouldn't trust someone who misleads with: "Now, 70% of the CVEs originating at Microsoft are memory safety issues, Levick said. “There is no real trend it’s just staying the exact same,” he said. “Despite massive efforts on our part to fix this issue it still seems to be a common thing.”" There is a trend, and the trend has been a MASSIVE reduction in memory errors. The percentage of security issues that are memory errors says nothing of the totals, which is what matters. Microsoft could go through 2021 with only one CVE and we'd see articles like this saying "100% of security vulnerabilities are caused by memory errors, we must switch to Rust!"

snazz 6y ago
I agree. While Rust is clearly a step forward in programming languages (and we need a step forward), modern tools built to deal with the issues common in C and C++ like coverage-guided fuzzers, ASan, Valgrind, etc. have made huge improvements in finding memory safety vulnerabilities and other issues. Mitigations have also improved, making it more difficult to exploit memory safety issues. While I think that unmanaged (i.e. no runtime or interpreter) memory-safe programming languages are a good idea, they're not magic and we shouldn't ignore the other major security tooling improvements recently.
- saagarjha 6y ago
  I should note that fuzzers/address sanitizer/Valgrind are very good tools, but they are not perfect. Projects that use them heavily still see issues due to memory-safety bugs. (And yes, memory-safe programming languages are not a magic bullet to fix all bugs; they just help with memory safety…)
zozbot234 6y ago
> There is a trend, and the trend has been a MASSIVE reduction in memory errors. The percentage of security issues that are memory errors says nothing of the totals, which is what matters. Except that CVE's seem to be increasing over time across the industry, not falling. There is no overall reduction in the numbers.
- userbinator 6y ago
  Indeed, software quality is in decline across the industry. I think this is more of a "make an idiot-proof system and the world will make a better idiot" type of situation.
  - rightbyte 6y ago
    How about designing the web around accepting arbitrary code from remote computers and executing it the browser VM.
rattray 6y ago
> the trend has been a MASSIVE reduction in memory errors What is the evidence for this? I'm not familiar.
puzzledobserver 6y ago
> There is a trend, and the trend has been a MASSIVE reduction in memory errors. Citation?
- anmsl 6y ago
  How about joining a well maintained OSS C or C++ project? It is obvious that Asan/Valgrind etc. massively reduce issues. Also, why do people always accept the original submission as truth but comments have to be verified? At least in Rust agitprop.
  - joshuamorton 6y ago
    Because Microsoft (and Google, who shares this opinion on memory safety in c++) are the groups that develop asan and similar tools, and who have some of the best practices around code review, testing, and CI. Yet they still say rust is better in the long run. Why so you think that is?
    - alTzxY 6y ago
      Valgrind was not developed by Google and probably accounts for the main reduction. Asan has seen a way more recent takeup in OSS. Microsoft and Google are not monoliths, and this particular submission is from a "cloud developer advocate", so I attach exactly zero importance to it as far as C/C++ are concerned.
      - joshuamorton 6y ago
        > , so I attach exactly zero importance to it as far as C/C++ are concerned. He's repeating statistics that members of the C++ committee will agree with, so I'm not sure what's controversial. > Valgrind was not developed by Google and probably accounts for the main reduction. Asan has seen a way more recent takeup in OSS. You miss the point. Despite these things, and despite aggressive compiler flags and everything else, the majority of bugs are memory safety issues. Whether you look at windows, chrome, or the Linux Kernal itself (KASAN). That seems fairly conclusive. And yet here you are arguing, what exactly? That the person making the statement isn't technical enough for you, so it's all lies? As a bit of an aside, > Valgrind was not developed by Google and probably accounts for the main reduction I never said it was. However one of the most active maintainers works at Mozilla, who's opinion on the safety of C++ is also probably in line with Google and Microsoft here, given their relationship with Rust.
deleted 6y ago
[deleted]
dtolnay 6y ago
Here is the data from Microsoft Security Response Center on this [1]. Slide 5 shows double as many CVEs as 4-5 years prior, triple as many as 6-8 years prior. Further, slide 10 shows 70% of CVEs are memory safety and that ratio has been constant since 2006. It follows that the absolute number of CVEs which are memory safety is double 4-5 years prior and triple 6-8 years prior, which is inconsistent with a massive reduction. The magnitudes here are in the many hundreds of CVEs per year from Microsoft (and growing), not "one CVE in 2021". 70% there is not a negligible number. [1] https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_02_BlueHatIL/2019_01%20-%20BlueHatIL%20-%20Trends%2C%20challenge%2C%20and%20shifts%20in%20software%20vulnerability%20mitigation.pdf https://github.com/microsoft/MSRC-Security-Research/blob/mas...
- fourcommas 6y ago
  Now divide the number of CVEs in each year by the LoC maintained. You're again looking at completely the wrong number. Line-for-line, C/C++ written today at MSFT/Goog has less memory errors than 10 years ago, and even less exploitation of memory errors. Anyone who lived through the rise and fall of Internet Explorer intuitively knows this.
  - dtolnay 6y ago
    Dividing by the LoC is a mistaken way of looking at this data. Regardless of the amount of code in a browser or in Windows, an attacker may only need ONE exploitable bug to cause mischief. If the amount of code in Windows grows by a factor X from year to year, the CVEs per LoC better be shrinking by at least factor X (this is where Rust comes in) or else the system is getting less secure. Thus the absolute number is the relevant metric, and indeed is the number reported by Microsoft Security Response Center.
    - fourcommas 6y ago
      > the CVEs per LoC better be shrinking by at least factor X (this is where Rust comes in) or else the system is getting less secure Do you really think windows 95 was more secure than win 10? Or that IE6 was more secure than the latest IE? The newer versions are way more secure, it's not even close. Your data is giving you incorrect conclusions, because you're combining and cutting the data in ways that don't make sense.

snazz 6y ago

I agree. While Rust is clearly a step forward in programming languages (and we need a step forward), modern tools built to deal with the issues common in C and C++ like coverage-guided fuzzers, ASan, Valgrind, etc. have made huge improvements in finding memory safety vulnerabilities and other issues. Mitigations have also improved, making it more difficult to exploit memory safety issues. While I think that unmanaged (i.e. no runtime or interpreter) memory-safe programming languages are a good idea, they're not magic and we shouldn't ignore the other major security tooling improvements recently.

saagarjha 6y ago
I should note that fuzzers/address sanitizer/Valgrind are very good tools, but they are not perfect. Projects that use them heavily still see issues due to memory-safety bugs. (And yes, memory-safe programming languages are not a magic bullet to fix all bugs; they just help with memory safety…)

zozbot234 6y ago

> There is a trend, and the trend has been a MASSIVE reduction in memory errors. The percentage of security issues that are memory errors says nothing of the totals, which is what matters. Except that CVE's seem to be increasing over time across the industry, not falling. There is no overall reduction in the numbers.

userbinator 6y ago
Indeed, software quality is in decline across the industry. I think this is more of a "make an idiot-proof system and the world will make a better idiot" type of situation.
- rightbyte 6y ago
  How about designing the web around accepting arbitrary code from remote computers and executing it the browser VM.

rattray 6y ago

> the trend has been a MASSIVE reduction in memory errors What is the evidence for this? I'm not familiar.

puzzledobserver 6y ago

> There is a trend, and the trend has been a MASSIVE reduction in memory errors. Citation?

anmsl 6y ago
How about joining a well maintained OSS C or C++ project? It is obvious that Asan/Valgrind etc. massively reduce issues. Also, why do people always accept the original submission as truth but comments have to be verified? At least in Rust agitprop.
- joshuamorton 6y ago
  Because Microsoft (and Google, who shares this opinion on memory safety in c++) are the groups that develop asan and similar tools, and who have some of the best practices around code review, testing, and CI. Yet they still say rust is better in the long run. Why so you think that is?
  - alTzxY 6y ago
    Valgrind was not developed by Google and probably accounts for the main reduction. Asan has seen a way more recent takeup in OSS. Microsoft and Google are not monoliths, and this particular submission is from a "cloud developer advocate", so I attach exactly zero importance to it as far as C/C++ are concerned.
    - joshuamorton 6y ago
      > , so I attach exactly zero importance to it as far as C/C++ are concerned. He's repeating statistics that members of the C++ committee will agree with, so I'm not sure what's controversial. > Valgrind was not developed by Google and probably accounts for the main reduction. Asan has seen a way more recent takeup in OSS. You miss the point. Despite these things, and despite aggressive compiler flags and everything else, the majority of bugs are memory safety issues. Whether you look at windows, chrome, or the Linux Kernal itself (KASAN). That seems fairly conclusive. And yet here you are arguing, what exactly? That the person making the statement isn't technical enough for you, so it's all lies? As a bit of an aside, > Valgrind was not developed by Google and probably accounts for the main reduction I never said it was. However one of the most active maintainers works at Mozilla, who's opinion on the safety of C++ is also probably in line with Google and Microsoft here, given their relationship with Rust.

deleted 6y ago

[deleted]

dtolnay 6y ago

Here is the data from Microsoft Security Response Center on this [1]. Slide 5 shows double as many CVEs as 4-5 years prior, triple as many as 6-8 years prior. Further, slide 10 shows 70% of CVEs are memory safety and that ratio has been constant since 2006. It follows that the absolute number of CVEs which are memory safety is double 4-5 years prior and triple 6-8 years prior, which is inconsistent with a massive reduction. The magnitudes here are in the many hundreds of CVEs per year from Microsoft (and growing), not "one CVE in 2021". 70% there is not a negligible number. [1] https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_02_BlueHatIL/2019_01%20-%20BlueHatIL%20-%20Trends%2C%20challenge%2C%20and%20shifts%20in%20software%20vulnerability%20mitigation.pdf https://github.com/microsoft/MSRC-Security-Research/blob/mas...

fourcommas 6y ago
Now divide the number of CVEs in each year by the LoC maintained. You're again looking at completely the wrong number. Line-for-line, C/C++ written today at MSFT/Goog has less memory errors than 10 years ago, and even less exploitation of memory errors. Anyone who lived through the rise and fall of Internet Explorer intuitively knows this.
- dtolnay 6y ago
  Dividing by the LoC is a mistaken way of looking at this data. Regardless of the amount of code in a browser or in Windows, an attacker may only need ONE exploitable bug to cause mischief. If the amount of code in Windows grows by a factor X from year to year, the CVEs per LoC better be shrinking by at least factor X (this is where Rust comes in) or else the system is getting less secure. Thus the absolute number is the relevant metric, and indeed is the number reported by Microsoft Security Response Center.
  - fourcommas 6y ago
    > the CVEs per LoC better be shrinking by at least factor X (this is where Rust comes in) or else the system is getting less secure Do you really think windows 95 was more secure than win 10? Or that IE6 was more secure than the latest IE? The newer versions are way more secure, it's not even close. Your data is giving you incorrect conclusions, because you're combining and cutting the data in ways that don't make sense.