9 ms·

Is the solution against such attacks in the future only to scrutinize more, or are there other reasonable options in terms of hardening?

JanisErdmanis 2y ago
The lesson here seems to not depend on tools written in languages that have complex, obscure build systems and no one is either able or interested to read. Using tools rewritten in Rust, Go or any other languege which resolves dependencies within project seems the only way to do hardening here.
- stbenjam 2y ago
  I agree there's safer languages than C, but nobody reads the 50,000 lines changed when you update the vendoring in a random golang project. It would be easy to introduce something there that nobody notices too.
  - JanisErdmanis 2y ago
    It is generally harder to introduce vulnerabilities in readable language even more when it is memory safe. Sure life is not perfect and bad actors would have found a ways to inject vulnerabilities also in Rust, Go codebase. The benefit of modern languages is that there is one way to build things and the source code is the only thing that needs to be auditted.
    - mstef 2y ago
      this backdoor had nothing at all to do with memory safety.
- Lichtso 2y ago
  Once somebody actually does this people are gonna complain the same as always: "The sole purpose of your project is to rewrite perfectly fine stuff in Rust for the sake of it" or something along these lines.
- ok123456 2y ago
  Wouldn't a supply chain attack like this be much worse with Rust and Cargo because of the fact it's not just a single dynamic library that needs to be reinstalled system-wise, but, instead, every binary would require a new release?
  - gpm 2y ago
    It would mean rebuilding more packages. I don't think that's meaningfully "much worse", package mangers are perfectly capable of rebuilding the world and the end-user fix is the same "pacman -Syu"/"apt-get update && apt-get upgrade"/... On the flip side the elegant/readable build system means that the place this exploit was hidden wouldn't exist. Though I wouldn't confidently say that 'no hiding places exist' (especially with the parts of the ecosystem that wrap dependencies in other languages).
    - ok123456 2y ago
      It's much worse because it requires repackaging every affected system package instead of a single library. Knowing which packages are affected is difficult because that information isn't exposed to the larger system package manager. After all, it's all managed by the build system.
      - packetlost 2y ago
        In the era of modern CI and build infrastructure, I don't really think that's materially an issue.
        ok123456 2y ago
        Those CI and build infrastructures rely on the Debian and RedHat being able to build system packages. How would an automated CI or build infrastructure stop this attack? It was stopped because the competent package maintainer noticed a performance regression. In this case, this imagined build system would have to track every rust library used in every package to know which packages to perform an emergency release for.
- klysm 2y ago
  People are going to be upset with this perspective but I completely agree. The whole autoconf set of tools is a complete disaster.
- arp242 2y ago
  You don't need a complex obscure build system for most C code. There's a lot of historical baggage here, but many projects (including xz, I suspect) can get away with a fairly straight-forward Makefile. Double so when using some GNU make extensions.
  - buserror 2y ago
    Thanks for that post, I wish people stopped pushing ever so more complicated build systems, opaque, non-backward compatible between their own versions when a 2 pages Makefile would work just fine, and still work in 20 years time.
- bonzini 2y ago
  Rust is the worst in terms of build system transparency. Ever heard of build.rs? You can hide backdoors in any crate, or in any crate's build.rs, or the same recursively.
  - timschmidt 2y ago
    Most build systems are turing-complete. Rust, at least, drastically reduces the need for custom build scripts (most of my projects have empty build.rs files or lack one entirely), and build.rs being in the same language as the rest of the codebase aids transparency immensely.
    - bonzini 2y ago
      That doesn't make build.rs any less of a juicy target for a supply chain attack. Arbitrary code downloaded from the internet and run at build time? That's a nightmare scenario for auditing, much worse than anything Autotools or CMake can offer.
      - timschmidt 2y ago
        You're not wrong about arbitrary code execution. It's just that your statement applies to most of the packages on any linux distribution, Autotools and Cmake included, regardless of language. Many moreso than Rust due to the aforementioned features of Cargo and build.rs not requiring me to be an expert in a second language just to audit it.
        bonzini 2y ago
        Packages in a Linux distro are not built on my machine, they are built by the distro in a sandbox. Every time I type "cargo build" I am potentially running arbitrary code downloaded from the internet. Every time I type "make" in an Autotools program only my code runs. > not requiring me to be an expert in another language just to audit it. Do you do that every time your Cargo.lock changes?
- msm_ 2y ago
  Is this really the lesson here? We are talking about a maintainer here, who had access to signing keys and a full access to the repository. Deb packages which were distributed are also different than the source code. Do you honestly believe that the (arguably awful) autotools syntax is the single root cause of this mess, Rust will save us from everything, and this is what we should take away from this situation?
- delfinom 2y ago
  I call bullshit. The fundamental problem here was a violation of chain of trust. Open source is only about the source being open. But if users are just downloading blobs with prebuilt binaries or even _pre-generated scripts_ that aren't in the original source, there is nothing a less-obscure build system will save you from as you are putting your entire security on the chain of trust being maintained.
joshhansen 2y ago
Am I crazy thinking libraries shouldn't be able to provide _other libraries'_ symbols without the other libraries' "permission"? What am I missing?

JanisErdmanis 2y ago

The lesson here seems to not depend on tools written in languages that have complex, obscure build systems and no one is either able or interested to read. Using tools rewritten in Rust, Go or any other languege which resolves dependencies within project seems the only way to do hardening here.

stbenjam 2y ago
I agree there's safer languages than C, but nobody reads the 50,000 lines changed when you update the vendoring in a random golang project. It would be easy to introduce something there that nobody notices too.
- JanisErdmanis 2y ago
  It is generally harder to introduce vulnerabilities in readable language even more when it is memory safe. Sure life is not perfect and bad actors would have found a ways to inject vulnerabilities also in Rust, Go codebase. The benefit of modern languages is that there is one way to build things and the source code is the only thing that needs to be auditted.
  - mstef 2y ago
    this backdoor had nothing at all to do with memory safety.
Lichtso 2y ago
Once somebody actually does this people are gonna complain the same as always: "The sole purpose of your project is to rewrite perfectly fine stuff in Rust for the sake of it" or something along these lines.
ok123456 2y ago
Wouldn't a supply chain attack like this be much worse with Rust and Cargo because of the fact it's not just a single dynamic library that needs to be reinstalled system-wise, but, instead, every binary would require a new release?
- gpm 2y ago
  It would mean rebuilding more packages. I don't think that's meaningfully "much worse", package mangers are perfectly capable of rebuilding the world and the end-user fix is the same "pacman -Syu"/"apt-get update && apt-get upgrade"/... On the flip side the elegant/readable build system means that the place this exploit was hidden wouldn't exist. Though I wouldn't confidently say that 'no hiding places exist' (especially with the parts of the ecosystem that wrap dependencies in other languages).
  - ok123456 2y ago
    It's much worse because it requires repackaging every affected system package instead of a single library. Knowing which packages are affected is difficult because that information isn't exposed to the larger system package manager. After all, it's all managed by the build system.
    - packetlost 2y ago
      In the era of modern CI and build infrastructure, I don't really think that's materially an issue.
      - ok123456 2y ago
        Those CI and build infrastructures rely on the Debian and RedHat being able to build system packages. How would an automated CI or build infrastructure stop this attack? It was stopped because the competent package maintainer noticed a performance regression. In this case, this imagined build system would have to track every rust library used in every package to know which packages to perform an emergency release for.
klysm 2y ago
People are going to be upset with this perspective but I completely agree. The whole autoconf set of tools is a complete disaster.
arp242 2y ago
You don't need a complex obscure build system for most C code. There's a lot of historical baggage here, but many projects (including xz, I suspect) can get away with a fairly straight-forward Makefile. Double so when using some GNU make extensions.
- buserror 2y ago
  Thanks for that post, I wish people stopped pushing ever so more complicated build systems, opaque, non-backward compatible between their own versions when a 2 pages Makefile would work just fine, and still work in 20 years time.
bonzini 2y ago
Rust is the worst in terms of build system transparency. Ever heard of build.rs? You can hide backdoors in any crate, or in any crate's build.rs, or the same recursively.
- timschmidt 2y ago
  Most build systems are turing-complete. Rust, at least, drastically reduces the need for custom build scripts (most of my projects have empty build.rs files or lack one entirely), and build.rs being in the same language as the rest of the codebase aids transparency immensely.
  - bonzini 2y ago
    That doesn't make build.rs any less of a juicy target for a supply chain attack. Arbitrary code downloaded from the internet and run at build time? That's a nightmare scenario for auditing, much worse than anything Autotools or CMake can offer.
    - timschmidt 2y ago
      You're not wrong about arbitrary code execution. It's just that your statement applies to most of the packages on any linux distribution, Autotools and Cmake included, regardless of language. Many moreso than Rust due to the aforementioned features of Cargo and build.rs not requiring me to be an expert in a second language just to audit it.
      - bonzini 2y ago
        Packages in a Linux distro are not built on my machine, they are built by the distro in a sandbox. Every time I type "cargo build" I am potentially running arbitrary code downloaded from the internet. Every time I type "make" in an Autotools program only my code runs. > not requiring me to be an expert in another language just to audit it. Do you do that every time your Cargo.lock changes?
msm_ 2y ago
Is this really the lesson here? We are talking about a maintainer here, who had access to signing keys and a full access to the repository. Deb packages which were distributed are also different than the source code. Do you honestly believe that the (arguably awful) autotools syntax is the single root cause of this mess, Rust will save us from everything, and this is what we should take away from this situation?
delfinom 2y ago
I call bullshit. The fundamental problem here was a violation of chain of trust. Open source is only about the source being open. But if users are just downloading blobs with prebuilt binaries or even _pre-generated scripts_ that aren't in the original source, there is nothing a less-obscure build system will save you from as you are putting your entire security on the chain of trust being maintained.

joshhansen 2y ago

Am I crazy thinking libraries shouldn't be able to provide _other libraries'_ symbols without the other libraries' "permission"? What am I missing?