7 ms·

I can sympathize with the need to have all required source code in the repository and not having to fetch a bunch of dependencies at build time. Thankfully, car

I can sympathize with the need to have all required source code in the repository and not having to fetch a bunch of dependencies at build time. Thankfully, cargo already offers a solution here: `cargo vendor` will download all the specified dependencies once into a local directory, which can then be checked into the source tree. This maintains cargo's dependency resolution/update checking/etc, but also allows for all dependency code to be kept alongside the kernel code and audited accordingly.

lodovic 4y ago
Requiring packages to build the kernel source feels like a circular dependency to me. The kernel should contain all of its own source code.
- nrabulinski 4y ago
  So now a kernel cannot depend on anything? What about all the tooling that’s required like python, perl and many others or the fact that you have to have a working machine with the kernel already on it to build the kernel?
  - yjftsjthsd-h 4y ago
    Obviously you have to bootstrap from something. A more interesting case is NetBSD, which can bootstrap from just about anything; take a NetBSD src checkout, run ./build.sh, and it will first use the local tooling to build its own dependencies, then use those to build the system. So in some OSs, yes, you can more or less vendor in the universe. But that only works because *BSDs are developed as full systems, not just a kernel; Linux probably can't (and arguably shouldn't) do that.
humanrebar 4y ago
Will there be a massive review process every time someone updates the vendored dependencies? Or will each dependency change be reviewed one release increment at a time? What happens if a dependency adds a system call something? Libraries intended for kernel-friendly use cases really need that scope to be an intentional goal.
- kibwen 4y ago
  Of course, they would certainly review any updates to vendored code. Just because most companies are too lazy to audit their dependencies doesn't mean the kernel needs to be. As for syscalls, Rust has a whole ecosystem of no_std crates for kernel and microcontroller development that already assume the lack of an OS. We use (and contribute to) such crates extensively in our product (which is already developed jointly with the Linux Foundation, though we're not working on the Linux kernel (well we are a little bit, but all of that work is still in C :P )) and can vouch for their quality.
  - jupp0r 4y ago
    > Just because most companies are too lazy to audit their dependencies They are not “too lazy”. It’s not economically viable for them to do so for various reasons.
    - geodel 4y ago
      Ah this makes total sense. Other day I read how some factories are dumping pollutant chemicals in rivers. But it turned out to be all fine as there was no wrong intentions it just was not economically viable for them to properly dispose waste.
      - notriddle 4y ago
        Being “not lazy” isn’t the same thing as being “fine.” What it means is that the problem is with the game, not the players.
      - jupp0r 4y ago
        Congratulations, you found the solution to the problem. Hold companies liable for the damages their sloppy security practices create. Bruce Schneier wrote an excellent article on the topic in 2003 [1]. [1] https://www.schneier.com/essays/archives/2003/11/liability_changes_ev.html https://www.schneier.com/essays/archives/2003/11/liability_c...
    - 4y ago
jerf 4y ago
I don't think it's even remotely likely the kernel will end up supporting Cargo in any form. As said in the article, "the world is changing". However, the way the world is changing is that more and more people are aware of supply-chain attacks. While I acknowledge that Cargo isn't npm, it is still the case that of all the software that can not afford to just sorta grab things from the internet, the Linux kernel is arguably #1, straight up. There is no chance that the kernel developers are ever going to accept "well, I wanted some async stuff so I grabbed a v0.5.23 of a package that I found appealing". Even if they pull some stuff in, it will be through a review process and it won't be through cargo in general. The argument against cargo or any equivalent being used in the kernel is stronger today than it was 10 years ago, and the first derivative is also positive. Probably the second one too, honestly. This isn't about kernel developers being old fogey sticks in the mud, this is about the kernel being such a high-assurance environment, the biggest, fattest target in the world, that things that make sense for most software packages don't make sense for it. And it has nothing to do with cargo specifically or Rust. It's just that the entire workflow afforded by cargo, or npm, or go modules, or the half-dozen Python package managers, or anything else resembling those things is simply not appropriate for the Linux kernel. The only way to make such a thing work would be to pin the versions so hard that you're effectively only using them as a downloader, not a package manager, and you might as well just have vendored code in the kernel repo anyhow.
- CuriousCosmic 4y ago
  > There is no chance that the kernel developers are ever going to accept "well, I wanted some async stuff so I grabbed a v0.5.23 of a package that I found appealing". Even if they pull some stuff in, it will be through a review process and it won't be through cargo in general. > The only way to make such a thing work would be to pin the versions so hard that you're effectively only using them as a downloader, not a package manager, and you might as well just have vendored code in the kernel repo anyhow. This is the exact thing the person you replied to said should be done. `cargo vendor` fetches the sources for the package @ the version stated and embeds them in the repo. After that all deps would be sourced from the repository itself. Nothing in the review process would have to change beyond adding a dependencies directory to the repo with a dedicated set of CODEOWNERS to handle reviewing patches with new dependencies.
akira2501 4y ago
I'm suspicious of someone who says they need to download and run /complex math/ libraries inside of a kernel module. I seriously wonder what they're planning on developing, and if a hybrid approach with most of that code being in user space wouldn't be a better idea here. I wonder if this is a real and considered need, or a knee-jerk response to try to put everything in rust and directly in ring 0 just for the sake of it. I see no reason to try to break down the "divide between kernel and user space" in this way, and I wonder what's actually driving it here. I wish this article went a little deeper into _what specifically_ these Rust users are trying to make modules _for_.

lodovic 4y ago

Requiring packages to build the kernel source feels like a circular dependency to me. The kernel should contain all of its own source code.

nrabulinski 4y ago
So now a kernel cannot depend on anything? What about all the tooling that’s required like python, perl and many others or the fact that you have to have a working machine with the kernel already on it to build the kernel?
- yjftsjthsd-h 4y ago
  Obviously you have to bootstrap from something. A more interesting case is NetBSD, which can bootstrap from just about anything; take a NetBSD src checkout, run ./build.sh, and it will first use the local tooling to build its own dependencies, then use those to build the system. So in some OSs, yes, you can more or less vendor in the universe. But that only works because *BSDs are developed as full systems, not just a kernel; Linux probably can't (and arguably shouldn't) do that.

humanrebar 4y ago

Will there be a massive review process every time someone updates the vendored dependencies? Or will each dependency change be reviewed one release increment at a time? What happens if a dependency adds a system call something? Libraries intended for kernel-friendly use cases really need that scope to be an intentional goal.

kibwen 4y ago
Of course, they would certainly review any updates to vendored code. Just because most companies are too lazy to audit their dependencies doesn't mean the kernel needs to be. As for syscalls, Rust has a whole ecosystem of no_std crates for kernel and microcontroller development that already assume the lack of an OS. We use (and contribute to) such crates extensively in our product (which is already developed jointly with the Linux Foundation, though we're not working on the Linux kernel (well we are a little bit, but all of that work is still in C :P )) and can vouch for their quality.
- jupp0r 4y ago
  > Just because most companies are too lazy to audit their dependencies They are not “too lazy”. It’s not economically viable for them to do so for various reasons.
  - geodel 4y ago
    Ah this makes total sense. Other day I read how some factories are dumping pollutant chemicals in rivers. But it turned out to be all fine as there was no wrong intentions it just was not economically viable for them to properly dispose waste.
    - notriddle 4y ago
      Being “not lazy” isn’t the same thing as being “fine.” What it means is that the problem is with the game, not the players.
    - jupp0r 4y ago
      Congratulations, you found the solution to the problem. Hold companies liable for the damages their sloppy security practices create. Bruce Schneier wrote an excellent article on the topic in 2003 [1]. [1] https://www.schneier.com/essays/archives/2003/11/liability_changes_ev.html https://www.schneier.com/essays/archives/2003/11/liability_c...
  - 4y ago

jerf 4y ago

I don't think it's even remotely likely the kernel will end up supporting Cargo in any form. As said in the article, "the world is changing". However, the way the world is changing is that more and more people are aware of supply-chain attacks. While I acknowledge that Cargo isn't npm, it is still the case that of all the software that can not afford to just sorta grab things from the internet, the Linux kernel is arguably #1, straight up. There is no chance that the kernel developers are ever going to accept "well, I wanted some async stuff so I grabbed a v0.5.23 of a package that I found appealing". Even if they pull some stuff in, it will be through a review process and it won't be through cargo in general. The argument against cargo or any equivalent being used in the kernel is stronger today than it was 10 years ago, and the first derivative is also positive. Probably the second one too, honestly. This isn't about kernel developers being old fogey sticks in the mud, this is about the kernel being such a high-assurance environment, the biggest, fattest target in the world, that things that make sense for most software packages don't make sense for it. And it has nothing to do with cargo specifically or Rust. It's just that the entire workflow afforded by cargo, or npm, or go modules, or the half-dozen Python package managers, or anything else resembling those things is simply not appropriate for the Linux kernel. The only way to make such a thing work would be to pin the versions so hard that you're effectively only using them as a downloader, not a package manager, and you might as well just have vendored code in the kernel repo anyhow.

CuriousCosmic 4y ago
> There is no chance that the kernel developers are ever going to accept "well, I wanted some async stuff so I grabbed a v0.5.23 of a package that I found appealing". Even if they pull some stuff in, it will be through a review process and it won't be through cargo in general. > The only way to make such a thing work would be to pin the versions so hard that you're effectively only using them as a downloader, not a package manager, and you might as well just have vendored code in the kernel repo anyhow. This is the exact thing the person you replied to said should be done. `cargo vendor` fetches the sources for the package @ the version stated and embeds them in the repo. After that all deps would be sourced from the repository itself. Nothing in the review process would have to change beyond adding a dependencies directory to the repo with a dedicated set of CODEOWNERS to handle reviewing patches with new dependencies.

akira2501 4y ago

I'm suspicious of someone who says they need to download and run /complex math/ libraries inside of a kernel module. I seriously wonder what they're planning on developing, and if a hybrid approach with most of that code being in user space wouldn't be a better idea here. I wonder if this is a real and considered need, or a knee-jerk response to try to put everything in rust and directly in ring 0 just for the sake of it. I see no reason to try to break down the "divide between kernel and user space" in this way, and I wonder what's actually driving it here. I wish this article went a little deeper into _what specifically_ these Rust users are trying to make modules _for_.