5 ms·

It doesn't just download random things. Cargo generates a Cargo.lock file with checksums and will make sure that those checksums match when building later on. I

by thecrm 4y ago

It doesn't just download random things. Cargo generates a Cargo.lock file with checksums and will make sure that those checksums match when building later on. It's about as safe as vendoring all dependencies while being far easier to work with (though tools like cargo-vendor do exist, of course). Edit: for things like the kernel, vendoring dependencies is still probably not a bad idea, of course

humanrebar 4y ago
What prevents a given URL from disappearing? Does that just break a particular source version of the Linux kernel? What happens when a given dependency adds new kernel-inappropriate features? Are kernel devs going to act like distro maintainers and decide between forking, maintaining patch sets, etc.?
- roca 4y ago
  All crate sources are stored in the crates.io package archive, which never deletes packages. A dependency veering off in a direction you don't like is one of the risks of using someone else's code instead of writing it yourself. Cargo makes it easy to use forked dependencies, and forking a dependency is almost always less work than if you'd never used it and written the code yourself from the beginning. (And to be clear this is only a problem for future evolution; a crate author cannot remove or modify an already-published version of their crate.)
  - marginalia_nu 4y ago
    This is still fairly short sighted. Websites shut down, large websites with big storage demands are especially vulnerable to attrition. Who wants to pay the mounting bill for keeping decades of revisions of historical rust packages online? I can grab the kernel sources from 1997 and build them today. Will I be able to build rust code from 2022 in 2047, because the 1997 kernel will still build at that date.
    - Gwypaas 4y ago
      I would imagine the kernel would use cargo vendor, or similar, to lock all dependencies into their chosen source control and quality requirements. https://doc.rust-lang.org/cargo/commands/cargo-vendor.html https://doc.rust-lang.org/cargo/commands/cargo-vendor.html
    - varajelle 4y ago
      > I can grab the kernel sources from 1997 and build them today. Can you? Do they still compiler with current compiler? You'll probably need to find a compiler of that time... And also all the interpreter for all the build scripts. Was that using bash or some old Perl? Maybe something more esoteric like m4 or tcl? The point is that it always had many external dependencies to bootstrap. And adding one is not such a big deal, it just add another thing to archive among the many other things. The crates.io archive is probably not even that big.
      - marginalia_nu 4y ago
        I'm not sure why that would be a problem given most of these languages and standards are older than the Linux kernel. The thing about mature technology is specifically that it doesn't have breaking changes every couple of months. This is the way it used to be for a fairly long time. But even if it has broken, I can just download an old linux distro. They effectively form a cohesive snapshot of the state of the toolchain whenever they were assembled. Slackware 3.1 from 1996 might be appropriate.
        estebank 4y ago
        > But even if it has broken, I can just download an old linux distro. They effectively form a cohesive snapshot of the state of the toolchain whenever they were assembled. Slackware 3.1 from 1996 might be appropriate. You will also need era-appropriate hardware to get that software to install.
    - pdw 4y ago
      "I can grab the kernel sources from 1997 and build them today." Where would you be grabbing it from? ...From a website? "Websites shut down, large websites with big storage demands are especially vulnerable to attrition. Who wants to pay the mounting bill for keeping decades of revisions of historical Linux kernels online?"
  - humanrebar 4y ago
    "Never" is a long time, just saying. It'll be impossible to beat the "availability" guarantees of a local mirror (like a thumb drive) of a kernel source tarball. What happens when a crate version has to be removed due to a critical CVE or court order (IP Law violation, perhaps)? There may come a day where crates.io becomes torn between not breaking Linux source and not hosting actively bad source code. Note that some of those concerns do apply to vendoring source as well, but the additional download step also removes options that the kernel maintainers have as long as they ship all the source for the kernel in one tarball. Like more control over the timing of inevitable decisions.
    - 3836293648 4y ago
      Does crates.io actively host any code? I thought it was all just readmes and links to github and docs.rs
      - conradludgate 4y ago
        They do host it. It's registry info is mirrored on github though
    - mcherm 4y ago
      What happens today when a kernel module has to be removed due to a critical CVE or court order? That's not just a rhetorical flourish, I'm actually curious what the answer is. As far as I know, (1) it almost never happens and (2) when it does, the change is made in upstream repos and as a practical matter, everyone downloads those changes and their up-to-date local copies lose that code.
      - humanrebar 4y ago
        Fixing it in the future isn't the point. Breaking previous releases is. The previous tarballs still work and contain the relevant code. Your build wouldn't rely on hosts complying with court orders in countries you might not live in. If the code isn't vendored, just referenced with URLs, the old tarballs stop working.
- sanderjd 4y ago
  To the first question, obviously the sources of dependencies would be brought into the tree. This is easy and there's no reason I'm aware of not to do it for something like the Linux kernel. To the second set of questions, how is this any different than any other dependency the kernel has? If the answer is "the kernel has no dependencies" then yeah, I'm very sympathetic to the argument that bringing in rust libraries is not a good reason to start having dependencies when none previously existed at all, but is that the case?
CraigJPerry 4y ago
The lock file is created but is not used by default. You must specify --locked to get that behaviour
- heftig 4y ago
  No, it is. Even without `--locked`, the Cargo.lock file is only updated when it no longer fulfills the Cargo.toml because the latter was edited (and then only making the minimal changes necessary), or explicitly using `cargo update`.
  - CraigJPerry 4y ago
    I don’t follow - I’m saying the cargo.lock isn’t read unless you specify —locked - I’m not talking about when it gets refreshed?
    - heftig 4y ago
      Yes, it's always read. If the file didn't require updating, a build with and without `--locked` will be identical. If it did require updating, `--locked` will make cargo exit with an error.
      - CraigJPerry 4y ago
        Gotcha, makes sense
- duckerude 4y ago
  That's true when running `cargo install` to install an application directly from crates.io, but not when running `cargo build` in an already checked-out repository.
  - CraigJPerry 4y ago
    I might be misreading this on an iphone screen but as i follow the logic here: https://github.com/rust-lang/cargo/blob/a77ed9ba87bfeaf3c2758a5b291876c65f734c39/src/cargo/ops/cargo_compile.rs#L385 https://github.com/rust-lang/cargo/blob/a77ed9ba87bfeaf3c275... A cargo build ends up there calling into the resolver’s resolve_ws_with_opts() which would refresh the lockfile. Not resolve_with_previous() which would use the lock file as-is. The only reason this sticks in my mind is i ran into an issue building bat after i made some changes, i obviously assumed it was my changes so went through the process of debugging and backing out my changes until finally i was back to a virgin branch and still failing - passing —frozen —locked fixed it.
deleted 4y ago
[deleted]
goodpoint 4y ago
> It doesn't just download random things. That's exactly what it does. The developer is not really expected to thoroughly review the codebase of every dependency. Just like javascript, all sort of supply chain attacks are made possible. A single malicious library can sneak into large ecosystems easily.
yw3410 4y ago
You're forgetting about custom build scripts. Thankfully most of the core ones have moved off cloning dependencies for ffi purposes (think cloning an alsa-lib version for ffi), but it used to be super common.

by thecrm 4y ago

humanrebar 4y ago

What prevents a given URL from disappearing? Does that just break a particular source version of the Linux kernel? What happens when a given dependency adds new kernel-inappropriate features? Are kernel devs going to act like distro maintainers and decide between forking, maintaining patch sets, etc.?

roca 4y ago
All crate sources are stored in the crates.io package archive, which never deletes packages. A dependency veering off in a direction you don't like is one of the risks of using someone else's code instead of writing it yourself. Cargo makes it easy to use forked dependencies, and forking a dependency is almost always less work than if you'd never used it and written the code yourself from the beginning. (And to be clear this is only a problem for future evolution; a crate author cannot remove or modify an already-published version of their crate.)
- marginalia_nu 4y ago
  This is still fairly short sighted. Websites shut down, large websites with big storage demands are especially vulnerable to attrition. Who wants to pay the mounting bill for keeping decades of revisions of historical rust packages online? I can grab the kernel sources from 1997 and build them today. Will I be able to build rust code from 2022 in 2047, because the 1997 kernel will still build at that date.
  - Gwypaas 4y ago
    I would imagine the kernel would use cargo vendor, or similar, to lock all dependencies into their chosen source control and quality requirements. https://doc.rust-lang.org/cargo/commands/cargo-vendor.html https://doc.rust-lang.org/cargo/commands/cargo-vendor.html
  - varajelle 4y ago
    > I can grab the kernel sources from 1997 and build them today. Can you? Do they still compiler with current compiler? You'll probably need to find a compiler of that time... And also all the interpreter for all the build scripts. Was that using bash or some old Perl? Maybe something more esoteric like m4 or tcl? The point is that it always had many external dependencies to bootstrap. And adding one is not such a big deal, it just add another thing to archive among the many other things. The crates.io archive is probably not even that big.
    - marginalia_nu 4y ago
      I'm not sure why that would be a problem given most of these languages and standards are older than the Linux kernel. The thing about mature technology is specifically that it doesn't have breaking changes every couple of months. This is the way it used to be for a fairly long time. But even if it has broken, I can just download an old linux distro. They effectively form a cohesive snapshot of the state of the toolchain whenever they were assembled. Slackware 3.1 from 1996 might be appropriate.
      - estebank 4y ago
        > But even if it has broken, I can just download an old linux distro. They effectively form a cohesive snapshot of the state of the toolchain whenever they were assembled. Slackware 3.1 from 1996 might be appropriate. You will also need era-appropriate hardware to get that software to install.
  - pdw 4y ago
    "I can grab the kernel sources from 1997 and build them today." Where would you be grabbing it from? ...From a website? "Websites shut down, large websites with big storage demands are especially vulnerable to attrition. Who wants to pay the mounting bill for keeping decades of revisions of historical Linux kernels online?"
- humanrebar 4y ago
  "Never" is a long time, just saying. It'll be impossible to beat the "availability" guarantees of a local mirror (like a thumb drive) of a kernel source tarball. What happens when a crate version has to be removed due to a critical CVE or court order (IP Law violation, perhaps)? There may come a day where crates.io becomes torn between not breaking Linux source and not hosting actively bad source code. Note that some of those concerns do apply to vendoring source as well, but the additional download step also removes options that the kernel maintainers have as long as they ship all the source for the kernel in one tarball. Like more control over the timing of inevitable decisions.
  - 3836293648 4y ago
    Does crates.io actively host any code? I thought it was all just readmes and links to github and docs.rs
    - conradludgate 4y ago
      They do host it. It's registry info is mirrored on github though
  - mcherm 4y ago
    What happens today when a kernel module has to be removed due to a critical CVE or court order? That's not just a rhetorical flourish, I'm actually curious what the answer is. As far as I know, (1) it almost never happens and (2) when it does, the change is made in upstream repos and as a practical matter, everyone downloads those changes and their up-to-date local copies lose that code.
    - humanrebar 4y ago
      Fixing it in the future isn't the point. Breaking previous releases is. The previous tarballs still work and contain the relevant code. Your build wouldn't rely on hosts complying with court orders in countries you might not live in. If the code isn't vendored, just referenced with URLs, the old tarballs stop working.
sanderjd 4y ago
To the first question, obviously the sources of dependencies would be brought into the tree. This is easy and there's no reason I'm aware of not to do it for something like the Linux kernel. To the second set of questions, how is this any different than any other dependency the kernel has? If the answer is "the kernel has no dependencies" then yeah, I'm very sympathetic to the argument that bringing in rust libraries is not a good reason to start having dependencies when none previously existed at all, but is that the case?

CraigJPerry 4y ago

The lock file is created but is not used by default. You must specify --locked to get that behaviour

heftig 4y ago
No, it is. Even without `--locked`, the Cargo.lock file is only updated when it no longer fulfills the Cargo.toml because the latter was edited (and then only making the minimal changes necessary), or explicitly using `cargo update`.
- CraigJPerry 4y ago
  I don’t follow - I’m saying the cargo.lock isn’t read unless you specify —locked - I’m not talking about when it gets refreshed?
  - heftig 4y ago
    Yes, it's always read. If the file didn't require updating, a build with and without `--locked` will be identical. If it did require updating, `--locked` will make cargo exit with an error.
    - CraigJPerry 4y ago
      Gotcha, makes sense
duckerude 4y ago
That's true when running `cargo install` to install an application directly from crates.io, but not when running `cargo build` in an already checked-out repository.
- CraigJPerry 4y ago
  I might be misreading this on an iphone screen but as i follow the logic here: https://github.com/rust-lang/cargo/blob/a77ed9ba87bfeaf3c2758a5b291876c65f734c39/src/cargo/ops/cargo_compile.rs#L385 https://github.com/rust-lang/cargo/blob/a77ed9ba87bfeaf3c275... A cargo build ends up there calling into the resolver’s resolve_ws_with_opts() which would refresh the lockfile. Not resolve_with_previous() which would use the lock file as-is. The only reason this sticks in my mind is i ran into an issue building bat after i made some changes, i obviously assumed it was my changes so went through the process of debugging and backing out my changes until finally i was back to a virgin branch and still failing - passing —frozen —locked fixed it.

deleted 4y ago

[deleted]

goodpoint 4y ago

> It doesn't just download random things. That's exactly what it does. The developer is not really expected to thoroughly review the codebase of every dependency. Just like javascript, all sort of supply chain attacks are made possible. A single malicious library can sneak into large ecosystems easily.

yw3410 4y ago

You're forgetting about custom build scripts. Thankfully most of the core ones have moved off cloning dependencies for ffi purposes (think cloning an alsa-lib version for ffi), but it used to be super common.