5 ms·

This is pretty dumb on Debian's part. First of all I don't understand why they insist crate dependencies must be pulled from their repository. They are just sou

by yshui 2y ago

This is pretty dumb on Debian's part. First of all I don't understand why they insist crate dependencies must be pulled from their repository. They are just source code, not built binary. AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that. Arch does this, Gentoo does this, NixOS does this, why does Debian has to be different? Secondly, even if they have to use crates from their repository, I don't understand what's so hard to just have multiple versions of the same crate? That will solve the problem too. This is just all-around weird what Debian is doing. (Full disclosure, I am the one who introduced the first piece of Rust code into bcachefs-tools)

wojciii 2y ago
Perhaps they want to have builds they can reproduce years later? When libraries are removed you have to update your build. This will not be the case if you use your own infrastructure (crates.io mirror).
acka 2y ago
Debian has a Social Contract[1] as well as guidelines {the DFSG}[2] regarding the commitment to only distribute free and open source software that all package maintainers must adhere to. This means that package maintainers must check the licenses of source code and documentation files, clear up any ambiguities by talking to upstream, and (as a last resort) even excise code and/or documentation from Debian's copy of the codebase if it doesn't meet the requirements of the DFSG. In practise, this means that Debian has to make its own copy of the source code available from a Debian-controlled repository, to ensure that no (accidental or otherwise) change to an upstream source archive can cause non-DFSG compliant Debian source or binary packages to be distributed. [1] https://www.debian.org/social_contract https://www.debian.org/social_contract [2] https://wiki.debian.org/DebianFreeSoftwareGuidelines https://wiki.debian.org/DebianFreeSoftwareGuidelines
- koverstreet 2y ago
  That justification doesn't work here - I provide vendored tarballs, so the source code availability argument is moot.
  - righthand 2y ago
    But it’s not guaranteed. The Debian way provides a method of allocating responsibility. So if anything does go wrong they can point to a responsible party, the package maintainer. By providing tarball source you’re trying to placate responsibility of some code. You could build those tarballs on a different machine/different OS and any issues wouldn’t technically be your problem because “it’s just deps”.
Palomides 2y ago
debian is substantially older than all of those distros, and you named three that happen to, in my view, been designed specifically in reaction against the debian style of maintenance (creating an harmonious, stable set of packages that's less vulnerable to upstream changes), so it's strange to say that debian is the odd one out keeping a debian-hosted copy of all source used to build packages seems like a very reasonable defensive move to minimize external infrastructure dependencies and is key in the reproducible builds process there's definitely a conflict with the modern style of volatile language-specific package management, and I don't think debian's approach is ideal, but there's a reason so many people use debian as a base system also it seems like the idea of maintaining stable branches of software has fallen out of vogue in general
cesarb 2y ago
> AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that. AFAIK, most traditional distributions do that, not just Debian. They consider it important that software can be rebuilt, even in the far future, with nothing more than a copy of the distribution's binary and source packages. Doing anything which depends on network access during a build of the software is verboten (and AFAIK the automated build hosts block the network to enforce that requirement). Keep also in mind that these distributions are from before our current hyper-connected time; it was common for a computer to be offline most of the time, and only dial up to the Internet when necessary. You can still download full CD or DVD sets containing all of the Debian binaries and source code, and these should be enough to rebuild anything from that distribution, even on an air-gaped computer. > Secondly, even if they have to use crates from their repository, I don't understand what's so hard to just have multiple versions of the same crate? That will solve the problem too. That is often done for C libraries; for instance, Debian stable has both libncurses5 and libncurses6 packages. But it's a lot of work, since for technical reasons, each version has to be an independent package with a separate name, and at least for Debian, each new package has to be reviewed by the small ftpmaster team before being added to the distribution. I don't know whether there's anything Rust-specific that makes this harder (for C libraries, the filenames within the library packages are different, and the -dev packages with the headers conflict with each other so only one can be installed at a time). There's also the issue that having multiple versions means maintaining multiple versions (applying security fixes and so on).
- kijin 2y ago
  > There's also the issue that having multiple versions means maintaining multiple versions (applying security fixes and so on). This is the most important part. Debian LTS maintains packages for 5 years. Canonical takes Debian sources, and offers to maintain their LTS for 10 years. Red Hat also promises 10 years of support. They don't want anything in the core part of their stable branches that they can't promise to maintain for the next 5-10 years, when they have no assurance that upstream will even exist that long. If you want to move fast and break things, that's also fine. Just build and distribute your own .deb or .rpm. No need to bother distro maintainers who are already doing so much thankless work.
ouEight12 2y ago
> Arch does this, Gentoo does this, NixOS does this, why does Debian has to be different? I say this as someone who ran Gentoo for years and daily drives Arch today. Because sometimes you don't want entire swaths of your server being rebuilt/tinkered with on a regular basis under the hood. "Move fast, break everything" is great in dev/test land, or a prod environment where the entire fleet is just containers treated like cattle, but contrary to what the SREs of the valley would believe, there's a whole ecosystem of 'stuff' out there that will never be containerized, where servers are still treated like pets, or rather, at least "cherished mules", that just do their job 24/7 and get the occasional required security updates/patches and then go right back to operating the same way they did last year.
Brian_K_White 2y ago
It's pretty dumb (your words if you don't like them) not to understand (your words if you don't like them) that, how, and why, different distributions "has to be different". Debian is not Arch or Nix. A tractor is not a race car is not a submarine, even though they are all vehicles. This seems like a very basic concept for anyone purporting to do any sort of engineering or designing in any field. If it were me, I would not be so eager to advertize my failure to understsnd such basics, let alone presume to call anyone else pretty dumb.
Twirrim 2y ago
>AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that. Note your examples are all bleeding edge / rolling distributions. Debian and the non-bleeding edge distributions go a different route and focus on reproduce-ability and security, among other things. With the "get from crates.io" route, if someone compromises/hijacks a crate upstream, you're in trouble, immediately. By requiring vendoring of sources, that requires at least some level of manual actions by maintainers to get that compromised source in to the debian repositories to then be distributed out to the users. As you get in towards distributions like RHEL, they get even more cautious on this front.
- doublepg23 2y ago
  NixOS is not a rolling distro.
jonhohle 2y ago
I’d consider the issue to be the opposite. Why does every programming language now have a package manager and all of the infrastructure around package management rather than rely on the OS package manager? As a user I have to deal with apt, ports, pkg, opkg, ipkg, yum, flatpak, snap, docker, cpan, ctan, gems, pip, go modules, cargo, npm, swift packages, etc., etc., which all have different opinions of how and where to package files. On packaged operating systems (Debian, FreeBSD) - you have the system’s package manager to deal with (apt, pkg respectively). I can have an offline snapshot of _all_ packages that can be mirrors from one place. IMHO, every programming language having its own package system is the weird thing.
- MobiusHorizons 2y ago
  If you are a developer you almost always eventually need some dependencies that don’t ship with the os package manager, and once some of your dependencies are upstream source, you very quickly find that some dependencies of the sources you download rely on features from newer versions of libraries. If you have multiple clients, you may also need to support both old and new versions of the same dependencies depending on who the work is for. Package managers for a Linux distribution have incompatible goals to these (except maybe nix)
- epage 2y ago
  We want to make our software available to any system without every library maintainer being a packaging expert in every system. The user experience is much better when working within tkese packaging systems. You can control versions of software independent of the machine (or what distros ship). Or in other words, the needs of software development and software distribution are different. You can squint and see similarities but the fill different roles.
  - jonhohle 2y ago
    So every user has to be an expert in every package manager instead? Makes sense. Make life easy for the developer and pass the pain on to thousands of users. 20 years ago you may or may not support RPM and DEB and for everyone else a tarball with a make file that respected PREFIX was enough. (Obviously a tarball doesn’t support dependencies.)
cozzyd 2y ago
Part of a distros job is to audit all the packages (even if just to a minimal extent), and in many cases patch for various reasons. This is much harder if the source is external and there are N copies of everything.
tsimionescu 2y ago
Because Debian and similar distros have a goal of maintaining all of the software that users are expected to use. And this means they commit to fixing security issues in every single piece of software they distribute. A consequence of this is that they need to fix security issues in every library version that any application they distribute uses, including any statically-linked library. So, if they allow 30 applications to each have their own version of the same library, even Rust crates, and those versions all have a security issue, then the Debian team needs to find or patch themselves 30 different pieces of code. If instead they make sure that all 30 of those applications use the same version of a Rust crate, then they only need to patch one version. Maybe it's not 30 times less work, but it's definitely 10 times work. At the size of the Debian repos, this is an extremely significant difference. Now, it could be that this commitment from Debian is foolish and should be done away with. I certainly don't think my OS vendor should be the one that maintains all the apps I use - I don't even understand the attraction of that. I do want the OS maintainer to handle the packaging of the base system components, and patch those as needed for every supported version and so on - and so I understand this requirement for them. And I would view bcachefs-tools as a base system component, so this requirement seems sane for it.

by yshui 2y ago

wojciii 2y ago

Perhaps they want to have builds they can reproduce years later? When libraries are removed you have to update your build. This will not be the case if you use your own infrastructure (crates.io mirror).

acka 2y ago

Debian has a Social Contract[1] as well as guidelines {the DFSG}[2] regarding the commitment to only distribute free and open source software that all package maintainers must adhere to. This means that package maintainers must check the licenses of source code and documentation files, clear up any ambiguities by talking to upstream, and (as a last resort) even excise code and/or documentation from Debian's copy of the codebase if it doesn't meet the requirements of the DFSG. In practise, this means that Debian has to make its own copy of the source code available from a Debian-controlled repository, to ensure that no (accidental or otherwise) change to an upstream source archive can cause non-DFSG compliant Debian source or binary packages to be distributed. [1] https://www.debian.org/social_contract https://www.debian.org/social_contract [2] https://wiki.debian.org/DebianFreeSoftwareGuidelines https://wiki.debian.org/DebianFreeSoftwareGuidelines

koverstreet 2y ago
That justification doesn't work here - I provide vendored tarballs, so the source code availability argument is moot.
- righthand 2y ago
  But it’s not guaranteed. The Debian way provides a method of allocating responsibility. So if anything does go wrong they can point to a responsible party, the package maintainer. By providing tarball source you’re trying to placate responsibility of some code. You could build those tarballs on a different machine/different OS and any issues wouldn’t technically be your problem because “it’s just deps”.

Palomides 2y ago

debian is substantially older than all of those distros, and you named three that happen to, in my view, been designed specifically in reaction against the debian style of maintenance (creating an harmonious, stable set of packages that's less vulnerable to upstream changes), so it's strange to say that debian is the odd one out keeping a debian-hosted copy of all source used to build packages seems like a very reasonable defensive move to minimize external infrastructure dependencies and is key in the reproducible builds process there's definitely a conflict with the modern style of volatile language-specific package management, and I don't think debian's approach is ideal, but there's a reason so many people use debian as a base system also it seems like the idea of maintaining stable branches of software has fallen out of vogue in general

cesarb 2y ago

> AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that. AFAIK, most traditional distributions do that, not just Debian. They consider it important that software can be rebuilt, even in the far future, with nothing more than a copy of the distribution's binary and source packages. Doing anything which depends on network access during a build of the software is verboten (and AFAIK the automated build hosts block the network to enforce that requirement). Keep also in mind that these distributions are from before our current hyper-connected time; it was common for a computer to be offline most of the time, and only dial up to the Internet when necessary. You can still download full CD or DVD sets containing all of the Debian binaries and source code, and these should be enough to rebuild anything from that distribution, even on an air-gaped computer. > Secondly, even if they have to use crates from their repository, I don't understand what's so hard to just have multiple versions of the same crate? That will solve the problem too. That is often done for C libraries; for instance, Debian stable has both libncurses5 and libncurses6 packages. But it's a lot of work, since for technical reasons, each version has to be an independent package with a separate name, and at least for Debian, each new package has to be reviewed by the small ftpmaster team before being added to the distribution. I don't know whether there's anything Rust-specific that makes this harder (for C libraries, the filenames within the library packages are different, and the -dev packages with the headers conflict with each other so only one can be installed at a time). There's also the issue that having multiple versions means maintaining multiple versions (applying security fixes and so on).

kijin 2y ago
> There's also the issue that having multiple versions means maintaining multiple versions (applying security fixes and so on). This is the most important part. Debian LTS maintains packages for 5 years. Canonical takes Debian sources, and offers to maintain their LTS for 10 years. Red Hat also promises 10 years of support. They don't want anything in the core part of their stable branches that they can't promise to maintain for the next 5-10 years, when they have no assurance that upstream will even exist that long. If you want to move fast and break things, that's also fine. Just build and distribute your own .deb or .rpm. No need to bother distro maintainers who are already doing so much thankless work.

ouEight12 2y ago

> Arch does this, Gentoo does this, NixOS does this, why does Debian has to be different? I say this as someone who ran Gentoo for years and daily drives Arch today. Because sometimes you don't want entire swaths of your server being rebuilt/tinkered with on a regular basis under the hood. "Move fast, break everything" is great in dev/test land, or a prod environment where the entire fleet is just containers treated like cattle, but contrary to what the SREs of the valley would believe, there's a whole ecosystem of 'stuff' out there that will never be containerized, where servers are still treated like pets, or rather, at least "cherished mules", that just do their job 24/7 and get the occasional required security updates/patches and then go right back to operating the same way they did last year.

Brian_K_White 2y ago

It's pretty dumb (your words if you don't like them) not to understand (your words if you don't like them) that, how, and why, different distributions "has to be different". Debian is not Arch or Nix. A tractor is not a race car is not a submarine, even though they are all vehicles. This seems like a very basic concept for anyone purporting to do any sort of engineering or designing in any field. If it were me, I would not be so eager to advertize my failure to understsnd such basics, let alone presume to call anyone else pretty dumb.

Twirrim 2y ago

>AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that. Note your examples are all bleeding edge / rolling distributions. Debian and the non-bleeding edge distributions go a different route and focus on reproduce-ability and security, among other things. With the "get from crates.io" route, if someone compromises/hijacks a crate upstream, you're in trouble, immediately. By requiring vendoring of sources, that requires at least some level of manual actions by maintainers to get that compromised source in to the debian repositories to then be distributed out to the users. As you get in towards distributions like RHEL, they get even more cautious on this front.

doublepg23 2y ago
NixOS is not a rolling distro.

jonhohle 2y ago

I’d consider the issue to be the opposite. Why does every programming language now have a package manager and all of the infrastructure around package management rather than rely on the OS package manager? As a user I have to deal with apt, ports, pkg, opkg, ipkg, yum, flatpak, snap, docker, cpan, ctan, gems, pip, go modules, cargo, npm, swift packages, etc., etc., which all have different opinions of how and where to package files. On packaged operating systems (Debian, FreeBSD) - you have the system’s package manager to deal with (apt, pkg respectively). I can have an offline snapshot of _all_ packages that can be mirrors from one place. IMHO, every programming language having its own package system is the weird thing.

MobiusHorizons 2y ago
If you are a developer you almost always eventually need some dependencies that don’t ship with the os package manager, and once some of your dependencies are upstream source, you very quickly find that some dependencies of the sources you download rely on features from newer versions of libraries. If you have multiple clients, you may also need to support both old and new versions of the same dependencies depending on who the work is for. Package managers for a Linux distribution have incompatible goals to these (except maybe nix)
epage 2y ago
We want to make our software available to any system without every library maintainer being a packaging expert in every system. The user experience is much better when working within tkese packaging systems. You can control versions of software independent of the machine (or what distros ship). Or in other words, the needs of software development and software distribution are different. You can squint and see similarities but the fill different roles.
- jonhohle 2y ago
  So every user has to be an expert in every package manager instead? Makes sense. Make life easy for the developer and pass the pain on to thousands of users. 20 years ago you may or may not support RPM and DEB and for everyone else a tarball with a make file that respected PREFIX was enough. (Obviously a tarball doesn’t support dependencies.)

cozzyd 2y ago

Part of a distros job is to audit all the packages (even if just to a minimal extent), and in many cases patch for various reasons. This is much harder if the source is external and there are N copies of everything.

tsimionescu 2y ago

Because Debian and similar distros have a goal of maintaining all of the software that users are expected to use. And this means they commit to fixing security issues in every single piece of software they distribute. A consequence of this is that they need to fix security issues in every library version that any application they distribute uses, including any statically-linked library. So, if they allow 30 applications to each have their own version of the same library, even Rust crates, and those versions all have a security issue, then the Debian team needs to find or patch themselves 30 different pieces of code. If instead they make sure that all 30 of those applications use the same version of a Rust crate, then they only need to patch one version. Maybe it's not 30 times less work, but it's definitely 10 times work. At the size of the Debian repos, this is an extremely significant difference. Now, it could be that this commitment from Debian is foolish and should be done away with. I certainly don't think my OS vendor should be the one that maintains all the apps I use - I don't even understand the attraction of that. I do want the OS maintainer to handle the packaging of the base system components, and patch those as needed for every supported version and so on - and so I understand this requirement for them. And I would view bcachefs-tools as a base system component, so this requirement seems sane for it.