5 ms·

Dependencies should be fetched directly from VCS

54 points by mrngm 2mo ago

arcatek 2mo ago
Packages are typically different once published than they were inside their original repositories. Call it transpilation, build, compilation, packaging, etc, most popular projects require some level of support for dynamic code execution before reaching their usable state. As much as I'd have liked Git to be a viable option compared to centralized registries, last couple of years demonstrated running arbitrary commands during install is too much of a risk for it to work at scale.
- AlotOfReading 2mo ago
  ...most popular projects require some level of support for dynamic code execution before reaching their usable state. None of your examples require arbitrary script execution. You can specify them all declaratively, like Bazel forces you to do. I don't think that package managers should be doing the job of a build system though.
tikkabhuna 2mo ago
There’s no perfect solution here. Publishing to a separate registry can survive a Git repo rename, migration or deletion. Locking into a Git host seems undesirable. By separating VCS and registry they can offer different feature sets. There’s also nothing stopping someone from publishing to multiple registries.
- klntsky 2mo ago
  The almost perfect solution is Nix
  - iririririr 2mo ago
    if you mean nix the state declaration, it is a ilusion. when you have packge for debian 12... you just install it. when you have debian 13, you need the package for debian 13 and there's no way around it. nix lies that you don't have to worry about all that. which is only as true as is true with packages. if you can replace -12 with -13 in a non nix setup, your nix "package x" will still work. otherwise you will have to deal with it, just with more layers. if you mean nixos, that's just starting yet another distro maintenance issue from scratch
    - tiffanyh 2mo ago
      I’m super interested in your comment about nix but am not following your example. Would you mind elaborating why (presumably) nixos (since you gave a Debian example), doesn’t help with this.
      - iririririr 2mo ago
        it is a distro. it packages code from upstream. it have the same problems as other distros when it came from depencies and managing breaking changes.
  - NewJazz 2mo ago
    Why almost?
- chickensong 2mo ago
  FWIW, Go provides a mechanism to abstract the import source, so you can use a vanity domain under your control that resolves to your host of choice.
kibwen 2mo ago
Maybe everyone else is too young to remember left-pad, but in the wake of left-pad everyone learned that one of the primary selling points of dedicated dependency repositories is that they can refuse to support "unpublishing" a dependency, which is not a guarantee that Github (or any other popular forge) makes.
- chickensong 2mo ago
  > Maybe everyone else is too young to remember left-pad It wasn't that long ago...
  - cygx 2mo ago
    Ten years as of March 22...
  - SoftTalker 2mo ago
    In the AI age, 6 months seems to be percieved as a long time ago. 6 months feels like yesterday to me.
- inigyou 2mo ago
  There's no such thing as refusing to unpublish, not if you want to avoid prison time. Package contains child porn, what now?
  - mook 2mo ago
    I think it's implied to be refusing to unpublish at the request of the author; unpublishing due to outside forces is probably inevitable. If some unsupportable content got introduced in a new version (e.g. xz), at least the historical versions would be kept. That of course brings in questions of cost, trust, and governance, so there's definitely a trade-off…
nijave 2mo ago
Not sure relying on a bunch of various VCS to stay online is necessarily a great approach either. I think go is also a little more amenable to source library distribution since there's a pretty broad pure go ecosystem. For interpreted languages, a lot of performance sensitive stuff tends to be offloaded to arbitrary compiled languages so you end up needing a bunch of different toolchains to get everything working. A statically linked binary library is a useful abstraction layer.
- theamk 2mo ago
  2nd paragraph of blogpost: > A hash of all files is checked against known hashes on sum.golang.org to prevent tags from being replaced, and it uses a proxy to prevent repos from being left-pad’d. there is no need for 3rd party hosts to stay online. It's the best combination of de-centralized and centralized approaches. Re pre-buiild packages: Python makes a difference between source and binary wheels. This approach is obviously for source wheels only.
  - nijave 2mo ago
    Proxies are still 3rd party hosts. I've ran into issues in my very limited go experience with them missing things and having hash mismatches (maybe someone changed a tag and it was working as intended? I don't remember the exact chain of events but I ended up with a hash that could no longer be found anywhere)
jchw 2mo ago
People keep replying seeming to miss what makes Go's solution so good. I think it deserves more attention. In many cases I think it's impossible to avoid the pain that comes with decentralization without defeating the point. It's challenging because certain things are hard to do decentralized so it's easier to just give up and rely on central authorities and components. When you install Go, it does just that, to some approximation; it will try to fetch from the module proxy first and fall back after that fails. But what makes Go different and more interesting is that you can just turn that off with no consequence. As long as all of the Git hosts are up, you can still use Go just fine with no proxy. I'm pretty sure Nix builds do this by default with Go so I believe most dependencies are usually online. That means the proxy approach and the decentralized approach are both fully redundant with eachother, while allowing the decentralized VCS hosts to remain the source of truth. Go also separately has the sumdb to enforce immutability, so you can know that when you go grab a specific package that the tag name corresponds to one and only one exact source code. This is also optional, and even without it you get the benefit of local sum checking via your own cache and go.sum files in source checkouts. But having this be separate is great because it also means you could use another third party module proxy or a third party sumdb, or use just the sumdb and no proxy. It gives you many options to not have to rely on a single party while still getting some of the benefits of centralization. Systems like this are rare. Many systems try very hard to have the cake and eat it too, but it usually fails somewhere. For me Go's approach works, and that is worth some attention.
- throwaway27448 2mo ago
  All package managers I've used allow specifying where the package is pulled from. What am I missing? What ecosystems are you comparing go to?
  - jchw 2mo ago
    Sure, you can tell your other package managers to pull a module from VCS, usually with some limitations and negative side effects. The Go module syste is not the same model. You can, for example, run an NPM proxy that caches locally, but what you're missing is that with Go, the proxy part is the centralized part. There is no NPM. There is no crates.io, no PyPI, no RubyGems, no Packigist, no PECL, no CPAN. The proxy's entire job is just to stand in for requests that would be made to individual VCSes using their original source repositories. The sumdb's entire job is to ensure that whether proxied or not, the source code you get isn't tampered with. The source of truth remains decentralized, in the individual VCSes. With NPM, uv, composer, Cargo, etc. you can indeed choose to make your dependencies pull from VCS, but: - Not having universal module proxies with caching means you will run into the usual problems. Repos can disappear, history can be rewritten, hosts can go down. Even if you invented an equivalent caching service, without making it an ecosystem default it won't achieve the same effectiveness as Go because less would be in the cache. - Because it isn't an ecosystem standard, dependencies you pull would then still reference dependencies from centralized repos anyways, more or less defeating the point. You'd have to go and override every recursive dependency to really make it decentralized. (And again, it usually comes with other downsides depending on the specific ecosystem. Performance is a big one, the module cache definitely helps make fetching dependencies in Go faster.) Not having centralized package repos comes with its ups and downs, but the lack of one existing means there are a limited number of interesting attacks you can really pull in the Go ecosystem and they are heavily tamper-evident. You can try to poison the module cache with a given revision for a dependency with malicious code then overwrite it with an inoccuous commit, but this will probably be detected when someone (possibly in Nixpkgs or another packaging system) runs GOPROXY=direct and the sumdb doesn't line up. So the least noticeable thing you can really do is just leave your malicious code in the repo. Go also has the kind of nice property that fetching and building modules is supposed to be safe with untrusted code. That's pretty good, it would be hard to do better than that. It also means there really isn't any central namespace to poison. You can still trick someone into downloading and using malicious software but you're mostly limited to doing it the old fashioned way. Again, hard to do much better than that. While everyone has been losing their minds over supply chain issues, the Go ecosystem has been concerned but less so. It's still a huge risk still, dependencies can get compromised still, but attackers have no real central target to go after and get a bunch of damage at once. Nothing realistical, anyway. So, it's back to just trying to pwn individual GitHub repos.
brunoarueira 2mo ago
Multiple times the VCS was poisoned, we'll need to improve a lot the security around them to be able to go that route!
cygx 2mo ago
Not sure this is the right solution for 2 reasons: First, I'm uncomfortable with making a package creator's VCS provider part of the language's module infrastructure. Second, not all files under version control necessarily belong in a tarball, and not all files in a tarball necessarily need to be under version control. However, the point that there should be an easy way to track changes in your dependencies is well taken. One possible approach would be that publishing a packge boils down to importing the files that normally would go in your tarball into a community-controlled VCS.
- g-b-r 2mo ago
  > First, I'm uncomfortable with making a package creator's VCS provider part of the language's module infrastructure. Why can't you store a (possibly shallow) clone of the repository in the "package" registry? > Second, not all files under version control necessarily belong in a tarball How likely is it that files under version control but "not belonging in a tarball" (not needed to compile the current version) take so much space to justify making a tarball? At least if you make a shallow clone. > not all files in a tarball necessarily need to be under version control Which wouldn't (besides huge files that can be handled with git-lfs and similar)? If you have configure in mind, let's not do it please. The risks introduced by including them vastly overcome their convenience, in my opinion. And maybe let's move towards getting rid of autotools
  - cygx 2mo ago
    Why can't you store a (possibly shallow) clone of the repository in the "package" registry? Yes, that would be an option. As to your question, you generally want to version control input files, but distribute the generated artifacts. Some possible scenarios would be documentation, data tables in source or binary form, generated code including maybe even configure scripts if you cannot avoid it (though if I had to 'vendor' 3rdparty code that uses autotools, I think I probably would just unpack a tarball and commit the relevant artifacts).
    - g-b-r 2mo ago
      If you distribute generated artifacts you're not really distributing the source (and thus xz can happen) Even for documentation, you'd think it would be harmless to generate it, but it's far from a given that the tools (e.g. browsers) used to read it are safe from malicious documentation files
Cedricgc 2mo ago
IIRC Go had to deal with a bunch of issue related to their initial source based package management system. The URL imports remain, but I believe fetches are served through the pkg.go.dev proxy now. Andrew Nesbitt has a good write [0] up on why using Git as a database is a bad idea [0] https://nesbitt.io/2025/12/24/package-managers-keep-using-git-as-a-database.html https://nesbitt.io/2025/12/24/package-managers-keep-using-gi...
- rendaw 2mo ago
  Most of those deal with downloading large centralized indexes using git, not individual packages. > The problem was that go get needed to fetch each dependency’s source code just to read its go.mod file and resolve transitive dependencies. Cloning entire repositories to get a single file. Then it needs to compile the module, right? It's not downloading it twice. > Grab’s engineering team went from 18 minutes for go get to 12 seconds after deploying a module proxy. > ... > GOPROXY became the default in Go 1.13. The way this is written it sounds like go builds were extremely slow (18 minutes) so go adopted a proxy to speed things up. But Go 1.13 was released in 2020, and the blog post about 18 minutes is from 2023. There's no link to anything about GOPROXY. The vcpkg example also isn't a git issue, it sounds like just bad engineering from Microsoft. I'm willing to believe there are issues with using git, but that article is a letdown.
  - rendaw 2mo ago
    https://go.dev/blog/module-mirror-launch https://go.dev/blog/module-mirror-launch seems like a better source. It seems to imply the reasons were optimization and to protect from disappearing dependencies. It doesn't imply that there were any hidden dangers to git. I think it's very possible that google simply thinks speed is more important than decentralization. And for google's purposes, a google-owned cache is probably a safe enough hedge against disappearing repos (read: if you're not google you'd need to evaluate this risk yourself). Edit: Discussion from the time here: https://news.ycombinator.com/item?id=20837753 https://news.ycombinator.com/item?id=20837753 - in that thread too people don't really know why the proxy was needed (or like, "is it really just for speed? but it isn't particularly slow right now"). One additional reason is analytics collection.
Ozzie-D 2mo ago
[flagged]
kpcyrd 2mo ago
This makes the problem harder instead of easier. The most practical conclusion is "the package registry is the VCS, but with stricter rules". Those stricter rules are useful, the registry acts as a authority to ensure no version string is ever reused in a way that would make it non-canonical. It also ensures availability of the source code, so your software won't suddenly fail to compile because somebody decided to delete the repository from their server, delete their Github/codeberg user, or they can't pay for their domain anymore (which is a fail-open identity system, not a fail-close one like crates.io). Sure there's also the risk of crates.io ceasing to exist, but that's less likely than "some software engineer gets layed off, decides tech is a dead-end and becomes a domain-less goose farmer".

54 points by mrngm 2mo ago

Packages are typically different once published than they were inside their original repositories. Call it transpilation, build, compilation, packaging, etc, most popular projects require some level of support for dynamic code execution before reaching their usable state. As much as I'd have liked Git to be a viable option compared to centralized registries, last couple of years demonstrated running arbitrary commands during install is too much of a risk for it to work at scale.

AlotOfReading 2mo ago
...most popular projects require some level of support for dynamic code execution before reaching their usable state. None of your examples require arbitrary script execution. You can specify them all declaratively, like Bazel forces you to do. I don't think that package managers should be doing the job of a build system though.

tikkabhuna 2mo ago

There’s no perfect solution here. Publishing to a separate registry can survive a Git repo rename, migration or deletion. Locking into a Git host seems undesirable. By separating VCS and registry they can offer different feature sets. There’s also nothing stopping someone from publishing to multiple registries.

klntsky 2mo ago
The almost perfect solution is Nix
- iririririr 2mo ago
  if you mean nix the state declaration, it is a ilusion. when you have packge for debian 12... you just install it. when you have debian 13, you need the package for debian 13 and there's no way around it. nix lies that you don't have to worry about all that. which is only as true as is true with packages. if you can replace -12 with -13 in a non nix setup, your nix "package x" will still work. otherwise you will have to deal with it, just with more layers. if you mean nixos, that's just starting yet another distro maintenance issue from scratch
  - tiffanyh 2mo ago
    I’m super interested in your comment about nix but am not following your example. Would you mind elaborating why (presumably) nixos (since you gave a Debian example), doesn’t help with this.
    - iririririr 2mo ago
      it is a distro. it packages code from upstream. it have the same problems as other distros when it came from depencies and managing breaking changes.
- NewJazz 2mo ago
  Why almost?
chickensong 2mo ago
FWIW, Go provides a mechanism to abstract the import source, so you can use a vanity domain under your control that resolves to your host of choice.

kibwen 2mo ago

Maybe everyone else is too young to remember left-pad, but in the wake of left-pad everyone learned that one of the primary selling points of dedicated dependency repositories is that they can refuse to support "unpublishing" a dependency, which is not a guarantee that Github (or any other popular forge) makes.

chickensong 2mo ago
> Maybe everyone else is too young to remember left-pad It wasn't that long ago...
- cygx 2mo ago
  Ten years as of March 22...
- SoftTalker 2mo ago
  In the AI age, 6 months seems to be percieved as a long time ago. 6 months feels like yesterday to me.
inigyou 2mo ago
There's no such thing as refusing to unpublish, not if you want to avoid prison time. Package contains child porn, what now?
- mook 2mo ago
  I think it's implied to be refusing to unpublish at the request of the author; unpublishing due to outside forces is probably inevitable. If some unsupportable content got introduced in a new version (e.g. xz), at least the historical versions would be kept. That of course brings in questions of cost, trust, and governance, so there's definitely a trade-off…

nijave 2mo ago

Not sure relying on a bunch of various VCS to stay online is necessarily a great approach either. I think go is also a little more amenable to source library distribution since there's a pretty broad pure go ecosystem. For interpreted languages, a lot of performance sensitive stuff tends to be offloaded to arbitrary compiled languages so you end up needing a bunch of different toolchains to get everything working. A statically linked binary library is a useful abstraction layer.

theamk 2mo ago
2nd paragraph of blogpost: > A hash of all files is checked against known hashes on sum.golang.org to prevent tags from being replaced, and it uses a proxy to prevent repos from being left-pad’d. there is no need for 3rd party hosts to stay online. It's the best combination of de-centralized and centralized approaches. Re pre-buiild packages: Python makes a difference between source and binary wheels. This approach is obviously for source wheels only.
- nijave 2mo ago
  Proxies are still 3rd party hosts. I've ran into issues in my very limited go experience with them missing things and having hash mismatches (maybe someone changed a tag and it was working as intended? I don't remember the exact chain of events but I ended up with a hash that could no longer be found anywhere)

jchw 2mo ago

People keep replying seeming to miss what makes Go's solution so good. I think it deserves more attention. In many cases I think it's impossible to avoid the pain that comes with decentralization without defeating the point. It's challenging because certain things are hard to do decentralized so it's easier to just give up and rely on central authorities and components. When you install Go, it does just that, to some approximation; it will try to fetch from the module proxy first and fall back after that fails. But what makes Go different and more interesting is that you can just turn that off with no consequence. As long as all of the Git hosts are up, you can still use Go just fine with no proxy. I'm pretty sure Nix builds do this by default with Go so I believe most dependencies are usually online. That means the proxy approach and the decentralized approach are both fully redundant with eachother, while allowing the decentralized VCS hosts to remain the source of truth. Go also separately has the sumdb to enforce immutability, so you can know that when you go grab a specific package that the tag name corresponds to one and only one exact source code. This is also optional, and even without it you get the benefit of local sum checking via your own cache and go.sum files in source checkouts. But having this be separate is great because it also means you could use another third party module proxy or a third party sumdb, or use just the sumdb and no proxy. It gives you many options to not have to rely on a single party while still getting some of the benefits of centralization. Systems like this are rare. Many systems try very hard to have the cake and eat it too, but it usually fails somewhere. For me Go's approach works, and that is worth some attention.

throwaway27448 2mo ago
All package managers I've used allow specifying where the package is pulled from. What am I missing? What ecosystems are you comparing go to?
- jchw 2mo ago
  Sure, you can tell your other package managers to pull a module from VCS, usually with some limitations and negative side effects. The Go module syste is not the same model. You can, for example, run an NPM proxy that caches locally, but what you're missing is that with Go, the proxy part is the centralized part. There is no NPM. There is no crates.io, no PyPI, no RubyGems, no Packigist, no PECL, no CPAN. The proxy's entire job is just to stand in for requests that would be made to individual VCSes using their original source repositories. The sumdb's entire job is to ensure that whether proxied or not, the source code you get isn't tampered with. The source of truth remains decentralized, in the individual VCSes. With NPM, uv, composer, Cargo, etc. you can indeed choose to make your dependencies pull from VCS, but: - Not having universal module proxies with caching means you will run into the usual problems. Repos can disappear, history can be rewritten, hosts can go down. Even if you invented an equivalent caching service, without making it an ecosystem default it won't achieve the same effectiveness as Go because less would be in the cache. - Because it isn't an ecosystem standard, dependencies you pull would then still reference dependencies from centralized repos anyways, more or less defeating the point. You'd have to go and override every recursive dependency to really make it decentralized. (And again, it usually comes with other downsides depending on the specific ecosystem. Performance is a big one, the module cache definitely helps make fetching dependencies in Go faster.) Not having centralized package repos comes with its ups and downs, but the lack of one existing means there are a limited number of interesting attacks you can really pull in the Go ecosystem and they are heavily tamper-evident. You can try to poison the module cache with a given revision for a dependency with malicious code then overwrite it with an inoccuous commit, but this will probably be detected when someone (possibly in Nixpkgs or another packaging system) runs GOPROXY=direct and the sumdb doesn't line up. So the least noticeable thing you can really do is just leave your malicious code in the repo. Go also has the kind of nice property that fetching and building modules is supposed to be safe with untrusted code. That's pretty good, it would be hard to do better than that. It also means there really isn't any central namespace to poison. You can still trick someone into downloading and using malicious software but you're mostly limited to doing it the old fashioned way. Again, hard to do much better than that. While everyone has been losing their minds over supply chain issues, the Go ecosystem has been concerned but less so. It's still a huge risk still, dependencies can get compromised still, but attackers have no real central target to go after and get a bunch of damage at once. Nothing realistical, anyway. So, it's back to just trying to pwn individual GitHub repos.

brunoarueira 2mo ago

Multiple times the VCS was poisoned, we'll need to improve a lot the security around them to be able to go that route!

cygx 2mo ago

Not sure this is the right solution for 2 reasons: First, I'm uncomfortable with making a package creator's VCS provider part of the language's module infrastructure. Second, not all files under version control necessarily belong in a tarball, and not all files in a tarball necessarily need to be under version control. However, the point that there should be an easy way to track changes in your dependencies is well taken. One possible approach would be that publishing a packge boils down to importing the files that normally would go in your tarball into a community-controlled VCS.

g-b-r 2mo ago
> First, I'm uncomfortable with making a package creator's VCS provider part of the language's module infrastructure. Why can't you store a (possibly shallow) clone of the repository in the "package" registry? > Second, not all files under version control necessarily belong in a tarball How likely is it that files under version control but "not belonging in a tarball" (not needed to compile the current version) take so much space to justify making a tarball? At least if you make a shallow clone. > not all files in a tarball necessarily need to be under version control Which wouldn't (besides huge files that can be handled with git-lfs and similar)? If you have configure in mind, let's not do it please. The risks introduced by including them vastly overcome their convenience, in my opinion. And maybe let's move towards getting rid of autotools
- cygx 2mo ago
  Why can't you store a (possibly shallow) clone of the repository in the "package" registry? Yes, that would be an option. As to your question, you generally want to version control input files, but distribute the generated artifacts. Some possible scenarios would be documentation, data tables in source or binary form, generated code including maybe even configure scripts if you cannot avoid it (though if I had to 'vendor' 3rdparty code that uses autotools, I think I probably would just unpack a tarball and commit the relevant artifacts).
  - g-b-r 2mo ago
    If you distribute generated artifacts you're not really distributing the source (and thus xz can happen) Even for documentation, you'd think it would be harmless to generate it, but it's far from a given that the tools (e.g. browsers) used to read it are safe from malicious documentation files

Cedricgc 2mo ago

IIRC Go had to deal with a bunch of issue related to their initial source based package management system. The URL imports remain, but I believe fetches are served through the pkg.go.dev proxy now. Andrew Nesbitt has a good write [0] up on why using Git as a database is a bad idea [0] https://nesbitt.io/2025/12/24/package-managers-keep-using-git-as-a-database.html https://nesbitt.io/2025/12/24/package-managers-keep-using-gi...

rendaw 2mo ago
Most of those deal with downloading large centralized indexes using git, not individual packages. > The problem was that go get needed to fetch each dependency’s source code just to read its go.mod file and resolve transitive dependencies. Cloning entire repositories to get a single file. Then it needs to compile the module, right? It's not downloading it twice. > Grab’s engineering team went from 18 minutes for go get to 12 seconds after deploying a module proxy. > ... > GOPROXY became the default in Go 1.13. The way this is written it sounds like go builds were extremely slow (18 minutes) so go adopted a proxy to speed things up. But Go 1.13 was released in 2020, and the blog post about 18 minutes is from 2023. There's no link to anything about GOPROXY. The vcpkg example also isn't a git issue, it sounds like just bad engineering from Microsoft. I'm willing to believe there are issues with using git, but that article is a letdown.
- rendaw 2mo ago
  https://go.dev/blog/module-mirror-launch https://go.dev/blog/module-mirror-launch seems like a better source. It seems to imply the reasons were optimization and to protect from disappearing dependencies. It doesn't imply that there were any hidden dangers to git. I think it's very possible that google simply thinks speed is more important than decentralization. And for google's purposes, a google-owned cache is probably a safe enough hedge against disappearing repos (read: if you're not google you'd need to evaluate this risk yourself). Edit: Discussion from the time here: https://news.ycombinator.com/item?id=20837753 https://news.ycombinator.com/item?id=20837753 - in that thread too people don't really know why the proxy was needed (or like, "is it really just for speed? but it isn't particularly slow right now"). One additional reason is analytics collection.

Ozzie-D 2mo ago

[flagged]

kpcyrd 2mo ago

This makes the problem harder instead of easier. The most practical conclusion is "the package registry is the VCS, but with stricter rules". Those stricter rules are useful, the registry acts as a authority to ensure no version string is ever reused in a way that would make it non-canonical. It also ensures availability of the source code, so your software won't suddenly fail to compile because somebody decided to delete the repository from their server, delete their Github/codeberg user, or they can't pay for their domain anymore (which is a fail-open identity system, not a fail-close one like crates.io). Sure there's also the risk of crates.io ceasing to exist, but that's less likely than "some software engineer gets layed off, decides tech is a dead-end and becomes a domain-less goose farmer".