8 ms·

Git Submodules as a Package Manager

91 points by ErenayDev 18d ago

orliesaurus 14d ago
I just have a feeling the developer-experience wouldn't be the same - but that's just a temporary issue perhaps?
AlphaSite 14d ago
I know it sa toy, but if someone gets the bright idea to do this for real: please learn from npm and dont duplicate packages for every clone of a project. Share them between projects like every sane package manager.
- matsz 14d ago
  > Share them between projects like every sane package manager. I'd argue the other extreme is less sane. Pip installs everything system wide and you can't have different versions of the same package without venv. Even early NPM was much much better than the pre-uv Python situation. (TBH even with uv it still feels hacky at times.)
  - brabel 14d ago
    The global repository should support versions obviously.
  - diversen7 14d ago
    Virtual venv existed before uv. So you could still just generate a virtual venv and then use pip to install packages in the local venv. (2007: virtualenv was released as a third-party tool).
  - zelphirkalt 14d ago
    Before uv there was poetry, and before poetry there was pipenv, before pipenv there was virtualenv. I wish people would stop portraying Python tooling completely wrong like that. uv is not the first tool to solve most problems in this space. It may be the best performing though and may be the best overall currently.
    - dlisboa 13d ago
      I think you’re underselling it. You’ve listed several projects with many thousand man hours in development. It’s telling that in Python there have been multiple package managers over the years to fix the broken status quo. Typescript/Javascript notwithstanding, in other languages and environments you get the one tool that’s been there for over a decade without competitors. Very few languages have suffered from the same package manager issues as JS/TS and Python.
      - no_wizard 13d ago
        Very few languages have also suffered their popularity
      - 0x20cowboy 13d ago
        Wouldn't yarn, grunt and gulp fall into this category?
        c-hendricks 13d ago
        > Typescript/Javascript notwithstanding Also, gulp / grunt weren't package managers.
  - steveklabnik 13d ago
    > the other extreme These aren't the only two options. This > Pip installs everything system wide does not inherently mean > you can't have different versions of the same package
  - Lvl999Noob 12d ago
    You don't want to install all dependencies globally. You only want to cache them globally. Each project should resolve its own dependency tree independently. If the wanted version is already in cache, use that, otherwise download from the repository even if an older / newer version is present in cache (unless the user specifically says to use the cache only). As long as your dependencies are inert (they don't mess with the system just by being present), this will work perfectly fine. For non-inert packages, such as applications installed via the language's dependency manager, that's a different use case and should be handled differently.
unqueued 14d ago
FYI A submodule doesn't have to use a gitfile and a corresponding `$GIT_DIR/modules/<name>` and there are good reasons not to. As long as the submodule has a '.git' it can be a symlink, or regular .git directory for a self contained embedded repo. You can still use the same gitlink in the parent repo representing it's commit id and git will still manage it. If I have a 20GB submodule I'd usually just do a `git clone <url> <path> && git submodule add <path>` and it will be treated the same by git. But now I can just delete it and it's purged. And it is more portable less fragile in some ways because it's not de-referencing a gitfile. I prefer my repos to be more bottom-heavy and to not clog my modules folder. I have a rough script that's the inverse of `git submodule absorbgitdirs` but it's a bit fragile. It would be cool if there was some plumbing to expose this a bit more.
DarmokTanagra 14d ago
[dead]
bellowsgulch 14d ago
I find https://github.com/andrewmcwattersandco/git-fetch-file https://github.com/andrewmcwattersandco/git-fetch-file more useful. It's manifest is nearly the same as a .gitmodules file, but having specific control over what files you want to pull, or what commit or tag, or just the equivalent of latest is way more practical day-to-day.
- bionade24 14d ago
  How does this compare to `git subtree`, which is part of git itself ?
  - unqueued 13d ago
    This isn't really like subtree it's like submodules, but for specific files. This could really be useful, you may want to keep track of a specific file in a repo but not need it as a submodule. There's a whole umbrella of similar-ish utilities. The one that I think does almost the same thing is: https://github.com/mullvad/git-carbon https://github.com/mullvad/git-carbon
    - bellowsgulch 13d ago
      Yes, basically this. I think git-fetch-file has been used in some larger projects based on Googling it, so it seems to be different in that it's been vetted by other orgs, but like Mullvad, theirs is built by a corporation.
  - bellowsgulch 13d ago
    It seems like something more in-between. Like a git-based npm install, but generic to any repository or language. It's also "pointer" based, but like vendoring, it's up to you whether or not you want to keep files in the repository. It also seems to be intentionally built for the use case where you modify something from upstream, because if you do have local changes, it detects them and won't overwrite them without your permission.
byako 14d ago
[dead]
dissent 14d ago
If you can point a package manager at a git repo and use it like package, this is accidental convenience. Source code repositories should be factored as source code, with a build step to transform them into packages. For some languages that means compilation, for others it means transpiling, minifying, or just copying files. The resulting artifact is a different shape, designed for consumption. If you use git for packages, then your repo becomes the package boundary. You no longer have the option of producing multiple packages from one repo, or even one package from multiple repos.
- fireflash38 13d ago
  > If you use git for packages, then your repo becomes the package boundary. You no longer have the option of producing multiple packages from one repo, or even one package from multiple repos. What does this even mean? I'd argue that having the source to build the thing is more important than the artifacts. Release artifacts are more of a convenience. If the thing doesn't build from the source given, what use is it?
  - dissent 13d ago
    Exactly my point. You assume the source is the thing you want to consume, but it isn't. Package registries are much more than a convenience and packages don't necessarily map 1:1 with source code repositories. Nor should they need to. You shouldn't have to build every dependency and care where their source code is. Sure you want the source available, otherwise it's closed source, but ideally you never need to look at it unless you're a contributor. git has really conflated the two concepts.
    - dezgeg 13d ago
      Correct way is to have everything as-if a source dependency, but a globally shared cache system that can automatically provide artifacts if that exact version of source has already been compiled. Any system that can only consume manually built (and even in worst case, only tagged and globally pushed) dependencies is horrible for cross-module development, like large scale refactorings and the like.
deleted 13d ago
[deleted]
Myrmornis 13d ago
A lot of the confusion and criticism surrounding submodules comes from people not understanding that you do not need to learn a lot of new commands to use them: you just cd into the submodule and use it as a normal git repo. Then you cd out and commit the submodule hash change if that's what you want to do. Some things are annoying, like removing them. But in general they're extremely useful and designed basically exactly how you'd want: it's extremely consistent since to a good approximation, a submodule behaves the same as a normal git repo once you're inside it.
- IshKebab 13d ago
  I disagree, even when you understand them there are still big problems with them: 1. Git doesn't support them very well. Bugs are disappointing frequent. Try changing a submodule to a directory or vice versa and feel the pain. 2. They are often used to split up projects (especially commercially) but this makes testing much harder and cross-repo changes way way harder. Even changes that only touch the submodule become way more tedious because you have to always do another PR to update the pointer. 3. The submodules are referenced by URL, which means you can't easily move or copy projects that have them because they still point to the old submodule location. Sometimes they're even misconfiguration to always use ssh:// or whatever, breaking unauthenticated clones. 4. Switching branches where submodules have been added/removed is a mess. 5. Having to constantly remember `git submodule update --init --recursive` is a right pain. 6. When used as a crap dependency manager you easily end up with duplicates. A project I worked on had 12 copied of a common submodule via transitive dependencies. IMO they're almost always the wrong solution.
  - lucasoshiro 13d ago
    > Git doesn't support them very well. Bugs are disappointing frequent I can't remember any "bug" in the submodule machinery. If you find one, report to the Git mailing list, they very receptive. > Having to constantly remember `git submodule update --init --recursive` Set this config: `git config submodule.recurse true` > IMO they're almost always the wrong solution. Yeah... but sometimes they are the only feasible solution, especially when the dependency needs to be shared across projects that use incompatible dependency managers and Git is the only common denominator between them...
jonmoore 13d ago
This appears to be ai-generated based on my impression, pangram's "100% of this text is ai", and the blog's posting history.
DonHopkins 13d ago
So a fully automatic Gatling Footgun.

91 points by ErenayDev 18d ago

I just have a feeling the developer-experience wouldn't be the same - but that's just a temporary issue perhaps?

I know it sa toy, but if someone gets the bright idea to do this for real: please learn from npm and dont duplicate packages for every clone of a project. Share them between projects like every sane package manager.

matsz 14d ago
> Share them between projects like every sane package manager. I'd argue the other extreme is less sane. Pip installs everything system wide and you can't have different versions of the same package without venv. Even early NPM was much much better than the pre-uv Python situation. (TBH even with uv it still feels hacky at times.)
- brabel 14d ago
  The global repository should support versions obviously.
- diversen7 14d ago
  Virtual venv existed before uv. So you could still just generate a virtual venv and then use pip to install packages in the local venv. (2007: virtualenv was released as a third-party tool).
- zelphirkalt 14d ago
  Before uv there was poetry, and before poetry there was pipenv, before pipenv there was virtualenv. I wish people would stop portraying Python tooling completely wrong like that. uv is not the first tool to solve most problems in this space. It may be the best performing though and may be the best overall currently.
  - dlisboa 13d ago
    I think you’re underselling it. You’ve listed several projects with many thousand man hours in development. It’s telling that in Python there have been multiple package managers over the years to fix the broken status quo. Typescript/Javascript notwithstanding, in other languages and environments you get the one tool that’s been there for over a decade without competitors. Very few languages have suffered from the same package manager issues as JS/TS and Python.
    - no_wizard 13d ago
      Very few languages have also suffered their popularity
    - 0x20cowboy 13d ago
      Wouldn't yarn, grunt and gulp fall into this category?
      - c-hendricks 13d ago
        > Typescript/Javascript notwithstanding Also, gulp / grunt weren't package managers.
- steveklabnik 13d ago
  > the other extreme These aren't the only two options. This > Pip installs everything system wide does not inherently mean > you can't have different versions of the same package
- Lvl999Noob 12d ago
  You don't want to install all dependencies globally. You only want to cache them globally. Each project should resolve its own dependency tree independently. If the wanted version is already in cache, use that, otherwise download from the repository even if an older / newer version is present in cache (unless the user specifically says to use the cache only). As long as your dependencies are inert (they don't mess with the system just by being present), this will work perfectly fine. For non-inert packages, such as applications installed via the language's dependency manager, that's a different use case and should be handled differently.

unqueued 14d ago

FYI A submodule doesn't have to use a gitfile and a corresponding `$GIT_DIR/modules/<name>` and there are good reasons not to. As long as the submodule has a '.git' it can be a symlink, or regular .git directory for a self contained embedded repo. You can still use the same gitlink in the parent repo representing it's commit id and git will still manage it. If I have a 20GB submodule I'd usually just do a `git clone <url> <path> && git submodule add <path>` and it will be treated the same by git. But now I can just delete it and it's purged. And it is more portable less fragile in some ways because it's not de-referencing a gitfile. I prefer my repos to be more bottom-heavy and to not clog my modules folder. I have a rough script that's the inverse of `git submodule absorbgitdirs` but it's a bit fragile. It would be cool if there was some plumbing to expose this a bit more.

DarmokTanagra 14d ago

[dead]

bellowsgulch 14d ago

I find https://github.com/andrewmcwattersandco/git-fetch-file https://github.com/andrewmcwattersandco/git-fetch-file more useful. It's manifest is nearly the same as a .gitmodules file, but having specific control over what files you want to pull, or what commit or tag, or just the equivalent of latest is way more practical day-to-day.

bionade24 14d ago
How does this compare to `git subtree`, which is part of git itself ?
- unqueued 13d ago
  This isn't really like subtree it's like submodules, but for specific files. This could really be useful, you may want to keep track of a specific file in a repo but not need it as a submodule. There's a whole umbrella of similar-ish utilities. The one that I think does almost the same thing is: https://github.com/mullvad/git-carbon https://github.com/mullvad/git-carbon
  - bellowsgulch 13d ago
    Yes, basically this. I think git-fetch-file has been used in some larger projects based on Googling it, so it seems to be different in that it's been vetted by other orgs, but like Mullvad, theirs is built by a corporation.
- bellowsgulch 13d ago
  It seems like something more in-between. Like a git-based npm install, but generic to any repository or language. It's also "pointer" based, but like vendoring, it's up to you whether or not you want to keep files in the repository. It also seems to be intentionally built for the use case where you modify something from upstream, because if you do have local changes, it detects them and won't overwrite them without your permission.

byako 14d ago

[dead]

dissent 14d ago

If you can point a package manager at a git repo and use it like package, this is accidental convenience. Source code repositories should be factored as source code, with a build step to transform them into packages. For some languages that means compilation, for others it means transpiling, minifying, or just copying files. The resulting artifact is a different shape, designed for consumption. If you use git for packages, then your repo becomes the package boundary. You no longer have the option of producing multiple packages from one repo, or even one package from multiple repos.

fireflash38 13d ago
> If you use git for packages, then your repo becomes the package boundary. You no longer have the option of producing multiple packages from one repo, or even one package from multiple repos. What does this even mean? I'd argue that having the source to build the thing is more important than the artifacts. Release artifacts are more of a convenience. If the thing doesn't build from the source given, what use is it?
- dissent 13d ago
  Exactly my point. You assume the source is the thing you want to consume, but it isn't. Package registries are much more than a convenience and packages don't necessarily map 1:1 with source code repositories. Nor should they need to. You shouldn't have to build every dependency and care where their source code is. Sure you want the source available, otherwise it's closed source, but ideally you never need to look at it unless you're a contributor. git has really conflated the two concepts.
  - dezgeg 13d ago
    Correct way is to have everything as-if a source dependency, but a globally shared cache system that can automatically provide artifacts if that exact version of source has already been compiled. Any system that can only consume manually built (and even in worst case, only tagged and globally pushed) dependencies is horrible for cross-module development, like large scale refactorings and the like.

deleted 13d ago

[deleted]

Myrmornis 13d ago

A lot of the confusion and criticism surrounding submodules comes from people not understanding that you do not need to learn a lot of new commands to use them: you just cd into the submodule and use it as a normal git repo. Then you cd out and commit the submodule hash change if that's what you want to do. Some things are annoying, like removing them. But in general they're extremely useful and designed basically exactly how you'd want: it's extremely consistent since to a good approximation, a submodule behaves the same as a normal git repo once you're inside it.

IshKebab 13d ago
I disagree, even when you understand them there are still big problems with them: 1. Git doesn't support them very well. Bugs are disappointing frequent. Try changing a submodule to a directory or vice versa and feel the pain. 2. They are often used to split up projects (especially commercially) but this makes testing much harder and cross-repo changes way way harder. Even changes that only touch the submodule become way more tedious because you have to always do another PR to update the pointer. 3. The submodules are referenced by URL, which means you can't easily move or copy projects that have them because they still point to the old submodule location. Sometimes they're even misconfiguration to always use ssh:// or whatever, breaking unauthenticated clones. 4. Switching branches where submodules have been added/removed is a mess. 5. Having to constantly remember `git submodule update --init --recursive` is a right pain. 6. When used as a crap dependency manager you easily end up with duplicates. A project I worked on had 12 copied of a common submodule via transitive dependencies. IMO they're almost always the wrong solution.
- lucasoshiro 13d ago
  > Git doesn't support them very well. Bugs are disappointing frequent I can't remember any "bug" in the submodule machinery. If you find one, report to the Git mailing list, they very receptive. > Having to constantly remember `git submodule update --init --recursive` Set this config: `git config submodule.recurse true` > IMO they're almost always the wrong solution. Yeah... but sometimes they are the only feasible solution, especially when the dependency needs to be shared across projects that use incompatible dependency managers and Git is the only common denominator between them...

jonmoore 13d ago

This appears to be ai-generated based on my impression, pangram's "100% of this text is ai", and the blog's posting history.

DonHopkins 13d ago

So a fully automatic Gatling Footgun.