7 ms·

Found this a lot easier to follow. https://blog.howardjohn.info/posts/go-tools-command/ https://blog.howardjohn.info/posts/go-tools-command/ And didn't quite u

Found this a lot easier to follow. https://blog.howardjohn.info/posts/go-tools-command/ https://blog.howardjohn.info/posts/go-tools-command/ And didn't quite understand the euphoria.

0x696C6961 2y ago
Yeah it's a nice QOL improvement. Not some game changer ...
- deleted 2y ago
  [deleted]
Groxx 2y ago
"Shared dependency state" was my very first thought when I heard about how it was built. Yeah I want none of that. I'll stick with my makefiles and a dedicated "internal/tools" module. Tools routinely force upgrades that break other things, and allowing that is a feature.
- nikolayasdf123 2y ago
  same. tools are not part of the codebase, nor dependencies. you got to have isolation of artefact and tools around to work with it. it is bonkers to start versioning tools used to build project mixed with artefact dependencies itself. should we include version of VSCode used to type code? how about transitive dependencies of VSCode? how about OS itself to edit files? how about version of LLM model that generated some of this code? where does this stop?
  - rob74 2y ago
    No one says that this is a one-size-fits-all solution, but for some use cases (small tools that are intimately connected to the rest of the codebase, even reuse some internal code/libraries) it's probably helpful...
  - MathMonkeyMan 2y ago
    The state of things with some projects I've touched is "We have a giant CI thing that does a bunch of things. It is really very large. It might or might not be broken, but we work around it and it's fine." I think some of the euphoria around tracking tooling in the repo is "yes, now I can run a command in the repo and it's as if I spun up a docker container with the CI locally, but it's just regular software running on my machine!" This is a huge improvement if you're used to the minutes-or-hours-long CI cycle being your only interaction with the "real environment." The reductio ad absurdum that you describe is basically "a snapshot of the definitions of all the container images in our CI pipeline." It's not a ridiculous example, it's how many large projects run.
    - nikolayasdf123 2y ago
      I am with you on the same boat that this better be versioned and reproducible and standardised. my key concern whether tools to build project have to be in the same pool as project itself (that may or may not use tools to build/edit/maintain/debug it).
    - SOLAR_FIELDS 2y ago
      It makes sense to some extent when the toolchain can tap into native language specific constructs when you need that REPL-like iteration loop to be tight and fast. But that kind of thing is probably only required in a small subset of the kind of tooling that gets implemented. The tradeoff with this approach is that you lose any sort of agnosticism when you drop into the language specific tooling. So now if you work at a corporation and have to deal with multiple toolchains every engineer now needs to learn and work with new build tooling X times for each supported language. This always happens to some extent - there’s always going to be some things that use the language’s specific task runner constructs - but keeping that minimal is usually a good idea in this scenario. Your complaint feels to me that it is about poorly implemented CI systems that heavily leverage container based workflows (of which there are many in the wild). If implemented properly with caching, really the main overhead you are paying in these types of setups is the virtualization overhead (on macs) and the cold start time for the engine. For most people and in most cases neither will make a significant difference in the wall clock time of their loop, comparatively.
  - Ferret7446 2y ago
    This take absolute boggles the mind. You don't want people compiling your code with different versions of tools so you have to debug thousands of potential combinations of everything. You don't want people running different versions of formatters/linters that leave conflicting diffs throughout your commit history.
    - nikolayasdf123 2y ago
      so where does it stop? let's include version of OS on laptop of people who edit code? it is getting ridiculous. you got to draw a line somewhere. in my opinion, "if dependency code is not linked nor compiled-into nor copied as a source (e.g. model weights, or other artefacts) then it must not be included into dependency tree of project source code" that still means, you are free to track versions/hashes/etc. of tools and their dependencies. just do it separately.
      - arccy 2y ago
        yes let's have a meta project that can track the version of my tools "separately", and the version of my repo. linters, formatters, reproducible codegen should be tracked. their output is deterministic and you want to enforce that in CI in case people forget. the rest doesn't really affect the code (well windows OS and their CRLF do but git has eol attributes to control that).
        nikolayasdf123 2y ago
        agree it has to be deterministic. my major concern was whether tools dependencies are mixed with actual software they are used to build. hopefully they are not mixed, so that you can have guarantees that everything you see in dependency tree is actually used. because otherwise, there is not much point in the tree (since you can't say if it is used or not). again, v1.24 seems to be okay here. go mod pruning should keep nodes in tree clear from polluting each other.
      - broken-kebab 2y ago
        It's not that uncommon to have OS version fixed in CI/CD pipelines. E.g. a build process intended to produce artefacts for Apple's Appstore is dependent on XCode, and XCode maybe forced to upgrade in case of MacOS upgrade, and it may break things. So the OS version becomes a line in requirements. It's kinda disappointing, but it's the real state of affairs.
- 0x696C6961 2y ago
  The way it's implemented is the way that almost everyone already does it. It's just more convenient now.
  - deleted 2y ago
    [deleted]
  - Groxx 2y ago
    "Popular" and "good" have no relation to each other. They correlate fairly well, but that's all. Blending those dependencies already causes somewhat frequent problems for library owners / for users of libraries that do this. Encouraging it is not what I would consider beneficial.
mook 2y ago
There's also the (draft) release notes: https://go.dev/doc/go1.24 https://go.dev/doc/go1.24 And the docs: https://go.dev/doc/modules/managing-dependencies#tools https://go.dev/doc/modules/managing-dependencies#tools I've done the blank import thing before, it was kinda awkward but not _that_ bad.
gr4vityWall 2y ago
The feature itself seems reasonable and useful. Specially if most of your tooling is written is Go as well. But this part caught my attention: > user defined tools are compiled each time they are used Why compile them each time they are used? Assuming you're compiling them from source, shouldn't they be compiled once, and then have the 'go tool' command reuse the same binaries? I don't see why it compiles them at the time you run the tool, rather than when you're installing dependencies. The benchmarks show a significant latency increase. The author also provided a different approach which doesn't seem to have any obvious downsides, besides not sharing dependency versions (which may or may not be a good thing - that's a separate discussion IMO).
- kbolino 2y ago
  There is a cache and they're aren't re-compiled unless they change or the cache is cleared.
  - jamietanna 2y ago
    cached* as of Go 1.24, prior to that they were re-compiled each time
    - kbolino 2y ago
      Ok, I'm trying to suss out what this means, since `go tool` didn't even exist before 1.24. The functionality of `go tool` seems to build on the existing `go run` and the latter already uses the same package compilation cache as `go build`. Subsequent invocations of `go run X` were notably faster than the first invocation long before 1.24. However, it seems that the final executable was never cached before, but now it will be as of 1.24. This benefits both `go run` and `go tool`. However, this raises the question: do the times reported in the blog reflect the benefit of executable caching, or were they collected before that feature was implemented (and/or is it not working properly)?
      - howardjohn 2y ago
        (Author of the blog here (the on in this thread, not the original submission)) Great question! I wasn't aware of the 1.24 optimization when I first wrote the post, but was interested to dig into this deeper. I updated the post: https://blog.howardjohn.info/posts/go-tools-command/#digging-into-timings https://blog.howardjohn.info/posts/go-tools-command/#digging.... The tl;dr is...: 1. The new caching impacts the new `go tool` and the existing `go run`. 2. It has a massive benefit. 3. `go tool` is a bit faster than `go run` due to skipping some module resolution phases. 4. Caching is still (relatively) slow for large packages
pragma_x 2y ago
Ah, okay. So this achieves yet more feature parity with npm/yarn? Sweet.

Found this a lot easier to follow. https://blog.howardjohn.info/posts/go-tools-command/ https://blog.howardjohn.info/posts/go-tools-command/ And didn't quite understand the euphoria.

0x696C6961 2y ago

Yeah it's a nice QOL improvement. Not some game changer ...

deleted 2y ago
[deleted]

Groxx 2y ago

"Shared dependency state" was my very first thought when I heard about how it was built. Yeah I want none of that. I'll stick with my makefiles and a dedicated "internal/tools" module. Tools routinely force upgrades that break other things, and allowing that is a feature.

nikolayasdf123 2y ago
same. tools are not part of the codebase, nor dependencies. you got to have isolation of artefact and tools around to work with it. it is bonkers to start versioning tools used to build project mixed with artefact dependencies itself. should we include version of VSCode used to type code? how about transitive dependencies of VSCode? how about OS itself to edit files? how about version of LLM model that generated some of this code? where does this stop?
- rob74 2y ago
  No one says that this is a one-size-fits-all solution, but for some use cases (small tools that are intimately connected to the rest of the codebase, even reuse some internal code/libraries) it's probably helpful...
- MathMonkeyMan 2y ago
  The state of things with some projects I've touched is "We have a giant CI thing that does a bunch of things. It is really very large. It might or might not be broken, but we work around it and it's fine." I think some of the euphoria around tracking tooling in the repo is "yes, now I can run a command in the repo and it's as if I spun up a docker container with the CI locally, but it's just regular software running on my machine!" This is a huge improvement if you're used to the minutes-or-hours-long CI cycle being your only interaction with the "real environment." The reductio ad absurdum that you describe is basically "a snapshot of the definitions of all the container images in our CI pipeline." It's not a ridiculous example, it's how many large projects run.
  - nikolayasdf123 2y ago
    I am with you on the same boat that this better be versioned and reproducible and standardised. my key concern whether tools to build project have to be in the same pool as project itself (that may or may not use tools to build/edit/maintain/debug it).
  - SOLAR_FIELDS 2y ago
    It makes sense to some extent when the toolchain can tap into native language specific constructs when you need that REPL-like iteration loop to be tight and fast. But that kind of thing is probably only required in a small subset of the kind of tooling that gets implemented. The tradeoff with this approach is that you lose any sort of agnosticism when you drop into the language specific tooling. So now if you work at a corporation and have to deal with multiple toolchains every engineer now needs to learn and work with new build tooling X times for each supported language. This always happens to some extent - there’s always going to be some things that use the language’s specific task runner constructs - but keeping that minimal is usually a good idea in this scenario. Your complaint feels to me that it is about poorly implemented CI systems that heavily leverage container based workflows (of which there are many in the wild). If implemented properly with caching, really the main overhead you are paying in these types of setups is the virtualization overhead (on macs) and the cold start time for the engine. For most people and in most cases neither will make a significant difference in the wall clock time of their loop, comparatively.
- Ferret7446 2y ago
  This take absolute boggles the mind. You don't want people compiling your code with different versions of tools so you have to debug thousands of potential combinations of everything. You don't want people running different versions of formatters/linters that leave conflicting diffs throughout your commit history.
  - nikolayasdf123 2y ago
    so where does it stop? let's include version of OS on laptop of people who edit code? it is getting ridiculous. you got to draw a line somewhere. in my opinion, "if dependency code is not linked nor compiled-into nor copied as a source (e.g. model weights, or other artefacts) then it must not be included into dependency tree of project source code" that still means, you are free to track versions/hashes/etc. of tools and their dependencies. just do it separately.
    - arccy 2y ago
      yes let's have a meta project that can track the version of my tools "separately", and the version of my repo. linters, formatters, reproducible codegen should be tracked. their output is deterministic and you want to enforce that in CI in case people forget. the rest doesn't really affect the code (well windows OS and their CRLF do but git has eol attributes to control that).
      - nikolayasdf123 2y ago
        agree it has to be deterministic. my major concern was whether tools dependencies are mixed with actual software they are used to build. hopefully they are not mixed, so that you can have guarantees that everything you see in dependency tree is actually used. because otherwise, there is not much point in the tree (since you can't say if it is used or not). again, v1.24 seems to be okay here. go mod pruning should keep nodes in tree clear from polluting each other.
    - broken-kebab 2y ago
      It's not that uncommon to have OS version fixed in CI/CD pipelines. E.g. a build process intended to produce artefacts for Apple's Appstore is dependent on XCode, and XCode maybe forced to upgrade in case of MacOS upgrade, and it may break things. So the OS version becomes a line in requirements. It's kinda disappointing, but it's the real state of affairs.
0x696C6961 2y ago
The way it's implemented is the way that almost everyone already does it. It's just more convenient now.
- deleted 2y ago
  [deleted]
- Groxx 2y ago
  "Popular" and "good" have no relation to each other. They correlate fairly well, but that's all. Blending those dependencies already causes somewhat frequent problems for library owners / for users of libraries that do this. Encouraging it is not what I would consider beneficial.

mook 2y ago

There's also the (draft) release notes: https://go.dev/doc/go1.24 https://go.dev/doc/go1.24 And the docs: https://go.dev/doc/modules/managing-dependencies#tools https://go.dev/doc/modules/managing-dependencies#tools I've done the blank import thing before, it was kinda awkward but not _that_ bad.

gr4vityWall 2y ago

The feature itself seems reasonable and useful. Specially if most of your tooling is written is Go as well. But this part caught my attention: > user defined tools are compiled each time they are used Why compile them each time they are used? Assuming you're compiling them from source, shouldn't they be compiled once, and then have the 'go tool' command reuse the same binaries? I don't see why it compiles them at the time you run the tool, rather than when you're installing dependencies. The benchmarks show a significant latency increase. The author also provided a different approach which doesn't seem to have any obvious downsides, besides not sharing dependency versions (which may or may not be a good thing - that's a separate discussion IMO).

kbolino 2y ago
There is a cache and they're aren't re-compiled unless they change or the cache is cleared.
- jamietanna 2y ago
  cached* as of Go 1.24, prior to that they were re-compiled each time
  - kbolino 2y ago
    Ok, I'm trying to suss out what this means, since `go tool` didn't even exist before 1.24. The functionality of `go tool` seems to build on the existing `go run` and the latter already uses the same package compilation cache as `go build`. Subsequent invocations of `go run X` were notably faster than the first invocation long before 1.24. However, it seems that the final executable was never cached before, but now it will be as of 1.24. This benefits both `go run` and `go tool`. However, this raises the question: do the times reported in the blog reflect the benefit of executable caching, or were they collected before that feature was implemented (and/or is it not working properly)?
    - howardjohn 2y ago
      (Author of the blog here (the on in this thread, not the original submission)) Great question! I wasn't aware of the 1.24 optimization when I first wrote the post, but was interested to dig into this deeper. I updated the post: https://blog.howardjohn.info/posts/go-tools-command/#digging-into-timings https://blog.howardjohn.info/posts/go-tools-command/#digging.... The tl;dr is...: 1. The new caching impacts the new `go tool` and the existing `go run`. 2. It has a massive benefit. 3. `go tool` is a bit faster than `go run` due to skipping some module resolution phases. 4. Caching is still (relatively) slow for large packages

pragma_x 2y ago

Ah, okay. So this achieves yet more feature parity with npm/yarn? Sweet.