7 ms·

Taking Go modules for a spin

229 points by 101km 8y ago

deadprogram 8y ago
Thanks, Dave!
ainar-g 8y ago
I am eagerly awaiting the arrival of the module system, but please note, that it's still experimental, so not all fun and rainbows. As of now, the Go issue tracker still has 53 open issues with the "modules" label[1], and Go 1.11 should ship around August. The system works well for most cases, but there are still rough edges, especially around vendoring, undocumented or not-so-well documented behaviours, and features which may or may not be "in the scope" for the module system. Overall, I am trying to be optimistic about the future of Go dependency management, but I am not planning to switch the projects I work on in my company from dep to Go modules until most of those rough edges are either smoothed, or officially recognised as "works as intended", with viable workarounds. [1] https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+label%3Amodules https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+l...
- 101km 8y ago
  Agreed. With that said, some none negligible chunk of the issues you linked fall into cosmetic (better error message), proposals, works as intended (docs could improve) or self inflicted misconfigurations of running a go beta. 53 sounds worse than it is. I don't know why this had so much drama around it to be honest. And yes, it is far from perfect, it will be approximately as crummy as python/ruby/node which is still an improvement.
  - deleted 8y ago
    [deleted]
chipaca 8y ago
Given it's 18.04, you could've `snap install go --channel=edge` to get go from master. (all praise mwhudson for maintaining the Go snaps -- `snap info go` for the whole story).
- codetrotter 8y ago
  > `snap info go` for the whole story It lists the channels and some versions, and has a short description; "This snap provides an assembler, compiler, linker, and compiled libraries for the Go programming language." When you said "the whole story" I expected there to be some sort of story but I guess I might have misunderstood what you meant.
- e12e 8y ago
  I've recently revisited asdf, and based on some testing, started moving my various compilers/interpreters to that. It's a "general" version manager - that works like rbenv for "all" languages. I'm not sure if I'd use it for deployment - but for development it's quite versatile. https://github.com/asdf-vm/asdf https://github.com/asdf-vm/asdf
bsaul 8y ago
I haven’t followed the history of the feature, but could anyone explain why we had to wait until version 11 to be able to build a project from any location we wanted ? I’m having a hard time believing it’s purely for technical reasons, but then i don’t understand either what has changed that makes it a desirable feature now rather than before.
- cyphar 8y ago
  Go has always had a very strong dependency on GOPATH and the other path-based semantics that are quite core to the language's import system, so it's understandable that a large change would be quite hard to pull off (for political reasons if nothing else). Personally (having been burned by this mis-feature regularly ever since I started using Go almost 5 years ago) I don't really like this design, but to quote George Lucas "it's stylistically designed to be that way".
- jakoblorz 8y ago
  Go has always been extensively opinionated (gofmt is just one example of that). This is also the beauty of Go - everything is just as expected once you understand the "opinions". The rather late support of "out-of-tree" building might be caused by a conflict between groups pressing to refrain from the $GOPATH approach (rising in size due to rising Go popularity itself) and Go Dev Team / early-adopters especially as the $GOPATH approach is a central part of Go. While I can see the intention of Go modules, I believe that many people which belong to the first group migrated from other languages like JavaScript and expect to migrate the workflow as well. I highly encourage everyone to try the $GOPATH approach before rooting for any side.
  - bsaul 8y ago
    I'm having a hard time believing the go core team changed its mind upon popular pressure only. Are you sure there wasn't a real-life painful scenario that appeared or grew in importance , and convinced them an alternative system had to be provided ?
    - jen20 8y ago
hellcow 8y ago
I've been using go modules in my company for several months now. Everything has "just worked." It's at least 1 order of magnitude faster than dep. It's worth noting that go modules use a novel dependency resolution algorithm which is extremely simple to reason about/implement, fast, and produces more reliable builds than npm/bundler/cargo. That's why I was excited about it, anyway. It removes the ever-present NP-complete assumptions in this space, so from a computer science perspective it's extremely interesting.
- weberc2 8y ago
  Can you elaborate on why it’s easier to reason about than Cargo et al? I’ve heard a lot of theoretical criticism of Go modules for not taking Cargo’s approach so I’m surprised to hear an experience report to the contrary.
  - hellcow 8y ago
    I can't do much justice to the topic in a HN post, but this post specifically describes Go module's dependency resolution algorithm and compares it against SAT solvers like Cargo's: https://research.swtch.com/vgo-mvs https://research.swtch.com/vgo-mvs. Click the "Go & Versioning" link in the header, and you'll find a whole series explaining the design decisions.
  - dilap 8y ago
    From cargo's source: "Actually solving a constraint graph is an NP-hard problem. This algorithm is basically a nice heuristic to make sure we get roughly the best answer most of the time." (https://github.com/rust-lang/cargo/blob/master/src/cargo/core/resolver/mod.rs https://github.com/rust-lang/cargo/blob/master/src/cargo/cor...) vgo's more-constrained specification for dependencies means there is exactly one right answer, and it can be easily and quickly calculated by both computer and human. Whether or not this will turn out to matter in practice is still an open question.
    - flurrything 8y ago
      Is there a catch to vgo's approach? If not, why aren't the cargo people copying it?
azhenley 8y ago
I recently started to learn Go and everything has gone smoothly... except the bizarre, unfriendly GOPATH and forced project structure. It was not fun when VS Code (with the Go plugin) would automatically remove my imports every time I saved the file because it couldn’t find it.
- weberc2 8y ago
  I like the consistent, sane project structure, but GOPATH is an odd duck to be sure.
  - majewsky 8y ago
    I actually started using the GOPATH structure for all my repos after discovering three full checkouts of github.com/torvalds/linux [1] on my drive. [1] I know it's not the official upstream remote, but I find this one the easiest to remember.
    - weberc2 8y ago
      I do the same, but I don't like that it is searched by default when building a Go program. It's too easy for the versions of dependencies in those directories to have changed. And even if you vendor, it falls back to GOPATH so you can still bring things into your build by accident.
- neuland 8y ago
  Removing imports sounds like a gofmt issue. That is the linter / formatter for all Go code. If you don't run the linter and try to build the code, it won't work anyway. For example, I added the net/http package to a file without using it and got this error when running `go build` without doing `go fmt` first: ./example.go:8:9: imported and not used: "net/http" So VS Code's Go plugin removes the unused import because otherwise the code is invalid. I'm a vim user and `vim-go` has the same behavior.
dilap 8y ago
Huh, vendor/ won't be honored unless you're in GOPATH? Anyone know any more about this? Seems like an odd choice.
- ainar-g 8y ago
  See "go help modules". Modules and vendoring When using modules, the go command completely ignores vendor directories. By default, the go command satisfies dependencies by downloading modules from their sources and using those downloaded copies (after verification, as described in the previous section). To allow interoperation with older versions of Go, or to ensure that all files used for a build are stored together in a single file tree, 'go mod -vendor' creates a directory named vendor in the root directory of the main module and stores there all the packages from dependency modules that are needed to support builds and tests of packages in the main module. To build using the main module's top-level vendor directory to satisfy dependencies (disabling use of the usual network sources and local caches), use 'go build -getmode=vendor'. Note that only the main module's top-level vendor directory is used; vendor directories in other locations are still ignored.
  - dilap 8y ago
    Thx. OK, so that makes it sound like it has nothing to do with GOPATH. But interesting (annoying?) that vendor won't kick in unless you specifically ask for it.
    - ainar-g 8y ago
      So far, I get a feeling that vendoring will be a second-class citizen in the world of modules. Unfortunately, I will not be surprised if the team considers removing the support entirely in a future release. I just hope that a viable alternative will be proposed.
      - dilap 8y ago
        yeah, i feel like it's really handy to not be reliant on 3p servers to do a build.
irq-1 8y ago
Go is forever changing the pathing and configuration. Even here we have: > Very nice, go build ignored the vendor/ folder in this repository (because we’re outside $GOPATH) and > Oddly these are stored in $HOME/go/src/mod not the $GOCACHE variable that was added in Go 1.10 Maybe Go 2.0 will be stable.
- lclarkmichalek 8y ago
  I mean, the point of modules is to get rid of vendor. And if you were building outside of GOPATH previously, you were likely broken (or using a specialized build tool that handled this for you). The second one is because it's a cache. Storing them there isn't a breaking change, it's just a bit odd
  - majewsky 8y ago
    > you were likely broken (or using a specialized build tool that handled this for you) Or you're using the isolated-GOPATH trick. $ mkdir -p .gopath/src/github.com/foo $ ln -s ../../../.. .gopath/src/github.com/foo/bar $ GOPATH=$PWD/.gopath go install github.com/foo/bar For example, this is one of my Go projects where you can unpack the release tarball anywhere and `make && make install` just works: https://github.com/sapcc/swift-http-import https://github.com/sapcc/swift-http-import
deleted 8y ago
[deleted]

229 points by 101km 8y ago

Thanks, Dave!

I am eagerly awaiting the arrival of the module system, but please note, that it's still experimental, so not all fun and rainbows. As of now, the Go issue tracker still has 53 open issues with the "modules" label[1], and Go 1.11 should ship around August. The system works well for most cases, but there are still rough edges, especially around vendoring, undocumented or not-so-well documented behaviours, and features which may or may not be "in the scope" for the module system. Overall, I am trying to be optimistic about the future of Go dependency management, but I am not planning to switch the projects I work on in my company from dep to Go modules until most of those rough edges are either smoothed, or officially recognised as "works as intended", with viable workarounds. [1] https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+label%3Amodules https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+l...

101km 8y ago
Agreed. With that said, some none negligible chunk of the issues you linked fall into cosmetic (better error message), proposals, works as intended (docs could improve) or self inflicted misconfigurations of running a go beta. 53 sounds worse than it is. I don't know why this had so much drama around it to be honest. And yes, it is far from perfect, it will be approximately as crummy as python/ruby/node which is still an improvement.
- deleted 8y ago
  [deleted]

chipaca 8y ago

Given it's 18.04, you could've `snap install go --channel=edge` to get go from master. (all praise mwhudson for maintaining the Go snaps -- `snap info go` for the whole story).

codetrotter 8y ago
> `snap info go` for the whole story It lists the channels and some versions, and has a short description; "This snap provides an assembler, compiler, linker, and compiled libraries for the Go programming language." When you said "the whole story" I expected there to be some sort of story but I guess I might have misunderstood what you meant.
e12e 8y ago
I've recently revisited asdf, and based on some testing, started moving my various compilers/interpreters to that. It's a "general" version manager - that works like rbenv for "all" languages. I'm not sure if I'd use it for deployment - but for development it's quite versatile. https://github.com/asdf-vm/asdf https://github.com/asdf-vm/asdf

bsaul 8y ago

I haven’t followed the history of the feature, but could anyone explain why we had to wait until version 11 to be able to build a project from any location we wanted ? I’m having a hard time believing it’s purely for technical reasons, but then i don’t understand either what has changed that makes it a desirable feature now rather than before.

cyphar 8y ago
Go has always had a very strong dependency on GOPATH and the other path-based semantics that are quite core to the language's import system, so it's understandable that a large change would be quite hard to pull off (for political reasons if nothing else). Personally (having been burned by this mis-feature regularly ever since I started using Go almost 5 years ago) I don't really like this design, but to quote George Lucas "it's stylistically designed to be that way".
jakoblorz 8y ago
Go has always been extensively opinionated (gofmt is just one example of that). This is also the beauty of Go - everything is just as expected once you understand the "opinions". The rather late support of "out-of-tree" building might be caused by a conflict between groups pressing to refrain from the $GOPATH approach (rising in size due to rising Go popularity itself) and Go Dev Team / early-adopters especially as the $GOPATH approach is a central part of Go. While I can see the intention of Go modules, I believe that many people which belong to the first group migrated from other languages like JavaScript and expect to migrate the workflow as well. I highly encourage everyone to try the $GOPATH approach before rooting for any side.
- bsaul 8y ago
  I'm having a hard time believing the go core team changed its mind upon popular pressure only. Are you sure there wasn't a real-life painful scenario that appeared or grew in importance , and convinced them an alternative system had to be provided ?
  - jen20 8y ago

hellcow 8y ago

I've been using go modules in my company for several months now. Everything has "just worked." It's at least 1 order of magnitude faster than dep. It's worth noting that go modules use a novel dependency resolution algorithm which is extremely simple to reason about/implement, fast, and produces more reliable builds than npm/bundler/cargo. That's why I was excited about it, anyway. It removes the ever-present NP-complete assumptions in this space, so from a computer science perspective it's extremely interesting.

weberc2 8y ago
Can you elaborate on why it’s easier to reason about than Cargo et al? I’ve heard a lot of theoretical criticism of Go modules for not taking Cargo’s approach so I’m surprised to hear an experience report to the contrary.
- hellcow 8y ago
  I can't do much justice to the topic in a HN post, but this post specifically describes Go module's dependency resolution algorithm and compares it against SAT solvers like Cargo's: https://research.swtch.com/vgo-mvs https://research.swtch.com/vgo-mvs. Click the "Go & Versioning" link in the header, and you'll find a whole series explaining the design decisions.
- dilap 8y ago
  From cargo's source: "Actually solving a constraint graph is an NP-hard problem. This algorithm is basically a nice heuristic to make sure we get roughly the best answer most of the time." (https://github.com/rust-lang/cargo/blob/master/src/cargo/core/resolver/mod.rs https://github.com/rust-lang/cargo/blob/master/src/cargo/cor...) vgo's more-constrained specification for dependencies means there is exactly one right answer, and it can be easily and quickly calculated by both computer and human. Whether or not this will turn out to matter in practice is still an open question.
  - flurrything 8y ago
    Is there a catch to vgo's approach? If not, why aren't the cargo people copying it?

azhenley 8y ago

I recently started to learn Go and everything has gone smoothly... except the bizarre, unfriendly GOPATH and forced project structure. It was not fun when VS Code (with the Go plugin) would automatically remove my imports every time I saved the file because it couldn’t find it.

weberc2 8y ago
I like the consistent, sane project structure, but GOPATH is an odd duck to be sure.
- majewsky 8y ago
  I actually started using the GOPATH structure for all my repos after discovering three full checkouts of github.com/torvalds/linux [1] on my drive. [1] I know it's not the official upstream remote, but I find this one the easiest to remember.
  - weberc2 8y ago
    I do the same, but I don't like that it is searched by default when building a Go program. It's too easy for the versions of dependencies in those directories to have changed. And even if you vendor, it falls back to GOPATH so you can still bring things into your build by accident.
neuland 8y ago
Removing imports sounds like a gofmt issue. That is the linter / formatter for all Go code. If you don't run the linter and try to build the code, it won't work anyway. For example, I added the net/http package to a file without using it and got this error when running `go build` without doing `go fmt` first: ./example.go:8:9: imported and not used: "net/http" So VS Code's Go plugin removes the unused import because otherwise the code is invalid. I'm a vim user and `vim-go` has the same behavior.

dilap 8y ago

Huh, vendor/ won't be honored unless you're in GOPATH? Anyone know any more about this? Seems like an odd choice.

ainar-g 8y ago
See "go help modules". Modules and vendoring When using modules, the go command completely ignores vendor directories. By default, the go command satisfies dependencies by downloading modules from their sources and using those downloaded copies (after verification, as described in the previous section). To allow interoperation with older versions of Go, or to ensure that all files used for a build are stored together in a single file tree, 'go mod -vendor' creates a directory named vendor in the root directory of the main module and stores there all the packages from dependency modules that are needed to support builds and tests of packages in the main module. To build using the main module's top-level vendor directory to satisfy dependencies (disabling use of the usual network sources and local caches), use 'go build -getmode=vendor'. Note that only the main module's top-level vendor directory is used; vendor directories in other locations are still ignored.
- dilap 8y ago
  Thx. OK, so that makes it sound like it has nothing to do with GOPATH. But interesting (annoying?) that vendor won't kick in unless you specifically ask for it.
  - ainar-g 8y ago
    So far, I get a feeling that vendoring will be a second-class citizen in the world of modules. Unfortunately, I will not be surprised if the team considers removing the support entirely in a future release. I just hope that a viable alternative will be proposed.
    - dilap 8y ago
      yeah, i feel like it's really handy to not be reliant on 3p servers to do a build.

irq-1 8y ago

Go is forever changing the pathing and configuration. Even here we have: > Very nice, go build ignored the vendor/ folder in this repository (because we’re outside $GOPATH) and > Oddly these are stored in $HOME/go/src/mod not the $GOCACHE variable that was added in Go 1.10 Maybe Go 2.0 will be stable.

lclarkmichalek 8y ago
I mean, the point of modules is to get rid of vendor. And if you were building outside of GOPATH previously, you were likely broken (or using a specialized build tool that handled this for you). The second one is because it's a cache. Storing them there isn't a breaking change, it's just a bit odd
- majewsky 8y ago
  > you were likely broken (or using a specialized build tool that handled this for you) Or you're using the isolated-GOPATH trick. $ mkdir -p .gopath/src/github.com/foo $ ln -s ../../../.. .gopath/src/github.com/foo/bar $ GOPATH=$PWD/.gopath go install github.com/foo/bar For example, this is one of my Go projects where you can unpack the release tarball anywhere and `make && make install` just works: https://github.com/sapcc/swift-http-import https://github.com/sapcc/swift-http-import

deleted 8y ago

[deleted]