Go: cmd/go: go get should not add a dependency to go.mod

[natefinch]

go get is not the right command for adding a dependency to the current module. We have a command for dealing with modules, and it's go mod, that's the command that should be used to add dependencies.

This is the first paragraph of help on go get:

Get downloads the packages named by the import paths, along with their dependencies. It then installs the named packages, like 'go install'.

None of that says "add the library you specified to the dependencies of the current project". And why would it? it's for getting code from somewhere else and bringing it to this machine.

Notably, go get will even add go commands as a dependency... so like if you run go get golang.org/x/tools/cmd/goimports and your current directory happens to be a go module, guess what? Your go.mod now has

golang.org/x/tools v0.0.0-20180911133044-677d2ff680c1 // indirect

This UX is extremely confusing for the ~2 million people who used go before modules. It's also way too implicit and magic regardless of the experience. Adding a dependency to your project should be an explicit action:

go mod add golang.org/x/tools/cmd/goimports  

What did you do?

from inside a module directory

$ ls
go.mod
$ go get github.com/natefinch/lumberjack
go: finding github.com/natefinch/lumberjack latest
go: downloading github.com/natefinch/lumberjack v0.0.0-20180817145747-7d6a1875575e
~/dev/test$ more go.mod
module app

require github.com/natefinch/lumberjack v0.0.0-20180817145747-7d6a1875575e // indirect

What did you expect to see?

No change to my go.mod

What did you see instead?

go get added a library to my go.mod file.

System details

go version go1.11 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/finchnat/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/finchnat"
GOPROXY=""
GORACE=""
GOROOT="/Users/finchnat/sdk/go1.11"
GOTMPDIR=""
GOTOOLDIR="/Users/finchnat/sdk/go1.11/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
GOROOT/bin/go version: go version go1.11 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.11
uname -v: Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64
ProductName:    Mac OS X
ProductVersion: 10.13.6
BuildVersion:   17G65
lldb --version: lldb-902.0.79.7
  Swift-4.1

[myitcv]

From the bottom of go help get:

...

This text describes the behavior of get when using GOPATH
to manage source code and dependencies.
If instead the go command is running in module-aware mode,
the details of get's flags and effects change, as does 'go help get'.
See 'go help modules' and 'go help module-get'.

From go help module-get:

The 'go get' command changes behavior depending on whether the
go command is running in module-aware mode or legacy GOPATH mode.
This help text, accessible as 'go help module-get' even in legacy GOPATH mode,
describes 'go get' as it operates in module-aware mode.

...

[myitcv]

@natefinch I think the documentation on this covers the points you raise, but please do say if you think it could be better signposted.

cc @bcmills

[natefinch]

It's not about the documentation. It's a complete change to what go get has meant for the last 6+ years. Which has meant "download and compile some code so I have it locally". That's still a useful command to have in the universe of go modules.

Changing what an established command does, in such an important way, is a really bad idea.

I think it's pretty clear, if I go get golang.org/x/tools/cmd/goimports ... what I mean is "download and compile goimports, and put it somewhere for general use". The fact that this command also stealthfully adds golang.org/x/tools/ to my current directory's dependencies is surprising and extremely unwelcome.

Like I said in the bug report... we have a command for dealing with modules. Why would we not just add this functionality there? go mod add github.com/... is perfectly clear, and wouldn't clash with current usage.

[komuw]

if I go get golang.org/x/tools/cmd/goimports ... what I mean is "download and compile goimports, and put it somewhere for general use".

I think this is; https://github.com/golang/go/issues/25922 and https://github.com/golang/go/issues/24250

[cespare]

I agree with this. I've already been surprised three separate times now by running go get and having tools unexpectedly added to my go.mod, or having a new go.mod file dropped into a directory where I didn't intend.

Before modules, go get was used for two pretty distinct use cases: (1) downloading tools and (2) downloading libraries. It seems much, much less surprising and user-friendly for post-modules go get to specialize on use case (1) and leave (2) for other tools. In my experience with modules so far, I've handled (2) by simply adding new imports and running go test/go build.

[myitcv]

I think it's pretty clear, if I go get golang.org/x/tools/cmd/goimports ... what I mean is "download and compile goimports, and put it somewhere for general use".

In a GOPATH world I would agree. But we're talking a modules world here. From my perspective I'm fully anticipating any go command to do something with my go.mod.

Put another way (and this is where https://github.com/golang/go/issues/24250 comes in), I would expect, in a modules world, to have to do something "special" in order to performa a "global" install (by definition outside the context of my go.mod)

[vdemario]

I was also surprised by this behavior. The suggestion that go get will have a different meaning in a "modules world" doesn't solve the use case of go get golang.org/x/tools/cmd/goimports or installing any other tool. Do I need to change directories every time I need to install a new tool? What if a go.mod is created anywhere I am because in the modules world any place is a valid place for a Go project?

Regardless of these examples, what bothers me the most is that we're using the same command that has existed for several years to do a different thing. Are we all supposed to accept that every tutorial, every doc, everything that has ever been said about go get is not valid anymore?

The global place for installing any library or tool still exists, so why do we need to do something special to get back the behavior that already exists?

[kaedys]

Have to chime in that that I agree with @natefinch and @vdemario here. Using go get for installing _tools_ has been the standard practice since go get was created. Changing that semantic is extremely problematic from a usability perspective, especially since the Go tooling currently lacks a command equivalent to "download this Go library, compile it, and install it somewhere I can use, but do _not_ add it to my current project".

We're adding entirely new behavior via the go mod command. It seems to make the most sense to have explicitly adding a dependency to be a subset of that command, rather than hijacking an existing command and completely obliterating one of the two most common ways that command is currently used.

[vdemario]

Also, apparently trying to install a tool outside a module doesn't even work:

/tmp
$ set -gx GO111MODULE on # fish shell syntax
/tmp
$ go get golang.org/x/tools/cmd/goimports
go: cannot find main module; see 'go help modules'

[myitcv]

@vdemario please see, as linked above, https://github.com/golang/go/issues/24250

[kaedys]

That issue clarifies this one even more. Prior to Go 1.11, the semantics for go get were identical to go install, except that it pulled the code from a remote repository before running go install on them. The module functionality in Go 1.11 has completely abandoned the go install piece (actually, it's still there, but it's mostly an anachronism at this point) because go get has been hijacked for adding a dependency to a module.

And that's a bit of a problem, since go get was validly used just as commonly for the go install functionality as it was for the abstraction of git clone. Abandoning the install functionality is extremely jarring and leads to some very unexpected side-effects, like the necessity of running go mod tidy after installing a tool that's not also a dependency, and as @vdemario just pointed out, the complete inability to install tools unless you're currently in a module (which you then need to tidy).

[myitcv]

@kaedys

The module functionality in Go 1.11 has completely abandoned the go install piece (actually, it's still there, but it's mostly an anachronism at this point) because go get has been hijacked for adding a dependency to a module.

And that's a bit of a problem, since go get was validly used just as commonly for the go install functionality as it was for the abstraction of git clone. Abandoning the install functionality is extremely jarring and leads to some very unexpected side-effects, like the necessity of running go mod tidy after installing a tool that's not also a dependency

go get does still perform an install. Given:

export GOPATH=$(mktemp -d)
cd $(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello
export GOBIN=$PWD/.bin
go get golang.org/x/tools/cmd/stringer

Then:

$ ls $GOBIN
stringer

like the necessity of running go mod tidy after installing a tool that's not also a dependency, and as @vdemario just pointed out, the complete inability to install tools unless you're currently in a module (which you then need to tidy).

This falls under the umbrella of https://github.com/golang/go/issues/24250.

As @komuw linked above, for discussion on the current "best practice" for defining tool dependencies, please see https://github.com/golang/go/issues/25922

[kaedys]

But it's not about tool _dependencies_. It's about tools you want to install on a _global_ level, _unrelated_ to the current module. That's the issue at the root of this, that under the Go module framework, go get is _inherently and unavoidably_ linked to the module you are currently operating on (and completely fails if you're not on a module). There's no reason to abandon the ability to install tools unrelated to the current module (or outside the context of a module), and there's no reason to assume that every package retrieved is necessarily a dependency of the current module, especially given the pre-1.11 semantics and usage of go get.

[myitcv]

It's about tools you want to install on a global level

Indeed, that's why I linked to https://github.com/golang/go/issues/24250.

Just to emphasise what @rsc said in https://github.com/golang/go/issues/24250#issuecomment-377553022:

$ vgo get github.com/golang/dep/cmd/[email protected]
cannot determine module root; please create a go.mod file there

This clearly must work eventually.

(we can now substitute go get for vgo get)

So there is agreement that this "global" concept indeed very much has merit. It's just that this didn't make Go 1.11.

There's no reason to abandon the ability to install tools unrelated to the current module (or outside the context of a module), and there's no reason to assume that every package retrieved is necessarily a dependency of the current module, especially given the pre-1.11 semantics and usage of go get.

Hopefully the link above covers this point?

[natefinch]

Why aren't there go mod add and go mod update commands? That would seem to be the obvious place to go looking for the functionality of adding or updating a dependency. Then there would be no need to overload go get for adding and updating dependencies.

[bcmills]

@natefinch

We have a command for dealing with modules, and it's go mod, that's the command that should be used to add dependencies.

You are mistaken. The go mod command is for module-specific operations, not all operations “dealing with modules”. As its documentation says, “support for modules is built into all the go commands, not just go mod. For example, day-to-day adding, removing, upgrading, and downgrading of dependencies should be done using go get.”

go get in GOPATH mode already performs a grab-bag of tasks: it “downloads the packages named by the import paths, along with their dependencies. It then installs the named packages, like go install.” go get has the same behavior in module mode, except that it downloads entire modules and it does not yet work outside of a module (that's #24250).

In my opinion, it is unfortunate that go get conflates downloading packages with installing binaries, but we're essentially stuck with it: we need to maintain compatibility with existing, documented command lines, and existing command lines don't only use go get to install tools — they also use it to download package dependencies.

[cespare]

@bcmills

[...] existing command lines don't only use go get to install tools — they also use it to download package dependencies.

This is kind of a hobbyist workflow, I suppose, but most serious Go users are not using go get to get package dependencies; they are using vendoring, dep, or some other solution.

OTOH I suspect that many or most Go users make use of the "fetch/update a Go tool" functionality of go get.

[natefinch]

existing command lines don't only use go get to install tools — they also use it to download package dependencies.

....sure. I'm not sure what that has to do with go get foo/bar/baz modifying the local go.mod?

go get foo/bar/baz explicitly tells the go tool "ignore the directory that I'm in. Instead, find this package, make sure it and its dependencies are downloaded locally, build it, and install it."

There's no reason why it couldn't do the exact same thing in the go modules world. You're explicitly telling the go tool not to build the local directory.

I agree that go build (implying go build .) should build the local directory, and update go.mod etc. That's fine. You're explicitly telling it to work on the local directory.

When you give it a package path, though, you're explicitly telling it not to work on the local directory. That's the disconnect.

[vdemario]

I agree that go getin GOPATH mode performs a grab-bag of tasks and I also agree that it is unfortunate that go get conflates downloading packages with installing binaries.

Considering that, why does the module-aware go get also gets the new responsibilities of adding, removing, upgrading, and downgrading of dependencies?

It seems counter-intuitive that a command that already does too much is getting new responsibilities and side effects (such as updating go.mod).

[bcmills]

go get foo/bar/baz explicitly tells the go tool "ignore the directory that I'm in. Instead, find this package, make sure it and its dependencies are downloaded locally, build it, and install it."

But go get -u foo/bar/baz upgrades arbitrarily many dependencies: it may affect the current directory, depending on what else is involved.

As weird as it is for go get of a tool to update the current module but install the tool globally, it seems even weirder for go get -u not to upgrade dependencies depending on the mode, or for go get -u to affect the module where go get would not.

[bcmills]

It seems counter-intuitive that a command that already does too much is getting new responsibilities and side effects (such as updating go.mod).

Every command that loads packages updates go.mod, and go get -u already adds and upgrades dependencies even in GOPATH mode. We're not adding new responsibilities to go get so much as to go globally.

(We did add one new responsibility to go get: it can now downgrade dependencies, whereas the go command couldn't downgrade dependencies at all before.)

[natefinch]

it seems even weirder for go get -u not to upgrade dependencies, or for go get -u to affect the module where go get would not

I feel like this is a disconnect between people who use a single gopath for everything (i.e. google) and people who exclusively use vendored dependencies.

Every serious public project for a long time has used vendored dependencies. go get -u would never update any of these dependencies, because they all use vendoring.

[natefinch]

IMO go get -u <some library> just doesn't really make any sense in the modules world. It's a holdover from gopath. It works for binaries because it updates the binary in gobin.

I'd really prefer if the commands to update dependencies were explicit and obvious named. go update foo/bar or go add foo/bar.

Why hang on to this fairly vague word "get" when we're talking about actions that never existed before? go get has baggage that we can leave behind.

[bcmills]

I feel like this is a disconnect between people who use a single gopath for everything (i.e. google) and people who exclusively use vendored dependencies.

Most Google-internal users pretty much aren't using GOPATH (or the go tool) at all. This isn't a Google-vs.-outside issue so much as a hobbyist-vs.-enterprise one.

Every serious public project for a long time has used vendored dependencies. go get -u would never update any of these dependencies, because they all use vendoring.

Yes, we clearly need to work on the migration path for vendoring. That's one of my top priorities for 1.12.

[bcmills]

I completely agree that get is a weird holdover from the pre-modules world, but I don't think we can leave that baggage behind. Go takes a very serious approach to backward-compatibility, and breaking install instructions that use go get -u would be a pretty big incompatibility.

That said, maybe it's worth leaving the get baggage behind when we're talking about installing tools: for example, go install as it stands is very nearly useless in module mode. Perhaps go install pkg@version should install the named version of the tool without updating the local go.mod, and then anyone who wants to update their instructions can switch over to that.

[natefinch]

go get -u as instructions to install a library have been invalid ever since the widespread use of the vendor directory. I'd be interested to see how many widely used libraries actually put that in their readme.

[natefinch]

This isn't a Google-vs.-outside issue so much as a hobbyist-vs.-enterprise one.

Sorry! I almost deleted the google bit there. definitely was not trying to make this a google vs. the outside world.

[bcmills]

I'd be interested to see how many widely used libraries actually put that in their readme.

I'd be curious to see that too!

For a couple of other data points:

• Google hit # 4 for update go dependencies is this StackOverflow question, for which the “accepted” answer is go get -u — and none of the other answers mention modules or vendoring at all.
  
  
  
  • (The first three hits are dep, a blog post about dep, and #13332, so that's the first hit that actually mentions how to use the go tool itself.)
  
  
• I found almost 500 hits for go get -u in Google's codebase (which includes a mix of scripts, HTML documentation, and READMEs).

(To be clear, none of this contradicts your point that serious public projects use vendoring. Ideally, we want a solution that works for serious projects and hobbyists.)

[natefinch]

To be fair, that question was answered in 2012 :)

And go get -u is fine for binaries, just not really useful for libraries. but separating out which readmes are for binaries vs. libraries is pretty hard, I'd think.

[thepudds]

Here are three quick current examples of instructions using go get -u:

go-redis (https://github.com/go-redis/redis)

Install:

go get -u github.com/go-redis/redis

grpc (https://grpc.io/docs/quickstart/go.html)

Use the following command to install gRPC.

$ go get -u google.golang.org/grpc

aws-sdk (https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/setting-up.html)

Install the AWS SDK for Go
To install the SDK and its dependencies, run the following Go command.

go get -u github.com/aws/aws-sdk-go/...

Or another simple data point is that 3 of the first 5 packages listed under "Popular Packages" on https://godoc.org included go get -u instructions based on quick peek. That might not be representative, though.

Also, those might not be perfect examples (for example, aws-sdk also includes commands, I think), or maybe turns out to be uncommon after a more detailed search (given what I did was just very brief).

In any event, it still might be smart to change the current approach for what go get means, or to add go add or go update. Just wanted to provide some quick examples.

(minor edit, including added simple data point from "Popular Packages")

[bcmills]

To be fair, that question was answered in 2012 :)

A testament to Go 1 compatibility! 🙂

[myitcv]

As weird as it is for go get of a tool to update the current module but install the tool globally,

I actually think the absence of a "default module-local install location" falls into the same category of "missing from Go 1.11" as https://github.com/golang/go/issues/24250. Let me try to explain by covering my "workflow".

The tools I use on a day-to-day basis fall into two categories:

1. Tools I want to be controlled by a project's go.mod
2. Tools that I need globally

Dealing with these in reverse order.

Category 2: I think it's clear that with Go 1.11, there is a "gap" here and that this is covered by https://github.com/golang/go/issues/24250. Per the detail in that discussion, there are open questions on how to handle multiple versions, where the installed binaries should be put etc, but it all falls under that issue.

Category 1: by far the largest category of tools for me, made up largely of code generators that I use with go generate and the like. I absolutely want these to be version controlled. And I _don't_ want to be using (via my PATH) a "global" install of such a tool, even if the version just happens to match at that point in time. But both go get and go install currently (i.e. Go 1.11) have a target of $GOPATH/bin (ignoring multi-element `GOPATH values for now).

Hence the workflow I have effectively adopted is to create a module-local install target:

# create a new module
cd $(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello

# set GOBIN as a module-local install target
export GOBIN=$PWD/.bin

# update my PATH accordingly (I actually use https://github.com/cxreg/smartcd for this)
export PATH=$GOBIN:$PATH

# add a tool dependency (by definition, category 1 tool) following "best practice" laid out in 
# https://github.com/golang/go/issues/25922#issuecomment-412992431
cat <<EOD > tools.go
// +build tools

package tools

import (
        _ "golang.org/x/tools/cmd/stringer"
)
EOD

# install the tool
go install golang.org/x/tools/cmd/stringer

# verify we are using the module-local binary
which stringer

# which gives something like:
# /tmp/tmp.Hh0BNOF6k2/hello/.bin/stringer

As mentioned, one of the disconnects (in this thread) is that a go get or go install in a module context modifies the "local" go.mod but installs "globally". This is, as @bcmills put it, "weird". But is to my mind a gap in Go 1.11, just as there not being a "global" tool install is a gap (i.e. https://github.com/golang/go/issues/24250). Remember, module support is experimental and almost by definition incomplete.

go run is a potential alternative to the "local" install here (and a very attractive one to my mind), but it is a non-starter unless we can find a way to address https://github.com/golang/go/issues/25416.

There could be some convention that a .bin/ directory, alongside a go.mod, is the target for go get and go install (of main packages) for "local" installs? But this wouldn't obviate the need for everyone to update their PATH and indeed .gitignore the .bin directory for every module they work on.

And I'm sure there are many other potential solutions.

But I thought it worth pointing out that I see this as a gap, as opposed to a final decision on workflow.

@bcmills I don't know what your thoughts on this are, and whether it's worth creating a separate issue for the "local" install target/run solution?

[natefinch]

A note on local binaries, https://github.com/twitchtv/retool does exactly what you want, @myitcv ... the only drawback being that you need to run them by running retool do <binary>. So like retool do dep ensure

[bcmills]

I don't know what your thoughts on this are, and whether it's worth creating a separate issue for the "local" install target/run solution?

Yes, please file a separate issue.

(Let's try to stick to one decision per issue so we don't lose track when issues are closed. As I see it, the decision for this issue is “should go get update go.mod?”)

[natefinch]

one more thing I just encountered that is related.

running go build foo/bar/baz in a module directory ... adds foo/bar/baz to the go.mod. That seems really weird. Now, I'll grant you, go build foo/bar/baz in a module context.... sorta doesn't make any sense... in which case maybe we just error out? I don't really think there's any action that makes sense in a module context where foo/bar/baz is not part of the current module.

[mvdan]

Just a quick note - as far as I've seen, it's always been clear that all of cmd/go would be adapted to work with modules. This is why something like go dep ensure would have never been a thing. To me, go mod add or go mod update fall in the same category.

I agree that the semantics of go get and go install may get a bit confusing during this gopath-to-modules transition period. Still, I don't think it's confusing enough to warrant stuffing all of these operations as sub-sub-commands under go mod. That's not a great long-term solution either, in my opinion.

[myitcv]

@bcmills

(Let's try to stick to one decision per issue so we don't lose track when issues are closed. As I see it, the decision for this issue is “should go get update go.mod?”)

Absolutely agree. Was just using that example as a situation where I see a "gap" as opposed to a "decision" as a gentle reminder to everyone that modules are still experimental.

Just opened https://github.com/golang/go/issues/27653

[kaedys]

I agree that the semantics of go get and go install may get a bit confusing during this gopath-to-modules transition period. Still, I don't think it's confusing enough to warrant stuffing all of these operations as sub-sub-commands under go mod. That's not a great long-term solution either, in my opinion.

Ok, but that still leaves us without a convenient equivalent to "ignore my current directory, pull down this repository, compile it, and then install it for global use", which is a semantic that go get is very commonly used for today. The go mod version has _no_ single command equivalent to this use case, the only options are using go get followed by go mod tidy (which can have ancillary side effects, since tidy doesn't respect build tags), or changing directories (also not currently possible, since lack of a go.mod fail is a failure, but even if that's fixed, that's 3 commands, cd -> go get -> cd back), or by manually doing a git clone followed by go install. None of those are very clean.

Perhaps a solution would be to add a flag to go install (-u appears to be unused) to tell it to fetch and update from remote before compiling and installing (and that flag would _require_ that a package be named and it have a resolvable import path). That would bake the usage into another command with better-aligned semantics, would retain close to the same functional flow as go get does, and wouldn't change the default behavior of go install either (unless that ends up being the desired pathway).

[natefinch]

I still think that overloading go get and go install is a mistake. We could just leave those as not modifying go.mod and instead add more explicit commands like go add or go update.

People keep saying that module support is built into the commands and ignoring the fact that the way they're built in is the confusing part.

[thepudds]

Regarding possibly changing the behavior of go get or introducing new commands, one consideration raised is the degree to which some or most historical install instructions can continue to work (as discussed above for this issue, and as you can see that is also a very important consideration for #24250 to make sure go get some/command always works in a module-world, although the best full solution for #24250 might need to wait until modules are on by default -- see for example https://github.com/golang/go/issues/24250#issuecomment-377553022 and https://github.com/golang/go/issues/24250#issuecomment-411860657 comments by Russ, though of course further discussion about that aspect probably deserves to live in #24250).

Another consideration for the "should go get update go.mod?" issue here that I wanted to briefly highlight is what was termed the build "isolation rule", which was described in this blog post back in Feb:

https://research.swtch.com/vgo-cmd

If you are thinking about how to best modify the go command behavior (e.g., as discussed above by altering what go get does, or perhaps introduce a new go add and go update, or ___), it is worth reading that blog post. (It is not too long, and it is the first 2/3 or so that is most relevant to the discussion here).

Some of the key snippets (though still worth reading the post itself):

the 'isolation rule':

The result of a build command should depend only on the source files that are its logical inputs, never on hidden state left behind by previous build commands.)

That is, what a command does in isolation—on a clean system loaded with only the relevant input source files—is what it should do all the time, no matter what else has happened on the system recently.

and:

Of course, all the [new] vgo get variants record the effect of their additions and upgrades in the go.mod file. In a sense, we've made these commands follow the isolation rule by introducing go.mod as an explicit, visible input replaces a previously implicit, hidden input: the state of the entire GOPATH.

and:

every command should have only one meaning, no matter what other commands have preceded it.

and:

Plain [pre-module] go get, without -u, violates the command isolation rule and must be fixed. [The pre-module go get has three meanings]:

• If GOPATH is empty, go get rsc.io/quote downloads and builds the latest version of rsc.io/quote and its dependencies (for example, rsc.io/sampler).
• If there is already a rsc.io/quote in GOPATH, from a go get last year, then the new go get builds the old version.
• If rsc.io/sampler is already in GOPATH but rsc.io/quote is not, then go get downloads the latest rsc.io/quote and builds it against the old copy of rsc.io/sampler.

Overall, [pre-module] go get depends on the state of GOPATH, which breaks the command isolation rule. We need to fix that.

There are different behaviors that could be chosen while still satisfying the build isolation rule when it comes to go get (up to even "eliminate go get entirely and instead introduce new commands X and Y" could satisfy the build isolation rule), but at least wanted to highlight that blog post for people as a design consideration given that post describes some of the philosophy behind having go build by default download code as needed and some of the corresponding current changes in behavior for go get in a modules world.

[myitcv]

@natefinch

https://github.com/twitchtv/retool does exactly what you want

Thanks. I have local scripts/hackery that also get the job done; I guess I'm pushing for the go tool to do this for me and others so that you don't need to install another tool to use tools :)

[bcmills]

@natefinch

running go build foo/bar/baz in a module directory ... adds foo/bar/baz to the go.mod. That seems really weird.

It does seem a bit weird for go build, but much less so for go test. For example, I could imagine the following steps:

• go test example.com/foo/bar (and verify that it passes).
• Read the documentation for example.com/foo/bar in a local godoc (which now shows exactly the version that was just tested).
• Add call to bar.SomeFunction() in a source file (for which goimports adds an import of example.com/foo/bar, preferring it over other forks since it is in an active module).
• go test all to check the remaining tests against the new dependency.

In that workflow, it's useful for go test example.com/foo/bar to update go.mod as a hint to goimports (and other tooling), and it's easy enough to reverse the addition (if desired) by running go mod tidy.

[kaedys]

RE: isolation, I guess the jarring part for me is that, up until now, go get on a _tool_ has had no effect on the current project or repository unless that tool is also an import dependency of that project/repo. In the module system, go get _always_ affects the current module, even if the user is just trying to download and install a globally-useful tool rather than an import dependency for the current module.

To an extent, that's why I favor shifting that functionality to the rather more intuitively related go install with a new flag for remote fetching.

[natefinch]

I have never in 6 years of writing Go, run go test example.com/foo/bar. If I want to twiddle with example.com/foo/bar before I use it, I go get it, switch to that directory, and run local commands there, because it's so much easier (like being able to look at files etc).

Literally the only time I specify an absolute package path is using go get. Every other time, I cd to the directory and use relative package names.... because it just saves so much typing if I have to run more than one command.

In fact, this is one thing I worry about when changing go get. it was super useful for.... getting code. I would propose that we leave go get as it is for expressly that purpose - getting Go code onto your machine so you can look at and fiddle with it as needed (as well as a super easy way to install binaries).

Here's my workflow:

1. google for that package name that I can't quite remember
2. go to the github page, check out the readme, maybe peek at the code to make sure it looks sane
3. click on the godoc.org badge to get a better view of the API
4. run go get on my machine to get the code locally.
5. open that (now local) repo in my editor to look at the code / run tests etc.
6. go back to my project, add the import, add a little code so goimports won't take away the import ;)
7. rebuild my project.

Right now there's usually a 6.5 where I run dep ensure, but I'm happy to have go build do that for me.

The thing about "having modules built into the go tools" is that I expect that only packages the code actually depends on will be added to the go.mod. Adding something my code doesn't explicitly use should only happen through an explicit command that I run that says "yes, I know I'm not actually using this, but add it to my go.mod anyway".

[DeedleFake]

It should be noted that even using GO111MODULE=off go get <import path for a main package> isn't a solution, as this disables the versioning for the dependencies of that package's module. The only workarounds are to either clone the repository manually and use go install, or to navigate to an empty directory somewhere, such as by using cd $(mktemp -d), initialize that as a module, and then run go get.

[iand]

Here's my workflow:

1. google for that package name that I can't quite remember
2. go to the github page, check out the readme, maybe peek at the code to make sure it looks sane
3. click on the godoc.org badge to get a better view of the API
4. run `go get` on my machine to get the code locally.
5. open that (now local) repo in my editor to look at the code / run tests etc.
6. go back to my project, add the import, add a little code so goimports won't take away the import ;)
7. rebuild my project.

I think you just need to add a step before you commit:

1. run go mod tidy

2. commit

[natefinch]

You can't tell me that's not going to get checked in by accident all the time. Especially in large projects with a large list of dependencies.

You shouldn't need to run tidy, because the go tool should never implicitly add a dependency that my code doesn't actually use. There should be an explicit command for that with a name that accurately describes the action taking place.

The problem is that "get" is not the right verb for "please add this dependency to my project even though I'm not actually referencing it in my code". If we were building the go tool from scratch with module support, I don't think "get" is the verb that would be chosen for that command. And since the verb already has baggage from earlier use, I don't know why it should be used now. That seems like it counts against an already inaccurate verb.

And I don't see why a new verb is a problem at all. Yes, there's lots of documentation out there using go get .... if we don't muck with go get, that documentation doesn't have to be incorrect. You could still go get a library, then reference it in your code, and when you build or test, the tool adds it to the go.mod, because you're actually using the code. Basically the same workflow as it is now.

[iand]

@natefinch go mod tidy is going to be needed all the time since it also removes unused dependencies. Otherwise we'd need go mod remove to complement the hypothetical go mod add and go mod update.

I think the new behaviour of go get makes sense in a world without $GOPATH where modules are standalone work areas. A developer is getting the package because they presumably want to use it. Running go get records the action in go.mod so that go know which of the various versions that might be in the cache should be used in the module when the developer imports it.

Using go get to download code to inspect it and run tests seems like a behaviour forced on us by needing everything in $GOPATH. With modules we'll just git clone to a temporary area. I don't think we should be encouraging people to change and build code in their cache.

Downloading code into the cache for future use now seems like outlying behaviour and could use a different command e.g. go cache, although personally I don't think it's needed.

[bcmills]

You can't tell me that's not going to get checked in by accident all the time. Especially in large projects with a large list of dependencies.

We probably need a go mod tidy mode that can be used as a commit hook. I've filed that as #27682.

[bcmills]

Yes, there's lots of documentation out there using go get .... if we don't muck with go get, that documentation doesn't have to be incorrect.

As previously noted, go get -u is the existing command for “update dependencies” in the non-vendored workflow. I think it's important that we not break the meaning of that command either, and its equivalent in module mode seems pretty clear.

It seems like there are three properties we would ideally like, and we can choose at most two:

1. go get $pkg continues to mean “install $pkg as a global tool”.
2. go get -u $pkgs continues to mean “upgrade the transitive dependencies of $pkgs”.
3. go get and go get -u differ only in whether they upgrade dependencies, including in module mode.

(1) seems more important for existing users with the vendored (enterprise) workflow.
(2) seems more important for existing users with the non-vendored (hobbyist) workflow.
(3) seems more important for new users.

[rsc]

@natefinch, here's my mental model for why this makes sense. I hope it will help you.

There must be some answer to the question of "which versions of which libraries are available to a build?" The answer used to be "the specific copies in GOPATH". That is, the entire GOPATH tree was in some sense equivalent to a traditional dependency manager lock file. (Tools like godeps made this more explicit, providing conversion back and forth between the exploded GOPATH and a single file recording versions.) How did we add new libraries or modify which versions were being used? go get.

In the new module world, GOPATH is gone, and the answer to which versions of which libraries are available is the go.mod file. In essence, each go.mod is now a description of a "virtual GOPATH" containing the specific modules and versions listed in the go.mod and their dependencies. We used to say that GOPATH defined the workspace; now go.mod defines the same sense of workspace, although we no longer use that term.

Just as "go get" once pulled code into, or updated code already in, the GOPATH workspace, now it does the same for code in the go.mod-defined workspace. It used to operate by writing file trees in GOPATH. Now it effects the same changes by writing lines in go.mod.

I hope that viewed this way it's clear that "go get" is doing what it has always done: set up which specific packages are going to be used by a future build.

[cespare]

There must be some answer to the question of "which versions of which libraries are available to a build?" The answer used to be "the specific copies in GOPATH". That is, the entire GOPATH tree was in some sense equivalent to a traditional dependency manager lock file. (Tools like godeps made this more explicit, providing conversion back and forth between the exploded GOPATH and a single file recording versions.) How did we add new libraries or modify which versions were being used? go get.

No, not really.

I mean, some people did, but as I explained in https://github.com/golang/go/issues/27643#issuecomment-420767420 any serious team-based Go workflow before modules didn't involve using go get for managing application dependencies. People used vendoring, or godep, or some other system.

In that world, the GOPATH is a handy global workspace for downloading and experimenting with libraries, scratch code, and (most importantly) installing Go tools globally.

In the module world, there is no equivalent way of saying "please install this tool globally, unrelated to the working directory I happen to be in", and attempting to use the old method has the frustrating side effect of editing the current module.

[komuw]

In the module world, there is no equivalent way of saying "please install this tool globally, unrelated to the working directory I happen to be in

I think that is supposed to be resolved in https://github.com/golang/go/issues/24250 and/or https://github.com/golang/go/issues/25922

[bufdev]

I'm a little disappointed we didn't find a solution to the original author's problem here - go get does now have side effects it did not used to, that's not really debatable, and while potentially not breaking Golang's backwards compatibility promise in theory (although that is debatable), this certainly breaks this promise in spirit.

One of the common use cases for Golang is to write CLI tools, and given that Golang is already installed, being able to have users do GOBIN=/path/to/bin go get github.com/team/project/cmd/project@VERSION without side effects would have been a pretty useful pattern.

[bufdev]

@natefinch this is my script workaround to the problem, if it helps - could be cleaned up and formalized:

#!/bin/bash

set -euo pipefail

if [ ! -d "${2}" ]; then
  echo "usage: ${0} path/to/go/cmd@VERSION path/to/bin" >&2
  exit 1
fi

PKG="${1}"
BIN="$(cd "${2}" && pwd)"

TMP="$(mktemp -d)"
trap 'rm -rf "${TMP}"' EXIT

cd "${TMP}"
echo "module tmp" > go.mod
GOPATH="" GO111MODULE=on GOBIN="${BIN}" go get "${PKG}"

[komuw]

I have been using
https://github.com/myitcv/gobin to install tools globally

On Fri, 21 Dec 2018 at 01:52, Peter Edge [e-mail] wrote:

@natefinch https://github.com/natefinch this is my script workaround to
the problem, if it helps - could be cleaned up and formalized:

!/bin/bash

set -euo pipefail
if [ ! -d "${2}" ]; then
echo "usage: ${0} path/to/go/cmd@VERSION path/to/bin" >&2
exit 1fi

PKG="${1}"
BIN="$(cd "${2}" && pwd)"

TMP="$(mktemp -d)"trap 'rm -rf "${TMP}"' EXIT
cd "${TMP}"echo "module tmp" > go.mod
GOPATH="" GO111MODULE=on GOBIN="${BIN}" go get "${PKG}"

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/27643#issuecomment-449161211, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AE7LUS1OmJPwdQD6az1rqJzSmroYtBDyks5u7BSogaJpZM4Wlsp-
.

>

Thanks,

KomuW.
https://www.komu.engineer/about

[leighmcculloch]

I've been using something very similar to install Go binaries from source instead of using go get. From the examples posted it looks like we're all doing similar things to compensate for go gets behavior. https://github.com/leighmcculloch/gobin

[myitcv]

I support what Russ/Bryan are doing with the go tool and the approach they are taking (I am the author of https://github.com/myitcv/gobin). They have many different aspects to consider/balance, from supporting previous behaviour of the go tool, to not bloating the go tool or its sub commands, etc.

https://github.com/golang/go/issues/27653#issuecomment-440346711 is a good summary of how that balance is currently struck.

None of this precludes experiments in other tools, and https://github.com/myitcv/gobin is one such experiment. It may live on, it may die... any number of outcomes. And most importantly none of this precludes the go tool gaining extra capabilities in the future, based on these experiments.

@leighmcculloch - hopefully the usage docs for github.com/myitcv/gobin help to show what that tool is capable of.

[mvdan]

Those of you who followed this thread might be interested in #30515, to add a "global install" command of some sort.

[dcarbone]

Another point of concern here is the unexpected removal of values from go.mod.

I am building a WAMP server that utilizes https://github.com/gammazero/nexus, which in turn relies in https://github.com/ugorji/go. ugorji's module is atypical in that there is no code nor mod file in the root of the repo, rather everything is under /codec.

ugorji pushed a breaking change to their master branch in march 2019. The version nexus pins in its own checked-in vendor dir is from 2018. The ugorji repo does have tagged versions, but it is impossible to actually USE them with go modules, it seems. The code is under /codec, as is the mod file, but when attempting to pin v1.1.2 (the latest tag as of this post) via go get github.com/ugorji/[email protected], this entry is seemingly entirely ignored when executing go build, go mod vendor, or go mod tidy. In all cases, the latest version of master is added to the sum file, subsequently used, and then even more subsequently breaks everything and causes me to see red.

The only solution I have found to this extremely idiotic issue is to do the following:

replace (
        github.com/ugorji/go/codec v0.0.0-20190204201341-e444a5086c43 => github.com/ugorji/go/codec v0.0.0-20190126102652-8fd0f8d918c8
    github.com/ugorji/go/codec v0.0.0-20190309163734-c4a1c341dc93 => github.com/ugorji/go/codec v0.0.0-20190126102652-8fd0f8d918c8
    github.com/ugorji/go/codec v0.0.0-20190315113641-a70535d8491c => github.com/ugorji/go/codec v0.0.0-20190126102652-8fd0f8d918c8
    github.com/ugorji/go/codec v0.0.0-20190316192920-e2bddce071ad => github.com/ugorji/go/codec v0.0.0-20190126102652-8fd0f8d918c8
)

require (
        github.com/ugorji/go/codec v0.0.0-20190126102652-8fd0f8d918c8 // indirect
)

This works as I basically take every version since the last tag (v1.1.2) of the repo and replace it the tagged version.

However. When any command dealing with deps is run (get, vendor, tidy) the entry from the require block is removed for me, which then causes builds to fail with this exceptional error: build github.com/ugorji/go/codec: cannot find module for path github.com/ugorji/go/codec

I fully appreciate that I am stupid, am probably doing it wrong, etc., however it is still annoying. I would greatly appreciate a "no cleanup" or "no edit" option or _something_, so I am not bit by build errors at midnight because I forgot to re-un-edit my program's mod file.

[bcmills]

@dcarbone

When any command dealing with deps is run (get, vendor, tidy) the entry from the require block is removed for me, which then causes builds to fail with this exceptional error: build github.com/ugorji/go/codec: cannot find module for path github.com/ugorji/go/codec

go mod vendor and go mod tidy should never change your configuration from one in which everything builds to one in which something does not.

If you observe otherwise, please file a separate issue with steps to reproduce the error — this issue is closed, and closed issues are not tracked or triaged.