Go: cmd/go: add 'require test' section to go.mod

[OneOfOne]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version devel +479da24aac 2018-08-10 00:47:31 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

Linux/amd64

What did you do?

cd $GOPATH/github.com/OneOfOne/xxhash
gotip mod tidy

What did you expect to see?

● cat go.mod
module github.com/OneOfOne/xxhash

What did you see instead?

module github.com/OneOfOne/xxhash

require (
    github.com/cespare/xxhash v1.0.0
    github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72 // indirect
)

Those 2 deps are only used in *_test.go files.

This isn't a bug really but more of a feature request, is there any way to split those sections into require (...) and require test (...)?

Without that split, go get -t is pretty much meaningless.

If there's interest, I'd like to take a stab at it.

[OneOfOne]

@gopherbot modules

[davecheney]

Yes please. This mirrors maven's notion of scopes; the set of dependencies for a build scope are different to the set of dependencies for a test scope.

[ianlancetaylor]

CC @rsc @bcmills

[rsc]

In a sense, go get -t is now the default.

The module download is split into two parts: downloading the go.mod and downloading the actual code. If you have dependencies only needed for tests, then they will show up in your go.mod, and go get will download their go.mods, but it will not download their code. The test-only dependencies get downloaded only when you need it, such as the first time you run go test.

This applies not just to test-only dependencies but also os-specific dependencies. For example if you use logrus but don't use Solaris, then the golang.org/x/sys go.mod needs to be downloaded, but not the code in that repo.

When you do split things out explicitly, then you'd have different version selection results for the different "scopes". You could potentially be using one version for a build and then get a different version for a test. That would be quite unfortunate. Having one unified go.mod avoids that potential problem.

So you're getting the fine-grained separation you want already, for free*, without having to maintain any explicit scopes, and without any possibility of inconsistency.

* The asterisk is that right now because we're still fetching git repos, even to get the go.mod, what I said isn't true. But once we have a better story for a proxy, it will become true, all with no effort. Because we do have a plan to get there, though, we're not planning to add any of these kinds of scopes as a temporary workaround. They'd just cause needless pain in the long run.

[posener]

@rsc , wouldn't it be helpful to mark in any way dependencies that are only used for tests? I find it quite confusing to have dependencies in the go.mod that are not actually present in run time.

[bcmills]

I find it quite confusing to have dependencies in the go.mod that are not actually present in run time.

They are present at run time, if you run go test all, and go mod why should help clear up the confusion for any given dependency.

[wsc1]

I think it would be more clear if there were a way to know from the go tool or the go.mod which requirements are only for testing, and that doing this would not violate the problem of inconsistent scopes. It is in any event helpful to be clear about really why a requirement is there, as requirements without test fall under a different set of considerations for a module maintainer than requirements for tests. (The consumer of the former is more the end user and the later more developers). Similarly for build tags...

Note, this is not to say that I think splitting the requirements is good, as @rsc noted that causes problems. But explaining why requirements are there without mentioning scopes like testing or build tags is not ideal.

Also there is a question on go-nuts where there is some concern about vendoring overhead, but the justification from @rsc
that they are or will be free
"""

• The asterisk is that right now because we're still fetching git repos, even to get the go.mod, what I said isn't true. But once we have a better story for a proxy, it will become true, all with no effort. Because we do have a plan to get there, though, we're not planning to add any of these kinds of scopes as a temporary workaround. They'd just cause needless pain in the long run.
  """
  does not address the relationship to vendoring. Sorry I don't have an answer for whether there is vendoring overhead.

CC @rsc

[wsc1]

redirecting my comment above to related #26955

[gdamore]

Having visibility into this is really important. Some of my test dependencies transitive packages are for things that I think I really really don't want in my runtime, and I would prefer my downstreams not see the pile of cruft that my use of GoConvey brings in, or at least see it with the explicit understanding that I'm not using this in the standard build.

For now this is actually driving me away from using certain 3rd party testing packages, because I consider their rather large dependency graph as pollution in my go.mod file.

[bcmills]

@gdamore, this issue is closed, which means that nobody will be revisiting it when it is time to plan work. Please continue discussion on one of the related issues that are still open (such as #26955).