Go: cmd/go: don't include the dependencies of external tests in the dependency graph

[leighmcculloch]

What version of Go are you using (go version)?

$ go version
go version go1.13rc1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env

GO111MODULE=""

GOARCH="amd64"

GOBIN="/home/leighmcculloch/local/bin"

GOCACHE="/home/leighmcculloch/.cache/go-build"

GOENV="/home/leighmcculloch/.config/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GONOPROXY=""

GONOSUMDB=""

GOOS="linux"

GOPATH="/home/leighmcculloch/go"

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/home/leighmcculloch/local/bin/go/1.13rc1"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/home/leighmcculloch/local/bin/go/1.13rc1/pkg/tool/linux_amd64"

GCCGO="gccgo"

AR="ar"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD=""

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build523664316=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Migrate a repository from Dep to Modules (stellar/go#1634) that is currently dependent on:

• github.com/go-sql-driver/mysql@2e00b5cd7039
• github.com/jmoiron/[email protected]

What did you expect to see?

These dependencies to be selected by Modules at the same versions, as listed above.

What did you see instead?

Modules selected the following versions:

• github.com/go-sql-driver/[email protected]
• github.com/jmoiron/[email protected]

According to go mod why and go mod graph the reason Modules is requiring I use at least [email protected] is because sqlx.test depends on [email protected]. The sqlx package itself does not import mysql in any file, and it is only _test.go files that import the mysql dependency.

From what I can see the dependency graph looks like this:

                 | +------+      +-----------+ |
      +----------->+ sqlx +<-----+ sqlx.test | |
      |   v1.2.0 | +------+      +-----------+ |
      |          +-----------------------------+
+-----+-----+                          |
| mypackage |                          |
+-----+-----+                          |
      |                                |
      |            +-------+           |
      +----------->+ mysql +<----------+
                   +-------+ v1.4.0

Wrangling with dependencies that are being restricted by test-only dependencies is something I've been struggling with over the last week and this is just one example.

Given that mypackage is not dependent on sqlx.test it seems inconvenient, unintuitive, and unnecessary that mypackage would inherit the requirements that are unique only to the tests of sqlx. In the reading I did about my understanding of why this exists I could only find comments from folks mentioning a developer might want to run all the tests for all the dependencies of a package. But in this situation, won't the tests for each dependency be built separately, and executed as their own main, and could assume their own dependency graphs without infecting the dependency graph of mypackage? It seems like we could make this easier to work with by limiting the impact on the main modules dependency graph by not including external test dependencies.

[av86743]

Discussed before by @bcmills et al.

With a present "fluent" state of module support in go and policies of the project, I think that most you may be able to get out of your complaint is a split presentation of dependencies in go.mod where test-only dependencies are grouped separately, to address cluttering part of the problem:

require (
   ...
   /// test-only dependencies
   ...
)

[bcmills]

won't the tests for each dependency be built separately, and executed as their own main, and could assume their own dependency graphs without infecting the dependency graph of mypackage?

Then you wouldn't be able to diagnose problems in your dependencies by running their tests, because running the test would potentially result in a different configuration from the one in which you observed the problem.

[bcmills]

Duplicate of #26626

[bcmills]

most you may be able to get out of your complaint is a split presentation of dependencies

That's #26955.

[av86743]

most you may be able to get out of your complaint is a split presentation of dependencies

That's #26955.

@bcmills This is trivial to implement - only generated canonical external representation will be affected, with fully backward compatible content. I am surprised that you do not even have a CL for this yet. So that those willing could forward apply it - like for #32389.

[bcmills]

I am surprised that you do not even have a CL for this yet.

@av86743, remember that among our Gopher values is to “be charitable”.

The reason we don't have a CL for this yet is because we've been dealing with the (literally) hundreds of other cmd/go and module-related issues and feature requests. It's not from lack of effort or attention.

[leighmcculloch]

Thanks for responding so quickly @bcmills and pointing me to an issue that has comments discussing the why behind this behavior. Linking here for anyone else who needs it: https://github.com/golang/go/issues/26626#issuecomment-408236234.