Go: cmd/go: mod tidy removes lines that build seems to need

[zombiezen]

(I haven't dug into exactly what dependency is causing this or why, but I've managed to whittle the issue down to a fairly manageable no-code repro case.)

What version of Go are you using (go version)?

$ go version
go 1.12.1

Does this issue reproduce with the latest release?

Yes. I haven't checked against tip.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env

GOARCH="amd64"

GOBIN=""

GOCACHE="/home/rlight2/.cache/go-build"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GOOS="linux"

GOPATH="/home/rlight2"

GOPROXY=""

GORACE=""

GOROOT="/usr/local/go"

GOTMPDIR=""

GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"

GCCGO="gccgo"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD="/home/rlight2/src/mg-repro/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build158725968=/tmp/go-build -gno-record-gcc-switches"

What did you do?

git clone https://gist.github.com/5513fc51a179cfd757dcdd1d7c1e4995.git mod-tidy-issue
cd mod-tidy-issue
go mod tidy
go build -mod=readonly

main.go + go.mod gist

What did you expect to see?

A successful build of the binary.

What did you see instead?

go: updates to go.mod needed, disabled by -mod=readonly

Additional surrounding steps

Running go build will cause go build -mod=readonly to pass. Doing this adds lines (all marked "indirect") for:

• github.com/golang/mock
• gopkg.in/check.v1
• gopkg.in/yaml.v2

However, running go mod tidy on this go.mod will unceremoniously remove these added lines. Most confusingly, go mod why doesn't seem to know why these lines are there either:

$ go mod why -m gopkg.in/yaml.v2
# gopkg.in/yaml.v2
(main module does not need module gopkg.in/yaml.v2)

[andybons]

@bcmills @jayconrod

[bcmills]

We should check for similarities with #29773. (I don't know whether it will turn out to be related, but it's a known source of missing dependencies.)

[prymitive]

Hit the same issue - go build and go mod tidy keep fighting over what should be in go.mod
gist for reference - https://gist.github.com/prymitive/9f8ac72d9a88f1704efa8ebfdf133bd1

$ git st
On branch master
[...]
    new file:   go.mod
    new file:   go.sum
    new file:   main.go
$ go build
$ git diff
diff --git a/go.mod b/go.mod
index 6980ec7..7e79628 100644
--- a/go.mod
+++ b/go.mod
@@ -15,6 +15,7 @@ require (
        github.com/gin-gonic/contrib v0.0.0-20190526021735-7fb7810ed2a0
        github.com/gin-gonic/gin v1.4.0
        github.com/go-bindata/go-bindata v3.1.1+incompatible
+       github.com/go-logfmt/logfmt v0.4.0 // indirect
        github.com/go-openapi/errors v0.19.0
        github.com/go-openapi/runtime v0.19.0
        github.com/go-openapi/strfmt v0.19.0
$ go mod tidy
$ git diff
[no diff]
$ go build
$ git diff
diff --git a/go.mod b/go.mod
index 6980ec7..7e79628 100644
--- a/go.mod
+++ b/go.mod
@@ -15,6 +15,7 @@ require (
        github.com/gin-gonic/contrib v0.0.0-20190526021735-7fb7810ed2a0
        github.com/gin-gonic/gin v1.4.0
        github.com/go-bindata/go-bindata v3.1.1+incompatible
+       github.com/go-logfmt/logfmt v0.4.0 // indirect
        github.com/go-openapi/errors v0.19.0
        github.com/go-openapi/runtime v0.19.0
        github.com/go-openapi/strfmt v0.19.0

[bcmills]

@prymitive, can you provide a more compact example, or at least compare the output of go list -m all and go mod graph? As noted above, it would be good to rule out duplicates of #29773.

[prymitive]

Narrowed down needed modules to just two:

github.com/prometheus/client_golang v0.9.4
github.com/spf13/viper v1.4.0

https://gist.github.com/prymitive/9f8ac72d9a88f1704efa8ebfdf133bd1 updated to reflect that
Also added the output of these 2 commands

[gopherbot]

Change https://golang.org/cl/186537 mentions this issue: cmd/go/internal/mvs: retain modules required by older versions

[gopherbot]

Change https://golang.org/cl/186557 mentions this issue: cmd/go/internal/mvs: in Req, omit versions implied by older-than-selected versions already in the graph

[bcmills]

I found a minimal repro. A consistent reproducer requires two modules that depend on some module X. One of the modules must depend on the newer version of X and on the other module. The other module must depend on the older X, which must pull in some indirect dependency I that the newer X does not. Then mvs.Req normally adds I to the build list, but since I is not required by the _selected_ versions and is _also_ not needed for the import graph, go mod tidy removes it.

https://go-review.googlesource.com/c/go/+/186557/2/src/cmd/go/testdata/script/mod_indirect_tidy.txt

[jgiles]

I'm seeing an issue that looks like this on go1.12.7 . Is there anything I can do to confirm that I have a repro of this issue, or if I should file a separate bug?

[horgh]

You could see if the behaviour you're seeing is fixed using gotip as described here: https://github.com/golang/go/issues/33435#issuecomment-517752652

[vibridi]

@bcmills
Hello, sorry to revive a closed issue. I think I'm observing the same problem described here in Go1.13.3
In particular, go get disagrees with go mod tidy over which package should go into go.mod. In my case:

go get writes github.com/ugorji/go v1.1.7 // indirect, also

$ go mod why github.com/ugorji/go
# github.com/ugorji/go
(main module does not need package github.com/ugorji/go)

go mod tidy writes github.com/ugorji/go/codec v1.1.7 // indirect, also

$ go mod why github.com/ugorji/go/codec
# github.com/ugorji/go/codec
mycompany/mypackage
github.com/gin-gonic/gin
github.com/gin-gonic/gin/binding
github.com/ugorji/go/codec

[bcmills]

In particular, go get disagrees with go mod tidy over which package should go into go.mod.

That is #34086, fixed at head.

[vibridi]

Thank you very much

[thepudds]

@vibridi If you haven’t already done so, would you mind confirming the behavior you reported is indeed fixed by using the gotip command:

https://godoc.org/golang.org/dl/gotip

[vibridi]

@thepudds confirmed, it is fixed at head (680ed10) 👍

[ceejatec]

@bcmills Sorry to necropost, but I'm seeing a similar issue even with go 1.14.4. I have a package where "go mod tidy" removes an //indirect entry from go.mod, but that then causes "go build -mod=readonly" to fail with "cannot find module providing package github.com/dustin/gojson: import lookup disabled by -mod=readonly". On the other hand, running "go build" without "-mod=readonly" adds the same //indirect entries back into go.mod that "go mod tidy" just took out.

Before I attempt to reduce this to a test case and file a new ticket, could you tell me if the following statement should 100% be true?

If you run "go mod tidy", then immediately running "go build -mod=readonly"
should always succeed. (Well, at least if the code compiles -
it should not fail due to needing to write to go.mod or for other
reasons implied by -mod=readonly.)

If it isn't supposed to be 100% true, is there any statement that can be made about "go mod tidy" vs. "go build -mod=readonly" that is 100% true?

[ceejatec]

FYI, I also confirmed this behaviour running "gotip".

[jayconrod]

@ceejatec Your statement should be true. Please open a new issue, and we'll take a look.

[ceejatec]

@jayconrod Thanks for confirming. I've filed a new ticket #39570 with a reproducer.