Go: cmd/go: `go get -u` on main module bloats go.mod with unnecessary dependencies

[inliquid]

What did you do?

Consider this simple test app:

package main

import (
    "fmt"
    "time"

    "github.com/jinzhu/gorm"
    _ "github.com/lib/pq"
)

var db *gorm.DB

func init() {
    var err error

    db, err = gorm.Open("postgres", "postgres://gorm:[email protected]/gorm?sslmode=disable")
    if err != nil {
        panic(err)
    }
}
func main() {
    t := time.Time{}
    db.Debug().Raw("SELECT NOW()").Row().Scan(&t)
    fmt.Println(t)
}

go.mod is also pretty simple:

module SomeDBTest

go 1.12

require (
    github.com/jinzhu/gorm v1.9.9
    github.com/lib/pq v1.1.1
)

Lets try to update this app dependencies by running go get -u -d, resulting go.mod will look as follows:

module SomeDBTest

go 1.12

require (
    cloud.google.com/go v0.41.0 // indirect
    github.com/DataDog/zstd v1.4.0 // indirect
    github.com/Shopify/sarama v1.23.0 // indirect
    github.com/denisenkom/go-mssqldb v0.0.0-20190710001350-29e7b2419f38 // indirect
    github.com/eapache/go-resiliency v1.2.0 // indirect
    github.com/go-kit/kit v0.9.0 // indirect
    github.com/gogo/protobuf v1.2.1 // indirect
    github.com/golang/protobuf v1.3.2 // indirect
    github.com/gorilla/mux v1.7.3 // indirect
    github.com/jcmturner/gofork v1.0.0 // indirect
    github.com/jinzhu/gorm v1.9.10
    github.com/kisielk/errcheck v1.2.0 // indirect
    github.com/konsorten/go-windows-terminal-sequences v1.0.2 // indirect
    github.com/lib/pq v1.1.1
    github.com/onsi/ginkgo v1.8.0 // indirect
    github.com/onsi/gomega v1.5.0 // indirect
    github.com/openzipkin/zipkin-go v0.2.0 // indirect
    github.com/pkg/errors v0.8.1 // indirect
    github.com/prometheus/common v0.6.0 // indirect
    github.com/prometheus/procfs v0.0.3 // indirect
    github.com/rcrowley/go-metrics v0.0.0-20190706150252-9beb055b7962 // indirect
    github.com/sirupsen/logrus v1.4.2 // indirect
    github.com/stretchr/objx v0.2.0 // indirect
    golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4 // indirect
    golang.org/x/exp v0.0.0-20190627132806-fd42eb6b336f // indirect
    golang.org/x/image v0.0.0-20190703141733-d6a02ce849c9 // indirect
    golang.org/x/mobile v0.0.0-20190711165009-e47acb2ca7f9 // indirect
    golang.org/x/net v0.0.0-20190628185345-da137c7871d7 // indirect
    golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7 // indirect
    golang.org/x/tools v0.0.0-20190712213246-8b927904ee0d // indirect
    google.golang.org/genproto v0.0.0-20190708153700-3bdd9d9f5532 // indirect
    google.golang.org/grpc v1.22.0 // indirect
    gopkg.in/jcmturner/gokrb5.v7 v7.3.0 // indirect
)

Now let's try go mod tidy if it's possible to remove some redundant dependencies:

module SomeDBTest

go 1.12

require (
    cloud.google.com/go v0.41.0 // indirect
    github.com/denisenkom/go-mssqldb v0.0.0-20190710001350-29e7b2419f38 // indirect
    github.com/jinzhu/gorm v1.9.10
    github.com/lib/pq v1.1.1
    golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4 // indirect
)

What did you expect to see?

Result of running go get -u -d should match corresponding go mod tidy on updated dependencies in go.mod, or match those lines in go.mod being updated according to documented behavior of go get -u without adding any additional indirect dependencies.

What did you see instead?

go.mod is bloated with indirect dependencies, forcing user to run go mod tidy additionally, or edit it manually.

System details

go version go1.12.7 windows/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="C:\Users\***\AppData\Local\go-build"
GOEXE=".exe"
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="windows"
GOOS="windows"
GOPATH="C:\Users\***\go"
GOPROXY=""
GORACE=""
GOROOT="c:\go"
GOTMPDIR=""
GOTOOLDIR="c:\go\pkg\tool\windows_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="NUL"
GOROOT/bin/go version: go version go1.12.7 windows/amd64
GOROOT/bin/go tool compile -V: compile version go1.12.7

[thepudds]

@inliquid would you be able to try this with the Go 1.13 beta?

There were some bug fixes in this area (e.g., #29702), as well as the overall upgrade behavior changed in a way that often fairly dramatically decreases how many indirect dependencies get updated by go get -u, which also affected the meaning of several go get flags. This is summarized in the go get section of the draft Go 1.13 release notes:
https://tip.golang.org/doc/go1.13#go-get

[inliquid]

@thepudds
Tried with this version:

$ gotip version
go version devel +89d300b Sun Jul 14 20:15:47 2019 +0000 linux/amd64

After running go get -u -d there is clean go.mod file with updated versions of dependencies and no indirect entries:

module SomeDBTest

go 1.13

require (
        github.com/jinzhu/gorm v1.9.10
        github.com/lib/pq v1.1.1
)

Then I tried go mod tidy with same result. Very good!

[jayconrod]

Glad to hear this is working. This was fixed in CL 174099 and a few follow-up CLs. Unfortunately, it's too big of a change to backport.

Previously, go get -u meant "upgrade the named modules and everything they transitively require". Without arguments, it would start at the main module, so it would upgrade everything in the module graph (way more than what anyone wants).

In go1.13, go get -u means "upgrade the modules providing the named packages and everything they transitively import". This is much narrower, and more similar to what go get -u did in GOPATH mode.