Google-cloud-go: all: audit transitive dependencies

[broady]

As part of ongoing modules work...

• [x] list all transitive dependencies of non-test packages
• [x] check they are using semver, and preferably modules
• [x] check that we should actually depend on them (trust/security)

Majority of x packages (and I think all x packages we depend on) are currently untagged, but apparently they will be soon (no reference for this, just saying this from memory after talking to the core Go team)

[jadekler]

Transitive deps are encoded in go.sum AFAIK https://github.com/googleapis/google-cloud-go/blob/master/go.sum

[dmitshur]

Majority of x packages (and I think all x packages we depend on) are currently untagged, but apparently they will be soon (no reference for this, just saying this from memory after talking to the core Go team)

I understand that the current plan is to add go.mod files only, but there will not be new tagged releases (other than existing old ones in x/text) until some time later. The commit message when go.mod file was added elaborates on this (e.g., see https://github.com/golang/tools/commit/8dcb7bc8c7fe0a895995c76c721cef79419ac98a), and points to https://github.com/golang/go/issues/28136#issuecomment-462971974 as the current plan.

[jadekler]

For posterity: maybe we should write our go list / go mod graph to a file and audit that per commit?

[jadekler]

Transitive dependencies here: https://gist.github.com/jadekler/4b36975a804d49fab91a5a7e5c1e9f76

Unique deps:

$ go mod graph | awk 'BEGIN{ ORS="" } { for ( i=1; i<= NF ; i++){ print $i"\n"  }  }' | cut -f1 -d"@" | sort | uniq
cloud.google.com/go
cloud.google.com/go/bigquery
cloud.google.com/go/datastore
cloud.google.com/go/pubsub
cloud.google.com/go/storage
dmitri.shuralyov.com/gpu/mtl
github.com/BurntSushi/toml
github.com/BurntSushi/xgb
github.com/client9/misspell
github.com/go-gl/glfw
github.com/golang/glog
github.com/golang/mock
github.com/golang/protobuf
github.com/google/btree
github.com/google/go-cmp
github.com/google/martian
github.com/google/pprof
github.com/google/renameio
github.com/googleapis/gax-go/v2
github.com/hashicorp/golang-lru
github.com/jstemmer/go-junit-report
github.com/kisielk/gotool
github.com/kr/pretty
github.com/kr/pty
github.com/kr/text
github.com/rogpeppe/go-internal
go.opencensus.io
golang.org/x/crypto
golang.org/x/exp
golang.org/x/image
golang.org/x/lint
golang.org/x/mobile
golang.org/x/mod
golang.org/x/net
golang.org/x/oauth2
golang.org/x/sync
golang.org/x/sys
golang.org/x/text
golang.org/x/time
golang.org/x/tools
golang.org/x/xerrors
google.golang.org/api
google.golang.org/appengine
google.golang.org/genproto
google.golang.org/grpc
gopkg.in/check.v1
gopkg.in/errgo.v2
honnef.co/go/tools
rsc.io/binaryregexp

(I am irrationally proud of that piece of bash)

Minus google/golang:

$ go mod graph | awk 'BEGIN{ ORS="" } { for ( i=1; i<= NF ; i++){ print $i"\n"  }  }' | cut -f1 -d"@" | sort | uniq | grep -v google | grep -v golang
dmitri.shuralyov.com/gpu/mtl
github.com/BurntSushi/toml
github.com/BurntSushi/xgb
github.com/client9/misspell
github.com/go-gl/glfw
github.com/jstemmer/go-junit-report
github.com/kisielk/gotool
github.com/kr/pretty
github.com/kr/pty
github.com/kr/text
github.com/rogpeppe/go-internal
go.opencensus.io
gopkg.in/check.v1
gopkg.in/errgo.v2
honnef.co/go/tools
rsc.io/binaryregexp

All look reasonable to me. Most of those are tools, not actual dependencies. I think we can close this.

[dennwc]

@jadekler There is still at least one weird dependency that I believe is unused by any of the packages:

github.com/go-gl/glfw

It's definitely strange that the Cloud SDK has a transitive dependency on OpenGL helpers. And it may be a problem in contexts where the use of cgo is forbidden.

[tbpg]

Next steps for this:

1. Remove apidiff from tools.go, thereby removing golang.org/x/exp from the root go.mod.
2. Release cloud.google.com/go.
3. Update all of the cloud.google.com/go/... modules that depend on cloud.google.com/go to the new version.
4. Release new versions of those modules.

@codyoss, can you take care of this? Step 4 can probably happen on-demand. But, if you want to do it across the board, that works, too.

More detail about how I found this below.

---

$ go mod why github.com/go-gl/glfw
go: finding module for package github.com/go-gl/glfw
go: downloading github.com/go-gl/glfw v0.0.0-20200222043503-6f7a984d4dc4
go: downloading google.golang.org/genproto v0.0.0-20200304201815-d429ff31ee6c
go: downloading golang.org/x/tools v0.0.0-20200304193943-95d2e580d8eb
go: downloading google.golang.org/api v0.20.0
# github.com/go-gl/glfw
(main module does not need package github.com/go-gl/glfw)

(This is https://github.com/golang/go/issues/27900.)

$ go mod graph | grep github.com/go-gl/glfw       
golang.org/x/[email protected] github.com/go-gl/glfw/v3.3/[email protected]
golang.org/x/[email protected] github.com/go-gl/glfw/v3.3/[email protected]
golang.org/x/[email protected] github.com/go-gl/glfw/v3.3/[email protected]
golang.org/x/[email protected] github.com/go-gl/glfw/v3.3/[email protected]
golang.org/x/[email protected] github.com/go-gl/glfw/v3.3/[email protected]
golang.org/x/[email protected] github.com/go-gl/[email protected]
golang.org/x/[email protected] github.com/go-gl/glfw/v3.3/[email protected]

$ go mod why golang.org/x/exp
go: finding module for package golang.org/x/exp
# golang.org/x/exp
(main module does not need package golang.org/x/exp)

$ go mod graph | grep golang.org/x/exp
cloud.google.com/go golang.org/x/[email protected]
[many other lines]

:dizzy_face:

golang.org/x/exp is in the root go.mod. Why?

$ grep golang.org/x/exp **/*.go         
tools.go:   _ "golang.org/x/exp/cmd/apidiff"

Ahh. apidiff. I don't think we need this tool dependency anymore.

check_incompat_changes.sh uses it, but it also installs it directly, so we don't need the extra dependency in our go.mod.

[codyoss]

v0.54.0 is released, will let next regen update go.mod files.

[dennwc]

@codyoss @tbpg I think this is still an issue. I noticed a few autoregen commits, but GLFW is still listed in the updated files.

[tbpg]

go-genproto does not depend on exp: https://github.com/googleapis/go-genproto/blob/master/go.mod

However, it used to. Those older versions are still in the module graph, even if they never end up in the build.

Is that still an issue?

[dmitshur]

As @tbpg said, they are not in the build, but they are in the module graph and so they still need to be written down.

Also see the cmd/go proposal https://github.com/golang/go/issues/36460. It is planned for Go 1.15, and it should help with this. /cc @bcmills