Google-cloud-go: Error when updating the library with go modules

[ernestoalejo]

Describe Your Environment

Docker, Cloud Shell, vanilla Go installation in an Ubuntu machine, ...

Expected Behavior

Be able to install the library and then update the packages without error.

Actual Behavior

Possibly related to this issue: https://github.com/golang/lint/issues/436#issuecomment-472637498 , but they asked me to open the issue here instead to investigate it.

You can reproduce the error with these commands:

$ cd $(mktemp -d)

$ go mod init m
go: creating new go.mod: module m

$ go get cloud.google.com/go
go: finding github.com/prometheus/common v0.0.0-20180801064454-c7de2306084e
go: finding github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273
go: finding github.com/shurcooL/notifications v0.0.0-20181007000457-627ab5aea122
go: finding github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470
go: finding github.com/microcosm-cc/bluemonday v1.0.1
go: finding github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1
go: finding github.com/shurcooL/issuesapp v0.0.0-20180602232740-048589ce2241
go: finding github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
go: finding github.com/dustin/go-humanize v1.0.0
go: finding cloud.google.com/go v0.26.0
go: finding github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
go: finding golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
go: finding github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc
go: finding github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537
go: finding go4.org v0.0.0-20180809161055-417644f6feb5
go: finding github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1
go: finding github.com/russross/blackfriday v1.5.2
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding golang.org/x/net v0.0.0-20181029044818-c44066c5c816
go: finding github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b
go: finding github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4
go: finding github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133
go: finding github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07
go: finding github.com/kisielk/gotool v1.0.0
go: finding github.com/kr/pty v1.1.3
go: finding github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
go: finding google.golang.org/appengine v1.4.0
go: finding github.com/googleapis/gax-go v2.0.0+incompatible
go: finding github.com/shurcooL/httpgzip v0.0.0-20180522190206-b1c53ac65af9
go: finding github.com/gogo/protobuf v1.1.1
go: finding golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52
go: finding github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568
go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1
go: finding github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191
go: finding github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239
go: finding golang.org/x/net v0.0.0-20180826012351-8a410e7b638d
go: finding github.com/kr/pty v1.1.1
go: finding github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86
go: finding github.com/client9/misspell v0.3.4
go: finding honnef.co/go/tools v0.0.0-20180728063816-88497007e858
go: finding golang.org/x/exp v0.0.0-20190121172915-509febef88a4
go: finding golang.org/x/text v0.3.0
go: finding google.golang.org/api v0.0.0-20181030000543-1d582fd0359e
go: finding golang.org/x/net v0.0.0-20180724234803-3673e40ba225
go: finding gopkg.in/inf.v0 v0.9.1
go: finding cloud.google.com/go v0.31.0
go: finding dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c
go: finding dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3
go: finding github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d
go: finding golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852
go: finding github.com/shurcooL/httpfs v0.0.0-20171119174359-809beceb2371
go: finding github.com/ghodss/yaml v1.0.0
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: finding google.golang.org/genproto v0.0.0-20181202183823-bd91e49a0898
go: finding google.golang.org/grpc v1.17.0
go: finding google.golang.org/appengine v1.3.0
go: finding golang.org/x/net v0.0.0-20181106065722-10aee1819953
go: downloading cloud.google.com/go v0.37.0
go: extracting cloud.google.com/go v0.37.0

$ go get -u
go: finding github.com/prometheus/client_model latest
go: finding dmitri.shuralyov.com/service/change latest
go: finding github.com/sourcegraph/syntaxhighlight latest
go: finding dmitri.shuralyov.com/app/changes latest
go: finding github.com/gregjones/httpcache latest
go: finding dmitri.shuralyov.com/state latest
go: finding github.com/shurcooL/octicon latest
go: finding github.com/flynn/go-shlex latest
go: finding github.com/kisielk/errcheck v1.2.0
go: finding google.golang.org/api v0.2.0
go: finding github.com/shurcooL/github_flavored_markdown latest
go: finding golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: finding github.com/gliderlabs/ssh v0.1.3
go: finding github.com/shurcooL/httpfs latest
go: finding github.com/golang/protobuf v1.3.0
go: finding golang.org/x/net v0.0.0-20190311183353-d8887717615a
go: finding github.com/gopherjs/gopherjs latest
go: finding github.com/golang/lint latest
go: finding gopkg.in/yaml.v2 v2.2.2
go: finding github.com/prometheus/client_golang v0.9.2
go: finding gopkg.in/resty.v1 v1.12.0
go: finding github.com/google/btree latest
go: finding golang.org/x/crypto latest
go: finding golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8
go: finding golang.org/x/oauth2 latest
go: finding golang.org/x/build latest
go: finding golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2
go: finding golang.org/x/lint latest
go: finding honnef.co/go/tools v0.0.0-20180920025451-e3ad64cb4ed3
go: finding golang.org/x/net v0.0.0-20190125091013-d26f9f9a57f3
go: finding github.com/openzipkin/zipkin-go v0.1.3
go: github.com/golang/[email protected]: parsing go.mod: unexpected module path "golang.org/x/lint"
go: finding github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802
go: finding github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829
go: finding grpc.go4.org latest
go: finding github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af
go: finding github.com/hashicorp/golang-lru v0.5.0
go: finding golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846
go: finding github.com/shurcooL/go-goon latest
go: finding golang.org/x/tools v0.0.0-20190312170243-e65039ee4138
go: finding golang.org/x/tools v0.0.0-20180221164845-07fd8470d635
go: finding sourcegraph.com/sqs/pbtypes v1.0.0
go: finding golang.org/x/net v0.0.0-20181220203305-927f97764cc3
go: finding golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563
go: finding google.golang.org/genproto v0.0.0-20181219182458-5a97ab628bfb
go: finding google.golang.org/api v0.0.0-20181220000619-583d854617af
go: finding github.com/grpc-ecosystem/grpc-gateway v1.6.2
go: finding cloud.google.com/go v0.34.0
go: finding golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a
go: finding golang.org/x/image v0.0.0-20190227222117-0694c2d4d067
go: finding gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7
go: finding golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f
go: finding golang.org/x/tools v0.0.0-20181219222714-6e267b5cc78e
go: finding golang.org/x/lint v0.0.0-20181217174547-8f45f776aaf1
go: finding golang.org/x/sys v0.0.0-20181218192612-074acd46bca6
go: finding golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6
go get: error loading module requirements

$ echo $?
1

[broady]

$ go mod graph | grep golang.lint
google.golang.org/[email protected] github.com/golang/[email protected]

Looks like it was fixed in gRPC by https://github.com/grpc/grpc-go/pull/2393

@jadekler we probably still need rev our gRPC dep in a couple more repos? I think go.mod is on grpc v1.19 across the board but not tagged yet in some repos.

$ go mod graph | grep ' .*grpc' | grep -v 1.19
golang.org/x/[email protected] google.golang.org/[email protected]
golang.org/x/[email protected] [email protected]
github.com/googleapis/gax-go/[email protected] google.golang.org/[email protected]
google.golang.org/[email protected] google.golang.org/[email protected]
[email protected] github.com/grpc-ecosystem/[email protected]
[email protected] google.golang.org/[email protected]
google.golang.org/[email protected] google.golang.org/[email protected]

[broady]

That said, it's weird that even though [email protected] is unequivocally used for the build, that the Go tool still uses an older version during go get -u. That's probably worth looping back on the original bug.

[jadekler]

Seems to be an issue with how go modules interprets modules whose names have changed (github.com/x/lint and golang.org/x/lint). Tracking discussion in https://github.com/golang/go/issues/30831.

[jadekler]

Since this is a core issue and affects anyone depending on x/lint, I'm closing this in favour of https://github.com/golang/go/issues/30831.

[thepudds]

Since this is a core issue and affects anyone depending on x/lint, I'm closing this in favour of golang/go#30831.

I know there are multiple tracking issues at this point for the related set of changes, but is there a next step here on this issue? The original symptoms reported here still seem to happen, e.g., as of a few moments ago:

cd $(mktemp -d)
go mod init m
go get cloud.google.com/go
go get -u
...
go: github.com/golang/[email protected]:
 parsing go.mod: unexpected module path "golang.org/x/lint"
go get: error loading module requirements

For example, was there a change made to this repo to help, but a tagged release is still pending?

golang/go#30831 I think might be more about a medium or long term fix by changing the go tool, but that is not happening immediately, so probably still makes sense to solve this and similar issues in the Go 1.12 world that exists today.

CC @DukeAnn @dmitshur @bcmills

[thepudds]

FWIW, the same steps starting with cloud.google.com/go@master also seem to currently fail:

$ cd $(mktemp -d)
$ go mod init m
$ go get cloud.google.com/go@master
go: finding cloud.google.com/go master
go: finding golang.org/x/build v0.0.0-20190314133821-5284462c4bec
go: downloading cloud.google.com/go v0.37.1-0.20190315043659-2f0e77050eaa
go: extracting cloud.google.com/go v0.37.1-0.20190315043659-2f0e77050eaa
$ go get -u
...
go: github.com/golang/[email protected]:
 parsing go.mod: unexpected module path "golang.org/x/lint"
go get: error loading module requirements

[dmitshur]

I know there are multiple tracking issues at this point for the related set of changes, but is there a next step here on this issue?

I agree that it'd be worth it to re-open this one for tracking things happening in this repository. Any changes in the cmd/go command are unlikely to land very soon, and this specific problem instance with lint can be worked on before then. See https://github.com/golang/go/issues/30833#issuecomment-473058892.

I've been wanting to try to resolve the issue in [email protected] first, before tacking this repo, but it may not be possible to do independently after all. See https://github.com/googleapis/google-api-go-client/issues/342#issuecomment-473159138.

[jadekler]

Ack - re-opening. Update: v0.37.1 released https://github.com/googleapis/google-cloud-go/releases/tag/v0.37.1. This includes several updates to our dependencies. (https://code-review.googlesource.com/c/gocloud/+/3901)

Don't think this fixes our problems: continuing to investigate.

[blackknight467]

Any updates on when a fix for this might be out? I need to give my PM a timeline.

$ go get -u google.golang.org/api/jobs/v3
go: finding google.golang.org/api/jobs/v3 latest
go: finding google.golang.org/api/jobs latest
go: finding google.golang.org/genproto latest
go: finding honnef.co/go/tools latest
go: finding golang.org/x/lint latest
go: finding golang.org/x/sync latest
go: finding golang.org/x/oauth2 latest
go: finding golang.org/x/net latest
go: finding golang.org/x/tools latest
go: finding github.com/beorn7/perks latest
go: finding golang.org/x/exp latest
go: finding golang.org/x/crypto latest
go: finding golang.org/x/sys latest
go: finding github.com/prometheus/client_model latest
go: finding github.com/golang/glog latest
go: finding github.com/prometheus/procfs latest
go: finding github.com/alecthomas/units latest
go: finding github.com/jstemmer/go-junit-report latest
go: finding github.com/google/pprof latest
go: finding github.com/golang/lint latest
go: finding github.com/mwitkow/go-conntrack latest
go: finding github.com/kr/logfmt latest
go: finding github.com/alecthomas/template latest
go: finding github.com/rcrowley/go-metrics latest
go: finding golang.org/x/mobile latest
go: finding gopkg.in/check.v1 latest
go: finding github.com/eapache/go-xerial-snappy latest
go: finding golang.org/x/time latest
go: finding golang.org/x/image latest
go: finding github.com/tarm/serial latest
go: finding golang.org/x/perf latest
go: finding github.com/flynn/go-shlex latest
go: finding golang.org/x/build latest
go: github.com/golang/[email protected]: parsing go.mod: unexpected module path "golang.org/x/lint"
go: finding github.com/bradfitz/go-smtpd latest
go: finding grpc.go4.org latest
go: finding github.com/jellevandenhooff/dkim latest
go: finding github.com/gregjones/httpcache latest
go: finding github.com/coreos/go-systemd latest
go: finding go4.org latest
go get: error loading module requirements

[jadekler]

Any updates on when a fix for this might be out?

It's going to require a change in golang/go to fix the underlying problem - that's tracked in https://github.com/golang/go/issues/30831.

We (Google library maintainers) are going to do a few moderately complicated tag song-and-dances in the next few weeks to try to alleviate the problem until then. No hard ETA, though.

In the meantime, I would try to avoid doing go get -u and instead upgrade your dependencies by hand.

[jadekler]

This has been fixed. For whichever you depend on, please use:

• cloud.google.com/go v0.37.4
• google.golang.org/api v0.3.2
• go.opencensus.io v0.20.1