Google-cloud-go: spanner: Version upgrade caused build failure with Bazel

[rytswd]

Client

Spanner

Environment

Bazel build (have not seen otherwise)

Code

I upgraded spanner version from v1.2.0 to v1.4.0 in go.mod. This also meant the upgrade of cloud.google.com/go from v0.52.0 to v0.56.0.

Expected behavior

The upgrade goes smoothly without errors in build.

Actual behavior

Build error occurs.

Details

spanner v1.2.0 to v1.4.0

The upgrade in go.mod caused the following error when building with Bazel:

ERROR: /github/home/.cache/bazel/_bazel_***/337a54448fa7739f30c624151e3e44a1/external/com_google_cloud_go_spanner/apiv1/BUILD.bazel:3:1: GoCompilePkg external/com_google_cloud_go_spanner/apiv1/linux_amd64_stripped/go_default_library%/cloud.google.com/go/spanner/apiv1.a failed (Exit 1)
/b/f/w/external/com_google_cloud_go_spanner/apiv1/spanner_client.go:248:37: cannot use connPool (type "google.golang.org/api/internal".ConnPool) as type *"google.golang.org/grpc".ClientConn in argument to spanner.NewSpannerClient

I did find #1789 mentioned the similar error, and the version mismatch of genproto was the cause. The indirect dependency to genproto was upgraded from v0.0.0-20200212174721-66ed5ce911ce to v0.0.0-20200331122359-1ee6d9798940 - which meant both versions already had https://github.com/googleapis/go-genproto/pull/296.

The above error only suggests that I'm getting a wrong version of genproto being referenced when running build with Bazel.

Side note: I tried an upgrade of genproto (although it was indirectly updated already), for which I got an error - it was not relevant, but puzzled me a bit...:

$ go get -u google.golang.org/genproto
...
can't load package: package google.golang.org/genproto: build constraints exclude all Go files in /Users/ryota/go/pkg/mod/google.golang.org/[email protected]

Bazel rules_go v0.21.2 to v0.22.2

As I dug into it a bit further, I found https://github.com/bazelbuild/rules_go/issues/2376 mentioning how bazelbuild/rules_go had to be updated. I was on v0.21.2, so updated to v0.22.2, and that fixed one issue of ConnPool; but raised another.

ERROR: /github/home/.cache/bazel/_bazel_***/337a54448fa7739f30c624151e3e44a1/external/com_google_cloud_go_spanner/BUILD.bazel:3:1: GoCompilePkg external/com_google_cloud_go_spanner/linux_amd64_stripped/go_default_library%/cloud.google.com/go/spanner.a failed (Exit 1)
/b/f/w/external/com_google_cloud_go_spanner/batch.go:185:3: unknown field 'QueryOptions' in struct literal of type "google.golang.org/genproto/googleapis/spanner/v1".ExecuteSqlRequest
/b/f/w/external/com_google_cloud_go_spanner/client.go:286:20: undefined: "google.golang.org/genproto/googleapis/spanner/v1".ExecuteSqlRequest_QueryOptions
/b/f/w/external/com_google_cloud_go_spanner/pdml.go:76:3: unknown field 'QueryOptions' in struct literal of type "google.golang.org/genproto/googleapis/spanner/v1".ExecuteSqlRequest
/b/f/w/external/com_google_cloud_go_spanner/transaction.go:222:11: undefined: "google.golang.org/genproto/googleapis/spanner/v1".ExecuteSqlRequest_QueryOptions
/b/f/w/external/com_google_cloud_go_spanner/transaction.go:230:13: undefined: "google.golang.org/genproto/googleapis/spanner/v1".ExecuteSqlRequest_QueryOptions
/b/f/w/external/com_google_cloud_go_spanner/transaction.go:343:3: unknown field 'QueryOptions' in struct literal of type "google.golang.org/genproto/googleapis/spanner/v1".ExecuteSqlRequest

This error indicates the genproto is missing changes from https://github.com/googleapis/go-genproto/pull/311. This would be the error you'd get with spanner/v1.3.0 onward, even if you are on the latest version of rules_go v0.22.2 as of this writing.

Question

When a new genproto feature is used (in this case of Spanner, the use of QueryOptions in ExecuteSqlRequest which is missing in the previous version), Bazel user needs to wait until the new genproto is integrated back to bazelbuild/rules_go? How can I (and the community) better handle the upgrade path?

(Apologies if this is the wrong repo to raise an issue for.)

[codyoss]

Are you able to fix this with: https://github.com/bazelbuild/rules_go/blob/master/go/workspace.rst#overriding-dependencies ?

I am no familiar enough /w bazel to be super helpful here, honestly speaking. If that does not work I will try to cc someone in /w more knowledge in the space.

In general we have have changes made to genproto and cloud.google.com/go daily. These usually come out as official releases ~1 week.(cloud go that is). We in the client library space do not directly support bazelbuild/rules_go, although we do try to help out when we can. I will try to ask around to see if there is more we can do to keep these projects better aligned.

[rytswd]

Thanks for the info @codyoss, I have been trying the overriding option since I opened this ticket a few hours ago, but have not succeeded to override correctly. I'm still working on it.

I understand it wouldn't scale if changes here need to have a separate release on other repo... bazelbuild/rules_go side touches on the same concern here:

# * Most proto repos are updated more frequently than rules_go, and
#   we can't keep up.

I'm doing a further research on this, and will report if and when I find more.

[rytswd]

I have tried several steps, but the override doesn't seem to solve - I just cannot confirm if the override has any effect.

I have created a new repo to demo the simple setup (rytswd/simple-bazel), and the repro of the above error in a separate branch (and diff is here).

This may be better handled by Bazel folks. I will test a little more over the weekend, and create a ticket there if I cannot resolve it myself. (Also, I'll link this ticket for future reference for anyone who may stumble on the same problem.)

[rytswd]

The way to manage the dependency is discussed over on the bazelbuild/rules_go side, and I got to know that its quarterly release cycles would pick up the latest genproto at that point.

That means, in order to use the latest release from this repo, it may require some special handling with Bazel + rules_go + Gazelle. There won't be any involvement from this repo's perspective, and for anyone who may find this ticket in future, I'll make sure to link back to this ticket when I find more.
As to this ticket, I'm closing as there is no further action needed.

Apologies for incorrectly raising this here, and thanks for the prompt help!

[codyoss]

No worries, thanks for reporting back!