Rules_go: Unable to build a binary that uses cloud.google.com/go/storage

[dlorenc]

I trimmed this down to a pretty minimal example, but here's my main.go file:

package main

import (
    "context"
    "fmt"
    "log"

    "cloud.google.com/go/storage"
)

func main() {
    ctx := context.Background()
    client, err := storage.NewClient(ctx)
    if err != nil {
        log.Fatalln("Error creating storage client: ", err)
    }
    fmt.Println(client)
}

I installed cloud.google.com/go/storage with dep:

dep ensure --add cloud.google.com/go/storage

Everything builds and runs with the go tool:

go build .

I hit a few issues when trying to use this with the rules_go tooling. gazelle is able to generate build files, but when I try to use these to build, I get this:

.../home/dlorenc/newgo/src/github.com/dlorenc/bazel-cache-gcs/vendor/google.golang.org/grpc/grpclb/grpc_lb_v1/messages/BUILD.bazel:10:1: output 'vendor/google.golang.org/grpc/grpclb/grpc_lb_v1/messages/messages_go_proto/google.golang.org/grpc/grpclb/grpc_lb_v1/messages/messages.pb.go' was not created
ERROR: .../google/home/dlorenc/newgo/src/github.com/dlorenc/bazel-cache-gcs/vendor/google.golang.org/grpc/grpclb/grpc_lb_v1/messages/BUILD.bazel:10:1: not all outputs were created or valid
Target //:bazel-cache-gcs failed to build

It looks like some conflict with the files fetched from dep - messages.pb.go already exists in that directory.

Here's the generated BUILD.bazel file at fault:

load("@io_bazel_rules_go//go:def.bzl", "go_library")
load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library")

proto_library(
    name = "messages_proto",
    srcs = ["messages.proto"],
    visibility = ["//visibility:public"],
)

go_proto_library(
    name = "messages_go_proto",
    importpath = "google.golang.org/grpc/grpclb/grpc_lb_v1/messages",
    proto = ":messages_proto",
    visibility = ["//visibility:public"],
)

go_library(
    name = "go_default_library",
    importpath = "google.golang.org/grpc/grpclb/grpc_lb_v1/messages",
    library = ":messages_go_proto",
    visibility = ["//visibility:public"],
)

Changing it up to remove the proto generation:

load("@io_bazel_rules_go//go:def.bzl", "go_library")

go_library(
    name = "go_default_library",
    srcs = ["messages.pb.go"],
    deps = ["//vendor/github.com/golang/protobuf/proto:go_default_library"],
    importpath = "google.golang.org/grpc/grpclb/grpc_lb_v1/messages",
    visibility = ["//visibility:public"],
)

Gets past that error, but I hit this one:

dlorenc@dlorenc:~/newgo/src/github.com/dlorenc/bazel-cache-gcs$ bazel build //:bazel-cache-gcs
INFO: Analysed target //:bazel-cache-gcs (0 packages loaded).
INFO: Found 1 target...
ERROR: .../dlorenc/newgo/src/github.com/dlorenc/bazel-cache-gcs/BUILD.bazel:17:1: GoLink bazel-cache-gcs failed (Exit 1)
google.golang.org/grpc/naming.init: call to external function
google.golang.org/api/internal.init: relocation target google.golang.org/grpc/naming.init not defined
google.golang.org/api/internal.init: undefined: "google.golang.org/grpc/naming.init"
2017/10/30 09:53:53 error running linker: exit status 1
Target //:bazel-cache-gcs failed to build

Which looks like https://github.com/bazelbuild/rules_go/issues/813, but I'm fairly sure I haven't introduced any circular dependencies myself.

Any thoughts on how to proceed?

[dlorenc]

Note that downgrading to rules_go 0.5.5 works for me.

[byu]

Chiming in because I'm seeing the same thing, but for google cloud spanner. But I imagine that because the error is in a common package that it'll affect all of google cloud go client libs.

Env:

1. Using rules_go v0.7.0 .
2. Using the vendor/ directory with Gopkg.toml and dep ensure
3. Also using bazel && gazelle to generate the BUILD files.

Thoughts:

1. Not copy-pasting my own logs since they pretty much match what was reported above.

1. The error message reasoning:

It looks like some conflict with the files fetched from dep - messages.pb.go already exists in that directory.

I get the same type of "not all outputs were created or valid" with my own protofiles in cases where there some sort of mismatch among the gazelle generated BUILD.bazel directives, the protobuf's go_package name and the protobuf's actual path/filename.

I had two workarounds here to get a build to successfully compile. But since I'm not actually a bazel or rules_go or google cloud developer... this is just what I found to work without diving into any of that code.

1. Workaround 1: rm messages.proto before bazel run //:gazelle

Since the messages.pb.go was already pregenerated, no need to have a build rule for messages.proto.

1. Workaround 2: Edit messages.proto go_package option.

option go_package = "google.golang.org/grpc/grpclb/grpc_lb_v1/messages";

This way, the go_proto_library directive won't blow up, since the generated file will end up in the expected importpath.

go_proto_library( name = "messages_go_proto", importpath = "google.golang.org/grpc/grpclb/grpc_lb_v1/messages", proto = ":messages_proto", visibility = ["//visibility:public"],

1. Not sure what the right fix is, or who's responsibility it is...
   a. Is it a google problem that the released client library isn't immediately friendly to bazel builds (should they also include BUILD files?)
   b. Is it a bazel rules_go problem because it's gazelle trying to auto generate BUILD files from code?

[JensRantil]

I can confirm workaround 1 worked like a charm for me.

[JensRantil]

Proposal: Change the title to something like "Pregenerated protobuf files fail build".

[ianthehat]

This should all be fixed now.
The new proto rules are much better at finding the outputs, so they should work, and if they do not you can set build_file_proto_mode = "disable" on the go_repository rule to disable all proto handling (which will just used the checked in .pb.go files)

[raliste]

I'm unable to build with the latest go rules.

I vendored cloud.google.com/go/storage, gazelle probably deps for x/net/context/trace that collides with external/org_golang_x_net/trace/trace.go as I'm algo using @org_golang_google_grpc, thus generating:

panic: http: multiple registrations for /debug/requests

goroutine 1 [running]:
net/http.(*ServeMux).Handle(0x1c83a80, 0x17f26fc, 0xf, 0x1875220, 0x181e958)
    GOROOT/src/net/http/server.go:2353 +0x239
net/http.(*ServeMux).HandleFunc(0x1c83a80, 0x17f26fc, 0xf, 0x181e958)
    GOROOT/src/net/http/server.go:2368 +0x55
net/http.HandleFunc(0x17f26fc, 0xf, 0x181e958)
    GOROOT/src/net/http/server.go:2380 +0x4b
golang.org/x/net/trace.init.0()
    external/org_golang_x_net/trace/trace.go:115 +0x42

[jayconrod]

It doesn't seem like you can safely have multiple copies of golang.org/x/net/trace in the same build. If you have a vendored copy of it, I'd suggest adding # gazelle:exclude vendor/golang.org/x/net/trace to prevent Gazelle from visiting that directory. Gazelle will default to using external dependencies when it can't find a local library that provides a package.