Rules_go: Build protobufs with gogoproto?

Created on 8 Sep 2017 · 12Comments · Source: bazelbuild/rules_go

We're currently using Bazel, mostly, for a Go project where we want to build protobufs with gogoprotobuf so that we can get the improved marshaling code.

Currently, we have rules like this in our Makefile:

PROTO_DEFS := $(shell find . -name tools -prune -o -name vendor -prune -o -type f -name '*.proto' -print)
PROTO_GOS := $(patsubst %.proto,%.pb.go,$(PROTO_DEFS))

%.pb.go: build-image/$(UPTODATE)
    protoc -I ./vendor:./$(@D) --gogoslick_out=plugins=grpc:./$(@D) ./$(patsubst %.pb.go,%.proto,$@)

bazel: $(PROTOS_GO)
    bazel build //cmd/...

I'd like to be able to dispense with the makefile and just use Bazel.

Looking at the rules, I can't quite figure out how to get that critical --gogoslick_out option into the command that Bazel uses to generate the Go protocol library.

Source

jml

All 12 comments

In #803 I submitted a whole new set of proto rules.
I have not added much in the way of documentation or examples yet, but gogo support was specifically one of the goals of the system. It should be possible to substitute the go_proto_toolchain with one that uses the gogo plugin instead and everything else will just work. This would be very hard to do in the old proto rules.
If you want me to help you try it out and your project is on github we can do that, or you can try to do it and ask questions here, or you can wait a while until I write all the documentation for the new systems we have been adding recently (tracked in #810)

ianthehat on 8 Sep 2017

❤1

Thank you, that's very kind. The project is https://github.com/weaveworks/cortex. The Makefile snippets are from a PR https://github.com/weaveworks/cortex/pull/557 that I expect to be merged soon. Your help would be much appreciated.

Worst case though, I can puzzle out the new rules. Thanks!

jml on 8 Sep 2017

I have got a long way, but hit a couple of issues. The biggest is that you don't vendor enough of gogo to build the plugin you are using (or any of it's plugins for that matter).
I am thinking I will add it to your WORKSPACE and use that copy instead, and also possibly see if I can upstream the build files and rules to the gogo project.

ianthehat on 8 Sep 2017

👍1

Thanks! I've merged weaveworks/cortex#557 since it's an unambiguously positive step forward. Hope this isn't too inconvenient.

jml on 11 Sep 2017

I have not been able to work on this over the weekend, and will not be able to today or tomorrow either as I am working on a different computer. Just letting you know I have not forgotten and will come back to it later in the week.

ianthehat on 12 Sep 2017

👍1

And now I have put up weaveworks/cortex#564 which is fully working (I believe, bazel build //cmd/... works anyway)

ianthehat on 15 Sep 2017

@ianthehat with the way you did it, is it possible to mix gogo in one package and original protobuf in another?

ali5h on 22 Sep 2017

@shariat Sorry, I only just noticed the question. No, currently it is not. In theory the design allows it, in that you could write a mega toolchain that selected a different implementation based on options supplied on the go_proto_library rule, at the moment though I am not aware of anyone that really wants this.

There are a couple of places where I am not happy with the design, this is one, the other is the normal/grpc rule split. If anyone has suggestions for a cleaner mechanism that does not require knowing all possible implementations up front and does not require carrying dependencies you don't really have, I would love to hear them.

ianthehat on 4 Oct 2017

@ianthehat I see. well, I may be the one that will need it. we have a mono repo and moving to use bazel for building it and we have used different protobuf compilers for different projects. may be the solution is to have multiple workspaces in one repo for now?

ali5h on 4 Oct 2017

what is the status of this? is it possible to build the go_proto using gogo? the latest version of the rules_go repository seems to break the rules_protobuf ones.

Mistobaan on 13 Oct 2017

It is now possible to mix proto plugins, you can even build the same proto package with multiple plugins.
I have updated weaveworks/cortex#564 with the most up to date versions.

ianthehat on 5 Dec 2017

I still can't figure out how to build protobufs with gogoproto. Is https://github.com/pubref/rules_protobuf still needed with rules_go >= 0.8? I have seen there was a compiler attribute added. Would anyone be so kind to share an example?