Protobuf: APIv2: How to compile proto file that imports descriptor.proto?

[meling]

I’m playing with API-v2. When compiling a proto file that imports the descriptor.proto file, I get a compile error on this line (in the generated code):

// This is a compile-time assertion that a sufficiently up-to-date version
// of the legacy proto package is being used.
const _ = proto.ProtoPackageIsVersion4

The command I used to compile (gives no errors while generating):

protoc --go_out=Mgoogle/protobuf/descriptor.proto=github.com/golang/protobuf/protoc-gen-go/descriptor:. gorums.proto

PS: The descriptor.proto file that I’m mapping to here is in the v1.3.3 folder. However, the new API-v2 does not seem to contain this file…

The file I’m trying to compile is gorums.proto:

syntax = "proto3";

package gorums;

import "google/protobuf/descriptor.proto";

option go_package = "github.com/relab/gorums";

extend google.protobuf.MethodOptions {
  bool qc = 50000;
  bool correctable = 50001;
  bool correctable_stream = 50002;
  bool multicast = 50003;
  bool qc_future = 50004;
  bool qf_with_req = 50005;
  bool per_node_arg = 50006;

  string custom_return_type = 51000;
}

Any pointers would be much appreciated.

[neild]

Thanks for trying out APIv2!

The error is because the generated .pb.go is using a version of the github.com/golang/protobuf/proto package that's too old. Unfortunately, the version you need isn't released yet. Assuming you're using modules, you can add the necessary prerelease version to go.mod with:

go get -u github.com/golang/protobuf/proto@api-v1

[meling]

Just to clarify; did you mean v1 or v2 in the go get command? (Not at computer; will test tomorrow...)

[neild]

I mean @api-v1--this specifies a branch that includes a version of APIv1 that uses APIv2 under the hood.

[neild]

Promoting this to blocks-v2, since it raises an interesting question:

Should google.golang.org/protobuf have a dependency on github.com/golang/protobuf, at least in the short term, to ensure that anyone using APIv2 gets an appropriately new version of APIv1?

We've wanted to avoid that dependency, but perhaps it's worth it at least for a transitional period to avoid confusion.

[meling]

Just wanted to report back that the go get command above did solve the problem for me.

About the short term dependency issue: It could save people a fair amount of initial debug time for the transitional period. Obviously, once the dependency is removed, people may run into problems then. But maybe it would smooth out the pain.

Another alternative is of course to write up a migration/troubleshooting guide, and place it visibly on the github page and elsewhere... Maybe also write a blog post at golang.org about the release. Let me know if you plan to do so; I'd be happy to review it from an external perspective, prior to publication.

[meling]

Okay, so the first part worked, but now I want to also build a proto with a gRPC service, and I run into a weird problem with Go modules (probably VSCode) updating the go.mod file.

Here is the file I'm compiling:

syntax = "proto3";
package goproto.protoc.gorums;
option go_package = "github.com/relab/gorums/cmd/protoc-gen-gorums/testdata/basic";

service Storage {
  rpc Read(ReadRequest) returns (ReadResponse) {}
}

message ReadRequest {}
message ReadResponse {}

I have this in ...gorums/go.mod (project root):

module github.com/relab/gorums

go 1.13

require (
    github.com/golang/protobuf v1.2.1-0.20200212180908-0fd14f96108a
    google.golang.org/protobuf v0.0.0-20200213061406-709e7c8474b1
)

And this in .../gorums/cmd/protoc-gen-gorums/testdata/go.mod:

module github.com/relab/gorums/cmd/protoc-gen-gorums/testdata

go 1.13

require (
    google.golang.org/grpc v1.28.0-pre.0.20200205181625-597699c0ef29
    github.com/golang/protobuf v1.2.1-0.20200212180908-0fd14f96108a
    google.golang.org/protobuf v0.0.0-20200213061406-709e7c8474b1
)

When I run:

protoc --go_out=paths=source_relative:. --go-grpc_out=paths=source_relative:. basic/basic.proto

It generates the expected .pb.go and _grpc.pb.go files. Soon thereafter, I assume that VSCode detects the two files, and updates the go.mod file in the project root as follows:

module github.com/relab/gorums

go 1.13

require (
    github.com/golang/protobuf v1.3.2
    github.com/google/go-cmp v0.3.1 // indirect
    google.golang.org/grpc v1.27.1 // indirect
    google.golang.org/protobuf v0.0.0-20200213061406-709e7c8474b1
)

I should point out that I have no other .go files in the entire project. I also tried to close down VSCode and run go build in the directory with the generated files. In this case, the go.mod file in testdata folder gets updated in a similar way to the above (though some different versions; I did recompile the v2 protoc-gen-go and protoc-gen-grpc between to ensure I was actually using v2, which may or may not be related to the different versions):

module github.com/relab/gorums/cmd/protoc-gen-gorums/testdata

go 1.13

require (
    github.com/golang/protobuf v1.3.3
    github.com/google/go-cmp v0.3.1 // indirect
    google.golang.org/grpc v1.28.0-pre.0.20200205181625-597699c0ef29
    google.golang.org/protobuf v0.0.0-20200213061406-709e7c8474b1
)

And resulting in the following compiler error:

# github.com/relab/gorums/cmd/protoc-gen-gorums/testdata/basic
./basic.pb.go:26:11: undefined: "github.com/golang/protobuf/proto".ProtoPackageIsVersion4

I would very much appreciate any input towards resolving these issues. Thanks!

PS: Maybe this is related to my use of two go.mod files and the special directory testdata, but I'm following the template from the protobuf@api-v2 repo. Maybe that's a bad idea?

[neild]

The problem turns out to be that the pseudoversion for api-v1 sorted earlier than the version depended on by the gRPC module, leaving you in a state where it's impossible to satisfy all the dependencies at the same time.

I've just tagged github.com/golang/[email protected], which contains the proper revision at an appropriately high (but unreleased) version. I believe everything will work properly now if you go get -u github.com/golang/[email protected].

Sorry for the trouble, and thank you for trying this out!

[neild]

And I just submitted a change to add a dependency from APIv2 to APIv1, which means you should no longer need to manually update github.com/golang/protobuf: Any program using APIv2 will also automatically use a sufficiently new version of APIv1.

[meling]

Thanks a lot! It now seems to build correctly on command line... I only tried to run go build -a -v -x without any "users" of the generated code yet, so no binary is produced in the local folder, but the last line of the output is cp $WORK/b001/_pkg_.a /Users.../Library/Caches/.... So I assume this is ok.

For documentation; I provide a description of behaviors experienced...

For some weird reason VSCode / gopls is unable to detect the types ReadRequest and ReadResponse (the message types) from the grpc file, even though both files are in the same folder, and have the same package name. I get:

undeclared name: ReadRequest
undeclared name: ReadResponse
...

(this seems to be related to how the go tooling (VSCode and gopls) treats the testdata folder in which the source files were located.)

I've tried to restart the language server and reload the VSCode window to no avail. And after restarting VSCode entirely, I get:

Package github.com/golang/protobuf/proto is deprecated: Use the "google.golang.org/protobuf/proto" package instead.  (SA1019)

(seems this last problem may be the staticcheck linter. I changed to golint and the problem seems to have gone away.)

Thanks for all your help so far! Much appreciated.

[meling]

I updated the comment above to reflect my latest findings. Seems that following the template of golang/protobuf with the testdata folder for building a new protobuf generator is unwise, at least if you want to have tests inside testdata that can compile in VSCode (or with other go tooling).

(Edit: It "may" actually be related to a problem with VSCode in handling multiple modules in a monorepo https://github.com/microsoft/vscode-go/issues/2447)

[neild]

Thanks again for trying out APIv2!