Protobuf: Protocol Buffers 3 parser in Go?

[myitcv]

_Asking here as it seems the most relevant and specific audience._

Is there in existence a Go-based package for parsing protocol buffers 3 .proto files?

[dsymonds]

There's github.com/dsymonds/gotoc, but that's not an official project. Other than that, no, there's no Go-based .proto parsers that I know of.

[myitcv]

@dsymonds thanks very much, exactly what I was after.

[myitcv]

Having just seen https://github.com/golang/protobuf/issues/155, I thought I should note here that I am maintaining a copy of github.com/dsymonds/gotoc/internal/* here:

https://github.com/myitcv/protobuf

The code is quick and dirty for now but has added the following improvements:

• Fix parser file search algo
• Allow parsing of multi-line strings
• Parsing of custom field options
• Parsing of custom method options

Contributions welcomed and of course thanks to @dsymonds for the original code

[emicklei]

https://github.com/emicklei/proto is another package that also accepts version 2. Thanks to @myitcv for mentioning the @dsymonds package and this issue.

[sirkon]

@emicklei, thank you for the great work

Still, sometimes exact file positioning is needed to locate errors and warnings. For instance, we are thinking about specific documentation generation where each enum, fiield, service (grpc) and message(grpc) must have a comment and we should point exact position of an element what lacks it. Am I missed something, or does your version not having attached lexems indeed?

[emicklei]

Related to https://github.com/emicklei/proto/issues/7, someone is working on adding LineNumbers to the nodes. Column position would also be needed.

[sirkon]

Great!

[awalterschulze]

Someone recently told me that the bnf has become available:

https://developers.google.com/protocol-buffers/docs/reference/proto2-spec
https://developers.google.com/protocol-buffers/docs/reference/proto3-spec

something that I have been waiting for years.
Maybe you will find it useful.

[yoheimuta]

https://developers.google.com/protocol-buffers/docs/reference/proto3-spec

https://github.com/yoheimuta/go-protoparser is yet another package. It is designed to strictly follow the bnf. I hope this will help.

[neild]

FWIW, you can convert a .proto source file into a manipulable form using protoc:

protoc --include_imports --include_source_info -oFILE foo.proto

This will write a FileDescriptorSet message (defined in descriptor.proto) containing the input files to FILE. The --include_imports and --include_source_info flags will include dependencies and source positioning information.

[sirkon]

@neild this doesn't work that well. For instance, you don't have positions (file, line, column) for options.

[neild]

I see source_code_info annotations for options when I try it, FWIW.

[sirkon]

@neild

syntax = "proto3";

import "google/api/annotations.proto";

message Message {
        string id = 1;
};

service Service {
        rpc Method(Message) returns(Message) {
            option (google.api.http) = {
                   post: "/v1/message"
                   body: "*"
            };
        };
};

Here's is the protoc dump: json-dump.zip

Please point, what part of source_code_info.location refers to body key in google.api.http option.

[neild]

            {
                "path": [
                    6,
                    0,
                    2,
                    0,
                    4,
                    999,
                    0,
                    8
                ],
                "span": [
                    10,
                    51,
                    13,
                    25
                ]
            }

Doesn't drill into the option content, so if you want to identify the position of elements within the HttpRule message you still have work to do. Still, it gets you past the .proto file parsing.