Protobuf: Implement custom options

[kerinin]

Custom option types appear to be broken - generated types do not implement ExtensionMap. Example protobuf definition:

syntax = "proto3";

import "google/protobuf/descriptor.proto";

extend google.protobuf.MessageOptions {
  string foo = 50001;
}


message MyMessage {
  option (foo) = "bar";
}

Attempting to access the option fails:

func main() {
    m := test.MyMessage{}

    ex, err = proto.GetExtension(&m, test.E_Foo)
    fmt.Printf("MyMessage.GetExtension: %+v, %+v\n", *ex.(*string), err)
}

// ./test.go:18: cannot use &m (type *test.MyMessage) as type proto.extendableProto in argument to proto.GetExtension:
//  *test.MyMessage does not implement proto.extendableProto (missing ExtensionMap method)

GetExtension succeeds with the original descriptor types (MessageOptions etc) if the extension defines a default type, but that provides pretty limited utility.

[cv]

I just bumped into the exact same issue. Is there a workaround?

[zellyn]

ExtensionMap() was removed recently. Make sure you have the latest version of github.com/golang/protobuf/proto: it should work with both old and new generated proto code.

[danny-cassidy]

I am also not able to get custom options working correctly with proto3 in golang.

I used the same .proto file supplied in the original post and the same main() functionality. I get this error:

proto: not an extendable proto

Here is the auto-generated go file:

type MyMessage struct {
}

func (m *MyMessage) Reset()                    { *m = MyMessage{} }
func (m *MyMessage) String() string            { return proto.CompactTextString(m) }
func (*MyMessage) ProtoMessage()               {}
func (*MyMessage) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{0} }

var E_Foo = &proto.ExtensionDesc{
    ExtendedType:  (*google_protobuf.MessageOptions)(nil),
    ExtensionType: (*string)(nil),
    Field:         50001,
    Name:          "foo",
    Tag:           "bytes,50001,opt,name=foo",
}

func init() {
    proto.RegisterType((*MyMessage)(nil), "MyMessage")
    proto.RegisterExtension(E_Foo)
}

func init() { proto.RegisterFile("example.proto", fileDescriptor0) }

var fileDescriptor0 = []byte{
    // 129 bytes of a gzipped FileDescriptorProto
    0x1f, 0x8b, 0x08, 0x00, 0x00, 0x09, 0x6e, 0x88, 0x02, 0xff, 0xe2, 0xe2, 0x4d, 0xad, 0x48, 0xcc,
    0x2d, 0xc8, 0x49, 0xd5, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x97, 0x52, 0x48, 0xcf, 0xcf, 0x4f, 0xcf,
    0x49, 0xd5, 0x07, 0xf3, 0x92, 0x4a, 0xd3, 0xf4, 0x53, 0x52, 0x8b, 0x93, 0x8b, 0x32, 0x0b, 0x4a,
    0xf2, 0x8b, 0x20, 0x2a, 0x94, 0x44, 0xb8, 0x38, 0x7d, 0x2b, 0x7d, 0x53, 0x8b, 0x8b, 0x13, 0xd3,
    0x53, 0xad, 0xd8, 0xbb, 0xb6, 0x4a, 0x30, 0x27, 0x25, 0x16, 0x59, 0x19, 0x73, 0x31, 0xa7, 0xe5,
    0xe7, 0x0b, 0xc9, 0xeb, 0x41, 0xf4, 0xeb, 0xc1, 0xf4, 0xeb, 0x41, 0x55, 0xfa, 0x17, 0x94, 0x64,
    0xe6, 0xe7, 0x15, 0x4b, 0x5c, 0x6c, 0x63, 0x56, 0x60, 0xd4, 0xe0, 0x0c, 0x02, 0xa9, 0x4e, 0x62,
    0x03, 0xab, 0x32, 0x06, 0x04, 0x00, 0x00, 0xff, 0xff, 0x4c, 0xa2, 0x4c, 0x3c, 0x84, 0x00, 0x00,
    0x00,
}

Let me know if I am doing anything wrong or if you need more information.

[junghoahnsc]

I have the same issue. How can I get the descriptor of a proto?
I think I have to do something like MyMessage.GetDescriptor().GetExtension(E_Foo), but I couldn't find how to do that.

[junghoahnsc]

Does anyone know whether this is supported or not? I tried to look the source tree, but I couldn't find any clue.

[bcmills]

    m := test.MyMessage{}
    ex, err = proto.GetExtension(&m, test.E_Foo)

is definitely not the way to get options for the message. They're set as extensions on the MessageOptions, not the value returned by the Descriptor method itself.

I don't think the Descriptor method is intended for public consumption. It returns a serialized, gzip-compressed FileDescriptorProto and, I believe, a path of indices to follow (via the MessageType field of FileDescriptorProto and then perhaps the NestedType field of DescriptorProto?) to get to the message type definition. It should be possible to implement an accessor for the actual DescriptorProto on top of that, but I'm not aware of anything in the proto package API that does.

Once you have the DescriptorProto — and assuming that the protocol compiler correctly preserved all of its extensions, you should be able to call GetExtension on _that_ to access the message options.

[junghoahnsc]

Thanks for the information.

I could access MessageOptions by

1. get gzipped FileDescriptorProto and Index by MyMessage.Desctipor()
2. ungzip and Unmarshal to FileDescriptorProto
3. get DescriptorProtot from FileDescriptorProto's message_type with Index
4. Access MessageOptions by proto.GetExtension

Is this the right way to do that?
As @bcmills mentioned, Descriptor() is intended for public consumption. Is there any other safe way to do that?

[bcmills]

Yep, that sounds about right. I'm not sure it's a huge amount of work to just provide a library that does all that; I'll see what I can do this week.

[bcmills]

You should be able to use the new descriptor subpackage to extract custom options from the descriptor.

@kerinin's example should be something like:

import (
    …
    "github.com/golang/protobuf/descriptor"
)

func main() {
    m := test.MyMessage{}

    _, md := descriptor.ForMessage(&m)
    ex, err = proto.GetExtension(md, test.E_Foo)
    fmt.Printf("MyMessage.GetExtension: %+v, %+v\n", *ex.(*string), err)
}

[timleung22]

It's not working for me... fd and md returned by the ForMessage () func is always google/protobuf/descriptor.proto and DescriptorProto respectively because my proto files import that.

In my proto file Booking.proto:

package com.example
import "google/protobuf/descriptor.proto"
...

and calling ForMessage(descriptor) where descriptor is the *DescriptorProto of my message "BookingStatus", just gives me google/protobuf/descriptor.proto and DescriptorProto

[ericchiang]

For those following this thread the example above no longer works and should be:

import (
    …
    "github.com/golang/protobuf/descriptor"
)

func main() {
    m := test.MyMessage{}

    _, md := descriptor.ForMessage(&m)
    ex, err = proto.GetExtension(md.Options, test.E_Foo) // "md.Options" not "md"
    fmt.Printf("MyMessage.GetExtension: %+v, %+v\n", *ex.(*string), err)
}

[ghost]

Thanks @ericchiang
this whole pb options thing is horrible dx in go.

[mayank-avinetworks]

@bcmills
Do we have something like below for ENUMS generated via protobufs which are not the properties of any message and only have been defined and generated via protobuf.

'
import (
…
"github.com/golang/protobuf/descriptor"
)

func main() {
m := test.MyMessage{}

_, md := descriptor.ForMessage(&m)
ex, err = proto.GetExtension(md.Options, test.E_Foo) // "md.Options" not "md"
fmt.Printf("MyMessage.GetExtension: %+v, %+v\n", *ex.(*string), err)

}
'

[neild]

To look at enum options, you'll want to look in the *FileDescriptorProto returned by descriptor.Options (the first return parameter) to find the *EnumDescriptorProto you're interested in.

In the google.golang.org/protobuf API (still not officially released, but getting close), this is a bit simpler since generated enum types have a Descriptor method which can be used to get at the EnumOptions message.

[mayank-avinetworks]

Dear @neild ,
Thanks for you reply. I tried the approach but the exact understanding of the implementation is still vague. My use case is I just need to read the string assigned to 'e_format_str' to a particular enum value say 123 in my case .

below is the sample enum definition for me for which the e_format_str is registered as well

enum abc { 
    .
    .
    . 
     XYZ_IN_PROGRESS = 123 [
        (e_format_str) = "'XYZ in progress.'"
    ];
    .
    .
    . 
}

I get the generated enum go code. This type does not satisfy the interface to use descriptor.ForMessage() as the generate ENUM does not use Descriptor() method but EnumDescriptor() method. so I'm fetching the FileDescriptor with EnumDescriptor() and doing GUNZIP myself and able to reach upto EnumDescriptorProto and can get EnumValueDescriptor as well from it. But I'm getting the whole lot of information for the whole file with which to traverse by specifying indices and this approach will not help me as if there are more enums defined in same file or more enum values are added it can distort the indices.

Below is my debug raw code :

func callme(){
    enumVar := pb.XYZ__IN_PROGRESS.Enum()
    gz, _ := enumVar.EnumDescriptor() // success
    fd, lError := extractFile(gz)
    if lError != nil {
        glog.Infof("DEBUG: Error in GUNZIP: %+v", lError)
        return
    }
    glog.Infof("raw FD: %+v", *fd)
    enumType := fd.GetEnumType()
    glog.Infof("DEBUG: Enum Descr Proto: %+v", enumType)
    enumValue := enumType[0].GetValue()      // have to use index
    glog.Infof("DEBUG: Enum Value Descr: %+v", enumValue)      // cannot keep traversing like above with index as I'm not sure what will the index be in future.
}

The above snippet is giving me a good details but not exactly for the value which I want to as I'm going via FileDescriptorProto. I'm sure I've not been able to comprehend the godoc guide properly for protobuf structure and there would be a way to directly pin point to the 'options' for the enum value which I instantiated.

Is there any way with which I can get Options for the particular enum value just by passing the 'XYZ_IN_PROGRESS' or '123'

[neild]

Currently, you will need to traverse the FileDescriptorProto until you find the enum descriptor you're looking for. For example,

var fd *descpb.FileDescriptorProto // get this from somewhere, as above
for _, ed := range fd.EnumType {
  if ed.Name() == "proto.package.name.abc" {
    // use this descriptor
  }
}

Note that if the enum is nested in a message, you will need to traverse into the DescriptorProto for that message to find it. The definition of the descriptor protos is here: https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto)

Once you have an EnumDescriptorProto, you can fetch your option from it:

xo, err = proto.GetExtension(ed.Options, testpb.E_EFormatStr)

As I mentioned above, this is all quite a bit simpler in the upcoming google.golang.org/protobuf API revision:

ext := proto.GetExtension(somepb.Enum_VALUE.Descriptor().Options(), testpb.E_EFormatStr)

[mayank-avinetworks]

@neild : I understood it now, will give it a try. Thanks a lot for your help. Hope this conversation helps other people as well.

[mayank-avinetworks]

@all , I'll add one more hierarchy in conjunction to @neild 's response to my question so that Options for the enum are read without fail.

func GetEnumOption(ErrorEnum *pb.errEnum) string {
    gz, _ := ErrorEnum.EnumDescriptor()
    fd, err := extractFile(gz) // get FD from generated code and extract it
    if err != nil {
       // Error
    }
    enumStr := ErrorEnum.String()

    for _, ed := range fd.GetEnumType() {
        for _, enumValueDescriptor := range ed.GetValue() {
            if enumValueDescriptor.GetName() == enumStr {
                if enumValueDescriptor.Options != nil {     // In case any Enum has not implemented any option
                    enumStrDescr, err := proto.GetExtension(enumValueDescriptor.Options, pb.E_EFormatStr)
                    if err != nil {
                        glog.Errorf("Error while fetching option for enum: %s", enumStr)
                        //error
                    }
                    return *enumStrDescr.(*string)
                } else {
                    glog.Infof("Enum: %s does not have options ", enumStr)
                }
            }
        }
    }
    // return
}  

Thanks.