Protobuf: message with oneof cannot unmarshal from json string

[bh4rtp]

hi, i am using protobuf oneof, i.e. union. the message can be marshaled to json. but it cannot be unmarshaled back.

1. message definition

message Person {
    string name = 1;
    int32 age = 2;
    bool married = 3;
    oneof number {
        int32 i = 5;
        float f = 8;
    }
}

2. message initialization
msg := new(t.Person) msg.Name = "Allen" msg.Age = 30 msg.Married = true msg.Number = &t.Person_F{float32(14500.6)}
3. messsage output
name:"Allen" age:30 married:true f:14500.6
4. jsonpb marshal
{"name":"Allen","age":30,"married":true,"f":14500.6}
5. json marshal
{"name":"Allen","age":30,"married":true,"Number":{"F":14500.6}}
6. json & jsonpb unmarshal
{Allen 30 true <nil>}
as above line shows, the number.f is not unmarshaled. how to solve it?

[spraints]

I ran into this problem too. It's particularly frustrating because the marshaler produces json that the unmarshaler can't consume. https://gist.github.com/spraints/3cbd71e5f1ccbcd30dc1d266ec19ac78 is an example.

[cybrcodr]

@bh4rtp I just tried your example and it works for me on latest master or dev branches. Note that you have to use jsonpb. Here's what I have ...

package main

import (
    "fmt"
    "strings"

    "github.com/golang/protobuf/jsonpb"
    "github.com/golang/protobuf/proto"
)

func main() {
    msg := &Person{
        Name: "Allen",
        Age: 30,
        Married: true,
        Number: &Person_F{float32(14500.6)},
    }

    m := jsonpb.Marshaler{}
    js, err := m.MarshalToString(msg)
    if err != nil {
        fmt.Println(err)
        return
    }
    fmt.Printf("json output ==> %v\n", js)

    msg2 := &Person{}
    if err := jsonpb.Unmarshal(strings.NewReader(js), msg2); err != nil {
        fmt.Println(err)
        return
    }

    fmt.Printf("%+v\n", proto.MarshalTextString(msg2))
}

Outputs:

json output ==> {"name":"Allen","age":30,"married":true,"f":14500.6}
name: "Allen"
age: 30
married: true
f: 14500.6

[akutz]

Hi @cybrcodr,

I'm having a similar issue, and I wonder if it's because the OneOf I'm marshalling with jsonpb is an element in a slice?

    // The capabilities that the provisioned volume MUST have: the Plugin
    // MUST provision a volume that could satisfy ALL of the
    // capabilities specified in this list. The Plugin MUST assume that
    // the CO MAY use the  provisioned volume later with ANY of the
    // capabilities specified in this list. This also enables the CO to do
    // early validation: if ANY of the specified volume capabilities are
    // not supported by the Plugin, the call SHALL fail. This field is
    // REQUIRED.
    VolumeCapabilities []*VolumeCapability `protobuf:"bytes,4,rep,name=volume_capabilities,json=volumeCapabilities" json:"volume_capabilities,omitempty"`

// Specify a capability of a volume.
type VolumeCapability struct {
    // Specifies what API the volume will be accessed using. One of the
    // following fields MUST be specified.
    //
    // Types that are valid to be assigned to AccessType:
    //  *VolumeCapability_Block
    //  *VolumeCapability_Mount
    AccessType isVolumeCapability_AccessType `protobuf_oneof:"access_type"`
    // This is a REQUIRED field.
    AccessMode *VolumeCapability_AccessMode `protobuf:"bytes,3,opt,name=access_mode,json=accessMode" json:"access_mode,omitempty"`
}

// XXX_OneofFuncs is for the internal use of the proto package.
func (*VolumeCapability) XXX_OneofFuncs() (func(msg proto.Message, b *proto.Buffer) error, func(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error), func(msg proto.Message) (n int), []interface{}) {
    return _VolumeCapability_OneofMarshaler, _VolumeCapability_OneofUnmarshaler, _VolumeCapability_OneofSizer, []interface{}{
        (*VolumeCapability_Block)(nil),
        (*VolumeCapability_Mount)(nil),
    }
}

Everything marshals correctly, but on unmarshal I get the unknown field warning.

[cybrcodr]

Sorry for the late response on this. Can you provide a simple reproducible case with the proto definition and similar Go code as I've posted that marshals and unmarshals?

[zhangli623]

@bh4rtp I just tried your example and it works for me on latest master or dev branches. Note that you have to use jsonpb. Here's what I have ...

package main

import (
  "fmt"
  "strings"

  "github.com/golang/protobuf/jsonpb"
  "github.com/golang/protobuf/proto"
)

func main() {
  msg := &Person{
      Name: "Allen",
      Age: 30,
      Married: true,
      Number: &Person_F{float32(14500.6)},
  }

  m := jsonpb.Marshaler{}
  js, err := m.MarshalToString(msg)
  if err != nil {
      fmt.Println(err)
      return
  }
  fmt.Printf("json output ==> %v\n", js)

  msg2 := &Person{}
  if err := jsonpb.Unmarshal(strings.NewReader(js), msg2); err != nil {
      fmt.Println(err)
      return
  }

  fmt.Printf("%+v\n", proto.MarshalTextString(msg2))
}

Outputs:

json output ==> {"name":"Allen","age":30,"married":true,"f":14500.6}
name: "Allen"
age: 30
married: true
f: 14500.6

but why just marshaled json works, the native json can't unmarshaled however. How can I solve it?

[cybrcodr]

I'm not sure I fully understand your question. If you are asking how come using encoding/json works to marshal the proto message, why doesn't it work to unmarshal, that really depends on the field types being marshaled/unmarshaled. It may happen to work in one case, but not in another. By saying it "works" is also dependent on whether it simply outputs JSON or whether it conforms to the proto JSON spec. And hence that's the rationale for having the jsonpb package is to make sure the output conforms to the proto JSON spec.

This can be approached differently to make it work with encoding/json but sadly that wasn't designed properly back then.

[zhangli623]

I'm not sure I fully understand your question. If you are asking how come using encoding/json works to marshal the proto message, why doesn't it work to unmarshal, that really depends on the field types being marshaled/unmarshaled. It may happen to work in one case, but not in another. By saying it "works" is also dependent on whether it simply outputs JSON or whether it conforms to the proto JSON spec. And hence that's the rationale for having the jsonpb package is to make sure the output conforms to the proto JSON spec.

This can be approached differently to make it work with encoding/json but sadly that wasn't designed properly back then.

I'm sorry I mislead you. This is my problem. The proto message is like this:

 message BidRequest {
  // REQUIRED by the OpenRTB specification.
  required string id = 1;

  // At least 1 Imp object is required.
  repeated Imp imp = 2;

  oneof distributionchannel_oneof {
    // Only applicable and recommended for websites.
    Site site = 3;

    // (non-browser applications). Only applicable and recommended for apps.
    App app = 4;
  }
 ...
}

Then my go file is like this:

type BidRequest struct {
    // REQUIRED by the OpenRTB specification.
    Id *string `protobuf:"bytes,1,req,name=id" json:"id,omitempty"`
    // At least 1 Imp object is required.
    Imp []*BidRequest_Imp `protobuf:"bytes,2,rep,name=imp" json:"imp,omitempty"`
    // Types that are valid to be assigned to DistributionchannelOneof:
    //  *BidRequest_Site_
    //  *BidRequest_App_
    DistributionchannelOneof isBidRequest_DistributionchannelOneof `protobuf_oneof:"distributionchannel_oneof"`
 ...
}

And this is my test json string:

const js1 = `{
                    "id": "80ce30c53c16e6ede735f123ef6e32361bfc7b22",
                    "at": 1, "cur": [ "USD" ],
                    "imp": [
                        {
                            "id": "1", "bidfloor": 0.03,
                            "banner": {
                                "h": 250, "w": 300, "pos": 0
                            }
                        }
                    ],
                    "site": {
                        "id": "102855",
                        "cat": [ "IAB3-1" ],
                        "domain": "www.foobar.com",
                        "page": "http://www.foobar.com/1234.html ",
                        "publisher": {
                            "id": "8953", "name": "foobar.com",
                            "cat": [ "IAB3-1" ],
                            "domain": "foobar.com"
                        }
                    },
                    "device": {
                        "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13(KHTML, like Gecko) Version/5.1.7 Safari/534.57.2",
                        "ip": "123.145.167.10"
                    },
                    "user": {
                        "id": "55816b39711f9b5acf3b90e313ed29e51665623f"
                    }
                }`

It looks well, the site attribute in my json string should be unmarshaled correct like this:

 {
        Id:0xc0003ce940
        Imp:[id:"1" banner:<w:300 h:250 pos:UNKNOWN > bidfloor:0.03 ] 
        Site:[id:"102855" ...] 
        Device: ...
        ...
}

but when my test json string Unmarshale to the struct, it will be like this:

  openrtbReq := openrtb.BidRequest{}
  json.Unmarshal([]byte(js1), &openrtbReq)
  fmt.Printf("%+v\n",openrtbReq)

{
        Id:0xc0003ce940
        Imp:[id:"1" banner:<w:300 h:250 pos:UNKNOWN > bidfloor:0.03 ] 
        DistributionchannelOneof:<nil> 
        Device: ...
        ...
}

And when I tried to use the jsonpb, I got the same result.

  openrtbReq := openrtb.BidRequest{}
  jsonpb.Unmarshal(strings.NewReader(js1), &openrtbReq)
  fmt.Printf("%+v\n",openrtbReq)

{
        Id:0xc0003ce940
        Imp:[id:"1" banner:<w:300 h:250 pos:UNKNOWN > bidfloor:0.03 ] 
        DistributionchannelOneof:<nil> 
        Device: ...
        ...
}

Yes, the oneof type site is just be ignored.

[cybrcodr]

Your posted example is not complete, i.e. there's no message definition for Site and App. So, it won't be easy for me to reproduce. However, instead of providing those, would you be able to trim down a reproducible test case that still fails?

We do have unmarshaling testcases for oneofs here https://github.com/golang/protobuf/blob/master/jsonpb/jsonpb_test.go#L783-L787. So, oneofs should work in general unless there's a bug.