Protobuf: jsonpb: how to handle array of proto message

[Tommy-42]

Hello guys,

I am struggling with a use case that I don't find any reference on any issue.

My use case is :

I've a message request that contains :

message CreateObjectRequest {
    repeated Specificity specificities = 1;
}

message Specificity {
    oneof value {
        SpecA specA = 1;
        SpecB specB = 2;
    }
}

message SpecA { string code = 1; }

message SpecB { string code = 1; }

the generated code look like this :

type Specificity struct {
    // Types that are valid to be assigned to Value:
    //  *Specificity_SpecA
    //  *Specificity_SpecB
    Value isSpecificity_Value `protobuf_oneof:"value"`
}

func (m *Specificity) Reset()                    { *m = Specificity{} }
func (*Specificity) ProtoMessage()               {}
func (*Specificity) Descriptor() ([]byte, []int) { return fileDescriptorXXX, []int{8} }

type isSpecificity_Value interface {
    isSpecificity_Value()
    MarshalTo([]byte) (int, error)
    Size() int
}

When I create the object, I want to store it into a postgres db.

So I try to do this

var specificities *bytes.Buffer

// marshaler is init like this : 
//  var marshaler = new(jsonpb.Marshaler)
//  marshaler.EmitDefaults = true
//  marshaler.OrigName = true
//  marshaler.EnumsAsInts = true
err = marshaler.Marshal(specificities, req.Specificities) // req.Specificities == []*api.Specificity
if err != nil {
    return err
}

I got an error

cannot use req.Specificities (type []*api.Specificity) as type "myrepo/vendor/github.com/golang/protobuf/proto".Message in argument to marshaler.Marshal:
    []*api.Specificity does not implement "myrepo/vendor/github.com/golang/protobuf/proto".Message (missing ProtoMessage method)

same goes if use json.Marshal instead of jsonpb to bypass the create, when I want to json.Unmarshal it does not work.

Do I have to create a specific Marshal/UnMarshal to handle the array, Or I am missing something ( or do something wrong ) ?

PS:
I use gogoproto but I think it is linked to jsonpb

[awalterschulze]

Have you tried github.com/gogo/protobuf/jsonpb rather than github.com/golang/protobuf/jsonpb ?

[Tommy-42]

@awalterschulze thanks, I did miss the gogo/protobuf/jsonpb ( my fault I should have known that you did add jsonpb to gogo too )

but sadly even switching to gogo/jsonpb does not fix the error

[awalterschulze]

Nothing to be sorry about, it is a shame that there are jsonpb packages in the first place.

Ok so that is good to know. I suspect that you might have to pass &CreateObjectRequest{} instead of []*api.Specificity to jsonpb.

[Tommy-42]

with the &CreateObjectRequest it works fine, but the json will be in fact the whole CreateObjectRequest ( I have filtered out useless field to be clearer on the issue ) when I only want one field to be marshaled :/

[awalterschulze]

I don't think marshaling a list is supported by jsonpb.

You could also marshal each item in the list *api.Specificity and then concat them into a json compatible list yourself, not ideal, but it should work?

[Tommy-42]

Yep, this is what I though when trying to do this. I wasn't sure if it was the best idea, so I opened the issue to see if there was a native way to do it.

thanks

( it is worth the time to try and think about implementing a support of this case ? )

[awalterschulze]

If you want this to be supported I would suggest asking golang/protobuf.
Since we are a fork, we will merge back any change they make.
But if only we make the change then we need to maintain this diff, which is not fun, I promise you.
So this is only done in extreme cases.
I hope that makes sense?

[awalterschulze]

I sorry I see this is on golang/protobuf
I was confused, because of the mention of gogo/protobuf my bad

[Tommy-42]

yes no problem :) , I understand, I will let the issue opened, so they can judge if it is worth or not.

btw, I can think of a way to marshal the list ( like you suggest ).
but what about the unmarshalling, it is not the same, I know little about marshalling/unmarshalling.

I will have a json object ( valid tho ) but to unmarshal the list, I will have to create a dedicate struct and then iterate over it to be able to unmarshal the correct value ?

[cameronhatfield]

Another option, which the jsonpb interface almost seems built for, is to use the json.Decoder stuff. Sadly, you end up having to worry about tokens, but it does work:

jsonDecoder := json.NewDecoder(strings.NewReader(<...insert jsonpb here...>))
// read open bracket
_, err = jsonDecoder.Token()
if err != nil {
    log.Fatal(err)
}
var protoMessages []*my.ProtoMessage
for jsonDecoder.More() {
    protoMessage := my.ProtoMessage{}
    err := jsonpb.UnmarshalNext(jsonDecoder, &protoMessage)
    if err != nil {
        log.Fatal(err)
    }
    protoMessages = append(protoMessages, &protoMessage)
}

[dsnet]

What you are seeing is an oddity in the protobuf type system where "repeated" is a property of a field not a property of the type. Thus, unlike many other languages (e.g., Go) or serialization formats (e.g., JSON), there is no such thing as a top-level list type. The only top-level data structure in protobufs are messages. Since the jsonpb's purpose is to implement the JSON <-> proto specification, it also doesn't have a concept of a top-level list type.

Thus, you can't serialize out a JSON array directly with the jsonpb package. Your workarounds are to manually concatenate them as @awalterschulze suggested.

Also, if we end up adding an option to generate {Unmarshal,Marshal}JSON method (see https://github.com/golang/protobuf/issues/256), you can then use the standard encoding/json package on the []*api.Specificity.

Nothing to be sorry about, it is a shame that there are jsonpb packages in the first place.

The jsonpb exists because the standard library json package is unable to implement the full specification of the JSON<->proto specification. Issue https://github.com/golang/protobuf/issues/256 may make it nicer for users such that they won't need to directly use the jsonpb package, but that package will probably have to always exist.