Protobuf: JsonFormat will not ignore unknown field when parsing

[ghost]

The JsonFormat.parser().merge(json, builder) is not ignoring unknown fields in input json.
Quoting the documentation from JsonFormat in project protobuf-java-util

Utility classes to convert protobuf messages to/from JSON format. The JSON format follows Proto3 JSON specification and only proto3 features are supported. Proto2 only features (e.g., extensions and unknown fields) will be discarded in the conversion. That is, when converting proto2 messages to JSON format, extensions and unknown fields will be treated as if they do not exist. This applies to proto2 messages embedded in proto3 messages as well.

You can reproduce the issue if you clone this and check out branch proto3-decode-bug.

[xfxyjwf]

Unknown fields will be ignored when converting from proto2 messages to JSON but not the other way around. When parsing from JSON, unknown fields are treated as error by default. We can add an option to override this behavior, but could you explain your use case a bit more? Why are there unknown fields in the JSON input? Do you want to ignore these unknown fields or you actually want to retain them in the conversion?

[ghost]

I think the use case is very simple, suppose you have an app running with version 1, and later you updated the message to version 2 this specific app does not really need to know about new changes old ones are enough for it... when new apps send messages of version 2 it will fail even to parse them (so where is the backward compatibility?).

The very least i expect from it is to parse loosing unknown information so that the application could continue to work, would be awesome if it could retain unknown data so that when serialised again that data might be properly used by other apps running version 2... that would be perfect...

Is this the same for non json conversion? say for example i have version 2 binary and i deserialize to version 1 proto object will it fail?

[xfxyjwf]

We dropped unknown fields support in proto3, so if you are using proto3 and parsing a version 2 binray into a version 1 object, the new fields (unknown to version 1) will be ignored. The JSON parser is more strict than the binary parser because the JSON input might be manually created and we want to catch human errors (like typos) with more strict checking.

With the current implementation there is no way to retain unknown JSON fields. You probably need to design your system in a way that doesn't require unknown fields retention. If that's not feasible, an alternative is to use proto2 with proto binary format.

[ghost]

I can easily add necessary code in Parser so that you could choose if you want to drop fields or not... however i wont dive in as far as to retain fields, but i definitely need the first part.
So i'd rather contribute that to protobuf project rather than write for me alone, i'm having some issues with setting up the project, if someone could help me to configure it i'd be happy to contribute.
I use proto almost in all my project so i suppose this is not the only case i might need this.

[xfxyjwf]

We use maven to build protobuf java library. It's easy to build and test:

$ git clone https://github.com/google/protobuf
$ cd protobuf
$ cp /path/to/the/downloaded/protoc/binary src/protoc
$ cd java
$ mvn test

[mickeyreiss]

Seems that this has been resolved in https://github.com/google/protobuf/pull/2092.

[ghost]

Thanks closing.

[rohithcm]

@xfxyjwf :
You said 'When parsing from JSON, unknown fields are treated as error by default. We can add an option to override this behavior'.
How do we do that?

[xfxyjwf]

@rohithcm try:

JsonFormat.parser().ignoringUnknownFields().merge(...)

See: https://github.com/google/protobuf/blob/master/java/util/src/main/java/com/google/protobuf/util/JsonFormat.java#L260

[yuri-sergiichuk]

We dropped unknown fields support in proto3, so if you are using proto3 and parsing a version 2 binray into a version 1 object, the new fields (unknown to version 1) will be ignored. The JSON parser is more strict than the binary parser because the JSON input might be manually created and we want to catch human errors (like typos) with more strict checking.

With the current implementation there is no way to retain unknown JSON fields. You probably need to design your system in a way that doesn't require unknown fields retention. If that's not feasible, an alternative is to use proto2 with proto binary format.

So, the unknown fields support is now in the proto3 officially, maybe it's time to reconsider and allow storing/retrieving unknown values in/from JSON messages?