Protobuf: C++ BinaryToJsonString generates incorrect Json: CamelCase instead of underscore

[hillzeta]

Style Guide https://developers.google.com/protocol-buffers/docs/style tells:
Use CamelCase (with an initial capital) for message names – for example, SongServerRequest. Use underscore_separated_names for field names – for example, song_name.
message SongServerRequest {
required string song_name = 1;
}

Following the Style Guide part of my .proto file is:

//////////////////////////////////////////////
message MedService {
int32 service_idx = 1;
string id = 2; // uuid
string name = 3;
int32 available = 4;
int32 allocated = 5;
int32 lab_area = 6;
int32 wait_area = 7;
bool lab_shareable = 8;
}

message MedServiceArray {
repeated MedService services = 1;
}
////////////////////////////////////////////////

BinaryToJsonString grnerates CamelCase for lab_area, wait_area, lab_shareable:

/////////////////////////////////////////////
{
"services": [
{
"serviceIdx": 0,
"id": "9b0cf953-ffa5-4696-9c76-e6fc5ec46991",
"name": "INFORMATION",
"available": 100,
"allocated": 0,
"labArea": 100,
"waitArea": 100,
"labShareable": true
},
.....
////////////////////////////////////////////////

But node.js needs underscores:

//////////////////////////////////////////////
{
"services": [
{
"service_idx": 0,
"id": "9b0cf953-ffa5-4696-9c76-e6fc5ec46991",
"name": "INFORMATION",
"available": 100,
"allocated": 0,
"lab_area": 100,
"wait_area": 100,
"lab_shareable": true
},
//////////////////////////////////////////////

Also, when BinaryToJsonString generates files with white space and carriage return by setting JsonOptions to add_whitespace=true, then these files cannot be read back by FromJson.

My project uses Json files in the browser, which are then sent/received by node.js client, which, in turn, talks to C++ server. It is necessary for me to have consistent uniform Json in all the environments.

Thank you, Zach.

[xfxyjwf]

Protobuf JSON uses lowerCamelCase field names by default (see: https://developers.google.com/protocol-buffers/docs/proto3#json), but we can add an option to let you override this default.

When you say "these files cannot be read back by FromJson", do you mean a FromJson method in a protobuf JS implementation? Which proto JS implementation are you using?

[hillzeta]

Sorry, FromJson is my function, which I mimicked from a googe example. It calls:
JsonToBinaryString(resolver_.get(), GetTypeUrl(message->GetDescriptor()), json, &binary);

I am using proto3. We certainly need a consistent binary <-> Json in all the supported languages, C++ and node.js in particular.

A C++ example, which writes out sufficiently rich binary to Json with white space and reads it back in would be helpful.

Actually, a regression test, which writes Json in one language and reads it in another for NxN combinations with and without white space is needed.

I can provide my proto file and json files if it would speed up implementation.

Thanks, Zach.

[xfxyjwf]

We actually already have a cross-language conformance test suite:
https://github.com/google/protobuf/blob/master/conformance/conformance_test.cc#L682

A JSON parser is expected to be able to parse both "field_name: value" and "fieldName: value" (i.e., both snake_case and lowerCamelCase should be accepted).

We are not done with the JSON implementation in all languages yet though.

If you find a case where the JSON parser can not parse what the JSON printer generates, can you file a separate github issue and include more details?

[hillzeta]

I'd appreciate an option to generate snake_case Json. This will address my main issue: consistency between C++ and node.js.
I will work on a small whitespace testcase.

[hillzeta]

Also, when BinaryToJsonString generates files with white space and carriage return by setting JsonOptions to add_whitespace=true, then these files cannot be read back by FromJson.

The above is my mistake. I was incorrectly reading from a file with linefeeds. Apologies, Zach.

[xfxyjwf]

@hillzeta you mean the parser doesn't accept linefeeds? That's still a bug in the parser I think. Could you help create a separate issue for that?

[hillzeta]

No, sorry, the parser is good in this case, I was using:
std::stringstream buffer;
buffer << doc.rdbuf();
buffer >> line;
instead of:
std::stringstream buffer;
buffer << doc.rdbuf();
line = buffer.str();

[hillzeta]

I did not suspect my own error because my wrong code worked in the case with add_whitespace=false;
No excuses -:)!

[hillzeta]

In my tests C++ code can read both camelCase json and snake_case json.
However Python via suggested https://github.com/google/protobuf/blob/master/python/google/protobuf/json_format.py

reads only camelCase data and does not parse snake_case.

Should I create a separate issue with test case?
Thanks, Zach.

[mklingen]

I have also run into this issue. Apparently, the Python implementation of json_format only allows camelCase names, while the C++ writer/reader can handle both. This leads to the python reader silently failing (reporting a confusing "protobuf X does not contain field" exception) whenever a C++ writer writes snake_case JSON files that a Python reader then reads.

[xfxyjwf]

@mklingen The python issue should be fixed in the latest 3.1.0 release. Can you give it a try?

[mklingen]

@xfxyjwf my apologies, I have been using protobuf 3.0.

[hillzeta]

I am building grpc as instructed here:
https://github.com/grpc/grpc/blob/v1.0.x/INSTALL.md
i.e.

$ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc
$ cd grpc
$ git submodule update --init
$ make
$ [sudo] make install

Please help me with instructions how to update the protobuf directory to
3.1.0 so that I build grpc with 3.1.0
Thank you, Zach.
https://github.com/grpc/grpc/blob/v1.0.x/INSTALL.md#windows

On Fri, Sep 30, 2016 at 10:35 AM, mklingen [email protected] wrote:

@xfxyjwf https://github.com/xfxyjwf my apologies, I have been using
protobuf 3.0.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/protobuf/issues/1691#issuecomment-250805188,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AP97DYzz9o4n54IQTMiKa2GZBWDtbJEQks5qvUhRgaJpZM4I20D9
.

[hillzeta]

I replaced the grpc/third_party/protobuf directory with the untarred protobuf-cpp-3.1.0.tar.gz but the compilation fails because of incompatibility:
include/grpc++/ext/reflection.pb.h:51:2: error: #error This file was generated by an older version of protoc which is
#error This file was generated by an older version of protoc which is incompatible with your Protocol Buffer headers. Please regenerate this file with a newer version of protoc.

Do you know when grpc will move to protobuf 3.1.0 ?

[xfxyjwf]

@nicolasnoble Do you know when will grpc move to protobuf 3.1.0?

[hillzeta]

Gentlemen, any clue?

[nicolasnoble]

So, the presence of pre-generated files into our repository is a mistake. We'll be discussing to replace this. That's the crux of the problem, really.

[hillzeta]

Please incorporate the changes I submitted in https://github.com/grpc/grpc/pull/6982/files
When you compile grpc with make prefix=$MY_PREFIX in order to install it in the place of your choice, rather than into /usr you need to propagate this to protobuf configure by adding --prefix="$(prefix)" to the configure string. You also want to install protobuf into the same area as grpc:
$(Q)$(MAKE) -C third_party/protobuf install

There is also an obvious shadow bug, which has not been fixed.
These changes are easy to make by hand, no need to git merge.

[hillzeta]

Gentlemen,

it has been more than a month.
Is there an ETA when grpc will be made agnostic to the protobuf version?

Regards, Zach.

On Thu, Oct 13, 2016 at 11:10 AM, Nicolas Noble [email protected]
wrote:

So, the presence of pre-generated files into our repository is a mistake.
We'll be discussing to replace this. That's the crux of the problem, really.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/protobuf/issues/1691#issuecomment-253592589,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AP97DcCHF3MHTrVnjCfhuKS7yMfdvjXOks5qznQJgaJpZM4I20D9
.

[xfxyjwf]

@hillzeta for grpc questions, please create an issue on the grpc repo: https://github.com/grpc/grpc

[hillzeta]

Done:
Make grpc build agnostic to the protobuf version #8773

On Wed, Nov 16, 2016 at 10:29 AM, Feng Xiao [email protected]
wrote:

@hillzeta https://github.com/hillzeta for grpc questions, please create
an issue on the grpc repo: https://github.com/grpc/grpc

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/protobuf/issues/1691#issuecomment-261029836,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AP97DYJJ_fjbwQConh1uK5jU5eDXM59uks5q-0t9gaJpZM4I20D9
.

[m-ric]

I have a question about the original issue dealing with JsonFormat handling snake_case in python. I still have the issue with v.3.1.0. My case is the following:

• JSON -> PB
• PB -> wire
• wire -> PB
• PB -> JSON
  My JSON input uses snake_case. google.protobuf.json_format converts it to CamelCase. Thus my output JSON is CamelCase, whereas I need snake_case. How can I retrieve the original snake_case in my output JSON?

If you find a case where the JSON parser can not parse what the JSON printer generates, can you file a separate github issue and include more details?

@hillzeta have you logged such an issue?

[m-ric]

Allelujah! Found that I missed the preserving_proto_field_name field in MessageToJson().