Protobuf: Cant deserialize messages from C++ in Javascript

[CheyenneForbes]

Messages serialized in C++ can not be deserialized in JavaScript but it works the other way around

[xfxyjwf]

Can you post the proto definition you are using?

[CheyenneForbes]

syntax = "proto3";

package AoB;

message User{
int32 id = 1;
string name = 2;
}

message Message{
message Stat{
int32 delivered = 1;
int32 viewed = 2;
int32 opened = 3;
}
int32 cid = 1;
int32 id = 2;
int32 date_time = 3;
string text = 4;
int32 sender = 5;
Stat message_stat = 6;
}

message Chat{
int32 id = 1;
User user = 2;
repeated Message messages = 3;
}

message SessionRes {
User user = 1;
repeated Chat chats = 2;
string id = 3;
}

message LoginReq {
string email = 1;
string password = 2;
}

[CheyenneForbes]

@xfxyjwf do you know how can I deserialize data from c++ in javascript?

[CheyenneForbes]

anyone?

[xfxyjwf]

How does your C++ serialization code and Javascript code looks like? I don't see anything special in your proto so it's likely how you send the data from C++ to Javascript that has caused the issue.

[CheyenneForbes]

Javascript gets the data as a http POST response from a C++ webserver

[xfxyjwf]

Can you check if the length of the data is the same on the C++ server before serialization (use ByteSize()) and on the Javascript side before deserialization (check length of the input)? It happens quite often that people only send data partially to a remote end.

[CheyenneForbes]

the size before serialization in c++ (ByteSize()) is 60 and 59 in before deseralization javascript (.length)

[xfxyjwf]

That explains why the javascript parsing fails. My guess is that some data get trunked/corrupted while sending over http. Something to look out for example:
http://stackoverflow.com/questions/1509743/what-is-a-good-way-to-transfer-binary-data-to-a-http-rest-api-service

Closing as the problem doesn't seem to be in protobuf.

[CheyenneForbes]

I think there is an issue with protobuf because whenever the string from C++ is passed to "deserializeBinary()" in Javascript it think its base64 encoded, I'm not the only one with the issue (https://github.com/google/protobuf/issues/2932).

The error on the Javascript side is "Error: Unknown base64 encoding at char: ("... I also tried turning the text to an Uint8Array with:

function StringToUint8Array(bStr){
    var i, len = bStr.length, u8_array = new Uint8Array(len);
    for (var i = 0; i < len; i++) {     
        u8_array[i] = bStr.charCodeAt(i);
    }
    return u8_array;
}

but I get "TypeError: b.unshift is not a function"

[CheyenneForbes]

I got it fixed, responseType should be set to "arraybuffer" for the XHR, for example:

$.ajax({
    responseType: "arraybuffer",
    type: 'POST',
    url: url,
    dataType: 'application/octet-stream',
    contentType: 'application/octet-stream',
    cache: false,
    processData: false,
    data: Message.serializeBinary(),
    crossDomain: true,
    success: function(){}
})

[For-thewin]

responseType should be set to "arraybuffer" for the XHR,

Hello, i meet the same problem. And i am comfused about your solution. Can you explain it in detail? Thank you very much.

[CheyenneForbes]

ArrayBuffer is used to store binary data in javascript (for example, you would use it to store a video file you want to process in javascript)

protocol buffers uses binary data.

responseType: "arraybuffer" is telling it to not "process" the data and read it raw just as it is.... as an example responseType: "json" is telling it to create a json object from the data meaning it wouldn't require you to use 'JSON.parse(data)to access json data sent by the server, while usingresponseType: "text"would requireJSON.parse(data)`.

[For-thewin]

ArrayBuffer is used to store binary data in javascript (for example, you would use it to store a video file you want to process in javascript)

protocol buffers uses binary data.

responseType: "arraybuffer" is telling it to not "process" the data and read it raw just as it is.... as an example responseType: "json" is telling it to create a json object from the data meaning it wouldn't require you to use 'JSON.parse(data)to access json data sent by the server, while usingresponseType: "text"would requireJSON.parse(data)`.

Well, thanks. In my project, i choose "webSocket" as my communication way. Here is my javascript client:

var ws = require("ws");
var sock = new ws("ws://127.0.0.1:8888");
sock.on("message", function(data) {

});

Now the type of data from webSocket is "string". I don't know where i can set the data type as "arraybuffer". And as you said, there is an error if i try turning the text to an Uint8Array.

[CheyenneForbes]

was arraybuffer set on the client side?