Protobuf.js: How to convert base64 string to Message (protobuf.js v6)

[sulliwane]

version: [email protected]

grpc-gateway (not open sourced by google yet) is sending a base64 string encoded protobuf message to me (fetch() request). I'd like to convert this base64_string to the protobuf message.

Here is my attempt, but with no luck:

const base64_string = "my_base64_string";
const Message = root.lookup("package.Message");
var message = Message.decode(base64_string);

I also tried converting the base64 string to Uint8array:

const base64_string = "my_base64_string";
const buffer = Uint8Array.from(atob(base64_string), c => c.charCodeAt(0))
const Message = root.lookup("package.Message");
var message = Message.decode(buffer);

But I get scrumbled characters in the wrong field.

v5 had decode64(), how to do that with v6?

Many thanks!

note: I'm getting the base64 string by doing:

fetch().then(response => response.text()).then(base64 => { ... })

[dcodeIO]

But I get scrumbled characters in the wrong field.

You might be experiencing The "Unicode Problem" or something like that. See also.

v5 had decode64(), how to do that with v6?

There is no such method anymore, but a base64 library is included. It's available as protobuf.util.base64.

var util = protobuf.util;

var string = "...";
var buffer = util.newBuffer(util.base64.length(string));
util.base64.decode(string, buffer, 0);
// use buffer

[sulliwane]

Ok, so I was doing it correctly I suppose, cause I'm still getting scrambled code:

This is probably my base64 string which is not why I expect it to be (it is generated by google grpc-nginx-gateway, soon to be opensourced. So I will ask over there).

[sulliwane]

Still trying to figure out how to decode the response：

import protobuf from 'protobufjs';
protobuf.load("package.proto")
.then(function(root) {
  const MessageReq = root.lookup("package.MessageReq");
  const messageReq = MessageReq.create({ symbols: ['IF1609'] })

  fetch('http://myhost:port/package.Service/Method', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/x-protobuf',
      'X-Accept-Content-Transfer-Encoding': 'base64',
      'X-Accept-Response-Streaming': 'true'
    },
    body: MessageReq.encode(messageReq).finish(),
  })
  .then(response => response.text())
  .then(base64_string => {
    // Is that the proper way to convert the base64 string to a TypedArray?
    const buffer = Uint8Array.from(atob(base64_string), c => c.charCodeAt(0))

    const MessageRes = root.lookup("package.MessageRes");
    var messageRes = MessageRes.decode(buffer);

    console.log('messageRes', messageRes) // not OK, scrambled code
  })
});

Here is the base64 string response:

Cl4KXAoGSUYxNjA5OgVDRkZFWFoT5rKq5rexMzAw5oyH5pWwMTYwOWmamZmZmZnZP7kBmpmZmZmZyT/CAQExygECSUbZAZqZmZmZmdk/iAKsApECAAAAAADGy0CoAucHEgQSAk9L

My goal is to write a fetch transport (rpcImpl method like described here) compatible with the the specs of grpc-web.

[dcodeIO]

var base64_string = "Cl4KXAoGSUYxNjA5OgVDRkZFWFoT5rKq5rexMzAw5oyH5pWwMTYwOWmamZmZmZnZP7kBmpmZmZmZyT";
var buffer = Uint8Array.from(atob(base64_string), c => c.charCodeAt(0));
console.log(Array.prototype.toString.call(buffer));

// 10,94,10,92,10,6,73,70,49,54,48,57,58,5,67,70,70,69,88,90,19,230,178,170,230,183,177,51,48,48,230,140,135,230,149,176,49,54,48,57,105,154,153,153,153,153,153,217,63,185,1,154,153,153,153,153,153,201

10  id 1, wireType 2 (A.a)
94  length = 94
10  id 1, wireType 2 (B.b)
92  length = 92
10  id 1, wireType 2 (C.c)
6   length = 6
73 70 49 54 48 57 = "IF1609"
...

To me it seems, if you get the following taken from your comment above, that your .proto definition is not describing your data, because "IF1609" should be its own string:

Structure is something like...

message A {
  B a = 1;
  message B {
    C b = 1;
    message C {
      string c = 1;
    }
  }
}

Might be helpful: Hot to reverse engineer a buffer by hand

[sulliwane]

this is basically my proto definition:

service myService {
  rpc GetInstruments(MessageReq) returns (MessageRes) {}
}
message MessageReq {
  repeated string symbols = 1; // ex: ['IH1609'']
}
message MessageRes {
  repeated Instrument instruments = 1;
}
message Instrument {
  string instrumentid = 1;
  ...
}

I think my proto definition is correct as it's working for my other grpc clients (node, python).
thank you for your reply!

[dcodeIO]

All I can tell is, from your code, that you are decoding a MessageRes that contains a single field with id 1 of repeated Instruments. Regarding the buffer you are decoding, this corresponds to:

10  id 1, wireType 2 (A.a)
94  length = 94

So the buffer starts with some field with id 1 that has a value of 94 bytes length, which is then interpreted as an Instrument. An Instrument contains a string field with id 1, which corresponds to:

10  id 1, wireType 2 (B.b)
92  length = 92

These 92 bytes are interpreted as a string (Instrument#instrumentid), but actually aren't a string but another level of nested messages:

10  id 1, wireType 2 (C.c)
6   length = 6
73 70 49 54 48 57 = "IF1609"

Hence, when decoding, your instrumentid starts with two scrambled bytes, then the string IF1609 and then more scrambled bytes.

The decoding step works because nested messages and strings use the same wire type (2, ldelim) + the field id still matches and hence all this looks the same on the wire. There is clearly something missing there, like another outer message. Maybe this outer message is some sort of envelope implicitly used by grpc?

You can actually test this theory by skipping the first 2 bytes of the buffer:

    const buffer = Uint8Array.from(atob(base64_string), c => c.charCodeAt(0))
    const MessageRes = root.lookup("package.MessageRes");
    var reader = protobuf.Reader.create(buffer)
    reader.pos += 2
    var messageRes = MessageRes.decode(reader);

Alternatively, introducing a dummy envelope like the following one, and decoding it instead of MessageRes, effectively does the same:

...

message Envelope {
  MessageRes res = 1;
}

[sulliwane]

Many thanks. I will try this tomorrow.

Notes:
Here is what the JS object looks like once decoded on the grpc-node client:

[{ instrumentid: 'IF1709', otherprops: 'string' }]

There is maybe some extra informations added by grpc-web. Here is an implementation of grpc-web for reference: https://github.com/improbable-eng/grpc-web/tree/master/ts/src/transports

[sulliwane]

it works! :1st_place_medal:

So will ask what is the purpose of the first 2 bytes. I'll update here when I understand more clearly the grpc-web specs.

Many thanks for your help!

[dcodeIO]

Alright, feel free to reopen if necessary!

[metled]

@sulliwane maybe the first 2 bytes is the "Length-Prefixed-Message" of gPRC