Protobuf.js: Decode a list of messages

[pmosconi]

protobuf.js version: 6.8.0

Hi,
I have the following protocol buffer definition that was provided to me:

package proto;

option optimize_for = LITE_RUNTIME;

message Device {
    required string deviceId = 1;
    required int32 signalStrength = 2;
    required DeviceType deviceType = 3;
    optional string uuid = 4;
}

message DevicesFrame {
    required int64 timestamp = 1;
    required int32 frameSize = 2;
    repeated Device devices = 3; 
}

enum DeviceType {
    BT = 0;
    BLE = 1;
    WIFI = 2;
}

The buffer I am reading contains several DevicesFrame instances, each one preceded by its lenght and I can read the first one:

  protobuf.load('DevicesFrame.proto', (err, root) => {
    if (err) {
     throw err;
    }
   const DevicesFrame = root.lookupType('proto.DevicesFrame');
   const message = DevicesFrame.decodeDelimited(buf);
   // ...
}

I don't know how to skip to the next DevicesFrame and loop through the buffer till the end.

I was given the following sample Python code that loops through the data buffer using the Google protobuffer library, but I don't know how to do the same thing in node.js (and I'd rather avoid Google's protobuffer implementation because of it's messy installation):

        while n < len(data):
            msg_len, new_pos = _DecodeVarint32(data, n)
            n = new_pos  
            msg_buf = data[n:n+msg_len]
            n += msg_len

            message = DevicesFrame.DevicesFrame()
            message.ParseFromString(msg_buf)

Thanks in advance for any help and sorry for the seemingly trivial question!

[dcodeIO]

Try:

var reader = protobuf.Reader.create(buf);
while (reader.pos < reader.len) {
  var frame = DevicesFrame.decodeDelimited(reader);
  ...
}

[pmosconi]

test.zip
Thanks a lot. I already tried that but it throws an exception:

Error: invalid wire type 6 at offset 17
reader.js:375
    at Error (native)
    at BufferReader.Reader.skipType (c:\xxx\node_modules\protobufjs\src\reader.js:375:19)
    at Type.DevicesFrame$decode [as decode] (eval at Codegen (c:\xxx\node_modules\@protobufjs\codegen\index.js:50:33), <anonymous>:22:5)
    at Type.decode_setup [as decode] (c:\xxx\node_modules\protobufjs\src\type.js:510:25)
    at Type.decodeDelimited (c:\xxx\node_modules\protobufjs\src\type.js:523:17)
    at module.exports.protobuf.load (c:\xxx\lib\insertTimingsWiFi\index.js:62:36)
    at finish (c:\xxx\node_modules\protobufjs\src\root.js:95:9)
    at process (c:\xxx\node_modules\protobufjs\src\root.js:123:13)
    at c:\xxx\node_modules\protobufjs\src\root.js:182:17
    at fetchReadFileCallback (c:\xxx\node_modules\@protobufjs\fetch\index.js:51:19)

Attached is a test file I'm trying to work with.

[vjpr]

@dcodeIO

It would be good to have a definitive example in the documentation of how to read from a stream and decode.

I found a few different examples spread throughout the issues:

• https://github.com/dcodeIO/protobuf.js/issues/115#issuecomment-73363831
• https://github.com/dcodeIO/protobuf.js/issues/385#issuecomment-170306329

[alessandro-dallatorre]

This code actullay works for me with the dependency "protobufjs": "^6.8.6":

 protobuf.load("wifi.proto", function (err, root) {
        if (err)
          throw err;

        var DevicesFrame = root.lookupType("proto.DevicesFrame");
        var reader = protobuf.Reader.create(req.raw);
        while (reader.pos < reader.len) {
          var frame = DevicesFrame.decodeDelimited(reader);
          console.log("Decoded frame:" + JSON.stringify(frame))
        }
      });

[kirillgroshkov]

Spent few hours looking for solution, and here's my working version:

function transformFromProtobuf<OUT>(type: protobuf.Type): TransformTyped<Buffer, OUT> {
  let remainder = Buffer.alloc(0)

  return new Transform({
    objectMode: false,
    readableObjectMode: true,
    async transform(chunk: Buffer, _encoding, cb) {
      // console.log(chunk)
      const buf = Buffer.concat([remainder, chunk])

      let lastPos = 0
      const reader = protobuf.Reader.create(buf)
      try {
        while (reader.pos < reader.len) {
          lastPos = reader.pos
          const frame = type.decodeDelimited(reader)
          this.push(frame)
        }

        remainder = Buffer.alloc(0)
      } catch (err) {
        // console.log({pos: reader.pos, lastPos, len: reader.len}, 'got err')
        remainder = buf.slice(lastPos)
      }
      // console.log(`remainder size: ${remainder.length}`)
      cb() // finished processing
    },
  })
}