Protobuf.js: Upgrade guide

[bennycode]

I am using protobuf.js v5 and I want to upgrade to protobuf.js v6.

Is there any guide on how to do this?

My unit test so far looks like this:

messages.proto

option java_package = "com.example";

message GenericMessage {
  required string message_id = 1;
  oneof content {
    Text text = 2;
  }
}

message Text {
  required string content = 1;
}

protobuf.js v5.0.1

var protobuf = require('protobufjs');

describe('protobuf.js v5.0.1', function() {

  var buffers = undefined;

  beforeAll(function(done) {
    var file = 'messages.proto';

    protobuf.loadProtoFile(file, function(error, builder) {
      if (error) {
        done.fail(error);
      } else {
        buffers = builder.build();
        done();
      }
    });
  });

  it('creates a generic message with text content', function() {
    var text = new buffers.Text('Hello');
    var genericMessage = new buffers.GenericMessage('id');    
    genericMessage.set('text', text);

    expect(genericMessage.content).toBe('text');
    expect(genericMessage.message_id).toBe('id');
    expect(genericMessage.text.content).toBe('Hello');
  });
});

protobuf.js v6.6.5

var protobuf = require('protobufjs');

describe('protobuf.js v6.6.5', function() {

  var buffers = undefined;

  beforeAll(function(done) {
    var file = 'messages.proto';

    protobuf.load(file)
      .then(function(root) {
        buffers = root;
        done();
      })
      .catch(done.fail);
  });

  it('creates a generic message with text content', function() {
    var text = buffers.nested.Text.create({content: 'Hello'});
    var genericMessage = buffers.nested.GenericMessage.create({message_id: 'id', text: text});

    expect(genericMessage.content).toBe('text');
    expect(genericMessage.message_id).toBe('id');
    expect(genericMessage.text.content).toBe('Hello');
  });
});

Can you tell me if this is the correct way to use the v6 API?

[dcodeIO]

There is a wiki page covering some of the differences.

Your example looks good to me. Instead of going through the nested hierarchy, you could also just use var Text = root.lookup("Text");.

[bennycode]

Thanks for your very fast reply!

Can you also help me with the serialization?

With v5 I used:

var buffer = genericMessage.toArrayBuffer();
var typedArray = new Uint8Array(buffer);
var expectedArray = new Uint8Array([10, 2, 105, 100, 18, 7, 10, 5, 72, 101, 108, 108, 111]);
expect(typedArray).toEqual(expectedArray);

With v6 I tried to use this:

var GenericMessage = buffers.lookup("GenericMessage");
var buffer = GenericMessage.encode(genericMessage).finish();
var typedArray = new Uint8Array(buffer);
var expectedArray = new Uint8Array([10, 2, 105, 100, 18, 7, 10, 5, 72, 101, 108, 108, 111]);
expect(typedArray).toEqual(expectedArray); // fails

But now my test fails telling me:

Expected 10,0,18,7,10,5,72,101,108,108,111 to equal 10,2,105,100,18,7,10,5,72,101,108,108,111.

So to me it looks like that the serialization format is not the same as it was before. 😢

[dcodeIO]

In v6, this already returns an Uint8Array:

var buffer = GenericMessage.encode(genericMessage).finish();

Other than that there are three differing bytes: 0 vs. 2,105,100.

10 = 1010 = id 1, wiretype 2 (ldelim)
0 = length 0

differing

10 = 1010 = id 1, wiretype 2 (ldelim)
2 = length 2
105, 100

like if the field with id 1 had an empty value. Could be a submessage with a field (id 13, wireType 1 varint), that has a default value of 100, which is (now) omitted because it's the default - or something like that.

[bennycode]

Thanks @dcodeIO for pointing me to the different bytes. I made them up in a comparison table:

Uint8Array (with protobuf.js v5) | Uint8Array (with protobuf.js v6)
:---:|:---:
10 | 10
2,105,100 | 0
18 | 18
7 | 7
10 | 10
5 | 5
72 | 72
101 | 101
108 | 108
108 | 108
111 | 111

Unfortunately I couldn't figure out why the serialization looks different (0 vs. 2,105,100).

Basically all I changed was:

var text = new buffers.Text('Hello');
var genericMessage = new buffers.GenericMessage('id');    
genericMessage.set('text', text);

→

var text = buffers.nested.Text.create({content: 'Hello'});
var genericMessage = buffers.nested.GenericMessage.create({message_id: 'id', text: text})

So it should generate the same Uint8Array? Maybe you can help me finding the missing piece. I created a sample project here: https://github.com/bennyn/protobufjs-upgrade

The "master" branch in my project uses protobuf.js v5.0.1 and the "version-6" branch uses protobuf.js v6.6.5.

I also created a Pull Request which now fails (because of the differences in the Uint8Array): https://github.com/bennyn/protobufjs-upgrade/pull/1

Are you also 100% sure that GenericMessage.encode(genericMessage).finish() creates already an Uint8Array? Because your documentation says that it creates a Uint8Array only for Browsers but a Buffer on Node.js and that's what I am experiencing:

var buffer = GenericMessage.encode(genericMessage).finish();
var typedArray = new Uint8Array(buffer);
console.log('A', buffer); // A <Buffer 0a 00 12 07 0a 05 48 65 6c 6c 6f>
console.log('B', typedArray); // B Uint8Array [10, 0, 18, 7, 10, 5, 72, 101, 108, 108, 111]

[dcodeIO]

Unfortunately I couldn't figure out why the serialization looks different (0 vs. 2,105,100).

It would help if you could paste the .proto definitions in question - including submessages, if applicable.

Are you also 100% sure that GenericMessage.encode(genericMessage).finish() creates already an Uint8Array? Because your documentation says that it creates a Uint8Array only for Browsers but a Buffer on Node.js and that's what I am experiencing:

On recent versions of node, Buffer extends Uint8Array - so yes, but: Also, on node the library uses BufferWriter (returned by .encode), which is a subclass of Writer, which returns a Buffer from .finish as of the documentation. It's still also an Uint8Array, of course.

[bennycode]

@dcodeIO The .proto file is referenced in my sample project: https://github.com/bennyn/protobufjs-upgrade/blob/master/package.json#L25

It looks like this:

// syntax = "proto2";
option java_package = "com.waz.model";

message GenericMessage {
  required string message_id = 1; // client generated random id, preferably UUID
  oneof content {
    Text text = 2;
    ImageAsset image = 3;
    Knock knock = 4;
    LastRead lastRead = 6;
    Cleared cleared = 7;
    External external = 8;
    ClientAction clientAction = 9;
    Calling calling = 10;
    Asset asset = 11;
    MessageHide hidden = 12;
    Location location = 13;
    MessageDelete deleted = 14;
    MessageEdit edited = 15;
    Confirmation confirmation = 16;
    Reaction reaction = 17;
    Ephemeral ephemeral = 18;
  }
}

message Ephemeral {
  required int64 expire_after_millis = 1;
  oneof content {
    Text text = 2;
    ImageAsset image = 3;
    Knock knock = 4;
    Asset asset = 5;
    Location location = 6;
  }
}

message Text {
  required string content = 1;
  repeated Mention mention = 2;
  repeated LinkPreview link_preview = 3;
}

message Knock {
  required bool hot_knock = 1 [default = false];
}

message LinkPreview {
  required string url = 1;
  required int32 url_offset = 2; // url offset from beginning of text message

  oneof preview {
    Article article = 3;  // deprecated - use meta_data
  }

  optional string permanent_url = 5;
  optional string title = 6;
  optional string summary = 7;
  optional Asset image = 8;

  oneof meta_data {
    Tweet tweet = 9;
  }
}

message Tweet {
  optional string author = 1;
  optional string username = 2;
}

// deprecated - use the additional fields in LinkPreview
message Article {
  required string permanent_url = 1;
  optional string title = 2;
  optional string summary = 3;
  optional Asset image = 4;
}

message Mention {
  required string user_id = 1;
  required string user_name = 2;
}

message LastRead {
  required string conversation_id = 1;
  required int64 last_read_timestamp = 2;
}

message Cleared {
  required string conversation_id = 1;
  required int64 cleared_timestamp = 2;
}

message MessageHide {
  required string conversation_id = 1;
  required string message_id = 2;
}

message MessageDelete {
  required string message_id = 1;
}

message MessageEdit {
  required string replacing_message_id = 1;
  oneof content {
    Text text = 2;
  }
}

message Confirmation {
  enum Type {
    DELIVERED = 0;
    READ = 1;
  }

  required string message_id = 1;
  required Type type = 2;
}

message Location {
  required float longitude = 1;
  required float latitude = 2;
  optional string name = 3; // location description/name
  optional int32 zoom = 4; // google maps zoom level (check maps api documentation)
}

message ImageAsset {
  required string tag = 1;
  required int32 width = 2;
  required int32 height = 3;
  required int32 original_width = 4;
  required int32 original_height = 5;
  required string mime_type = 6;
  required int32 size = 7;
  optional bytes otr_key = 8;
  optional bytes mac_key = 9;    // deprecated - use sha256
  optional bytes mac = 10;       // deprecated - use sha256
  optional bytes sha256 = 11;      // sha256 of ciphertext
}

message Asset {
  message Original {
    required string mime_type = 1;
    required uint64 size = 2;
    optional string name = 3;
    oneof meta_data {
      ImageMetaData image = 4;
      VideoMetaData video = 5;
      AudioMetaData audio = 6;
    }
  }

  message Preview {
    required string mime_type = 1;
    required uint64 size = 2;
    optional RemoteData remote = 3;
    oneof meta_data {
      ImageMetaData image = 4;
    }
  }

  message ImageMetaData {
    required int32 width = 1;
    required int32 height = 2;
    optional string tag = 3;
  }

  message VideoMetaData {
    optional int32 width = 1;
    optional int32 height = 2;
    optional uint64 duration_in_millis = 3;
  }

  message AudioMetaData {
    optional uint64 duration_in_millis = 1;
    // repeated float normalized_loudness = 2 [packed=true]; // deprecated - Switched to bytes instead
    optional bytes normalized_loudness = 3;     // each byte represent one loudness value as a byte (char) value.
                                                // e.g. a 100-bytes field here represents 100 loudness values.
                                                // Values are in chronological order and range from 0 to 255.
  }

  enum NotUploaded {
    CANCELLED = 0;
    FAILED = 1;
  }

  message RemoteData {
    required bytes otr_key = 1;
    required bytes sha256 = 2;
    optional string asset_id = 3;
//    optional bytes asset_token = 4; // deprecated - changed type to string
    optional string asset_token = 5;
  }

  optional Original original = 1;
//  optional Preview preview = 2;  // deprecated - preview was completely replaced
  oneof status {
    NotUploaded not_uploaded = 3;
    RemoteData uploaded = 4;
  }
  optional Preview preview = 5;
}

// Actual message is encrypted with AES and sent as additional data
message External {
  required bytes otr_key = 1;
  optional bytes sha256 = 2;      // sha256 of ciphertext
}

message Reaction {
  optional string emoji = 1; // some emoji reaction or the empty string to remove previous reaction(s)
  required string message_id = 2;
}

enum ClientAction {
  RESET_SESSION = 0;
}

message Calling {
  required string content = 1;
}

[dcodeIO]

Thanks, so let's see what v6 encodes:

10  id 1, wireType 2 (ldelim)
0   length = 0

messageId = ""

and v5:

10  id 1, wireType 2 (ldelim)
2   length = 2
105 i
100 d

messageId = "id"

Looks like GenericMessage#messageId isn't set in your v6 code (uses message_id instead of messageId - v6 uses camelCase by default):

var genericMessage = buffers.nested.GenericMessage.create({message_id: 'id', text: text});

To detect this condition, you could do:

var GenericMessage = buffers.lookup("GenericMessage");
var obj = {message_id: 'id', text: text};
var err = GenericMessage.verify(obj);
if (err)
  throw Error(err);
var genericMessage = GenericMessage.create(obj);
...

which should return an error noting that a required field is missing (.encode alone doesn't throw there).

[bennycode]

Thanks @dcodeIO! Your tip with GenericMessage.verify(obj) was worth gold. 🥇

I made the test pass now by using:

var GenericMessage = buffers.lookup("GenericMessage");

var payload = {
  message_id: 'id',
  messageId: 'id',
  text: Text.create({content: 'Hello'})
};

var genericMessage = GenericMessage.create(payload);

But it's a bit inconvenient to declare message_id and also messageId.

With protobuf.js v5 I could just do genericMessage = new buffers.GenericMessage('id');. Is there anything like that in v6? I am thinking of something like genericMessage = GenericMessage.create('id');...

[dcodeIO]

Argument-style constructors have been removed, but you can still use underscore_case, if you prefer:

  beforeAll(function(done) {
    var file = 'messages.proto';
    var root = new protobuf.Root();
    root.load(file, { keepCase: true }, function(err, root) {
      if (err)
        done.fail(err);
      else {
        buffers = root;
        done();
      }
    })
  });

Note that there is a bug in 6.6.5 (fixed in master) that prevents promise-style loading throughRoot#load with options.

[bennycode]

Ok, this helped! Thank you very much!!

We are on track now. 🎃

[bennycode]

P.S. Your project is mentioned here: https://wire.com/en/legal/#licenses

[dcodeIO]

Ironically, I also attempted launching a privacy focused messenger in 2013, but it didn't make it. Though it's good to know that I am still contributing to the good cause 😄

[bennycode]

Hey, that's cool!! It's also a bit funny that the taglines are always the same ("Messenger from Germany wants to beat WhatsApp"). 😄

P.S. We are still hiring: https://wire.softgarden.io/job/616102 ;)