Protobuf.js: Cannot get struct to work correctly

[vespakoen]

Hey there, I am using ProtoBuf.js through grpc and have the following protos:

datainfo.proto

syntax = "proto3";
import "google/protobuf/struct.proto";

message DataInfo {
  required string projectId = 1;
  required string projectName = 2;
  required uint32 projectVersion = 3;
  required string schemaId = 4;
  required string schemaName = 5;
  required google.protobuf.Struct item = 6;
}

google/protobuf/struct.proto

syntax = "proto3";

package google.protobuf;

option java_generate_equals_and_hash = true;
option java_multiple_files = true;
option java_outer_classname = "StructProto";
option java_package = "com.google.protobuf";
option csharp_namespace = "Google.Protobuf.WellKnownTypes";
option objc_class_prefix = "GPB";

// `Struct` represents a structured data value, consisting of fields
// which map to dynamically typed values. In some languages, `Struct`
// might be supported by a native representation. For example, in
// scripting languages like JS a struct is represented as an
// object. The details of that representation are described together
// with the proto support for the language.
//
// The JSON representation for `Struct` is JSON object.
message Struct {
  // Map of dynamically typed values.
  map<string, Value> fields = 1;
}

// `Value` represents a dynamically typed value which can be either
// null, a number, a string, a boolean, a recursive struct value, or a
// list of values. A producer of value is expected to set one of that
// variants, absence of any variant indicates an error.
//
// The JSON representation for `Value` is JSON value.
message Value {
  // The kind of value.
  oneof kind {
    // Represents a null value.
    NullValue null_value = 1;
    // Represents a double value.
    double number_value = 2;
    // Represents a string value.
    string string_value = 3;
    // Represents a boolean value.
    bool bool_value = 4;
    // Represents a structured value.
    Struct struct_value = 5;
    // Represents a repeated `Value`.
    ListValue list_value = 6;
  }
}

// `NullValue` is a singleton enumeration to represent the null value for the
// `Value` type union.
//
//  The JSON representation for `NullValue` is JSON `null`.
enum NullValue {
  // Null value.
  NULL_VALUE = 0;
}

// `ListValue` is a wrapper around a repeated field of values.
//
// The JSON representation for `ListValue` is JSON array.
message ListValue {
  // Repeated field of dynamically typed values.
  repeated Value values = 1;
}

I can load the datainfo.proto file just fine, however when I try to serialize the struct value isnt working

import grpc from 'grpc';
var dataInfoProto = grpc.load(__dirname + '/protos/datainfo.proto');
dataInfoProto.DataInfo.encode({
  projectId: 'test',
  projectName: 'test',
  projectVersion: 1,
  schemaId: 'test',
  schemaName: 'test',
  item: {
    test: 'string'
  }
});

the error:

Error: .google.protobuf.Struct#test is not a field: undefined
    at Error (native)
    at MessagePrototype.set (/home/........./node_modules/grpc/node_modules/protobufjs/dist/ProtoBuf.js:2451:58)

I have updated protobufjs inside of the grpc node module, but to no avail.

Thanks!

[dcodeIO]

Try constructing a ProtoBuf.Map instance from {test: 'string'}, specifying this as the value for item.

Try:

var struct = new google.protobuf.Struct({
   test: {
      string_value: 'string'
   }
});

dataInfoProto.DataInfo.encode({
  ...
  item: struct
});

[vespakoen]

Unfortunately, no success:

var grpc = require('grpc');
var proto = grpc.load(__dirname + '/protos/datainfo.proto');

var item = new proto.google.protobuf.Struct({
  string_value: 'string'
});
console.log(item);

/home/koen/prototest/node_modules/grpc/node_modules/protobufjs/dist/ProtoBuf.js:2440
                            throw Error(this+"#"+keyOrObj+" is not a field: un
                                                         ^
Error: .google.protobuf.Struct#string_value is not a field: undefined
    at Error (native)
    at MessagePrototype.set (/home/koen/prototest/node_modules/grpc/node_modules/protobufjs/dist/ProtoBuf.js:2440:58)
    at MessagePrototype.set (/home/koen/prototest/node_modules/grpc/node_modules/protobufjs/dist/ProtoBuf.js:2434:38)
    at Message (/home/koen/prototest/node_modules/grpc/node_modules/protobufjs/dist/ProtoBuf.js:2363:34)
    at Object.<anonymous> (/home/koen/prototest/index.js:4:12)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:501:10)

[vespakoen]

Same with this:

var item = new proto.google.protobuf.Struct({
  test: {
    string_value: 'string'
  }
});
console.log(item);

/home/koen/prototest/node_modules/grpc/node_modules/protobufjs/dist/ProtoBuf.js:2440
                            throw Error(this+"#"+keyOrObj+" is not a field: un
                                                         ^
Error: .google.protobuf.Struct#test is not a field: undefined
    at Error (native)
    at MessagePrototype.set (/home/koen/prototest/node_modules/grpc/node_modules/protobufjs/dist/ProtoBuf.js:2440:58)
    at MessagePrototype.set (/home/koen/prototest/node_modules/grpc/node_modules/protobufjs/dist/ProtoBuf.js:2434:38)
    at Message (/home/koen/prototest/node_modules/grpc/node_modules/protobufjs/dist/ProtoBuf.js:2363:34)
    at Object.<anonymous> (/home/koen/prototest/index.js:4:12)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:501:10)

[vespakoen]

This works!

var item = new proto.google.protobuf.Struct({
  fields: {
    test: {
      string_value: 'test'
    }
  }
});
console.log(item);

The "fields" was the trick here (pretty obvious looking at it now)

Thanks a lot for the help!!!

[vespakoen]

For other stumbling on this, here is a more complete example:

jsonToStruct.js

module.exports = function (google) {
  var toString = Object.prototype.toString;
  var toFields = function (data) {
    var result = {};
    Object.keys(data).forEach(function (key) {
      var valueRep = {};
      var value = data[key];
      var typeString = toString.call(data[key]);
      switch (typeString) {
        case '[object Null]':
        case '[object Undefined]':
          valueRep.null_value = 0;
          break;
        case '[object Object]':
          valueRep.struct_value = new proto.google.protobuf.Struct({
            fields: toFields(value)
          });
          break;
        case '[object Array]':
          var typed = toFields(value);
          var values = Object.keys(typed).map(function (key) {
            return typed[key];
          });
          valueRep.list_value = new proto.google.protobuf.ListValue(values);
          break;
        case '[object Number]':
          valueRep.number_value = value;
          break;
        case '[object Boolean]':
          valueRep.bool_value = value;
          break;
        case '[object String]':
          valueRep.string_value = value;
          break;type
        case '[object Date]':
          valueRep.date_value = value;
          break;
        default:
          throw new Error('Unsupported type: ' + typeString);
          break;
      }
      result[key] = valueRep;
    });
    return result;
  }

  return function jsontoStruct(json) {
    return new proto.google.protobuf.Struct({
      fields: toFields(json)
    });
  }
};

structToJson.js

function protoValueToJs(val) {
  var kind = val.kind;
  var value = val[kind];
  if (kind === 'list_value') {
    return value.values.map(function (value) {
      return protoValueToJs(value);
    });
  } else if (kind === 'struct_value') {
    return structToJson(value)
  } else {
    return value;
  }
}

function structToJson(struct) {
  var result = {};
  var map = struct.fields.map;
  Object.keys(map).forEach(function (key) {
    result[key] = protoValueToJs(map[key].value);
  });
  return result;
}

module.exports = structToJson;

[rphillips]

@vespakoen were you able to get an object with an array in it working? I keep getting the following error with this object:

{
  label: 'some label',
  anArray: ['a1', a2']
}

Error: Illegal value for Message.Field .proto.CheckUpdate.params: .google.protobuf.Struct (Error: Illegal value for Message.Field .google.protobuf.Struct.fields: [object Object] (Error: .google.protobuf.Value#kind is not a field: undefined))

[antho1404]

For the one interested, this solution wasn't working for me anymore so I came up with my own implementation based on the existing one that you can find here

https://github.com/mesg-foundation/mesg-js/blob/238e70e56cc8a35cfc8ffeb1ffa92c3160ff5d87/src/util/encoder.ts

[kpiyush17]

I have a Golang server with the client in JS. I am trying to send struct value like this

var item = new proto.google.protobuf.Struct({
  fields: {
    test: {
      string_value: 'test'
    }
  }
});

but on the golang side, I am receiving empty value.

[rockers7414]

@kpiyush17

Can you help to try this?

var item = new proto.google.protobuf.Struct({
  fields: {
    test: {
      stringValue: 'test'
    }
  }
});

[kpiyush17]

@rockers7414
It didn't work.

This works:

const query = { 'name':'piyush' };
const item = proto.google.protobuf.Struct.fromJavaScript(plainObj);

I am using static-loading of proto files.

[rockers7414]

It's weird that your solution doesn't work on my side.

const { Struct } = require('google-protobuf/google/protobuf/struct_pb');
const structData = Struct.fromJavaScript({ a: 1 });

I cannot find the object in structData.

[kpiyush17]

@rockers7414
I think only strings are supported in the object. Can you try {a: "1"}