Grpc-node: [question] How to generate `require('@grpc/grpc-js')` instead of `require('grpc')`

[kittaakos]

I am not sure if this is the correct place to ask...

I would like to use @grpc/grpc-js instead of grpc in the generated JS code to overcome an issue inside a bundled electron application.

Is it possible to staticly generate the JS code with require('@grpc/grpc-js') instead of require('grpc'). My current approach is to generate it with require('grpc'), then parse the generated files and replace the requires in a manual step 😞 Is there a flag I can tweak to adjust the generator?

I was unable to find the solution, so I set up a bare minimum example, perhaps someone can advise.

Thank you!

[murgatroid99]

Version 1.8.0 of grpc-tools added the generate_package_definition option to the grpc_out argument. It can be used like this: --grpc_out=generate_package_definition:<path>. This will generate a package definition object that can be passed to the loadPackageDefinition function in both grpc libraries.

[kittaakos]

Thank you, @murgatroid99. Is there an example somewhere I can check? I have found the proposal of this new feature, but not a single test or sample.

[murgatroid99]

No, there is not currently an example for this.

[paambaati]

@murgatroid99 I have a very similar usecase where I'm trying to statically generate pb files and TypeScript bindings, and simply replacing grpc with @grpc/grpc-js does not work for me (ServiceDefinition, MethodDefinition, etc. aren't defined/exported).

Is this possible currently? If no, is there a branch/PR I can follow or help with?

[ctrlshp]

I would really like to try this as well. I have added the generate_package_definition option but the command crashes with this error message "Plugin output is unparseable". I am using grpc-tools 1.8.0 on macOS 10.13.6 and node 12.10.0. Without the generate_package_definition option, it works just fine, but of course it doesn't generate a static proto definition object that can be used with @grpc/grpc-js.

Can anybody help ?

Here is the full command :

npx grpc_tools_node_protoc \
    --js_out=import_style=commonjs,binary:./src/grpc/proto \
    --grpc_out=generate_package_definition:./src/grpc/proto \
    --plugin=protoc-gen-grpc="`npm bin`/grpc_tools_node_protoc_plugin" \
    --proto_path=./proto \
    ./proto/*.proto

Thanks !

[murgatroid99]

@paambaati I'm not surprised some type definitions are missing from grpc-js, because the tests pass anyway. Those should be easy to fix because those types mentioned are defined in internal TypeScript files and they're just not exported from the library. I would accept a PR to properly export them.

@ctrlshp I'm not sure what could be causing that error. The first thing to try would be removing the --plugin argument. Recent versions of grpc-tools add that argument automatically, and duplicating it may be doing something weird.

[paambaati]

@murgatroid99 I did a quick scan and looks like ServiceDefinition and MethodDefinition are exported — https://github.com/grpc/grpc-node/blob/2ab15fef4ff19d02b78f33a0290ca10f6978262b/packages/grpc-js/src/make-client.ts

Is there some additional step to get the codegen to pick it up?

[murgatroid99]

They have to be re-exported in index.ts to actually be exported by the library.

[ctrlshp]

@murgatroid99 Thanks for your reply. Still the same problem with the amended command :

npx grpc_tools_node_protoc \
    --js_out=import_style=commonjs,binary:./src/grpc/proto \
    --grpc_out=generate_package_definition:./src/grpc/proto \
    --proto_path=./proto \
    ./proto/system.proto

Removing generate_package_definition: still works. After "Plugin output is unparseable" I get a dump of what appears to be the generated file :

// GENERATED CODE -- DO NOT EDIT!

'use strict';
var system_pb = require('./system_pb.js');

function serialize_SystemProxyHandleEventRequest(arg) {
  if (!(arg instanceof system_pb.SystemProxyHandleEventRequest)) {
    throw new Error('Expected argument of type SystemProxyHandleEventRequest');
  }
  return Buffer.from(arg.serializeBinary());
}

function deserialize_SystemProxyHandleEventRequest(buffer_arg) {
  return system_pb.SystemProxyHandleEventRequest.deserializeBinary(new Uint8Array(buffer_arg));
}

function serialize_SystemProxyHandleEventResponse(arg) {
  if (!(arg instanceof system_pb.SystemProxyHandleEventResponse)) {
    throw new Error('Expected argument of type SystemProxyHandleEventResponse');
  }
  return Buffer.from(arg.serializeBinary());
}

function deserialize_SystemProxyHandleEventResponse(buffer_arg) {
  return system_pb.SystemProxyHandleEventResponse.deserializeBinary(new Uint8Array(buffer_arg));
}


var SystemProxyService = exports['SystemProxy'] = {
  handleEvent: {
    path: '/SystemProxy/handleEvent',
    requestStream: false,
    responseStream: false,
    requestType: system_pb.SystemProxyHandleEventRequest,
    responseType: system_pb.SystemProxyHandleEventResponse,
    requestSerialize: serialize_SystemProxyHandleEventRequest,
    requestDeserialize: deserialize_SystemProxyHandleEventRequest,
    responseSerialize: serialize_SystemProxyHandleEventResponse,
    responseDeserialize: deserialize_SystemProxyHandleEventResponse,
  },
};

It looks a little bit different than the one generated without generate_package_definition: :

// GENERATED CODE -- DO NOT EDIT!

'use strict';
var grpc = require('grpc');
var system_pb = require('./system_pb.js');

function serialize_SystemProxyHandleEventRequest(arg) {
  if (!(arg instanceof system_pb.SystemProxyHandleEventRequest)) {
    throw new Error('Expected argument of type SystemProxyHandleEventRequest');
  }
  return Buffer.from(arg.serializeBinary());
}

function deserialize_SystemProxyHandleEventRequest(buffer_arg) {
  return system_pb.SystemProxyHandleEventRequest.deserializeBinary(new Uint8Array(buffer_arg));
}

function serialize_SystemProxyHandleEventResponse(arg) {
  if (!(arg instanceof system_pb.SystemProxyHandleEventResponse)) {
    throw new Error('Expected argument of type SystemProxyHandleEventResponse');
  }
  return Buffer.from(arg.serializeBinary());
}

function deserialize_SystemProxyHandleEventResponse(buffer_arg) {
  return system_pb.SystemProxyHandleEventResponse.deserializeBinary(new Uint8Array(buffer_arg));
}


var SystemProxyService = exports.SystemProxyService = {
  handleEvent: {
    path: '/SystemProxy/handleEvent',
    requestStream: false,
    responseStream: false,
    requestType: system_pb.SystemProxyHandleEventRequest,
    responseType: system_pb.SystemProxyHandleEventResponse,
    requestSerialize: serialize_SystemProxyHandleEventRequest,
    requestDeserialize: deserialize_SystemProxyHandleEventRequest,
    responseSerialize: serialize_SystemProxyHandleEventResponse,
    responseDeserialize: deserialize_SystemProxyHandleEventResponse,
  },
};

exports.SystemProxyClient = grpc.makeGenericClientConstructor(SystemProxyService);

Here is the proto file just in case :

syntax = "proto3";

service SystemProxy {

 rpc handleEvent (SystemProxyHandleEventRequest) returns (SystemProxyHandleEventResponse) {}

}

message SystemProxyEvent {

 string domain_id = 1;
 string credentials_id = 2;
 string device_id = 3;
 string connection_id = 4;

}

message SystemProxyHandleEventRequest {

    SystemProxyEvent event = 1;

}

message SystemProxyHandleEventResponse {

    bool success = 1;

}

Is there anything that I am doing wrong ? I tried to remove the "node_modules" directory and redo a npm install from scratch but I get the same behavior. Thanks again !

[murgatroid99]

I think the problem is the filename. I can reproduce the problem with the given code, but it works fine if I rename the proto file to system2.proto. I don't know why that is.

In the future when you encounter that kind of error it might help to include the full dump that comes with the "Plugin output is unparseable" error. That's a complete dump of the binary communication between the plugin and protoc and it could contain something useful for figuring this out.

[ctrlshp]

I renamed the file to "system2.proto" like you did. It doesn't work for me. Here is the full dump as per your request :

npx grpc_tools_node_protoc     --js_out=import_style=commonjs,binary:./src/grpc/proto     --grpc_out=generate_package_definition:./src/grpc/proto     --proto_path=./proto     ./proto/system2.proto
--grpc_out: protoc-gen-grpc: Plugin output is unparseable: z\357\013\n\022system2_grpc_pb.jsz\330\013// GENERATED CODE -- DO NOT EDIT!\n\n\'use strict\';\nvar system2_pb = require(\'./system2_pb.js\');\n\nfunction serialize_SystemProxyHandleEventRequest(arg) {\n  if (!(arg instanceof system2_pb.SystemProxyHandleEventRequest)) {\n    throw new Error(\'Expected argument of type SystemProxyHandleEventRequest\');\n  }\n  return Buffer.from(arg.serializeBinary());\n}\n\nfunction deserialize_SystemProxyHandleEventRequest(buffer_arg) {\n  return system2_pb.SystemProxyHandleEventRequest.deserializeBinary(new Uint8Array(buffer_arg));\n}\n\nfunction serialize_SystemProxyHandleEventResponse(arg) {\n  if (!(arg instanceof system2_pb.SystemProxyHandleEventResponse)) {\n    throw new Error(\'Expected argument of type SystemProxyHandleEventResponse\');\n  }\n  return Buffer.from(arg.serializeBinary());\n}\n\nfunction deserialize_SystemProxyHandleEventResponse(buffer_arg) {\n  return system2_pb.SystemProxyHandleEventResponse.deserializeBinary(new Uint8Array(buffer_arg));\n}\n\n\nvar SystemProxyService = exports[\'SystemProxy\'] = {\n  handleEvent: {\n    path: \'/SystemProxy/handleEvent\',\n    requestStream: false,\n    responseStream: false,\n    requestType: system2_pb.SystemProxyHandleEventRequest,\n    responseType: system2_pb.SystemProxyHandleEventResponse,\n    requestSerialize: serialize_SystemProxyHandleEventRequest,\n    requestDeserialize: deserialize_SystemProxyHandleEventRequest,\n    responseSerialize: serialize_SystemProxyHandleEventResponse,\n    responseDeserialize: deserialize_SystemProxyHandleEventResponse,\n  },\n};\n\n!
/Users/ctrlshp/api/node_modules/grpc-tools/bin/protoc.js:41
    throw error;
    ^

Error: Command failed: /Users/ctrlshp/api/node_modules/grpc-tools/bin/protoc --plugin=protoc-gen-grpc=/Users/ctrlshp/api/node_modules/grpc-tools/bin/grpc_node_plugin --js_out=import_style=commonjs,binary:./src/grpc/proto --grpc_out=generate_package_definition:./src/grpc/proto --proto_path=./proto ./proto/system2.proto
--grpc_out: protoc-gen-grpc: Plugin output is unparseable: z\357\013\n\022system2_grpc_pb.jsz\330\013// GENERATED CODE -- DO NOT EDIT!\n\n\'use strict\';\nvar system2_pb = require(\'./system2_pb.js\');\n\nfunction serialize_SystemProxyHandleEventRequest(arg) {\n  if (!(arg instanceof system2_pb.SystemProxyHandleEventRequest)) {\n    throw new Error(\'Expected argument of type SystemProxyHandleEventRequest\');\n  }\n  return Buffer.from(arg.serializeBinary());\n}\n\nfunction deserialize_SystemProxyHandleEventRequest(buffer_arg) {\n  return system2_pb.SystemProxyHandleEventRequest.deserializeBinary(new Uint8Array(buffer_arg));\n}\n\nfunction serialize_SystemProxyHandleEventResponse(arg) {\n  if (!(arg instanceof system2_pb.SystemProxyHandleEventResponse)) {\n    throw new Error(\'Expected argument of type SystemProxyHandleEventResponse\');\n  }\n  return Buffer.from(arg.serializeBinary());\n}\n\nfunction deserialize_SystemProxyHandleEventResponse(buffer_arg) {\n  return system2_pb.SystemProxyHandleEventResponse.deserializeBinary(new Uint8Array(buffer_arg));\n}\n\n\nvar SystemProxyService = exports[\'SystemProxy\'] = {\n  handleEvent: {\n    path: \'/SystemProxy/handleEvent\',\n    requestStream: false,\n    responseStream: false,\n    requestType: system2_pb.SystemProxyHandleEventRequest,\n    responseType: system2_pb.SystemProxyHandleEventResponse,\n    requestSerialize: serialize_SystemProxyHandleEventRequest,\n    requestDeserialize: deserialize_SystemProxyHandleEventRequest,\n    responseSerialize: serialize_SystemProxyHandleEventResponse,\n    responseDeserialize: deserialize_SystemProxyHandleEventResponse,\n  },\n};\n\n!

    at ChildProcess.exithandler (child_process.js:295:12)
    at ChildProcess.emit (events.js:209:13)
    at maybeClose (internal/child_process.js:1021:16)
    at Socket.<anonymous> (internal/child_process.js:430:11)
    at Socket.emit (events.js:209:13)
    at Pipe.<anonymous> (net.js:658:12) {
  killed: false,
  code: 1,
  signal: null,
  cmd: '/Users/ctrlshp/api/node_modules/grpc-tools/bin/protoc --plugin=protoc-gen-grpc=/Users/ctrlshp/api/node_modules/grpc-tools/bin/grpc_node_plugin --js_out=import_style=commonjs,binary:./src/grpc/proto --grpc_out=generate_package_definition:./src/grpc/proto --proto_path=./proto ./proto/system2.proto'
}

[murgatroid99]

The other difference is that I ran the code on Linux, not Mac. Since I saw a failure I assumed it was the same one you were seeing. Otherwise it could be a problem specific to Mac.

[paambaati]

@murgatroid99 I started working on https://github.com/grpc/grpc-node/issues/931#issuecomment-529702923, but looks like the the types from grpc are quite different to @grpc/grpc-js.

For example, ServiceDefinition is untyped in grpc-js, while it is ServiceDefinition<ImplementationType> in grpc. Similarly, I see a lot of differences in other types (like ServerUnaryCall taking 2 args in grpc-js while taking 1 in grpc).

How do you propose I go about reconciling these differences?

[mpminardi]

@murgatroid99 @paambaati has there been any progress on this?

I modified index.d.ts locally to export the types I needed (e.g., ServiceDefinition, handleUnaryCall, etc.) and changed ServiceDefinition to have MethodDefinition<any, any> instead of MethodDefinition<object, object> and it seems to be working (albeit with modifications still needed to the generated code to do the require('@grpc/grpc-js') change among other things).

I can try making a pull-request if there is still interest.

[paambaati]

@mpminardi Please do! AFAIK, PRs are being actively merged.

[jordanworner]

@murgatroid99 I came across the same error as @ctrlshp in https://github.com/grpc/grpc-node/issues/931#issuecomment-529693760 in both macOS and Linux. The fix seems to be quite simple, I have added a PR https://github.com/grpc/grpc-node/pull/1225

[Wambou]

Hi,

Since I am passing by and seeing this issues, I managed to generate code that uses require('@grpc/grpc-js') by using --grpc_out=grpc_js:<path>.

I still need to test if it works properly.

[kittaakos]

I am going to close it. It is supported by grpc-tools from 1.8.1 and above. The documentation is here. If something is not obvious, I propose comparing the examples.

Thank you for the help! 🙏