Protobuf.js: Generated static-module with grpc-node usage example

[MOZGIII]

protobuf.js version: 6.8.6

I've generated a bunch of code using pbjs and pbts (I'm using TypeScript), and want to use them from node.js to call a Go app using grpc.

I have no idea on how to actually perform the request. Can someone provide an example of how would I make a grpc call?

[MOZGIII]

I kind of found the solution. I'm using the generated static-module's js and ts, but I monkey-patch rpcCall methods in services upon construction (right after I call MyService.create). I replace them with a custom build method that uses client build with grpc and json-module. This is kind of a durty hack, however it has all the properties I need so far, so I'm very happy with the results.
If only RPCImpl was less opinionated on how thing should work, it would've been much easier.

I'd propose changing RPCImpl to allow for a more flexible implementaion. Or just allow for a cleaner way to override rpcCall so that it could be replaced with custom logic - that would improve the situation as well.

[gcarling]

I am having a similar issue - I want to use GRPC, which seems like the obvious choice for making RPCs using protobuf. However it seems like the existing rpcImpl stuff in this library kind of overlaps strangely with all of the examples of node-grpc, such as this: https://grpc.io/docs/tutorials/basic/node.html.

Specifically, node-grpc seems to expect you to load the grpc client from the .proto file itself, but using protobuf.js you are simply given a JS wrapper, and it seems odd to then have to reach back to the .proto file to use grpc.

I think that this library is set up to be RPC agnostic, but seeing as how GRPC is the de facto protobuf RPC it seem strange that this use case is not supported. @MOZGIII would you mind sharing some more information about the workaround you have built?

[MOZGIII]

Unfortunately, I can't share the code. The idea is like this:

// Create client with node-grpc
const client = new nodeGrpc.mypackage.MyService('localhost:50051', grpc.credentials.createInsecure());

// Create typed service
const dummy = () => {}
const svc = typed.mypackage.MyService.create(dummy)

// Replace rpcImpl that uses client for all the calls.
(svc as any).rpcImpl = makeRpcImpl(client)

// Now you have typed service that's capable to do gRPC calls.
return svc

The code above may have mistakes, but it should give you an idea.

The makeRpcImpl produces the function that has just the same interface that the protobuf.js's rpcImpl has, but it uses passed client and performs real calls though it.

[murgatroid99]

I am the maintainer of Node gRPC. The gRPC library isn't really set up to interoperate cleanly with Protobuf.js's rpcImpl interface, but with some work it should be possible. Here's a simple example of how to do it:

const grpc = require('grpc');
const client = new grpc.Client('localhost:50051', grpc.credentials.createInsecure());
function passThrough(argument) {
  return argument;
}
const rpcImpl = function(method, requestData, callback) {
  /* Conventionally in gRPC, the request path looks like 
     "/package.names.ServiceName/MethodName/", 
     so getPath would generate that from the method */
  const path = getPath(method);
  // Using passThrough as the serialize and deserialize functions
  client.makeUnaryRequest(path, passThrough, passThrough, requestData, callback);
};

I'm assuming here that the requestData argument will actually be a Buffer on node, and that callback will accept a Buffer. If both really have to be a Uint8Array, then the serializer will have to convert that to a Buffer and the deserializer will have to do the opposite, but that should be pretty simple.

Note that I specifically invoke the makeUnaryRequest method here. The rpcImpl interface doesn't really seem to be set up to handle streaming except in the very specific case where you only have a single active streaming call to any one method, and the stream has exactly one response message per request message.

[MOZGIII]

Ok, I messed up my code sample.

I really meant to replace rpcCall, not rpcImpl.

Fixed version:

// Create client with node-grpc
const client = new nodeGrpc.mypackage.MyService('localhost:50051', grpc.credentials.createInsecure());

// Create typed service
const dummy = () => {}
const svc = typed.mypackage.MyService.create(dummy)

// Replace rpcCall with the one that uses client for all the calls.
(svc as any).rpcCall = makeRpcCall(client)

// Now you have typed service that's capable to do gRPC calls.
return svc

and

// Simplified version that captures the point, but it not correct
const makeRpcCall = (client) => {
  return (method, requestCtor, responseCtor, request, callback) => {
    // requestCtor and responseCtor are both ignored
    return client[method.name](request).then(callback)
  }
}

Real version implements the API of https://github.com/dcodeIO/protobuf.js/blob/23f4b990375efcac2c144592cf4ca558722dcf2d/src/rpc/service.js#L79-L127

That includes emitting the events and stuff like that.

[MOZGIII]

@murgatroid99 my version is different in that I don't have to manually specify the request type (unary/etc) - it's all handled by the node-grpc client (that is not generic one like in your sample, but built from the json-module data generated by pbjs, and therefore has all the required data).

I acutally have a special factory that takes both static-module and json-module bundles (typed and nodeGrpc in my code samples) and is capable to produce node-grpc powered services that are properly typed with TypeScript definition generated by pbts from static-module.

UPD: I think if with the static-module pbjs generated information required to make a call using raw node-grpc client (I mean, not the node-grpc client that's build from the json-module or .proto file, but bare grpc.Client) - for example in a property of the generated method, like mypackage.MyService.myMethod.grpcMetadata - that would eliminate the need to generate the json-module, as it's only purpose is to construct proper grpc clients.

[dev-mraj]

I am the maintainer of Node gRPC. The gRPC library isn't really set up to interoperate cleanly with Protobuf.js's rpcImpl interface, but with some work it should be possible. Here's a simple example of how to do it:

const grpc = require('grpc');
const client = new grpc.Client('localhost:50051', grpc.credentials.createInsecure());
function passThrough(argument) {
  return argument;
}
const rpcImpl = function(method, requestData, callback) {
  /* Conventionally in gRPC, the request path looks like 
     "/package.names.ServiceName/MethodName/", 
     so getPath would generate that from the method */
  const path = getPath(method);
  // Using passThrough as the serialize and deserialize functions
  client.makeUnaryRequest(path, passThrough, passThrough, requestData, callback);
};

I'm assuming here that the requestData argument will actually be a Buffer on node, and that callback will accept a Buffer. If both really have to be a Uint8Array, then the serializer will have to convert that to a Buffer and the deserializer will have to do the opposite, but that should be pretty simple.

Note that I specifically invoke the makeUnaryRequest method here. The rpcImpl interface doesn't really seem to be set up to handle streaming except in the very specific case where you only have a single active streaming call to any one method, and the stream has exactly one response message per request message.

if its not "set up to interoperate" then i didn't understood yet, why you used protobuff.js and made grpc-tools accound protoc.

at run time you are using protobuff.js but if now try to use protoc (as in grpc-tools) then its's definitions won't be fit with protobuff.js's definitions (runtime)

[murgatroid99]

I was saying that gRPC does not interoperate with service interfaces generated by pbjs. gRPC instead uses a different kind of service interface and other protobuf.js types.

If you use grpc-tools/protoc to generate code, you should not be using any protobuf.js types, so there should not be any conflict.

grpc-tools/protoc and protobuf.js are simply two separate options for a data serialization and service declaration format that can be used with the gRPC library.