Grpc-node: Does not support multiple threads

[kondi]

Problem description

If I try to use the grpc library in multiple threads, I get Error: Module did not self-register. exception.

Reproduction steps

Run with node --experimental-worker:

import { isMainThread, Worker } from 'worker_threads';

if (isMainThread) {
  require('grpc');
  new Worker(__filename);
} else {
  require('grpc'); // throws: `Error: Module did not self-register.` 
}

Environment

• OS name, version and architecture: Windows or Linux
• Node version: 10.12.0
• Node installation method: nvm
• Package name and version: [email protected]

Additional context

Related issue in node repo with more info: https://github.com/nodejs/node/issues/21783

Proposed fix: https://github.com/nodejs/node/issues/21783#issuecomment-429637117 (The module has to be context-aware.)

This has been already fixed at least in these projects:

• node-lzo#10
• iltorb#78

Full stack trace:

Error: Module did not self-register.
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:717:18)
    at Module.load (internal/modules/cjs/loader.js:598:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:537:12)
    at Function.Module._load (internal/modules/cjs/loader.js:529:3)
    at Module.require (internal/modules/cjs/loader.js:636:17)
    at require (internal/modules/cjs/helpers.js:20:18)
    at Object.<anonymous> (...\node_modules\grpc\src\grpc_extension.js:32:13)
    at Module._compile (internal/modules/cjs/loader.js:688:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:699:10)
    at Module.load (internal/modules/cjs/loader.js:598:32)

[murgatroid99]

It looks like this is a relatively new feature; the Node 8 version of that addon documentation does not have that section. Because of this, we can only practically support it if nan supports it and the current nan documentation doesn't mention it.

In addition, I suspect that making this change is more complicated in this library than in others. We do have some global state, and we have an unusual amount of interaction with Node internals (mainly libuv). So, I don't think it's a good idea to just use some preprocessor logic to switch between the two initializers.

[kondi]

Thank you for the info! So this at least requires a nan feature, however, I am not competent enough to open a meaningful ticket there.

Looks like a kind of fix is now to use the @grpc/js instead which does not have native modules. Hopefully, it will graduate to not be experimental.

[murgatroid99]

Update: I have filed an issue for nan to support this at nodejs/nan#844.

[patriq]

Any news on this? Or still waiting on nan? Any work arounds we can do in our end in the meanwhile?

[murgatroid99]

Still waiting. It looks like there has been no activity on that linked issue, and that is a blocker for fixing that for grpc. In the meantime, @grpc/grpc-js is more mature.

[patriq]

Not really a direct dependecy. I want to use Hyperledger Fabric Node SDK with workers. They use grpc-node :( Thanks anyway, guess I will figure some other way! :)

[ctavan]

@murgatroid99 I have been running into this issue by using a combination of firebase/firestore and Next.js, see https://github.com/zeit/next.js/issues/7894#issuecomment-545410475

The Next.js build process uses workers to prebuild pages in parallel.

Are there any plans to fix this in grpc-node? Is there alternatively any way to make firebase/firestore make use of @grpc/grpc-js instead of grpc?

[murgatroid99]

As I have already stated on this issue, fixing this depends on the linked nan issue being fixed, and as you can see, that has not yet happened. You should ask in the firestore repository about using grpc-js instead.

[kirillgroshkov]

Affects us as well.

[kirillgroshkov]

We can't use grpc (C++) because of this issue.
We can't use @grpc/js because of Bandwidth exhausted / Deadline exceeded errors that are not fixed yet.
Catch 22 :(

[murgatroid99]

If the "Bandwidth exhausted" issue you're referring to is #1158, a couple of people have reported that that bug is fixed in Node 13. I'm not sure what "Deadline exceeded" error you're referring to, can you point to a specific GitHub issue?

[drizzd]

For users of Electron and Firestore coming here: The issue for moving @firebase/firestore from grpc to @gprc/js is being tracked here: https://github.com/firebase/firebase-js-sdk/issues/1783.

[akoita]

Hi,

I also had a similar error when trying to use worker threads with Hyperledger Fabric Node SDK client:

• fabric-node-sdk: 1.4.11
• NodeJS: 12.18.3

 error: uncaughtException: Failed to load node_modules/grpc/src/node/extension_binary/node-v79-linux-x64-glibc/grpc_node.node. Module did not self-register: 'node_modules/grpc/src/node/extension_binary/node-v79-linux-x64-glibc/grpc_node.node'.
 Error: Failed to load node_modules/grpc/src/node/extension_binary/node-v79-linux-x64-glibc/grpc_node.node. Module did not self-register: 'node_modules/grpc/src/node/extension_binary/node-v79-linux-x64-glibc/grpc_node.node'.
     at Object.Module._extensions..node (internal/modules/cjs/loader.js:1190:18)
     at Module.load (internal/modules/cjs/loader.js:976:32)
     at Function.Module._load (internal/modules/cjs/loader.js:884:14)
     at Module.require (internal/modules/cjs/loader.js:1016:19)
     at require (internal/modules/cjs/helpers.js:69:18)
     at Object.<anonymous> (/opt/api/node_modules/grpc/src/grpc_extension.js:32:13)
     at Module._compile (internal/modules/cjs/loader.js:1121:30)
     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1160:10)
     at Module.load (internal/modules/cjs/loader.js:976:32)
    at Function.Module._load (internal/modules/cjs/loader.js:884:14)

Is there a recommended workeround for this issue?