Better-sqlite3: Usage with Worker Threads

[instagendleg]

Hello,

I use a node.js app in order to benchmark an application server. I use for that a queue combined with multiple threads managed by Node Worker Threads.

better-sqlite3 is excellent for parallel access, so I wanted to use it in my app. The problem is when I include the module in my worker, an error Module did not self-register. appear.

Environments

better-sqlite: v5.4.0

Tested with node:

• 11.10.0
• 10.15.1
• 10.15.0

System: Windows 10 1709

Steps to reproduce

1. Write a new node module with the worker threads integration

const {
  Worker, isMainThread, parentPort, workerData
} = require('worker_threads');

if (isMainThread) {
  module.exports = function parseJSAsync(script) {
    return new Promise((resolve, reject) => {
      const worker = new Worker(__filename, {
        workerData: script
      });
      worker.on('message', resolve);
      worker.on('error', reject);
      worker.on('exit', (code) => {
        if (code !== 0)
          reject(new Error(`Worker stopped with exit code ${code}`));
      });
    });
  };
} else {
  const { parse } = require('some-js-parsing-library');
  const script = workerData;
  parentPort.postMessage(parse(script));
}

1. Include better-sqlite3 module

const sqlite = require('better-sqlite3'); // the new added module
const {
  Worker, isMainThread, parentPort, workerData
} = require('worker_threads');

if (isMainThread) {
...

1. See the error

Error: Module did not self-register.
    at Object.Module._extensions.node (internal/modules/cjs/loader.js:779:18)
    at Module.load (internal/modules/cjs/loader.js:630:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:570:12)
    at Function.Module._load (internal/modules/cjs/loader.js:562:3)
    at Module.require (internal/modules/cjs/loader.js:667:17)
    at require (internal/modules/cjs/helpers.js:20:18)
    at Object.<anonymous> ([XXX]\node_modules\\better-sqlite3\\lib\\database.js:5:21)
    at Module._compile (internal/modules/cjs/loader.js:738:30)
    at Object.Module._extensions.js (internal/modules/cjs/loader.js:749:10)
    at Module.load (internal/modules/cjs/loader.js:630:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:570:12)
    at Function.Module._load (internal/modules/cjs/loader.js:562:3)
    at Module.require (internal/modules/cjs/loader.js:667:17)
    at require (internal/modules/cjs/helpers.js:20:18)
    at Object.<anonymous> ([XXX]\node_modules\\better-sqlite3\\lib\\index.js:2:18)
    at Module._compile (internal/modules/cjs/loader.js:738:30)

Related sources

I have already trace some related articles:

• https://stackoverflow.com/questions/28486891/uncaught-error-module-did-not-self-register/28859693
• https://stackoverflow.com/questions/53434134/loading-the-same-native-module-into-two-node-js-threads

Regards,

[JoshuaWise]

The ability to use native addons within worker threads is very new (Node 11.11.0 and up). Node.js also requires the addon (better-sqlite3, in this case) to be cooperative, which means no existing addons will automatically have this ability. It's possible to support this, but it will require significant updates to better-sqlite3. I'll mark this as "enhancement", and perhaps support for it can be added some day in the future.

[wmertens]

Hijacking this to muse about how it would work with threads.

I really want to use this library, but I would have to run it in a different process to have a responsive webserver at all times, and worker threads look like a nice middle ground, even giving parallelization for read-only workloads.

Unfortunately, the communication with the worker needs to undergo a serialization/deserialization step. The most performant way to communicate with worker threads seems to be a SharedArrayBuffer.

I wonder what would be a nice way to serialize sqlite responses directly without having to go through the Node abstractions, and then do the deserialization in the main thread from the SharedArrayBuffer.

[micooz]

I encountered this problem too, here is cleaner code to reproduce:

const { Worker, isMainThread } = require('worker_threads');

if (isMainThread) {
  require('better-sqlite3');
  new Worker(__filename);
} else {
  require('better-sqlite3'); // Error: Module did not self-register
}

[JoshuaWise]

This is now supported in v7.0.0. See an example here.