Node-oracledb: Webpack: Error: NJS-045: cannot load a node-oracledb binary for Node.js 10.16.3 (win32 x64)

[yakov-rs]

Help me please resolve a problem with description below:

1. Describe the problem
   I developed server (express + oracledb) witch work perfect and have no problem with getting data from Oracle.
   Then I wanted to create a bundle (using webpack 4) and run my bundle separeted from directory where I working.
   And I've got the next message (my problem):

Error: NJS-045: cannot load a node-oracledb binary for Node.js 10.16.3 (win32 x64)
  Looked for D:\test\build\Release\oracledb-4.0.1-win32-x64.node, D:\test\build\Release\oracledb.node, D:\test\build\Debug\oracledb.node
  Node-oracledb installation instructions: https://oracle.github.io/node-oracledb/INSTALL.html

I placed my bundle into "D:\test" catalog.
I copied oracledb-4.0.1-win32-x64.node file into "D:\testbuild\Release" catalog (but error is still have).

1. Include a runnable Node.js script that shows the problem.
   My webpack config is really simple:

const path = require('path');

module.exports = {
  mode: 'production',
  entry: {
    server: './index.js'
  },
  output: {
    path: path.join(__dirname, '../dist'),
    publicPath: '/',
    filename: '[name].js'
  },
  target: 'node',
  node: {
    __dirname: false,
    __filename: false,
  }
};

1. Run node and show the output of:

Platform:  win32
Version:  v10.16.3
Arch:  x64
Oracledb:  4.0.1

node --version
v10.16.3

1. What is your Oracle Database version?
   Oracle Client version 12.2.0.1.0

Thanks for any future help or advice.

[cjbj]

Previous issues mentioning webpack found that changing the require() line in lib/oracledb.js helped. Hardcode the path and see if that helps: https://github.com/oracle/node-oracledb/blob/v4.0.1/lib/oracledb.js#L67

Or consider using pkg, see https://github.com/oracle/node-oracledb/issues/1098#issuecomment-493750589

[yakov-rs]

Thanks for your answer.
You're right it's how webpack works with dynamic paths in require.

[cjbj]

@yakov-rs what was the exact solution for you?

[cjbj]

@yakov-rs I came across my notes from a Slack discussion last year about (presumably) node-oracledb 3.0. Apologies to the uncredited author, but she/he said:

installed string-replace-loader and this webpack config works for me:

{
     test: /\.node$/,
     use: [{
         loader: 'native-ext-loader',
         options: {
         },
     }],
   },
   {
      test: /oracledb\.js$/,
      loader: 'string-replace-loader',
      options: {
          search: 'require(binaryReleasePath);',
          replace: "require('../build/Release/oracledb.node');",
      }
   }

[yakov-rs]

@cjbj
What I've tried:

1. "switch" for "require" depending on platform.
2. using NormalModuleReplacementPlugin

Both solution required to change "oracledb.js" file.
For me the second solution is the better because of no warning when webpack is building, less change in "oracledb.js" and more flexibility with "process.platform" and "process.arch".

[cjbj]

@yakov-rs thanks for sharing!

[MisterMX]

That is how I solved the problem (still hacky though):

// webpack.config.js
config = {
  plugins: [
    new CopyPlugin([
      {
        // Copy the binary Oracle DB driver to dist.
        // This needs to be used together with the string-replace-loader below.
        from: path.resolve(__dirname, 'node_modules/oracledb/build'),
        to: 'node_modules/oracledb/build',
      },
    ]),
  ],
  module: {
    rules: [
      {
        // Change the path the oracledb package is looking for the binary to the dist directory.
        // This is a very hacky solution that modifies the source code in 'node_modules/oracledb/lib/oracledb.js'.
        test: /oracledb\.js$/i,
        loader: 'string-replace-loader',
        options: {
            search: '../',
            replace: './node_modules/oracledb/',
        },
      },
      {
        // Use __non_webpack_require__ to look for the Oracle binary in the dist folder at runtime.
        test: /oracledb\.js$/i,
        loader: 'string-replace-loader',
        options: {
          search: 'require(binaryLocations[i])',
          replace: '__non_webpack_require__(binaryLocations[i])',
        },
      },
    ]
  }
}

[cjbj]

If there is a solution that doesn't involve copying, and is not dependent on other modules (eg not https://github.com/oracle/node-oracledb/pull/851), we'd easily be able to add new paths to the search list https://github.com/oracle/node-oracledb/blob/v4.1.0/lib/oracledb.js#L59-L63

[MisterMX]

A quick solution would be to add './node_modules/oracledb/' + nodbUtil.RELEASE_DIR + '/' + nodbUtil.BINARY_FILE as a fourth option to const binaryLocations in oracledb.js. Then you can use the webpack copy plugin to move the binaries to the dist folder.

For a solution to work with webpack out of the box you would have to hardcode all possible require(<path>) and surround each of them with a try catch clause to catch the module not found error.

[cjbj]

@MisterMX we can do that.

Can you post the exact copy plugin code that users would then need to use (I assume it is a cutdown version of what you posted)?

[MisterMX]

Actually, that's all it. The copy plugin copies the binaries into dist/node_modules/oracledb/<releasedir>. So they can be found using './node_modules/oracledb/' + nodbUtil.RELEASE_DIR + '/' + nodbUtil.BINARY_FILE. You could also just copy them to dist. But I prefer the full path as it rules out any (despite very unlikely) conflicts with other files.

Here is a full webpack.config.js from my code above with unnecessary stuff removed that works if the path is added to binaryLocations:

const baseConfig = {
  entry: './src/index.js',
  target: 'node',
  output: {
    path: path.join(__dirname, 'dist'),
    filename: 'index.js',
  },
  plugins: [
    new CopyPlugin([
      {
        // Copy the binary Oracle DB driver to dist.
        from: path.resolve(__dirname, 'node_modules/oracledb/build'),
        to: 'node_modules/oracledb/build',
      },
    ]),
  ],
  module: {
    rules: [
      {
        // Use __non_webpack_require__ to look for the Oracle binary in the dist folder at runtime.
        test: /oracledb\.js$/,
        loader: 'string-replace-loader',
        options: {
          search: 'require(binaryLocations[i])',
          replace: '__non_webpack_require__(binaryLocations[i])',
        },
      },
    ],
  },
};

[cjbj]

@MisterMX thanks for that.

The change planned for lib/oracledb.js in node-oracledb 4.2 will make the search path like:

const binaryLocations = [
  '../' + nodbUtil.RELEASE_DIR + '/' + nodbUtil.BINARY_FILE,  // pre-built binary
  '../' + nodbUtil.RELEASE_DIR + '/' + 'oracledb.node',       // binary built from source
  '../build/Debug/oracledb.node',                             // debug binary
  // Ease Webpack use, see https://github.com/oracle/node-oracledb/issues/1156
  './node_modules/oracledb/' + nodbUtil.RELEASE_DIR + '/' + nodbUtil.BINARY_FILE,
  './node_modules/oracledb/' + nodbUtil.RELEASE_DIR + '/' + 'oracledb.node'
];

[cjbj]

@MisterMX the load path change landed in node-oracledb 4.2, see https://github.com/oracle/node-oracledb/blob/v4.2.0/lib/oracledb.js#L59-L66

Let us know if this is optimal.

[MisterMX]

Sorry for the late response.

Unfortunately, your solution does not work because it still uses require. Webpack replaces this function with one of its own. To get the native JS function, you would have to use __non_webpack_require__.

I added a simple function requireBinary to the code, that uses __non_webpack_require__ if available and falls back to require otherwise:

function requireBinary(path) {
  if (__non_webpack_require__) {
    return __non_webpack_require__(path)
  }

  return require(path)
}

let oracledbCLib;
for (let i = 0; i < binaryLocations.length; i++) {
  try {
    oracledbCLib = requireBinary(binaryLocations[i]);
    break;
  } catch(err) {
    if (err.code !== 'MODULE_NOT_FOUND' || i == binaryLocations.length - 1) {
      let nodeInfo;
      if (err.code === 'MODULE_NOT_FOUND') {
        // a binary was not found in any of the search directories
        nodeInfo = `\n  Looked for ${binaryLocations.map(x => require('path').resolve(__dirname, x)).join(', ')}\n  ${nodbUtil.getInstallURL()}\n`;
      } else {
        nodeInfo = `\n  Node.js require('oracledb') error was:\n  ${err.message}\n  ${nodbUtil.getInstallHelp()}\n`;
      }
      throw new Error(nodbUtil.getErrorMessage('NJS-045', nodeInfo));
    }
  }
}

Just tested it with my code and it worked. Hope this helps.

Best

[cjbj]

I can add that. Doesn't the second require() also need changing, so the message is useful?

[cjbj]

@MisterMX also please check whether this works in Webpack:

function requireBinary(path) {
  if (typeof __non_webpack_require__ === 'function') 
    return __non_webpack_require__(path);
  else
    return require(path);
}

Or perhaps this is better:

const requireBinary = (typeof __non_webpack_require__ === 'function') ? __non_webpack_require__ : require;

because your method fails in vanilla Node.js.

With this change, does binaryLocations still need the newly added paths?

[MisterMX]

@cjbj your code also works with Webpack 👍

The extra binaryLocations are still necessary.

By "second require" you mean require('path') in

nodeInfo = `\n  Looked for ${binaryLocations.map(x => require('path').resolve(__dirname, x)).join(', ')}\n  ${nodbUtil.getInstallURL()}\n`;

?

You don't need to change that, because path is a node module that Webpack is able to resolve. __non_webpack_require__ is only needed for paths that are not included in the bundle.

What does cause a problem is the __dirname in the same line. By default, Webpack replaces every occurrence of __dirname with the root path / (https://webpack.js.org/configuration/node/#node-__dirname).

This behaviour can be changed in the config.js, but if not, the error message prints the wrong path that wasn't actually checked.

I am not exactly sure what would be the best way to fix this, as __dirname and the alternative process.cwd() are not always the same even without Webpack (see https://stackoverflow.com/questions/9874382/whats-the-difference-between-process-cwd-vs-dirname).

Since it is not really an issue, I would also agree to just ignore it, since Webpack developers should be aware of the __dirname behaviour.

[cjbj]

@MisterMX thanks for the info.

With the planned change to use requireBinary(), and the existing extra binaryLocations entries, will Webpack users still need to change webpack.config.js in any way?

The current diff I'm proposing is:

diff --git a/lib/oracledb.js b/lib/oracledb.js
index 23a96788..998d248d 100644
--- a/lib/oracledb.js
+++ b/lib/oracledb.js
@@ -56,6 +56,8 @@ const defaultPoolAlias = 'default';

 //  Load the Oracledb binary

+const requireBinary = (typeof __non_webpack_require__ === 'function') ? __non_webpack_require__ : require; // See Issue 1156
+
 const binaryLocations = [
   '../' + nodbUtil.RELEASE_DIR + '/' + nodbUtil.BINARY_FILE,  // pre-built binary
   '../' + nodbUtil.RELEASE_DIR + '/' + 'oracledb.node',       // binary built from source
@@ -68,13 +70,14 @@ const binaryLocations = [
 let oracledbCLib;
 for (let i = 0; i < binaryLocations.length; i++) {
   try {
-    oracledbCLib = require(binaryLocations[i]);
+    oracledbCLib = requireBinary(binaryLocations[i]);
     break;
   } catch(err) {
     if (err.code !== 'MODULE_NOT_FOUND' || i == binaryLocations.length - 1) {
       let nodeInfo;
       if (err.code === 'MODULE_NOT_FOUND') {
-        // a binary was not found in any of the search directories
+        // A binary was not found in any of the search directories.
+   // Note this message may not be accurate for Webpack users since Webpack changes __dirname
         nodeInfo = `\n  Looked for ${binaryLocations.map(x => require('path').resolve(__dirname, x)).join(', ')}\n  ${nodbUtil.getInstallURL()}\n`;
       } else {
         nodeInfo = `\n  Node.js require('oracledb') error was:\n  ${err.message}\n  ${nodbUtil.getInstallHelp()}\n`;

[cjbj]

@MisterMX I would love to see a basic, runnable example on using Webpack with node-oracledb. The hack Webpack example I threw together works the same with and without the above change.

[cjbj]

@MisterMX is the change above OK?

[MisterMX]

I created a simple Webpack example: https://github.com/MisterMX/node-oracledb-webpack-example

The project does not run with node-oracledb version 4.2.0, because it fails to load the binary.

If I add your changes from above, it works.

[cjbj]

@MisterMX thank you. I'll merge the patch to the master branch when I get a chance.

[cjbj]

@MisterMX would it be cleaner if we changed the paths in lib/oracledb.js:

diff --git a/lib/oracledb.js b/lib/oracledb.js
index 4bdbe828..441f6acc 100644
--- a/lib/oracledb.js
+++ b/lib/oracledb.js
@@ -64,8 +64,8 @@ const binaryLocations = [
   '../' + nodbUtil.RELEASE_DIR + '/' + 'oracledb.node',       // binary built from source
   '../build/Debug/oracledb.node',                             // debug binary
   // Ease Webpack use, see https://github.com/oracle/node-oracledb/issues/1156
-  './node_modules/oracledb/' + nodbUtil.RELEASE_DIR + '/' + nodbUtil.BINARY_FILE,
-  './node_modules/oracledb/' + nodbUtil.RELEASE_DIR + '/' + 'oracledb.node'
+  './oracledb/' + '/' + nodbUtil.BINARY_FILE,
+  './oracledb/' + '/' + 'oracledb.node'
 ];

 let oracledbCLib;

This would give a flatter dist like:

cjones@cjones-mac2:/tmp/dist$ lr -lR
total 72
drwxr-xr-x   4 wheel    128  6 Mar 15:59 ./
drwxrwxrwt  29 wheel    928  6 Mar 15:59 ../
-rw-r--r--   1 wheel  35519  6 Mar 15:59 index.js
drwxr-xr-x   3 wheel     96  6 Mar 16:00 oracledb/

./oracledb:
total 856
drwxr-xr-x  3 wheel      96  6 Mar 16:00 ./
drwxr-xr-x  4 wheel     128  6 Mar 15:59 ../
-rw-r--r--  1 wheel  438196  6 Mar 15:59 oracledb.node

Your webpack.config.js would need to be:

        new CopyPlugin([
            {
                // Copy the binary Oracle DB driver to dist.
                from: path.resolve(__dirname, 'node_modules/oracledb/build/Release'),
                to: 'oracledb',
            },
        ]),

What do you think?

[cjbj]

@MisterMX ping

[MisterMX]

Sorry, @cjbj, I was caught up with other projects.

Sure, you could do that. I am not really sure, if there is a correct way to do this.

In our project, we copy the files to dist/node_modules/oracledb/..., because we also had to copy some files from other libraries and this approach made it clear where the files came from.

If you think your way is cleaner, we can go this way. I don't think it would make much of a difference.

[cjbj]

@MisterMX that's a good argument. My thoughts were that currently on Windows you can copy the Instant Client DLLs to the same directory where oracledb-4.2.0-win32-x64.node is and have a complete, self-standing application that can be distributed without users needing to set PATH. (I'm still hopeful Oracle can change its build steps when creating libclntsh on Linux to also make this work on Linux in a future). Since Instant Client is not part of node-oracledb, it seems odd to put Instant Client in a node_modules subdirectory. Let me discuss it with the team and then make a decision.

[cjbj]

@MisterMX no one else had a strong opinion so I went with /node_modules/oracledb/build. I pushed the change to the master branch - take a look. (Sadly a distraction snafu made me commit it with an ODPI-C update and I don't think it's worth force-pushing an amended commit message).

For future readers, your webpack.config.js will need to be:

        new CopyPlugin([
            {
                // Copy the binary Oracle DB driver to dist.
                from: path.resolve(__dirname, 'node_modules/oracledb/build'),
                to: 'node_modules/oracledb/build',
            },
        ]),

[MisterMX]

Looks, good. Thanks for implementing it 👍

[dbertozzi]

I just went through this using Windows 10 x64, Oracle Instant Client 19.6, node v10.21.0, and oracle/node-oracledb.git#v4.2.0. It still required both CopyPlugin and string-replace-loader to get it to work.

Thanks for the solution.

[cjbj]

@dbertozzi thanks for the feedback. We just released node-oracledb 5.0; you could try with it. I'm expecting you'll just need the CopyPlugin.