Jest: Cannot load native module when running under jest

Created on 7 Dec 2018  ·  19Comments  ·  Source: facebook/jest

🐛 Bug Report

I can load a native module in a regular script just fine, but I cannot do the same in a spec being executed by jest.

To Reproduce

Steps to reproduce the behavior:

Install native module like this: 

$ npm config set @sap:registry https://npm.sap.com
$ npm install @sap/hana-client

Reproducible example:

$ cat package.json
{
  "name": "foo",
  "version": "1.0.0",
  "description": "",
  "main": "foobar.spec.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@sap/hana-client": "^2.3.123",
    "jest": "^23.6.0"
  }
}
$ cat foobar.spec.js
require("@sap/hana-client");
$ node -v
v8.12.0
$ DEBUG=*
$ node foobar.spec.js
  @sap/hana-client:index Starting index.js +0ms
  @sap/hana-client:index Attempting to load Hana node-hdbcapi driver +3ms
  @sap/hana-client:index ... Trying user-built copy... +0ms
  @sap/hana-client:index ... Looking for user-built copy in /Users/else/code/foo/node_modules/@sap/hana-client/build/Release/hana-client.node ...  +0ms
  @sap/hana-client:index Not found. +0ms
  @sap/hana-client:index ... Trying prebuilt copy... +0ms
  @sap/hana-client:index ... Looking for prebuilt copy in /Users/else/code/foo/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/hana-client_v8.node ...  +0ms
  @sap/hana-client:index Loaded. +50ms
  @sap/hana-client:index Success. +0ms
$ echo $?
0
$ node_modules/.bin/jest --runInBand
  @sap/hana-client:index Starting index.js +0ms
  @sap/hana-client:index Attempting to load Hana node-hdbcapi driver +1ms
  @sap/hana-client:index ... Trying user-built copy... +0ms
  @sap/hana-client:index ... Looking for user-built copy in /Users/else/code/foo/node_modules/@sap/hana-client/build/Release/hana-client.node ...  +0ms
  @sap/hana-client:index Not found. +0ms
  @sap/hana-client:index ... Trying prebuilt copy... +0ms
  @sap/hana-client:index ... Looking for prebuilt copy in /Users/else/code/foo/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/hana-client_v8.node ...  +0ms
  @sap/hana-client:index Failed to load DBCAPI. +2ms
  @sap/hana-client:index Could not load: Prebuilt copy did not satisfy requirements. +0ms
  @sap/hana-client:index Could not load modules for Platform: 'darwin', Process Arch: 'x64', and Version: 'v8.12.0' +0ms
 FAIL  ./foobar.spec.js
  ● Test suite failed to run

    Failed to load DBCAPI.

      at /Users/else/code/foo/node_modules/jest-runtime/build/index.js.requireModule (../node_modules/jest-runtime/build/index.js:372:31)
      at /Users/else/code/foo/node_modules/@sap/hana-client/lib/index.js.Object.<anonymous> (node_modules/@sap/hana-client/lib/index.js:127:14)

Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        1.022s
Ran all test suites.

Expected behavior

No error related to module loading

Link to repl or repo (highly encouraged)

https://transfer.sh/%28/JOeLK/example.tar.gz%29.tar.gz

Run npx envinfo --preset jest

Paste the results here:

$ npx envinfo --preset jest
npx: installed 1 in 3.457s

  System:
    OS: macOS 10.14
    CPU: (4) x64 Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
  Binaries:
    Node: 8.12.0 - ~/.nvm/versions/node/v8.12.0/bin/node
    Yarn: 1.10.1 - ~/.nvm/versions/node/v8.12.0/bin/yarn
    npm: 6.4.1 - ~/.nvm/versions/node/v8.12.0/bin/npm
  npmPackages:
    jest: ^23.6.0 => 23.6.0
Bug
Source
picture elsbrock

Most helpful comment

Thanks for sharing, IMO it's much easier though to just require("@sap/hana-client/build.js"); in your jest config.

picture elsbrock on 18 May 2019
👍2

All 19 comments

Hey @else, a few questions

  • What is @sap/hana-client?
  • Are you transpiling?
  • Have you tried the node test environment?
rickhanlonii picture rickhanlonii on 7 Dec 2018

I have managed to track down this bug.

@sap/hana-client's entrypoint is setting process.env['DBCAPI_API_DLL']. The native module checks for this env value and throws if it's not set. It runs as expected in normal node process, but Jest is running all of its tests in separate vms, which use a copies of process.env. If I change the code to use the real process.env, the bug disappears.
I have no idea how vm or native module require work under the hood, but it seems that native module, which was required from vm, is reading from process.env of the main process instead of vm's one. Will try to look deeper into the topic, but if someone knows more about it, any help is appreciated.
Anyways, for now setting env variable manually will solve your problem.

export DBCAPI_API_DLL="path/to/project/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/libdbcapiHDB.dylib"
grosto picture grosto on 8 Dec 2018

Thanks Giorgi, I can confirm that indeed makes it work!

If I change the code to use the real process.env, the bug disappears.

What do you mean by that?

@rickhanlonii regarding your questions:

  • It's a database client, you can get it from registry https://npm.sap.com
  • I am not (see example)
  • Yes, it does not make a difference.
elsbrock picture elsbrock on 8 Dec 2018

What do you mean by that?

I was just explained how I debugged. It's not an actual solution for this problem.

grosto picture grosto on 8 Dec 2018

I know, but I was trying to understand what you did.

elsbrock picture elsbrock on 8 Dec 2018

Oh my bad.

https://github.com/facebook/jest/blob/master/packages/jest-util/src/createProcessObject.js#L19
I changed a return statement of this function with return process.env

grosto picture grosto on 8 Dec 2018
👍1

Anything I can do to help get this fixed?

elsbrock picture elsbrock on 2 Jan 2019

Just to clarify, I am not actively working on this issue. I think it would be more suitable for someone who has more knowledge in native modules and vm module.

grosto picture grosto on 3 Jan 2019

Am aware, thanks 👍

elsbrock picture elsbrock on 3 Jan 2019

Hi,
We have experienced the same issue. Hope a solution will come soon!

nablaoperator picture nablaoperator on 18 Jan 2019

Anything I can do to help get this fixed?

elsbrock picture elsbrock on 25 Jan 2019

Hi,

we have a very similar issue that can be reproduced with this repository:
https://github.com/wwweidi/jest-jwt/
on all platforms (MAC,WIN,LINUX)

Ciao
Stefan

wwweidi picture wwweidi on 18 Mar 2019

You can workaround this by setting the environment variable again in the Jest config.

elsbrock picture elsbrock on 19 Mar 2019

We were having the same issues combining the SAP Hana client library with Jest.

Setting environment variables was no solution for us, since we are working on different systems (Windows, Linux, Mac). The following steps solved it for us:

  1. set the global setup parameter in the jest config file:
globalSetup: "<rootDir>/sapHana.config.js",
  1. create a config file (in this example: sapHana.config.js in the root folder) which sets the environment variable:
var path = require('path');

module.exports = async () => {
    var extensions = {
        'darwin': 'dylib',
        'linux': 'so',
        'win32': 'dll'
    };

    // Look for prebuilt binary and DBCAPI based on platform
    var pb_subdir = null;
    if (process.platform === 'linux') {
        if (process.arch === 'x64') {
            pb_subdir = 'linuxx86_64-gcc48';
        } else if (process.arch.toLowerCase().indexOf('ppc') != -1 && os.endianness() === 'LE') {
            pb_subdir = 'linuxppc64le-gcc48';
        } else {
            pb_subdir = 'linuxppc64-gcc48';
        }
    } else if (process.platform === 'win32') {
        pb_subdir = 'ntamd64-msvc2010';
    } else if (process.platform === 'darwin') {
        pb_subdir = 'darwinintel64-xcode7';
    }

    var modpath = path.dirname(require.resolve("@sap/hana-client/README.md"));
    var pb_path = path.join(modpath, 'prebuilt', pb_subdir);
    var dbcapi = process.env['DBCAPI_API_DLL'] || path.join(pb_path, 'libdbcapiHDB.' + extensions[process.platform]);

    process.env['DBCAPI_API_DLL'] = dbcapi;
  };

Please note that this was tested with version 2.4.126 of @sap/hana-client and things might change with newer versions.

FourSpotProject picture FourSpotProject on 16 May 2019
👍1

Thanks for sharing, IMO it's much easier though to just require("@sap/hana-client/build.js"); in your jest config.

elsbrock picture elsbrock on 18 May 2019
👍2

It seems the issue still exists.
It fails even when hard coding the DBCAPI_API_DLL and loading the hana-client like below.

process.env['DBCAPI_API_DLL'] = "/path_to_my_project/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/libdbcapiHDB.dylib";
require('/path_to_my_project/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/hana-client_v10.node');

seems that the native module loading done thru jest, behaves different than not via jest.

The stack trace is as below:

Error: Failed to load DBCAPI.
    at internal/modules/cjs/loader.js.Module._extensions..node (internal/modules/cjs/loader.js:807:18)
    at internal/modules/cjs/loader.js.Module.load (internal/modules/cjs/loader.js:653:32)
    at internal/modules/cjs/loader.js.tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at internal/modules/cjs/loader.js.Module._load (internal/modules/cjs/loader.js:585:3)
    at internal/modules/cjs/loader.js.Module.require (internal/modules/cjs/loader.js:692:17)
    at internal/modules/cjs/helpers.js.require (internal/modules/cjs/helpers.js:25:18)
    at /path_to_my_project/node_modules/jest-runtime/build/index.js._loadModule (/path_to_my_project/node_modules/jest-runtime/build/index.js:673:29)
    at /path_to_my_project/node_modules/jest-runtime/build/index.js.requireModule (/path_to_my_project/node_modules/jest-runtime/build/index.js:536:10)
    at /path_to_my_project/node_modules/jest-runtime/build/index.js.requireModuleOrMock (/path_to_my_project/node_modules/jest-runtime/build/index.js:699:21)
    at /path_to_my_project/node_modules/@sap/hana-client/lib/index.js.Object.<anonymous> (/path_to_my_project/node_modules/@sap/hana-client/lib/index.js:112:14)

Is there anything under the scenes that would explain why the native (nodejs) require for native modules would behave different via jest?
Or there are more variables required to be set for loading that hana-client module.

sassman picture sassman on 28 Feb 2020
👍1

Is there anything under the scenes that would explain why the native (nodejs) require for native modules would behave different via jest?

The problem here is with process.env.

Every test file is run inside VM module. But VM context does not have Node globals(require, process and etc), so Jest modifies these globals and then injects them in. VM and these modifications are needed to ensure all test files run in isolation. a.test.js should not affect the outcome of b.test.js.

process.env is one of the globals which is modified. Jest makes a new object from the prototype of process.env and reimplements the behaviour. Problem is that process.env is a bit special since Node stores env variables in C++ core and passes it down to child processes and native bindings if needed.

So if you do this in normal Node process

process.env['DBCAPI_API_DLL'] = "/path_to_my_project/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/libdbcapiHDB.dylib";

All the child processes and native bindings can see this value in their Environment Variables.
This is how hana-client works. It read this value from Environment Variables and then tries to load the file.

The reason it does not work with Jest is because Jest's process.env is just a Javascript object. When you set a value on it, this update only exists in Javascript's context. Child processes and Native bindings cannot see it. Since hana-client cannot see this value it fails. But if you do it outside of test context, like in global setup it should work.

I don't think this issue is solvable from Jest's codebase, you will have to use the workarounds described above. Nodejs only has one environment per process. if you run your code in VM module, you get a new Javascript Context but you cannot create new environment without creating a new process.

grosto picture grosto on 5 Mar 2020

Every test file is run inside VM module. But VM context does not have Node globals(require, process and etc), so Jest modifies these globals and then injects them in. VM and these modifications are needed to ensure all test files run in isolation. a.test.js should not affect the outcome of b.test.js.

IMHO the VM that you described should be created the from the node process at the very moment the test enters the describe (the very outer) callback, so that everything that did modify the process / environment etc. is kept for the whole test execution time but only modification from within the describe blocks are not affecting any other calls.

But in any case testing like this would give false confidence on side effects that are in the code and altering the environment. For example imaging function a modifies the process.env in some way function b might be affected because it uses this same variable. In production this code might break. But tested with jest it is all fine.

Anyways I switched away from jest using now nyc / mocha / chai and all these problems are gone.

sassman picture sassman on 5 Mar 2020

Edit: I just found this. Should help a bunch.

Hello. My issue does not involve jest, but @sap/hana-client is giving me trouble and I've not had much luck finding people using it, so here goes.

I'm trying to get this library to work in a container, with no luck. I'm getting this exact same error. I've set the DBCAPI_API_DLL env variable in my container, and I can confirm the value is there.
However, something else fails I guess, since I get this error: Failed to load DBCAPI.

Full error here. I've been able to confirm the env variable is set, and the folder does contain the files it should. I don't know what to do.

DBI API PATH /usr/app/node_modules/@sap/hana-client/prebuilt/linuxx86_64-gcc48/libdbcapiHDB.so
PREBUILT FOLDER CONTENTS:
-- hana-client.node
-- hana-client_v10.node
-- hana-client_v12.node
-- hana-client_v4.node
-- hana-client_v8.node
-- libdbcapiHDB.so
2020-03-05T23:43:08.366Z @sap/hana-client:index Starting index.js
2020-03-05T23:43:08.367Z @sap/hana-client:index Checking for existence of /usr/app/node_modules/@sap/hana-client/prebuilt/linuxx86_64-gcc48/hana-client_v12.node
2020-03-05T23:43:08.367Z @sap/hana-client:index Attempting to load Hana node-hdbcapi driver
2020-03-05T23:43:08.367Z @sap/hana-client:index ... Trying user-built copy...
2020-03-05T23:43:08.367Z @sap/hana-client:index ... Looking for user-built copy in /usr/app/node_modules/@sap/hana-client/build/Release/hana-client.node ...
2020-03-05T23:43:08.368Z @sap/hana-client:index Not found.
2020-03-05T23:43:08.368Z @sap/hana-client:index ... Trying prebuilt copy...
2020-03-05T23:43:08.368Z @sap/hana-client:index ... Looking for prebuilt copy in /usr/app/node_modules/@sap/hana-client/prebuilt/linuxx86_64-gcc48/hana-client_v12.node ...
2020-03-05T23:43:08.369Z @sap/hana-client:index Failed to load DBCAPI.
2020-03-05T23:43:08.369Z @sap/hana-client:index Could not load: Prebuilt copy did not satisfy requirements.
2020-03-05T23:43:08.369Z @sap/hana-client:index Could not load modules for Platform: 'linux', Process Arch:
'x64', and Version: 'v12.16.1'

/usr/app/node_modules/@sap/hana-client/lib/index.js:123
        throw ex;
        ^
{
  code: -20005,
  message: 'Failed to load DBCAPI.',
  sqlState: 'HY000',
  stack: 'Error: Failed to load DBCAPI.\n' +
    '    at internal/modules/cjs/loader.js.Module._extensions..node (internal/modules/cjs/loader.js:1208:18)\n' +
    '    at internal/modules/cjs/loader.js.Module.load (internal/modules/cjs/loader.js:1002:32)\n' +
    '    at internal/modules/cjs/loader.js.Module._load (internal/modules/cjs/loader.js:901:14)\n' +
    '    at internal/modules/cjs/loader.js.Module.require (internal/modules/cjs/loader.js:1044:19)\n' +
    '    at internal/modules/cjs/helpers.js.require (internal/modules/cjs/helpers.js:77:18)\n' +
    '    at /usr/app/node_modules/@sap/hana-client/lib/index.js (/usr/app/node_modules/@sap/hana-client/lib/index.js:115:14)\n' +
    '    at internal/modules/cjs/loader.js.Module._compile (internal/modules/cjs/loader.js:1158:30)\n' +
    '    at internal/modules/cjs/loader.js.Module._extensions..js (internal/modules/cjs/loader.js:1178:10)\n' +
    '    at internal/modules/cjs/loader.js.Module.load (internal/modules/cjs/loader.js:1002:32)\n' +
    '    at internal/modules/cjs/loader.js.Module._load (internal/modules/cjs/loader.js:901:14)\n'
}

Again, apologies for posting a non jest issue on this repo, I'm willing to remove this if inapropriate

Cheers!

BenoitRanque picture BenoitRanque on 6 Mar 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

samzhang111 picture samzhang111  ·  3Comments
hramos picture hramos  ·  3Comments
gustavjf picture gustavjf  ·  3Comments
Secretmapper picture Secretmapper  ·  3Comments
rosiakr picture rosiakr  ·  3Comments