Node-oracledb: yarn install from github not working (v2.0-dev not compiling, needed for LONG RAW.)

[pgkehle]

Answer the following questions:

1. What is your version of Node.js? Run examples/version.js to find versions.

v8.9.1

1. What version of node-oracledb are you using?

v2.0-dev

1. What is the version of your Oracle client (e.g. Instant Client)? How was it installed? Where it is installed?

v12.2
/opt/oracle/instantclient

1. What is the version of Oracle Database?

Not there yet

1. What is your OS and version?

Ubuntu 16.04.3 LTS
MacOS 10.13.1 (17B48)

1. What compiler version did you use? For example, with GCC, run gcc --version

On linux:
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.5) 5.4.0 20160609

On MacOS:
Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin17.2.0

1. What environment variables did you set? How exactly did you set them?

Nothing in particular

1. What exact command caused the problem (e.g. what command did you try to install with)? Who were you logged in as?

I am trying to get LONG RAW support. I submitted a PR that was accepted for strong-oracle, and now want to switch over to this native driver. I started a branch to attempt to shoe-horn in a read-only option in the v1 branch (killed). This was due the v2.0-dev branch not compiling, since it is unclear where we should manually export the odpi folder. I just noticed a comment in the INSTALL.md file for dev-2.0:

If you download or clone node-oracledb code
from GitHub, you need to
make sure the ODPI-C submodule is
also included. Otherwise the build will fail with an error like 'dpi.h' file not found.

The package URL referenced gives a 404.

1. What error(s) you are seeing?

Arguments:
Directory: /opt/servers/node/node_modules/oracledb
Output:
node-oracledb ERR! Error: Cannot find odpi/include/dpi.h.  For GitHub ZIP downloads you must separately download the odpi subdirectory from GitHub.
node-oracledb ERR! Error: See https://github.com/oracle/node-oracledb/blob/master/INSTALL.md#github

gyp: Call to 'INSTURL="https://github.com/oracle/node-oracledb/blob/master/INSTALL.md#github"; ERR="node-oracledb ERR! Error:"; if [ -f odpi/include/dpi.h ]; then echo "Has dpi.h"; else echo "$ERR Cannot find odpi/include/dpi.h.  For GitHub ZIP downloads you must separately download the odpi subdirectory from GitHub." >&2; echo "$ERR See $INSTURL" >&2; echo "" >&2; fi;' returned exit status 0 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/local/nvm/versions/node/v8.9.1/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:336:16)
gyp ERR! stack     at emitTwo (events.js:126:13)
gyp ERR! stack     at ChildProcess.emit (events.js:214:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Linux 4.4.0-98-generic
gyp ERR! command "/usr/local/nvm/versions/node/v8.9.1/bin/node" "/usr/local/nvm/versions/node/v8.9.1/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /opt/servers/node/node_modules/oracledb

Resolution Requests

I see two one option:

1. Update the installation documents to ensure that we can use the dev-2.0 branch. Be specific in how to install along side the instantclient.

[anthony-tuininga]

Indeed the link provided is broken, and the link to the documentation is for the master branch (which is still v1). I'm sure @cjbj will correct those shortly. In the meantime you can get ODPI-C from here. Or, if you want to use git, you can do the following:

git clone https://github.com/oracle/node-oracledb.git
cd node-oracledb
git checkout dev-2.0
git submodule init
git submodule update
npm install

That should get you going!

I can't speak for everyone on the team but I strongly doubt that we will add support for long raw in the v1 stream. The v2 branch is almost ready for a production release and is to be preferred. If you have any difficulties with using it, please let us know! Thanks.

[pgkehle]

Thank you @anthony-tuininga. I'd definitely prefer to use v2.0.

I think I will need to integrate the following into a pre or post-install step in my package.json.

cd node_modules
rm -rf oracledb
git clone https://github.com/oracle/node-oracledb.git
mv node-oracledb oracledb 
cd oracledb
git checkout dev-2.0
git submodule init
git submodule update
npm install

Running this manually does seem to be working very nicely. Thank you!

[pgkehle]

For completeness, postinstall script is here:

scripts/postinstall.js:

#!/usr/bin/env node

let fs            = require('fs');
let child_process = require('child_process');

try {
  console.log('Running postinstall script...');
  process.chdir('./node_modules');

  let exists = fs.existsSync('./oracledb');
  //console.info('oracle db exists: ' + exists);

  if (exists) {
    // go into the directory, update and install again
    console.info('update from github');
    process.chdir('./oracledb');
    child_process.execSync('git pull');
  }
  else {
    console.info('check out from github');
    child_process.execSync('git clone https://github.com/oracle/node-oracledb.git');
    child_process.execSync('mv node-oracledb oracledb');
    process.chdir('./oracledb');
    child_process.execSync('git checkout dev-2.0');
    child_process.execSync('git submodule init'); // only needed the first time
  }

  child_process.execSync('git submodule update');

  console.info('installing...');
  child_process.exec('npm install', (err) => {
    if (err) {
      console.error('error during install: ' + err.message);
    }
    else {
      console.info('postinstall complete.');
    }
  });
}
catch (error) {
  console.error('\n\x1b[41mError:\x1b[0m Could not complete the postinstall script.');
  return;
}

and in package.json:

{
  "scripts": {
    "postinstall": "node scripts/postinstall.js"
  }
}

[anthony-tuininga]

Thanks. FYI, the command "git submodule init" is only required the first time.

[pgkehle]

I updated that last comment to put the git submodule init line when installing the first time,

[cjbj]

Unless you're modifying node-oracledb code, just install it from the branch with npm install oracle/node-oracledb.git#dev-2.0

See https://github.com/oracle/node-oracledb/issues/716

[pgkehle]

@cjbj unfortunately that was not working due to errors in building due to the dpi.h file missing, and the error saying to install the odpi folder manually when installing from the github zip file.

[cjbj]

@pgkehle can you test again? The command npm install oracle/node-oracledb.git#dev-2.0 runs fine for me. It is an important part of our v2 install plan for users who can't install pre-built binaries (for whatever reasons). If the command above doesn't work they will have to do what you are doing with zips and clones, which is ugly.

[pgkehle]

OK, I'm sorry for the confusion here.

npm install oracle/node-oracledb.git#dev-2.0 runs fine, so it seems.

But, we are using yarn for our install process. Much faster than npm.

yarn add oracle/node-oracledb.git#dev-2.0 fails:

error .../node_modules/oracledb: Command failed.
Exit code: 1
Command: node-gyp rebuild
Arguments:
Directory: .../node_modules/oracledb
Output:
node-oracledb ERR! Error: Cannot find odpi/include/dpi.h.  For GitHub ZIP downloads you must separately download the odpi subdirectory from GitHub.
node-oracledb ERR! Error: See https://github.com/oracle/node-oracledb/blob/master/INSTALL.md#github

gyp: Call to 'INSTURL="https://github.com/oracle/node-oracledb/blob/master/INSTALL.md#github"; ERR="node-oracledb ERR! Error:"; if [ -f odpi/include/dpi.h ]; then echo "Has dpi.h"; else echo "$ERR Cannot find odpi/include/dpi.h.  For GitHub ZIP downloads you must separately download the odpi subdirectory from GitHub." >&2; echo "$ERR See $INSTURL" >&2; echo "" >&2; fi;' returned exit status 0 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/local/lib/node_modules/node-gyp/lib/configure.js:336:16)
gyp ERR! stack     at emitTwo (events.js:126:13)
gyp ERR! stack     at ChildProcess.emit (events.js:214:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Darwin 17.2.0
gyp ERR! command "/usr/local/Cellar/node/8.9.1/bin/node" "/usr/local/bin/node-gyp" "rebuild"
gyp ERR! cwd .../node_modules/oracledb

So maybe the real issue is to get this build process to work with yarn.

[cjbj]

@pgkehle thanks for the confirmation. I'll change this to an enhancement request. Hopefully you will be submitting a PR soon?!

[cjbj]

This seems to be a common issue with yarn: https://github.com/yarnpkg/yarn/issues/1488

[pgkehle]

@cjbj Do you see any workarounds? ie: will this be something that can be integrated directly into the packages rather than as a submodule?

[cjbj]

Copying the odpi code into the node-oracledb repo is ugly. Maybe you can stir up the yarn bug at bit?

Or maybe you can stage your own node-oracledb bundle on your own webserver?

The big question is whether yarn will cope with the future two-step install process to automatically download pre-built binaries, see https://github.com/oracle/node-oracledb/issues/18#issuecomment-322675513. The npm install oracledb command will install just the *.js files, and then run an install "script" (in package.json) to download a binary. Will yarn run the script? I'll be pushing out updates to the dev-2.0 branch soon, which will show the scripts - now that @dmcghan has been able to grab time to add support for the extra package format, and refactor the scripts.

[cjbj]

@pgkehle have you tested with the pre-built binaries, using oracledb@dev for the package specifier? See https://github.com/oracle/node-oracledb/issues/716.

[cjbj]

@pgkehle another thing, I noticed that older npm's (e.g. with Node 4) also had issues pulling odpi for npm install oracle/node-oracledb.git#v2.0.14 so I pushed a package with all the source including OPDI to https://github.com/oracle/node-oracledb/releases/download/v2.0.14/oracledb-src-2.0.14.tgz

[pgkehle]

@cjbj Yes, this is working for us. Thank you! 👍

[cjbj]

@pgkehle which one worked - oracledb@dev or the URL?

[pgkehle]

@cjbj yarn install oracledb@dev

[cjbj]

@pgkehle nice. And the prod release is coming soon. I'll close this out now.

[cjbj]

@pgkehle the production release is out, as you may have seen: https://blogs.oracle.com/opal/node-oracledb-20-with-pre-built-binaries-is-on-npm

[pgkehle]

Yes I did see it. Thank you. We have updated our build process to use the standard package.json method for installation. yarn add oracledb works as expected.

Thank you so much for handling this!