Node-sqlite3: 3.1.5 breaks global install of dependents on Windows

Created on 5 Oct 2016 · 10Comments · Source: mapbox/node-sqlite3

On Windows 10 (x86 and x64), installing a package that depends on [email protected] (or something that resolves to 3.1.5, like ^3.1.4) globally (npm install -g) fails with error EBUSY: resource busy or locked, rmdir from npm install node-pre-gyp. The same thing works as expected with 3.1.4. It also works with 3.1.5 on OS X. Installing locally also works as expected.

I have not understood yet how exactly the error comes about, but it appears to be caused by 9a038b50.

Minimal reproducing example:

C:\Users\Walther\sqtest315>dir
 Volume in drive C has no label.
 Volume Serial Number is 7CE2-F96E

 Directory of C:\Users\Walther\sqtest315

05.10.2016  16:32    <DIR>          .
05.10.2016  16:32    <DIR>          ..
05.10.2016  16:28                85 package.json
               1 File(s)             85 bytes
               2 Dir(s)  26'892'546'048 bytes free

C:\Users\Walther\sqtest315>type package.json
{
"name": "sqtest315",
"version": "1.0.0",
"dependencies": {"sqlite3": "3.1.5"}
}
C:\Users\Walther\sqtest315>npm pack
sqtest315-1.0.0.tgz

C:\Users\Walther\sqtest315>npm install -g sqtest315-1.0.0.tgz

> [email protected] preinstall C:\Users\Walther\node\node_modules\.staging\sqlite3-a06df773
> npm install node-pre-gyp

C:\Users\Walther\node
`-- [email protected]
  +-- [email protected]
  | `-- [email protected]
  +-- [email protected]
  | `-- [email protected]
  +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | | +-- [email protected]
  | | | | `-- [email protected]
  | | | `-- [email protected]
  | | +-- [email protected]
  | | | `-- [email protected]
  | | `-- [email protected]
  | `-- [email protected]
  +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | `-- [email protected]
  +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | +-- [email protected]
  | | +-- [email protected]
  | | | +-- [email protected]
  | | | +-- [email protected]
  | | | +-- [email protected]
  | | | `-- [email protected]
  | | +-- [email protected]
  | | | `-- [email protected]
  | | +-- [email protected]
  | | | +-- [email protected]
  | | | +-- [email protected]
  | | | | `-- [email protected]
  | | | +-- [email protected]
  | | | `-- [email protected]
  | | `-- [email protected]
  | |   `-- [email protected]
  | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | `-- [email protected]
  | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | | +-- [email protected]
  | | | +-- [email protected]
  | | | `-- [email protected]
  | | `-- [email protected]
  | |   +-- [email protected]
  | |   +-- [email protected]
  | |   +-- [email protected]
  | |   +-- [email protected]
  | |   | `-- [email protected]
  | |   +-- [email protected]
  | |   +-- [email protected]
  | |   | `-- [email protected]
  | |   +-- [email protected]
  | |   +-- [email protected]
  | |   `-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | `-- [email protected]
  +-- [email protected]
  | `-- [email protected]
  |   +-- [email protected]
  |   +-- [email protected]
  |   +-- [email protected]
  |   | `-- [email protected]
  |   |   +-- [email protected]
  |   |   `-- [email protected]
  |   +-- [email protected]
  |   `-- [email protected]
  +-- [email protected]
  +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | `-- [email protected]
  `-- [email protected]
    +-- [email protected]
    | `-- [email protected]
    +-- [email protected]
    +-- [email protected]
    | `-- [email protected]
    +-- [email protected]
    | +-- [email protected]
    | +-- [email protected]
    | +-- [email protected]
    | +-- [email protected]
    | +-- [email protected]
    | `-- [email protected]
    `-- [email protected]

npm ERR! Windows_NT 10.0.10240
npm ERR! argv "C:\\Users\\Walther\\node\\node.exe" "C:\\Users\\Walther\\node\\node_modules\\npm\\bin\\npm-cli.js" "install" "node-pre-gyp"
npm ERR! node v4.4.7
npm ERR! npm  v3.10.8
npm ERR! path C:\Users\Walther\node\node_modules\.staging\sqlite3-a06df773
npm ERR! code EBUSY
npm ERR! errno -4082
npm ERR! syscall rmdir

npm ERR! EBUSY: resource busy or locked, rmdir 'C:\Users\Walther\node\node_modules\.staging\sqlite3-a06df773'
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\Walther\node\node_modules\.staging\sqlite3-a06df773\npm-debug.log
npm ERR! Windows_NT 10.0.10240
npm ERR! argv "C:\\Users\\Walther\\node\\node.exe" "C:\\Users\\Walther\\node\\node_modules\\npm\\bin\\npm-cli.js" "install" "-g" "sqtest315-1.0.0.tgz"
npm ERR! node v4.4.7
npm ERR! npm  v3.10.8
npm ERR! code ELIFECYCLE

npm ERR! [email protected] preinstall: `npm install node-pre-gyp`
npm ERR! Exit status 4294963214
npm ERR!
npm ERR! Failed at the [email protected] preinstall script 'npm install node-pre-gyp'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the sqlite3 package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     npm install node-pre-gyp
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs sqlite3
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls sqlite3
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\Walther\sqtest315\npm-debug.log

C:\Users\Walther\sqtest315>

npm-debug.log.zip

Source

cwalther

👍2

Most helpful comment

@springmeyer 9a038b5 is broken, you can't list node-pre-gyp as a dep, _and_ install it in preinstall, I suspect what is happening is that sometimes npm is installing it at the same time as the preinstall script is trying.

The change was committed directly without a PR, so its hard to understand the justificatio.

I'm going to guess its because of the well known problem with npm 2.x that it does not guarantee a package's dependencies are installed before a package's install script runs, so its an attempt to work around this. The only robust fix is to bundle any deps that must be present at the time a package's install script runs. Which is what the code used to do... so why was it changed?

sam-github on 6 Oct 2016

👍3

All 10 comments

The change was committed directly without a PR, so its hard to understand the justificatio.

sam-github on 6 Oct 2016

👍3

you can't list node-pre-gyp as a dep, and install it in preinstall, I suspect what is happening is that sometimes npm is installing it at the same time as the preinstall script is trying.

@sam-github can you point me to npm documentation on this restriction? I've tried and been unable to replicate this problem.

The change was committed directly without a PR, so its hard to understand the justificatio.

Bundling is a headache because security issues in the bundled dependencies cannot be fixed without re-tagging node-pre-gyp, and every single module that depends on it upgrading. Across modules depending on node-pre-gyp this was resulting in a lot of unnecessary time lost and confusion from users that don't know why vulnerable modules are being included (since they are way downstream of the module that includes node-pre-gyp).

Secondarily the size of the package is then large and users were complaining. But this is going to be this way once installed of course, so this is less of a problem.

Some background at https://github.com/mapbox/node-pre-gyp/issues/162 and https://github.com/mapbox/node-pre-gyp/pull/163

springmeyer on 7 Oct 2016

@sam-github any idea what .sl-blip.js is. I see npm ERR! [email protected] preinstall: node .sl-blip.js in many of the error messages at https://github.com/strongloop/strong-arc/issues/1733.

In the meantime I'm working on a v3.1.6 release that bundles again to avoid the breakages.

springmeyer on 7 Oct 2016

The failing node .sl-blip.js is another example of annoying install script behaviour I think.

One thing you can try w.r.t. install scripts in the future like you did with node-pre-gyp is to do npm install --no-global node-pre-gyp. This will prevent it from being hoisted and installed globally when sqlite3 is being pulled in by a global module, but won't have any effect when not being pulled in via a global install. This is required because all npm CLI flags get turned into env vars, and the propagate to nested npm processes that way.

rmg on 7 Oct 2016

npm install --no-global node-pre-gyp

@rmg ah, that sounds really smart. Do you think that could potentially avoid this Windows-specific EBUSY error as well, or are you recommending as a good idea in general?

springmeyer on 7 Oct 2016

v3.1.6 is published. Can someone who can replicate confirm this is fixed and let me know here?

springmeyer on 7 Oct 2016

@springmeyer just as a general best practice. I'm not sure if it would help with the EBUSY or not, but it is possible it might.

I'll see if I can wrangle some people into testing 3.1.6.

rmg on 7 Oct 2016

@springmeyer I just tested on node 5.11.1 and npm 3.8.6 and npm install -g apiconnect works again.

superkhau on 7 Oct 2016

👍 Confirmed that things work again with 3.1.6 in both my minimal example (interesting that it didn’t reproduce the problem for you, btw – any differences in node/npm/os versions?) and my real case (a private package). Thanks!

I leave it to you whether to close this issue now or keep it open while looking for a better solution.

cwalther on 7 Oct 2016

I tried the npm install --no-global node-pre-gyp by hacking it into a copy of sqlite3-3.1.5.tgz. On the first attempt, my minimal example just failed with a different error (EPERM, see below), but subsequent attempts (deleting the installed package in between) were successful. I can’t make much sense of that, but I’ll just blame it on incomprehensible Windows behavior. So I’m unsure if that’s a viable solution.

Error output of first attempt

C:\Users\Walther\sqtest315cw>dir
 Volume in drive C has no label.
 Volume Serial Number is 7CE2-F96E

 Directory of C:\Users\Walther\sqtest315cw

07.10.2016  08:59    <DIR>          .
07.10.2016  08:59    <DIR>          ..
07.10.2016  09:00               132 package.json
               1 File(s)            132 bytes
               2 Dir(s)  26'830'405'632 bytes free

C:\Users\Walther\sqtest315cw>type package.json
{
"name": "sqtest315-cw",
"version": "1.0.0",
"dependencies": {"sqlite3": "C:\\Users\\Walther\\Desktop\\sqlite3-cw-3.1.5.tgz"}
}
C:\Users\Walther\sqtest315cw>npm pack
sqtest315-cw-1.0.0.tgz

C:\Users\Walther\sqtest315cw>npm install -g sqtest315-cw-1.0.0.tgz

> [email protected] preinstall C:\Users\Walther\node\node_modules\.staging\sqlite3-87af97aa
> npm install --no-global node-pre-gyp

[email protected] C:\Users\Walther\node\node_modules\.staging\sqlite3-87af97aa
`-- [email protected]
  +-- [email protected]
  | `-- [email protected]
  +-- [email protected]
  | `-- [email protected]
  +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | | +-- [email protected]
  | | | | `-- [email protected]
  | | | `-- [email protected]
  | | +-- [email protected]
  | | | `-- [email protected]
  | | `-- [email protected]
  | `-- [email protected]
  +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | `-- [email protected]
  +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | +-- [email protected]
  | | +-- [email protected]
  | | | +-- [email protected]
  | | | +-- [email protected]
  | | | +-- [email protected]
  | | | `-- [email protected]
  | | +-- [email protected]
  | | | `-- [email protected]
  | | +-- [email protected]
  | | | +-- [email protected]
  | | | +-- [email protected]
  | | | | `-- [email protected]
  | | | +-- [email protected]
  | | | `-- [email protected]
  | | `-- [email protected]
  | |   `-- [email protected]
  | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | `-- [email protected]
  | +-- [email protected]
  | | +-- [email protected]
  | | +-- [email protected]
  | | | +-- [email protected]
  | | | +-- [email protected]
  | | | `-- [email protected]
  | | `-- [email protected]
  | |   +-- [email protected]
  | |   +-- [email protected]
  | |   +-- [email protected]
  | |   +-- [email protected]
  | |   | `-- [email protected]
  | |   +-- [email protected]
  | |   +-- [email protected]
  | |   | `-- [email protected]
  | |   +-- [email protected]
  | |   +-- [email protected]
  | |   `-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | `-- [email protected]
  +-- [email protected]
  | `-- [email protected]
  |   +-- [email protected]
  |   +-- [email protected]
  |   +-- [email protected]
  |   | `-- [email protected]
  |   |   +-- [email protected]
  |   |   `-- [email protected]
  |   +-- [email protected]
  |   `-- [email protected]
  +-- [email protected]
  +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | `-- [email protected]
  `-- [email protected]
    +-- [email protected]
    | `-- [email protected]
    +-- [email protected]
    +-- [email protected]
    | `-- [email protected]
    +-- [email protected]
    | +-- [email protected]
    | +-- [email protected]
    | +-- [email protected]
    | +-- [email protected]
    | +-- [email protected]
    | `-- [email protected]
    `-- [email protected]

npm ERR! Windows_NT 10.0.10240
npm ERR! argv "C:\\Users\\Walther\\node\\node.exe" "C:\\Users\\Walther\\node\\node_modules\\npm\\bin\\npm-cli.js" "install" "-g" "sqtest315-cw-1.0.0.tgz"
npm ERR! node v4.4.7
npm ERR! npm  v3.10.8
npm ERR! path C:\Users\Walther\node\node_modules\sqtest315-cw
npm ERR! code EPERM
npm ERR! errno -4048
npm ERR! syscall rename

npm ERR! Error: EPERM: operation not permitted, rename 'C:\Users\Walther\node\node_modules\sqtest315-cw' -> 'C:\Users\Walther\node\node_modules\.sqtest315-cw.DELETE'
npm ERR!     at moveAway (C:\Users\Walther\node\node_modules\npm\lib\install\action\finalize.js:38:5)
npm ERR!     at destStatted (C:\Users\Walther\node\node_modules\npm\lib\install\action\finalize.js:27:7)
npm ERR!     at FSReqWrap.oncomplete (fs.js:82:15)
npm ERR!
npm ERR! Error: EPERM: operation not permitted, rename 'C:\Users\Walther\node\node_modules\sqtest315-cw' -> 'C:\Users\Walther\node\node_modules\.sqtest315-cw.DELETE'
npm ERR!     at Error (native)
npm ERR!  [Error: EPERM: operation not permitted, rename 'C:\Users\Walther\node\node_modules\sqtest315-cw' -> 'C:\Users\Walther\node\node_modules\.sqtest315-cw.DELETE']
npm ERR!
npm ERR! Please try running this command again as root/Administrator.

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\Walther\sqtest315cw\npm-debug.log

C:\Users\Walther\sqtest315cw>