Meteor: [1.4.0.1] node-pre-gyp needed when running `npm install` in bundle/programs/server

[ffxsam]

I'm running into some build errors on Galaxy. So I tried to create the problem on an EC2 instance, and succeeded. The npm install process fails when node npm-rebuild.js is run:

make: Entering directory '/home/ubuntu/bundle/programs/server/npm/node_modules/meteor/npm-bcrypt/node_modules/bcrypt/build'
  CXX(target) Release/obj.target/bcrypt_lib/src/blowfish.o
  CXX(target) Release/obj.target/bcrypt_lib/src/bcrypt.o
  CXX(target) Release/obj.target/bcrypt_lib/src/bcrypt_node.o
  SOLINK_MODULE(target) Release/obj.target/bcrypt_lib.node
  COPY Release/bcrypt_lib.node
make: Leaving directory '/home/ubuntu/bundle/programs/server/npm/node_modules/meteor/npm-bcrypt/node_modules/bcrypt/build'
[email protected] /home/ubuntu/bundle/programs/server/npm/node_modules/meteor/npm-bcrypt/node_modules/bcrypt
[email protected] /home/ubuntu/bundle/programs/server/npm/node_modules/meteor/npm-bcrypt/node_modules/bindings
[email protected] /home/ubuntu/bundle/programs/server/npm/node_modules/meteor/npm-bcrypt/node_modules/nan

> [email protected] install /home/ubuntu/bundle/programs/server/npm/node_modules/fsevents
> node-pre-gyp install --fallback-to-build

sh: 1: node-pre-gyp: not found

npm ERR! Linux 4.4.0-28-generic
npm ERR! argv "/node-v4.4.7-linux-x64/bin/node" "/node-v4.4.7-linux-x64/bin/npm" "rebuild" "--no-bin-links" "--update-binary"
npm ERR! node v4.4.7
npm ERR! npm  v3.10.5
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build`
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the [email protected] install script 'node-pre-gyp install --fallback-to-build'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the fsevents package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-pre-gyp install --fallback-to-build
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs fsevents
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls fsevents
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/ubuntu/bundle/programs/server/npm/npm-debug.log

npm ERR! Linux 4.4.0-28-generic
npm ERR! argv "/node-v4.4.7-linux-x64/bin/node" "/node-v4.4.7-linux-x64/bin/npm" "run" "install"
npm ERR! node v4.4.7
npm ERR! npm  v3.10.5
npm ERR! code ELIFECYCLE
npm ERR! [email protected] install: `node npm-rebuild.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script 'node npm-rebuild.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the meteor-dev-bundle package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node npm-rebuild.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs meteor-dev-bundle
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls meteor-dev-bundle
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/ubuntu/bundle/programs/server/npm-debug.log

[ffxsam]

Downgrading the project to 1.3.5.1 doesn't fix this. I'm theorizing that it's the meteor CLI tool in charge of making builds that might have an issue. @benjamn, any ideas?

[benjamn]

Try running npm install node-pre-gyp before running npm install?

[benjamn]

Actually it looks like fsevents depends on node-pre-gyp, so you can probably run METEOR_NPM_REBUILD_FLAGS=" " meteor deploy yourhostname.com to work around this problem on Galaxy.

Explanation: the npm-rebuild.js script runs npm rebuild --no-bin-links by default, but the --no-bin-links prevents node-pre-gyp from getting linked into bundle/programs/server/node_modules/.bin, where it would be visible to npm.

[ffxsam]

Cool, trying the METEOR_NPM_REBUILD_FLAGS=" " workaround now.

[ffxsam]

Same error. I ran this locally, is this incorrect?

METEOR_NPM_REBUILD_FLAGS=" " DEPLOY_HOSTNAME=galaxy.meteor.com meteor deploy --settings settings.staging.json staging.xxx.com

Or does "METEOR_NPM_REBUILD_FLAGS": "" belong in settings.json in the galaxy.meteor.com section??

[ffxsam]

Have you got another workaround for me? And will this be fixed in 1.4.0.2?

[abernix]

I believe METEOR_NPM_REBUILD_FLAGS should be in settings.json + galaxy.meteor.com in the same place that you would set MONGO_URL, etc. and not just run locally...

Per this documentation: http://galaxy-guide.meteor.com/environment-variables.html

Also, do note the space in the variable value! " "

[ffxsam]

@abernix sadly, this did not work:

2016-08-04 16:10:32-07:00> node-gyp rebuild
v040
2016-08-04 16:10:32-07:00
v040
2016-08-04 16:10:35-07:00make: Entering directory `/app/bundle/programs/server/npm/node_modules/meteor/npm-bcrypt/node_modules/bcrypt/build'
v040
2016-08-04 16:10:35-07:00 CXX(target) Release/obj.target/bcrypt_lib/src/blowfish.o
v040
2016-08-04 16:10:36-07:00 CXX(target) Release/obj.target/bcrypt_lib/src/bcrypt.o
v040
2016-08-04 16:10:36-07:00 CXX(target) Release/obj.target/bcrypt_lib/src/bcrypt_node.o
v040
2016-08-04 16:10:40-07:00 SOLINK_MODULE(target) Release/obj.target/bcrypt_lib.node
v040
2016-08-04 16:10:40-07:00 COPY Release/bcrypt_lib.node
v040
2016-08-04 16:10:40-07:00make: Leaving directory `/app/bundle/programs/server/npm/node_modules/meteor/npm-bcrypt/node_modules/bcrypt/build'
v040
2016-08-04 16:10:40-07:[email protected] /app/bundle/programs/server/npm/node_modules/meteor/npm-bcrypt/node_modules/bcrypt
v040
2016-08-04 16:10:40-07:[email protected] /app/bundle/programs/server/npm/node_modules/meteor/npm-bcrypt/node_modules/bindings
v040
2016-08-04 16:10:40-07:[email protected] /app/bundle/programs/server/npm/node_modules/meteor/npm-bcrypt/node_modules/nan
v040
2016-08-04 16:10:59-07:00
v040
2016-08-04 16:10:59-07:00
v040
2016-08-04 16:10:59-07:00> [email protected] install /app/bundle/programs/server/npm/node_modules/fsevents
v040
2016-08-04 16:10:59-07:00> node-pre-gyp install --fallback-to-build
v040
2016-08-04 16:10:59-07:00sh: 1: node-pre-gyp: not found
v040
2016-08-04 16:10:59-07:00
v040
2016-08-04 16:10:59-07:00npm ERR! Linux 4.4.0-28-generic
v040
2016-08-04 16:10:59-07:00npm ERR! code ELIFECYCLE
v040
2016-08-04 16:10:59-07:00npm ERR! argv "/node-v4.4.7-linux-x64/bin/node" "/node-v4.4.7-linux-x64/bin/npm" "rebuild" "--no-bin-links" "--update-binary"
v040
2016-08-04 16:10:59-07:00npm ERR! node v4.4.7
v040
2016-08-04 16:10:59-07:00npm ERR! npm v3.10.5

[ffxsam]

I see the first RC for 1.4.0.2 is out. If I upgrade my project, will this help me?

EDIT: Nope. 😢

[ffxsam]

This seems like a simple path issue. node-pre-gyp exists, but it's in node_modules/fsevents/node_modules/.bin.

[abernix]

This hasn't made it into any RC yet. I'd expect it to make it into the 1.4.1 release candidate though. Follow #7580 for its progress.

[ffxsam]

I wish I could wait that long. Is there really no workaround for this??

[abernix]

I'm not sure about Galaxy, but have you tried npm install --global node-pre-gyp on your Ubuntu box? Are you sure the errors you're getting are _exactly_ the same on Galaxy and Ubuntu?

Are you using fsevents directly yourself? Or is something else using it? Frankly, I'm confused why you're having this problem at all. I'm having no problems on Debian (jessie and wheezy) or Ubuntu (14.04 or 16.04) with binary dep compilation, but maybe I'm missing something for particular packages.

Can you set the environment variable V to 1 (i.e. export V=1 on Ubuntu) before you run the npm install command, and instead of npm install use npm install --verbose – it should provide a lot more output.

[ffxsam]

I'm not sure about Galaxy, but have you tried npm install --global node-pre-gyp on your Ubuntu box?

The thing is, Galaxy is literally all I care about in this case, because it's our chosen platform for hosting apps.

Are you using fsevents directly yourself? Or is something else using it? Frankly, I'm confused why you're having this problem at all.

So am I. 😉 No, not using fsevents directly. But it's being depended upon in my project:

│ │ ├─┬ [email protected]
│ │ │ ├── [email protected]
│ │ │ └─┬ [email protected]
│ │ │   ├── [email protected]
│ │ │   ├── [email protected]
│ │ │   ├─┬ [email protected]

Following this up leads to webpack which is being included by @kadira/storybook, and...

uh oh. There it is. @kadira/storybook is mistakenly in the dependencies section of package.json. I'm gonna fix that and see if a deploy works.

[ffxsam]

@abernix You, sir, are a hero. I would've never thought to trace down the npm dependencies, I kept looking in the wrong places. It's fixed!!

@benjamn I'm wondering if it makes sense to have Galaxy catch "user stupidity" errors? 😉 Like if certain npm packages are trying to be installed that don't make sense (like fsevents, which I had no idea what it was), it could throw a warning to double check package.json or something.

[benjamn]

I'm glad this ended up having a straightforward diagnosis and solution, even if it was tricky to track down. Closing now, but feel free to reopen if the issue occurs again in a more problematic form.

[vsivsi]

@benjamn If I've read this thread correctly, it looks as though this issue was closed because the OP managed to avoid the node-pre-gyp issue he was seeing by removing an unnecessary dependency that was triggering it, not because the issue was actually resolved at the time this issue was closed.

I've also noticed that you followed up by committing (and partially reversing) some changes to fix Galaxy node-pre-gyp related problems:
https://github.com/meteor/meteor/commit/a617ddaec288be4c77a3ff753b1e3f4dc2d58e2d
https://github.com/meteor/meteor/commit/b2e2d0651be672a4be085cf705d0eb6261650deb

These commits appear to have been released in Meteor 1.4.1:
https://github.com/meteor/meteor/pull/7580

Okay, with that history recapped, I'm either experiencing a regression of this issue using Meteor 1.4.2.3, or perhaps the problem was never really fixed. I'm unsure on which is the case.

I'm the author of the (relatively) popular job-collection Atmosphere package. I also maintain a simple "playground" application demonstrating this package, and MDG/Galaxy graciously provides hosting for a live version of it here: https://jcplayground.meteorapp.com/

It is this playground app that I can use to simply demonstrate the issue that I am experiencing.

If you just deploy this app to Galaxy from master, the following warning crops up in the Galaxy container log:

2016-12-20 19:08:42-08:00Note: you are using a pure-JavaScript implementation of bcrypt.
2016-12-20 19:08:42-08:00While this implementation will work correctly, it is known to be
2016-12-20 19:08:42-08:00approximately three times slower than the native implementation.
2016-12-20 19:08:42-08:00In order to use the native implementation instead, run
2016-12-20 19:08:42-08:00
2016-12-20 19:08:42-08:00 meteor npm install --save bcrypt
2016-12-20 19:08:42-08:00
2016-12-20 19:08:42-08:00in the root directory of your application.

I see the same warning on my MacOS dev box. If I follow the meteor npm install --save bcrypt install instructions on my dev machine, everything works great. But when I deploy it to Galaxy, the node-pre-gyp resolution issue breaks the container build. You can find the version that does this on the npm_bcrypt_test branch of the playground app repo.

I want to believe that I'm just missing something brain dead simple here, because this bcrypt binary npm package scenario must be incredibly common, and yet none of the information in this issue thread, nor the Galaxy docs, nor searching around for StackOverflow questions, etc has been any help. The failing container build log is below. Any thoughts on how to resolve this would be much appreciated, since my ability to debug a Galaxy container build process that I can't log into or accurately replicate in some other way limits my ability to make any further progress here on my own.

2016-12-21 14:48:47-08:00Building version 17.

[AWS stuff redacted]

2016-12-21 14:48:47-08:00 ---> [Warning] Your kernel does not support swap limit capabilities, memory limited without swap.
2016-12-21 14:48:47-08:00 ---> Running in 1fdc5ca07474
2016-12-21 14:48:47-08:00+ set -ex

[more AWS stuff redacted]

2016-12-21 14:48:48-08:00+ cd /app/bundle
2016-12-21 14:48:48-08:00+ export NODE_VERSION
2016-12-21 14:48:48-08:00++ /app/select_node_version.sh
2016-12-21 14:48:48-08:00+ NODE_VERSION=4.6.2
2016-12-21 14:48:48-08:00+ /app/install_node.sh
2016-12-21 14:48:48-08:00+ export PATH=/node-v4.6.2-linux-x64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2016-12-21 14:48:48-08:00+ PATH=/node-v4.6.2-linux-x64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2016-12-21 14:48:48-08:00+ npm -g install npm@latest
2016-12-21 14:48:48-08:00node version 4.6.2 already installed.
2016-12-21 14:48:55-08:00/node-v4.6.2-linux-x64/bin/npm -> /node-v4.6.2-linux-x64/lib/node_modules/npm/bin/npm-cli.js
2016-12-21 14:48:56-08:[email protected] /node-v4.6.2-linux-x64/lib/node_modules/npm
2016-12-21 14:48:56-08:00+ pushd programs/server
2016-12-21 14:48:56-08:00/app/bundle/programs/server /app/bundle
2016-12-21 14:48:56-08:00+ su meteor -c 'PATH='\''/node-v4.6.2-linux-x64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'\'' npm install'
2016-12-21 14:48:59-08:00
2016-12-21 14:48:59-08:00> [email protected] install /app/bundle/programs/server/node_modules/fibers
2016-12-21 14:48:59-08:00> node build.js || nodejs build.js
2016-12-21 14:48:59-08:00
2016-12-21 14:48:59-08:00`linux-x64-46` exists; testing
2016-12-21 14:48:59-08:00Binary is fine; exiting
2016-12-21 14:48:59-08:00
2016-12-21 14:48:59-08:00> [email protected] install /app/bundle/programs/server
2016-12-21 14:48:59-08:00> node npm-rebuild.js
2016-12-21 14:48:59-08:00
2016-12-21 14:49:00-08:00
2016-12-21 14:49:00-08:00> [email protected] install /app/bundle/programs/server/npm/node_modules/bcrypt
2016-12-21 14:49:00-08:00> node-pre-gyp install --fallback-to-build
2016-12-21 14:49:00-08:00
2016-12-21 14:49:00-08:00sh: 1: node-pre-gyp: not found
2016-12-21 14:49:00-08:00
2016-12-21 14:49:00-08:00npm ERR! Linux 4.4.0-45-generic
2016-12-21 14:49:00-08:00npm ERR! argv "/node-v4.6.2-linux-x64/bin/node" "/node-v4.6.2-linux-x64/bin/npm" "rebuild" "--update-binary"
2016-12-21 14:49:00-08:00npm ERR! node v4.6.2
2016-12-21 14:49:00-08:00npm ERR! npm v4.0.5
2016-12-21 14:49:00-08:00npm ERR! file sh
2016-12-21 14:49:00-08:00npm ERR! code ELIFECYCLE
2016-12-21 14:49:00-08:00npm ERR! errno ENOENT
2016-12-21 14:49:00-08:00npm ERR! syscall spawn
2016-12-21 14:49:00-08:00npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build`
2016-12-21 14:49:00-08:00npm ERR! spawn ENOENT
2016-12-21 14:49:00-08:00npm ERR!
2016-12-21 14:49:00-08:00npm ERR! Failed at the [email protected] install script 'node-pre-gyp install --fallback-to-build'.
2016-12-21 14:49:00-08:00npm ERR! Make sure you have the latest version of node.js and npm installed.
2016-12-21 14:49:00-08:00npm ERR! If you do, this is most likely a problem with the bcrypt package,
2016-12-21 14:49:00-08:00npm ERR! not with npm itself.
2016-12-21 14:49:00-08:00npm ERR! Tell the author that this fails on your system:
2016-12-21 14:49:00-08:00npm ERR! node-pre-gyp install --fallback-to-build
2016-12-21 14:49:00-08:00npm ERR! You can get information on how to open an issue for this project with:
2016-12-21 14:49:00-08:00npm ERR! npm bugs bcrypt
2016-12-21 14:49:00-08:00npm ERR! Or if that isn't available, you can get their info via:
2016-12-21 14:49:00-08:00npm ERR! npm owner ls bcrypt
2016-12-21 14:49:00-08:00npm ERR! There is likely additional logging output above.
2016-12-21 14:49:00-08:00
2016-12-21 14:49:00-08:00npm ERR! Please include the following file with any support request:
2016-12-21 14:49:00-08:00npm ERR! /app/bundle/programs/server/npm/npm-debug.log
2016-12-21 14:49:00-08:00
2016-12-21 14:49:00-08:00npm WARN [email protected] No description
2016-12-21 14:49:00-08:00npm WARN [email protected] No repository field.
2016-12-21 14:49:00-08:00npm WARN [email protected] No license field.
2016-12-21 14:49:00-08:00npm ERR! Linux 4.4.0-45-generic
2016-12-21 14:49:00-08:00npm ERR! argv "/node-v4.6.2-linux-x64/bin/node" "/node-v4.6.2-linux-x64/bin/npm" "install"
2016-12-21 14:49:00-08:00npm ERR! node v4.6.2
2016-12-21 14:49:00-08:00npm ERR! npm v4.0.5
2016-12-21 14:49:00-08:00npm ERR! code ELIFECYCLE
2016-12-21 14:49:00-08:00npm ERR! [email protected] install: `node npm-rebuild.js`
2016-12-21 14:49:00-08:00npm ERR! Exit status 1
2016-12-21 14:49:00-08:00npm ERR!
2016-12-21 14:49:00-08:00npm ERR! Failed at the [email protected] install script 'node npm-rebuild.js'.
2016-12-21 14:49:00-08:00npm ERR! Make sure you have the latest version of node.js and npm installed.
2016-12-21 14:49:00-08:00npm ERR! If you do, this is most likely a problem with the meteor-dev-bundle package,
2016-12-21 14:49:00-08:00npm ERR! not with npm itself.
2016-12-21 14:49:00-08:00npm ERR! Tell the author that this fails on your system:
2016-12-21 14:49:00-08:00npm ERR! node npm-rebuild.js
2016-12-21 14:49:00-08:00npm ERR! You can get information on how to open an issue for this project with:
2016-12-21 14:49:00-08:00npm ERR! npm bugs meteor-dev-bundle
2016-12-21 14:49:00-08:00npm ERR! Or if that isn't available, you can get their info via:
2016-12-21 14:49:00-08:00npm ERR! npm owner ls meteor-dev-bundle
2016-12-21 14:49:00-08:00npm ERR! There is likely additional logging output above.
2016-12-21 14:49:00-08:00
2016-12-21 14:49:00-08:00npm ERR! Please include the following file with any support request:
2016-12-21 14:49:00-08:00npm ERR! /app/bundle/programs/server/npm-debug.log
2016-12-21 14:49:01-08:00Removing intermediate container 1fdc5ca07474

[dschnelldavis]

I had almost the exact same problem, in that my app worked fine locally, but every time I tried to deploy it to Galaxy the deployment itself would appear to work, but then the build on Galaxy would fail with an error message nearly identical to the one posted above by vsivsi.

However, after beating my head against it for a couple hours, I stumbled across an unexpected solution. The problem turned out to be the "bcrypt": "^1.0.1", line in my package.json file (which was added after I followed Meteor's recommendation to run meteor npm install --save bcrypt). But to my surprise, when I moved that line from the "dependencies" section of package.json to "devDependencies", suddenly I could successfully deploy to Galaxy again.

Of course, Galaxy now complains it is "using a pure-JavaScript implementation of bcrypt" which is 3x slower—but I don't care about that, I just want my app to work, which it now does.

So for other developers having trouble deploying to Galaxy after installing bcrypt—try editing your package.json to move bcrypt from "dependencies" to "devDependencies", where it looks like Galaxy will ignore it, but at least it won't break your app.

And for the Meteor developers, if this is a recurring problem (which it appears to be), would a short-term solution be to change the recommendation message to run meteor npm install --save-dev bcrypt instead, which would put bcrypt in "devDependencies" to start with, where it won't prevent Galaxy deployments?

[igal1c0de4n]

not sure if this will be of assistance:
I've bumped into this error on a docker deployment. Eventually I managed to solve/workaround it by adding to my Dockerfile:
RUN npm install -g node-gyp node-pre-gyp

[abernix]

@vsivsi Thanks for the thorough update, but as this issue is closed and not necessarily an identical issue (similar, perhaps!). I'd recommend opening a new issue, but it appears someone else has opened #8188 earlier today so I might recommend following that for updates. Based on the +1's on your comment, this does seem interesting, but I don't want it getting lost on this old issue.

@dschnelldavis While that suggestion would work, you could also just remove the bcrypt dependency entirely – though this would have the side-effect of the bcrypt warning showing in your development . In case it's not clear, the reason this "fixes" it is because anything listed in devDependencies is not bundled or included in your production (e.g. Galaxy) build. Since it's not bundled, it's not attempted to be rebuilt. If you don't care about using the faster bcrypt implementation (it's entirely feasible that you're not), you don't need it listed in your package.json at all.

[vsivsi]

@abernix I have added my reproduction instructions above to the bottom of https://github.com/meteor/meteor/issues/8188

However, in the meantime, the OP of that issue has closed it without any explanation. Should I open yet another issue for this?!?

[glasser]

See #8188: your new issue is probably #8136 which is fixed on master and has a workaround.

[vsivsi]

@glasser I have confirmed that 1.4.3-beta.1 solves the issue I reported above. Thanks for the pointer.

[isathish]

Hi all,
following this commend working fine...:)
#npm install -g node-gyp node-pre-gyp or $sudo npm install -g node-gyp node-pre-gyp

[EKoetsjarjan]

@benjamn npm install node-pre-gyp
worked for me

[hatim-heffoudhi]

npm install node-pre-gyp worked for me too