Yarn: [0.17] Regression: Native modules (node-gyp) not built correctly after using "yarn add"

[gitawego]

Do you want to request a feature or report a bug?

bug
What is the current behavior?
failed to build bcrypt correctly. it gives error:

Error: Could not locate the bindings file. Tried:
 → path/to/node_modules/bcrypt/build/bcrypt_lib.node
 →path/to/node_modules/bcrypt/build/Debug/bcrypt_lib.node
 → path/to/node_modules/bcrypt/build/Release/bcrypt_lib.node
...

If the current behavior is a bug, please provide the steps to reproduce.
just install bcrypt
What is the expected behavior?
it should build correctly bcrypt. it works with npm. after npm rebuild, it works normally.
Please mention your node.js, yarn and operating system version.
node 6.8.0 yarn: 0.17.0 ubuntu 16.10

[bestander]

Can't reproduce it in master

 ~/work/yarn/artifacts/dist/bin/yarn
yarn install v0.18.0-0
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 3.66s.
bestander-mbp:yarn-init bestander$ cat package.json
{
  "name": "yarn-init",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "bcrypt": "^0.8.7"
  }
}

@gitawego, is it fixed now?

[holm]

I got similar on Yarn 0.17.0 installing libpq:

>> Error: Could not locate the bindings file. Tried:
>>  → path/to/node_modules/libpq/build/addon.node
>>  → path/to/node_modules/libpq/build/Debug/addon.node
>>  → path/to/node_modules/libpq/build/Release/addon.node
>>  → path/to/node_modules/libpq/out/Debug/addon.node
>>  → path/to/node_modules/libpq/Debug/addon.node
>>  → path/to/node_modules/libpq/out/Release/addon.node
>>  → path/to/node_modules/libpq/Release/addon.node
>>  → path/to/node_modules/libpq/build/default/addon.node
>>  → path/to/node_modules/libpq/compiled/6.6.0/darwin/x64/addon.node

[adamreisnz]

I am getting the same with Yarn 0.17.2, had to run npm rebuild to fix it.

node_modules/bindings/bindings.js:91
  throw err
  ^

Error: Could not locate the bindings file. 

[adamreisnz]

@bestander it's worth noting that this error only occurs at runtime, not at yarn install time. The yarn install works fine, just when you start your app/script it throws that error when you require bcrypt.

[daltones]

Same here installing bcrypt after upgrade Yarn from v0.16.1 to v0.17.2. yarn install finishes without errors, but the installation is broken.

[bestander]

Let's reopen this.
Please provide the minimum e2e example that reproduces the issue

[adamreisnz]

I've tried to create an isolated example, but have not succeeded on my laptop, which runs OS X El Capitan with yarn 0.17.2. The issue I had yesterday was on a different version I believe, will have to recheck later today.

[daltones]

package.json:

{
  "name": "test",
  "dependencies": {
    "bcrypt": "^0.8.7"
  }
}

index.js:

require('bcrypt');

After yarn install, run node index.js:

[...]/node_modules/bindings/bindings.js:91
  throw err
  ^

Error: Could not locate the bindings file. Tried:
 → [...]/node_modules/bcrypt/build/bcrypt_lib.node
 → [...]/node_modules/bcrypt/build/Debug/bcrypt_lib.node
 → [...]/node_modules/bcrypt/build/Release/bcrypt_lib.node
 → [...]/node_modules/bcrypt/out/Debug/bcrypt_lib.node
 → [...]/node_modules/bcrypt/Debug/bcrypt_lib.node
 → [...]/node_modules/bcrypt/out/Release/bcrypt_lib.node
 → [...]/node_modules/bcrypt/Release/bcrypt_lib.node
 → [...]/node_modules/bcrypt/build/default/bcrypt_lib.node
 → [...]/node_modules/bcrypt/compiled/7.1.0/linux/x64/bcrypt_lib.node
    at bindings ([...]/node_modules/bindings/bindings.js:88:9)
    at Object.<anonymous> ([...]/node_modules/bcrypt/bcrypt.js:3:35)
    at Module._compile (module.js:573:32)
    at Object.Module._extensions..js (module.js:582:10)
    at Module.load (module.js:490:32)
    at tryModuleLoad (module.js:449:12)
    at Function.Module._load (module.js:441:3)
    at Module.require (module.js:500:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> ([...]/index.js:1:63)

The whole directory node_modules/bcrypt/build/ isn't there.

node.js: v7.1.0
yarn: v0.17.2
OS: Arch Linux

[adamreisnz]

Yes, I was going to suggest the same reproduction steps, but it seems to work on my current device. Will try again on the problematic device and try to provide additional details.

[bestander]

Is it the same as #1852, i.e. install script not running on install?

[daltones]

@bestander sort of… except that bcrypt v0.8.7 doesn't have an explicit script install in package.json like the case in #1852. But it does some gyp stuff that I actually don't understand how it works.

[markstos]

We build bcrypt with yarn on our CI server frequently-- currently with 0.16.1-- and see intermittent failures. That could explain why some people see the issue here and some don't. I currently have no idea what triggers the failure, but will be looking at it some today, as I'm trying to run yarn in a Ubuntu 14.04 Docker container, and am getting bcrypt-related failures every time there, with bcrypt 0.8.7.

[markstos]

@gitawego I see the error _Could not locate the bindings file._ at _run time_. Do you see any errors at _build time_, when running yarn add [email protected]? There are a lot of dependencies for getting the binary bits built which could fail if expected packages haven't been installed in the OS. This is definitely an issue inside stripped-down container OS environments, but could also be an issue on new OS installs that don't have developer tools installs. bcrypt required node-gyp, python, make, `g++`` and possible more to be installed. The _build_ output should have details about the build failure, if any. While the * Could not locate the bindings file.* is a run-time error simply telling you the build failed.

[bestander]

Can anyone confirm if this is a regression from 0.16.1?

[hpurmann]

@markstos @bestander We run 0.16.1 locally in Docker and get the "Could not locate the bindings file" from time to time as well. Didn't test 0.17 though. Sometimes it happens right after a git pull of a changed yarn.lock, sometimes after docker-compose run --rm --entrypoint yarn <container> install.

Unfortunately, we cannot reproduce the cause. I suspect it to be yarn's cache. Maybe that's also tied to the inconsistent re-builds of native modules.

[markstos]

@hpurmann I also suspect there's an issue with the cache, but can't pin it down. Today I had luck replacing a yarn install with yarn install --force in a Docker container. I think the issue was that an earlier bcrypt build had failed because some of the binaries it required had been installed in the container yet. However, I think the build was still considered "successful" by yarn and was cached anyway. By fixing the binaries it depends on and using --force, I fixed the underlying issue and busted the cache. I think I'm I'm just going to use --force all the time in the CI server to maximize build reliability.

[markstos]

I had my last bcrypt failure today. I was also getting "Could not locate bindings file" error. It didn't seem to matter if I used yarn 0.16.x, or 0.17.x, whether I used add --force or install --force or yarn cache clean. It took about 5 minutes to replace my require calls with bcryptjs instead This native-JavaScript alternative avoids all the problems related to compiling native bindings that bcrypt has. Sure, this part of my code will be slightly slower, but now the team can get on with writing, testing and deploying our app again reliably!

[adamreisnz]

@markstos "slightly" slower is actually about 3-4 times slower, as I have tested several weeks ago. I would recommend that instead of switching to bryptjs, you instead switch from yarn back to npm. The issue here is not with bcrypt, but with yarn.

It just seems yarn is not mature and stable enough to be a complete drop-in replacement for npm yet. Hopefully this and other issues that prevent that will be addressed and fixed soon though, as I do love the speed of yarn.

[Daniel15]

I can repro on a fresh Ubuntu 16.04 Docker container

root@vps03:/# docker run -it ubuntu:16.04 /bin/bash
root@717f26973255:/# echo 'Acquire::http::proxy "http://192.168.122.1:3142";' > /etc/apt/apt.conf.d/02proxy
root@717f26973255:/# apt-key adv --fetch-keys http://dl.yarnpkg.com/debian/pubkey.gpg
Executing: /tmp/tmp.5jMC1GGtME/gpg.1.sh --fetch-keys
http://dl.yarnpkg.com/debian/pubkey.gpg
gpg: key 86E50310: public key "Yarn Packaging <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
root@717f26973255:/# echo "deb http://nightly.yarnpkg.com/debian/ nightly main" > /etc/apt/sources.list.d/yarn.list
root@717f26973255:/# apt-get update -y
Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [95.7 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial/main Sources [1103 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial/restricted Sources [5179 B]
Get:6 http://archive.ubuntu.com/ubuntu xenial/universe Sources [9802 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]
Get:9 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9827 kB]
Get:10 http://nightly.yarnpkg.com/debian nightly InRelease [5237 B]
Get:11 http://archive.ubuntu.com/ubuntu xenial-updates/main Sources [256 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/restricted Sources [1872 B]
Get:13 http://archive.ubuntu.com/ubuntu xenial-updates/universe Sources [134 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [532 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [11.7 kB]
Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [448 kB]
Get:17 http://archive.ubuntu.com/ubuntu xenial-security/main Sources [54.7 kB]
Get:18 http://archive.ubuntu.com/ubuntu xenial-security/restricted Sources [1872 B]
Get:19 http://archive.ubuntu.com/ubuntu xenial-security/universe Sources [13.8 kB]
Get:20 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 Packages [200 kB]
Get:21 http://archive.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [11.7 kB]
Get:22 http://archive.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [63.2 kB]
Get:23 http://nightly.yarnpkg.com/debian nightly/main amd64 Packages [12.6 kB]
Fetched 24.5 MB in 3s (7742 kB/s)
Reading package lists... Done

root@717f26973255:/# DEBIAN_FRONTEND=noninteractive apt-get install yarn ca-certificates nodejs-legacy -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libicu55 libssl1.0.0 libuv1 nodejs openssl
The following NEW packages will be installed:
  ca-certificates libicu55 libssl1.0.0 libuv1 nodejs nodejs-legacy openssl yarn
0 upgraded, 8 newly installed, 0 to remove and 11 not upgraded.
Need to get 14.5 MB of archives.
After this operation, 77.3 MB of additional disk space will be used.
Get:1 http://nightly.yarnpkg.com/debian nightly/main amd64 yarn all 0.18.0-20161117.1624-1 [1815 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libssl1.0.0 amd64 1.0.2g-1ubuntu4.5 [1082 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 openssl amd64 1.0.2g-1ubuntu4.5 [491 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial/main amd64 ca-certificates all 20160104ubuntu1 [191 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial/main amd64 libicu55 amd64 55.1-7 [7643 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libuv1 amd64 1.8.0-1 [57.4 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 nodejs amd64 4.2.6~dfsg-1ubuntu4.1 [3161 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 nodejs-legacy all 4.2.6~dfsg-1ubuntu4.1 [27.7 kB]
Fetched 14.5 MB in 0s (57.7 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libssl1.0.0:amd64.
(Reading database ... 7256 files and directories currently installed.)
Preparing to unpack .../libssl1.0.0_1.0.2g-1ubuntu4.5_amd64.deb ...
Unpacking libssl1.0.0:amd64 (1.0.2g-1ubuntu4.5) ...
Selecting previously unselected package openssl.
Preparing to unpack .../openssl_1.0.2g-1ubuntu4.5_amd64.deb ...
Unpacking openssl (1.0.2g-1ubuntu4.5) ...
Selecting previously unselected package ca-certificates.
Preparing to unpack .../ca-certificates_20160104ubuntu1_all.deb ...
Unpacking ca-certificates (20160104ubuntu1) ...
Selecting previously unselected package libicu55:amd64.
Preparing to unpack .../libicu55_55.1-7_amd64.deb ...
Unpacking libicu55:amd64 (55.1-7) ...
Selecting previously unselected package libuv1:amd64.
Preparing to unpack .../libuv1_1.8.0-1_amd64.deb ...
Unpacking libuv1:amd64 (1.8.0-1) ...
Selecting previously unselected package yarn.
Preparing to unpack .../yarn_0.18.0-20161117.1624-1_all.deb ...
Unpacking yarn (0.18.0-20161117.1624-1) ...
Selecting previously unselected package nodejs.
Preparing to unpack .../nodejs_4.2.6~dfsg-1ubuntu4.1_amd64.deb ...
Unpacking nodejs (4.2.6~dfsg-1ubuntu4.1) ...
Selecting previously unselected package nodejs-legacy.
Preparing to unpack .../nodejs-legacy_4.2.6~dfsg-1ubuntu4.1_all.deb ...
Unpacking nodejs-legacy (4.2.6~dfsg-1ubuntu4.1) ...
Processing triggers for libc-bin (2.23-0ubuntu3) ...
Setting up libssl1.0.0:amd64 (1.0.2g-1ubuntu4.5) ...
Setting up openssl (1.0.2g-1ubuntu4.5) ...
Setting up ca-certificates (20160104ubuntu1) ...
Setting up libicu55:amd64 (55.1-7) ...
Setting up libuv1:amd64 (1.8.0-1) ...
Setting up yarn (0.18.0-20161117.1624-1) ...
Setting up nodejs (4.2.6~dfsg-1ubuntu4.1) ...
update-alternatives: using /usr/bin/nodejs to provide /usr/bin/js (js) in auto mode
Setting up nodejs-legacy (4.2.6~dfsg-1ubuntu4.1) ...
Processing triggers for libc-bin (2.23-0ubuntu3) ...
Processing triggers for ca-certificates (20160104ubuntu1) ...
Updating certificates in /etc/ssl/certs...
173 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

root@717f26973255:/# cd /tmp
root@717f26973255:/tmp# mkdir yarntest
root@717f26973255:/tmp# cd yarntest
root@717f26973255:/tmp/yarntest# yarn add bcrypt
yarn add v0.18.0-20161117.1624
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 3 new dependencies.
├─ [email protected]
├─ [email protected]
└─ [email protected]
Done in 0.71s.
root@717f26973255:/tmp/yarntest# echo "require('bcrypt')" > test.js
root@717f26973255:/tmp/yarntest# nodejs test.js
/tmp/yarntest/node_modules/bindings/bindings.js:91
  throw err
  ^

Error: Could not locate the bindings file. Tried:
 → /tmp/yarntest/node_modules/bcrypt/build/bcrypt_lib.node
 → /tmp/yarntest/node_modules/bcrypt/build/Debug/bcrypt_lib.node
 → /tmp/yarntest/node_modules/bcrypt/build/Release/bcrypt_lib.node
 → /tmp/yarntest/node_modules/bcrypt/out/Debug/bcrypt_lib.node
 → /tmp/yarntest/node_modules/bcrypt/Debug/bcrypt_lib.node
 → /tmp/yarntest/node_modules/bcrypt/out/Release/bcrypt_lib.node
 → /tmp/yarntest/node_modules/bcrypt/Release/bcrypt_lib.node
 → /tmp/yarntest/node_modules/bcrypt/build/default/bcrypt_lib.node
 → /tmp/yarntest/node_modules/bcrypt/compiled/4.2.6/linux/x64/bcrypt_lib.node
    at bindings (/tmp/yarntest/node_modules/bindings/bindings.js:88:9)
    at Object.<anonymous> (/tmp/yarntest/node_modules/bcrypt/bcrypt.js:3:35)
    at Module._compile (module.js:410:26)
    at Object.Module._extensions..js (module.js:417:10)
    at Module.load (module.js:344:32)
    at Function.Module._load (module.js:301:12)
    at Module.require (module.js:354:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/tmp/yarntest/test.js:1:63)
    at Module._compile (module.js:410:26)

[Daniel15]

@bestander I confirmed that this is a regression from 0.16.1. It looks like bcrypt builds a native module after installation. This build runs in 0.16.1, but it's not even attempting to build it in 0.17.0, resulting in the error message. Perhaps it is related to #1852.

[wmertens]

_workaround_: run npm rebuild after any yarn install steps.

I have the same problem with 0.17.4 on El Capitan, even after clearing all caches (both the Library/Yarn one and .yarn-cache from the previous versions)

[markstos]

@wmertens, yarn install --force is supposed to be equivalent to npm rebuild, and it didn't always work for me to fix the bcrypt problem (but sometimes it did). https://yarnpkg.com/en/docs/migrating-from-npm#toc-cli-commands-comparison.

[bestander]

A possible fix https://github.com/yarnpkg/yarn/pull/1935

[gitawego]

it doesn't work neither with --force flag. hope the #1935 could fix the prob.

[bestander]

I'll push 0.17.5 in a couple of hours

On 18 November 2016 at 18:30, Hongbo LU [email protected] wrote:

it doesn't work neither with --force flag. hope the #1935
https://github.com/yarnpkg/yarn/pull/1935 could fix the prob.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/yarnpkg/yarn/issues/1848#issuecomment-261605400, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACBdWMhW0C0m3USHBenyx-q-juUAdmO-ks5q_e7KgaJpZM4KygTW
.

[bestander]

Should be fixed with https://github.com/yarnpkg/yarn/commit/3f765c354306bd8e6ca22b3bb333542d1a8fd6e2 in 0.17.5, please report if not

[hph]

@bestander This is unfortunately not fixed in 0.17.5. I still have to run npm rebuild after adding a package with yarn.

[bestander]

@hph, can you help me to sort it out then?
Could you open a separate issue with your exact case + all the data regarding what is missing.

[daltones]

I confirmed that v0.17.5 worked here (node v7.1.0, Arch Linux).

I also reverted to v0.17.3, did the same process, upgraded back to v0.17.5 and confirmed that it works now.

[holm]

I can also confirm that it works for me. I upgraded to 0.17.5, did a yarn install --force to get the bad cache sorted, and it works.

[bestander]

Ok, let's close this issue and open a new one for 0.17.5

[hph]

@bestander Sorry for the confusion, I can confirm that it does indeed work (on Mac, latest Node.js, latest Yarn). Thanks for addressing this so quickly and for the prompt release!

[adamreisnz]

No issues anymore building bcrypt with Yarn 0.17.6. Thanks for the quick response to this.

[eessex]

I'm seeing this issue come up in Yarn v0.20.0,
Locally and in production yarn misses the bcrpyt dependencies:

Feb 16 14:23:21 positron-staging app/web.1: Uncaught Exception Error: Could not locate the bindings file. Tried:
Feb 16 14:23:21 positron-staging app/web.1: → /app/node_modules/bcrypt/build/bcrypt_lib.node
Feb 16 14:23:21 positron-staging app/web.1: → /app/node_modules/bcrypt/build/Debug/bcrypt_lib.node
Feb 16 14:23:21 positron-staging app/web.1: → /app/node_modules/bcrypt/build/Release/bcrypt_lib.node
Feb 16 14:23:21 positron-staging app/web.1: → /app/node_modules/bcrypt/out/Debug/bcrypt_lib.node
Feb 16 14:23:21 positron-staging app/web.1: → /app/node_modules/bcrypt/Debug/bcrypt_lib.node
Feb 16 14:23:21 positron-staging app/web.1: → /app/node_modules/bcrypt/out/Release/bcrypt_lib.node
Feb 16 14:23:21 positron-staging app/web.1: → /app/node_modules/bcrypt/Release/bcrypt_lib.node
Feb 16 14:23:21 positron-staging app/web.1: → /app/node_modules/bcrypt/build/default/bcrypt_lib.node
Feb 16 14:23:21 positron-staging app/web.1: → /app/node_modules/bcrypt/compiled/7.5.0/linux/x64/bcrypt_lib.node
Feb 16 14:23:21 positron-staging app/web.1: at bindings (/app/node_modules/bindings/bindings.js:88:9)
Feb 16 14:23:21 positron-staging app/web.1: at Object. (/app/node_modules/bcrypt/bcrypt.js:3:35)

[markstos]

@eessex If you never want to deal with this issue again, switching to brcryptjs just takes a few minutes if you can deal with the speed hit. See my related comment above.

If you want to troubleshoot the issue with Yarn v0.20.0, please provide steps to reproduce including which version of bcrypt you were trying to install.

[eessex]

Thanks @markstos, using the bcryptjs npm package did the trick

[adamreisnz]

Switching to bcryptjs is unfortunately not an option for us due to the speed penalty (up to 5x slower in some cases)

[zaaack]

Same here, with the newest version v0.21.3. Not just bcrypt，node-sass，dtrace(bunyan), terk-capture， all need rebuild after install any new package.

[alexborisov]

Same here. Also bcrypt. Yarn v0.21.3

[holm]

This is currently blocking us from using yarn also. Can we have this reopened?

[bestander]

@holm, please open a new issue with repro steps

[MadRabbit]

still happening for me in v1.0.1 in alpine linux