Yarn: Does not compile from source when binary not available (like NPM does)

[swernerx]

I see a bug when trying to install a package which depends on node-zopfli. There currently does not seem to be a binrary available for my platform: NodeJS 7.10 + Mac OS 10.12.5.

Behavior of NPM (@5.0.0):

• Trying to download the package.
• Fails with 403
• Downloads & compiles source package

> [email protected] install /Users/Sebastian/Workspace/sebastian-software/edgeapp/node_modules/node-zopfli
> node-pre-gyp install --fallback-to-build

node-pre-gyp ERR! Tried to download(403): https://node-zopfli.s3.amazonaws.com/Release/zopfli-v1.4.0-node-v51-darwin-x64.tar.gz 
node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v51 ABI) (falling back to source compile with node-gyp) 
  CXX(target) Release/obj.target/zopfli/src/zopfli-binding.o
  CXX(target) Release/obj.target/zopfli/src/png/zopflipng.o
  CC(target) Release/obj.target/zopfli/zopfli/src/zopfli/blocksplitter.o
  CC(target) Release/obj.target/zopfli/zopfli/src/zopfli/cache.o
  CC(target) Release/obj.target/zopfli/zopfli/src/zopfli/deflate.o
  CC(target) Release/obj.target/zopfli/zopfli/src/zopfli/gzip_container.o
  CC(target) Release/obj.target/zopfli/zopfli/src/zopfli/hash.o
  CC(target) Release/obj.target/zopfli/zopfli/src/zopfli/katajainen.o
  CC(target) Release/obj.target/zopfli/zopfli/src/zopfli/lz77.o
  CC(target) Release/obj.target/zopfli/zopfli/src/zopfli/squeeze.o
  CC(target) Release/obj.target/zopfli/zopfli/src/zopfli/tree.o
  CC(target) Release/obj.target/zopfli/zopfli/src/zopfli/util.o
  CC(target) Release/obj.target/zopfli/zopfli/src/zopfli/zlib_container.o
  CC(target) Release/obj.target/zopfli/zopfli/src/zopfli/zopfli_lib.o
  CXX(target) Release/obj.target/zopfli/zopfli/src/zopflipng/zopflipng_lib.o
  CXX(target) Release/obj.target/zopfli/zopfli/src/zopflipng/lodepng/lodepng.o
  CXX(target) Release/obj.target/zopfli/zopfli/src/zopflipng/lodepng/lodepng_util.o
  SOLINK_MODULE(target) Release/zopfli.node
  COPY /Users/Sebastian/Workspace/sebastian-software/edgeapp/node_modules/node-zopfli/lib/binding/node-v51-darwin-x64/zopfli.node
  TOUCH Release/obj.target/action_after_build.stamp

Behavior of Yarn (@0.24.6):

• Trying to download the package.
• Fails with 403
• No attempt to download source package + exit 1 of yarn process

yarn install v0.24.6
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
[1/7] ⠄ fsevents
[2/7] ⠄ leveldown
[3/7] ⠄ iltorb
[4/7] ⠄ node-zopfli
error /Users/Sebastian/Workspace/sebastian-software/edgeapp/node_modules/node-zopfli: Command failed.
Exit code: 1
Command: sh
Arguments: -c node-pre-gyp install --fallback-to-build
Directory: /Users/Sebastian/Workspace/sebastian-software/edgeapp/node_modules/node-zopfli
Output:
node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using [email protected]
node-pre-gyp info using [email protected] | darwin | x64
node-pre-gyp info check checked for "/Users/Sebastian/Workspace/sebastian-software/edgeapp/node_modules/node-zopfli/lib/binding/node-v51-darwin-x64/zopfli.node" (not found)
node-pre-gyp http GET https://node-zopfli.s3.amazonaws.com/Release/zopfli-v1.4.0-node-v51-darwin-x64.tar.gz
node-pre-gyp http 403 https://node-zopfli.s3.amazonaws.com/Release/zopfli-v1.4.0-node-v51-darwin-x64.tar.gz
node-pre-gyp ERR! Tried to download(403): https://node-zopfli.s3.amazonaws.com/Release/zopfli-v1.4.0-node-v51-darwin-x64.tar.gz 
node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v51 ABI) (falling back to source compile with node-gyp) 
node-pre-gyp http 403 status code downloading tarball https://node-zopfli.s3.amazonaws.com/Release/zopfli-v1.4.0-node-v51-darwin-x64.tar.gz 
node-pre-gyp ERR! build error 
node-pre-gyp ERR! stack Error: Failed to execute 'node-gyp clean' (Error: spawn node-gyp ENOENT)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/Users/Sebastian/Workspace/sebastian-software/edgeapp/node_modules/node-pre-gyp/lib/util/compile.js:77:29)
node-pre-gyp ERR! stack     at emitOne (events.js:96:13)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:213:12)
node-pre-gyp ERR! stack     at onErrorNT (internal/child_process.js:367:16)
node-pre-gyp ERR! stack     at _combinedTickCallback (internal/process/next_tick.js:80:11)
node-pre-gyp ERR! stack     at process._tickCallback (internal/process/next_tick.js:104:9)
node-pre-gyp ERR! System Darwin 16.6.0
node-pre-gyp ERR! command "/usr/local/Cellar/node/7.10.0/bin/node" "/Users/Sebastian/Workspace/sebastian-software/edgeapp/node_modules/node-zopfli/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd /Users/Sebastian/Workspace/sebastian-software/edgeapp/node_modules/node-zopfli
node-pre-gyp ERR! node -v v7.10.0
node-pre-gyp ERR! node-pre-gyp -v v0.6.34
node-pre-gyp ERR! not ok 
Failed to execute 'node-gyp clean' (Error: spawn node-gyp ENOENT)
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

What should happen:

I figure Yarn should follow the behavior of NPM here and compile from source on any error while downloading binary packages.

[Daniel15]

It sounds like the node-gyp version bundled with your NPM version isn't working properly. Try yarn global add node-gyp and see if it fixes it.

[swernerx]

The workaround you mentioned indeed fixes the problem. The question remaining is: Why is this only affecting Yarn but not NPM itself.

[Daniel15]

I think it's just an issue with node-pre-gyp, packages that use the regular
node-gyp are fine. I remember seeing another task for node-pre-gyp; you
might be able to find it by searching.

Sent from my phone.

On May 28, 2017 3:20 AM, "Sebastian Werner" [e-mail]
wrote:

The workaround you mentioned indeed fixes the problem. The question
remaining is: Why is this only affecting Yarn but not NPM itself.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/yarnpkg/yarn/issues/3507#issuecomment-304465212, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAFnHd3zTmy56QQR2ZYeOhFL1pSF9Jevks5r-FtDgaJpZM4NnWoV
.

[chamini2]

It happens with node-gyp for me:

$ yarn add [email protected] --force                                                                                                                                                                             816ms
yarn add v0.24.6
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Rebuilding all packages...
[2/4] ⠂ sharp
[3/4] ⠂ fsevents
[-/4] ⠁ waiting...
[-/4] ⠄ waiting...
[-/4] ⢀ waiting...
[2/4] ⢀ sharp
[3/4] ⢀ fsevents: http GET https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.1.1/fse-v1.1.1-node-v51-darwin-x64.tar.gz
[4/4] ⢀ bcrypt
[4/4] 📃  Building fresh packages...
success Installed "[email protected]" with binaries:
      - node-gyp
warning No license field
error /Users/matteo/Projects/zoi/api/node_modules/sharp: Command failed.
Exit code: 127
Command: sh
Arguments: -c node-gyp rebuild
Directory: /Users/matteo/Projects/zoi/api/node_modules/sharp
Output:
sh: node-gyp: command not found
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

[BYK]

Cannot reproduce with the latest master so closing.