Grpc-node: gRPC sporadically doesn't build properly under node 11.

[lafama]

Problem description

A clear and concise description of what the problem is.
After upgrading to node 11.0.0 npm install fails due to @google-cloud/language dependency which has a dependency to google-gax which has a dependency to [email protected]

 node-pre-gyp install --fallback-to-build --library=static_library

node-pre-gyp WARN Using request for node-pre-gyp https download 
node-pre-gyp WARN Tried to download(404): https://node-precompiled-binaries.grpc.io/grpc/v1.15.1/node-v67-darwin-x64-unknown.tar.gz 
node-pre-gyp WARN Pre-built binaries not found for [email protected] and [email protected] (node-v67 ABI, unknown) (falling back to source compile with node-gyp) 
node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v67 ABI, unknown) (falling back to source compile with node-gyp) 
node-pre-gyp WARN Hit error Connection closed while downloading tarball file 

grpc/src/node/extension_binary/node-v67-darwin-x64-unknown --napi_version=3 --node_abi_napi=napi' (1)
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/backoff/backoff.oexec install script
../deps/grpc/src/core/lib/backoff/backoff.cc:19:10: fatal error: 'grpc/support/port_platform.h' file not found
#include <grpc/support/port_platform.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.░⸩ ⠸ install: info lifecycle [email protected]~install: Failed to exec install script
make: *** [Release/obj.target/grpc/deps/grpc/src/core/lib/backoff/backoff.o] Error 1

Reproduction steps

Give very precise steps you've discovered to reproduce your problem. If possible and applicable, provide us with a repository we can clone that contains a reproduction case. Also if possible and applicable, please include a Dockerfile that exhibits the problem if it's specific to a certain environment. Bug reports with no reproduction steps will be closed.
Npm install
-Node version 11.0.0
-Add reference to @google-cloud/language

Environment

• OS name, version and architecture: [e.g. Linux Ubuntu 18.04 amd64] Mac OS
• Node version [e.g. 8.10.0] 11.0.0
• Node installation method [e.g. nvm]
• If applicable, compiler version [e.g. clang 3.8.0-2ubuntu4]
• Package name and version [e.g. [email protected]] 1.15.1

Additional context

Add any other context about the problem here. If possible, attach full logs. Do not try to omit anything for brevity, but instead include absolutely everything. Please try and set your operating system's locale to English, so that logs contain error messages in the English language.

[nicolasnoble]

This looks like we have a broken source distribution for 1.15.1...

[murgatroid99]

I don't know what's going on with that source distribution. I tried it myself and it works, and that file is present and in the include paths.

[nicolasnoble]

Maybe it's a Node 11 thing? Did you try with this runtime?

[nicolasnoble]

So I just tried installing @google-cloud/language on my macbook with node 11 installed from nvm, and things worked properly fallbacking on compiling from source, so we're not able to reproduce this one.

[nicolasnoble]

From https://github.com/grpc/grpc/issues/17011 - there IS something going on, it's sporadic, and horribly feels like a race condition. I don't know where the race is yet, but it's definitely related to node 11, and not with grpc itself.

[nicolasnoble]

By the looks of it, there'll be a node 11.0.1 release soon, with issues in there that may be related to this. Please wait until this is released before attempting to use node 11.

[jkwlui]

Thanks for investigating, we'll wait and see if things change with node 11.0.1 :)

[ojordann]

Works fine with Node 10.12.0

[nodeful]

Running Node v10.13.0 today and GRPC fails to install using npm i -S grpc
Error:

node-pre-gyp ERR! Tried to download(403): https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v1.9.1/node-v64-linux-x64-musl.tar.gz 
node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v64 ABI, musl) (falling back to source compile with node-gyp) 
node-pre-gyp ERR! Tried to download(undefined): https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v1.9.1/node-v64-linux-x64-musl.tar.gz 
node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v64 ABI, musl) (falling back to source compile with node-gyp) 
make: Entering directory '/var/www/app/current/node_modules/grpc/build'
make: Entering directory '/var/www/app/current/node_modules/grpc/build'
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o
rm: can't remove './Release/.deps/Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o.d.raw': No such file or directory
make: *** [grpc.target.mk:394: Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o] Error 1
make: Leaving directory '/var/www/app/current/node_modules/grpc/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)
gyp ERR! System Linux 4.14.48-coreos-r2
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--library=static_library" "--module=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl/grpc_node.node" "--module_name=grpc_node" "--module_path=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl"
gyp ERR! cwd /var/www/app/current/node_modules/grpc
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/backoff/backoff.o
gyp ERR! node -v v10.13.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
node-pre-gyp ERR! build error 
node-pre-gyp ERR! stack Error: Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --library=static_library --module=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl/grpc_node.node --module_name=grpc_node --module_path=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/var/www/app/current/node_modules/grpc/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:962:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)
node-pre-gyp ERR! System Linux 4.14.48-coreos-r2
node-pre-gyp ERR! command "/usr/bin/node" "/var/www/app/current/node_modules/grpc/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build" "--library=static_library"
node-pre-gyp ERR! cwd /var/www/app/current/node_modules/grpc
node-pre-gyp ERR! node -v v10.13.0
node-pre-gyp ERR! node-pre-gyp -v v0.6.39
node-pre-gyp ERR! not ok 
Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --library=static_library --module=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl/grpc_node.node --module_name=grpc_node --module_path=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl' (1)
Assembler messages:
Error: can't open /var/www/app/current/node_modules/grpc/ccDekbog.s for reading: No such file or directory
make: *** [grpc.target.mk:394: Release/obj.target/grpc/deps/grpc/src/core/lib/backoff/backoff.o] Error 1
make: Leaving directory '/var/www/app/current/node_modules/grpc/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)
gyp ERR! System Linux 4.14.48-coreos-r2
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--library=static_library" "--module=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl/grpc_node.node" "--module_name=grpc_node" "--module_path=/var/www/app/current/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-musl"
gyp ERR! cwd /var/www/app/current/node_modules/grpc
gyp ERR! node -v v10.13.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build --library=static_library`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Seems like this file:
https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v1.9.1/node-v64-linux-x64-musl.tar.gz is not Publicly openned

[jggc]

ArchLinux nodejs 11.0 fails EVERY time.

But with node 10.13 from nvm it works fine.

Here is my failing build log

[84564221]

gyp ERR! configure error 
gyp ERR! stack Error: Command failed: /usr/bin/python -c import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack   File "<string>", line 1
gyp ERR! stack     import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack                                ^
gyp ERR! stack SyntaxError: invalid syntax
gyp ERR! stack 
gyp ERR! stack     at ChildProcess.exithandler (child_process.js:289:12)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at maybeClose (internal/child_process.js:962:16)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)

...

gyp ERR! command "/usr/bin/node" "/usr/bin/node-gyp" "build" 
  "--fallback-to-build" 
  "--library=static_library" 
  "--module=/home/user/grpc-test/node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc/grpc_node.node" 
  "--module_name=grpc_node" 
  "--module_path=/home/user/grpc-test/node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc" 
  "--napi_version=3" 
  "--node_abi_napi=napi"
gyp ERR! cwd /home/user/grpc-test/node_modules/grpc
gyp ERR! node -v v11.0.0
gyp ERR! node-gyp -v v3.8.0

Let's have a look

$ python --version
Python 2.7.15

Same issue when switching to python3

$ python --version
Python 3.7.1

[murgatroid99]

The original issue is about sporadic compilation failures under Node 11. If you are experiencing different errors under a different version of Node, please file a separate issue.

[hilongjw]

same issue here

node-pre-gyp ERR! node -v v11.0.0
node-pre-gyp ERR! node-pre-gyp -v v0.10.3
node-pre-gyp ERR! not ok
Failed to execute '/Users/awe/.nvm/versions/node/v11.0.0/bin/node /Users/awe/.nvm/versions/node/v11.0.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --library=static_library --module=/Users/awe/Desktop/code/project/era-cloud-proxy/node_modules/grpc/src/node/extension_binary/node-v67-darwin-x64-unknown/grpc_node.node --module_name=grpc_node --module_path=/Users/awe/Desktop/code/project/era-cloud-proxy/node_modules/grpc/src/node/extension_binary/node-v67-darwin-x64-unknown --napi_version=3 --node_abi_napi=napi' (1)
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/backoff/backoff.oexec install script
../deps/grpc/src/core/lib/backoff/backoff.cc:19:10: fatal error: 'grpc/support/port_platform.h' file not
      found
#include <grpc/support/port_platform.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.░⸩ ⠹ install: info lifecycle [email protected]~install: Failed to exec install script
make: *** [Release/obj.target/grpc/deps/grpc/src/core/lib/backoff/backoff.o] Error 1

[christopher-johnson]

reproduced using docker node:latest (11.1.0) and travis here. grpc compiles 2 times successfully, then on the third time it fails. The source binaries are not available yet as indicated with "Pre-built binaries not found for [email protected] and [email protected]"

[nicolasnoble]

Found the rootcause. The culprit is mapbox/node-pre-gyp#426. A workaround for now with node 11 is to always add the --build-from-source option on npm install, which won't attempt an http download, and won't trigger the problem.

[smtapp-smtio-cloud]

@nicolasnoble dude, you're a genius. You saved me from all the frustrations that I was building up due to this issue for the past week! Way to go!

[itaied246]

@nicolasnoble what about yarn?

[nicolasnoble]

We've never officially supported yarn as a distribution system.

[malkhalaf]

I have the same problem running MacOS High Sierra using homebrew to install node and the installed version is 11.0. Build from sources option did not work.
Solution: I force downgraded node to v10 and install went fine.

[jscti]

+1 here, impossible to build today :

NPM version : 6.4.1
Node version : v11.1.0

> [email protected] install /app/node_modules/grpc
> node-pre-gyp install --fallback-to-build --library=static_library

node-pre-gyp WARN Using request for node-pre-gyp https download
node-pre-gyp WARN Tried to download(403): https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v1.13.1/node-v67-linux-x64-musl.tar.gz
node-pre-gyp WARN Pre-built binaries not found for [email protected] and [email protected] (node-v67 ABI, musl) (falling back to source compile with node-gyp)
node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v67 ABI, musl) (falling back to source compile with node-gyp)
node-pre-gyp WARN Hit error Connection closed while downloading tarball file
gypgyp ERR! configure error

In the browser, the requested url show a :

<Error>
<Code>AccessDenied</Code>
<Message>Access denied.</Message>
<Details>
Anonymous caller does not have storage.objects.get access to grpc-precompiled-binaries/node/grpc/v1.13.1/node-v67-linux-x64-musl.tar.gz.
</Details>
</Error>

[rmarya-asu]

+1

node -v 
v11.0.0
npm -v
6.4.1

running the command to install 'grpc' with sudo doesnt seem to work either.

node-pre-gyp WARN Hit error EACCES: permission denied, mkdir '/usr/local/lib/node_modules/grpc/src/node' 
gyp ERR! configure error 
gyp ERR! stack Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/grpc/build'
gyp ERR! System Darwin 18.0.0

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build --library=static_library`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.

Here from, the Firebase install issue. I tried to update the grpc package and ran into this issue.

[nicolasnoble]

Did any of the last reports see my update and workaround?

[rmarya-asu]

I switched to node 10.13 and it works fine.

Thanks @nicolasnoble

[Djiit]

I'm using electron-builder to rebuild grpc. node 10.13 and 11.0 fail. How can I pass the relevant option (--build-from-source) ? (please tell me if you want me to open another detailed issue about that)

[jscti]

node 10.13 worked. Thanks @nicolasnoble

[nicolasnoble]

@Djiit electron-builder is a different issue altogether. Please read the relevant section of our readme about using grpc with electron: https://github.com/grpc/grpc-node/tree/master/packages/grpc-native-core#about-electron

[swoodivyesh]

Found the rootcause. The culprit is mapbox/node-pre-gyp#426. A workaround for now with node 11 is to always add the --build-from-source option on npm install, which won't attempt an http download, and won't trigger the problem.

would you please explain me what does npm install --build-from-source will do? I mean what is the use case of --build-from-source is for npm install in general?

[swoodivyesh]

Found the rootcause. The culprit is mapbox/node-pre-gyp#426. A workaround for now with node 11 is to always add the --build-from-source option on npm install, which won't attempt an http download, and won't trigger the problem.

why only this fix works, what is the actual issue happening? is it because whenever it goes to download the http download it gets the close connection error?

[Tsangares]

I used npm install --build-from-source @grpc/grpc-js and or npm install --build-from-source grpc both fail when building at:

CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/surface/channel.o
../deps/grpc/src/core/lib/surface/channel.cc: In function ‘grpc_channel* grpc_channel_create_with_builder(grpc_channel_stack_builder*, grpc_channel_stack_type)’:
../deps/grpc/src/core/lib/surface/channel.cc:103:38: error: ‘void* memset(void*, int, size_t)’ clearing an object of type ‘grpc_channel’ {aka ‘struct grpc_channel’} with no trivial copy-assignment; use assignment or value-initialization instead [-Werror=class-memaccess]
   memset(channel, 0, sizeof(*channel));
                                      ^
../deps/grpc/src/core/lib/surface/channel.cc:61:8: note: ‘grpc_channel’ {aka ‘struct grpc_channel’} declared here
 struct grpc_channel {
        ^~~~~~~~~~~~
cc1plus: all warnings being treated as errors
make: *** [grpc.target.mk:469: Release/obj.target/grpc/deps/grpc/src/core/lib/surface/channel.o] Error 1
make: Leaving directory ./node_modules/grpc/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:246:12)
gyp ERR! System Linux 4.18.16-arch1-1-ARCH
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--library=static_library" "--module=./node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc/grpc_node.node" "--module_name=grpc_node" "--module_path=./node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc" "--napi_version=3" "--node_abi_napi=napi"
gyp ERR! cwd ./node_modules/grpc
gyp ERR! node -v v11.1.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 

Note ArchLinux OS

[nicolasnoble]

@Tsangares that may be a tangential issue, where we don't properly support your compiler. It's in fact the same as https://github.com/grpc/grpc/issues/15576 - is your compiler gcc 8? If yes, please follow along on the other issue.

[nicolasnoble]

@swoodivyesh the explanations are on the various bugs I've filed against node-pre-gyp, and the gist of it is that there's a bug in this software that will trigger a race condition if we haven't published binaries for the platform you're trying to use grpc on. And we can't publish node 11 binaries because node-pre-gyp doesn't support it yet.

Using --build-from-source calls into another codepath of node-pre-gyp, and that avoids the race condition. Call it an ugly workaround until their bugs are fixed.

[spencerfontein]

Using

node 10.13.0
npm 6.4.1
OS: Ubuntu 18.04

And still getting error anyone else? I tried npm install --build-from-source but that also doesnt work

[nicolasnoble]

@spencerfontein please create a new issue with the full report of the failure you are experiencing.

[swoodivyesh]

@swoodivyesh the explanations are on the various bugs I've filed against node-pre-gyp, and the gist of it is that there's a bug in this software that will trigger a race condition if we haven't published binaries for the platform you're trying to use grpc on. And we can't publish node 11 binaries because node-pre-gyp doesn't support it yet.

Using --build-from-source calls into another codepath of node-pre-gyp, and that avoids the race condition. Call it an ugly workaround until their bugs are fixed.

okay so was it due to mixed content error? What I understood from it is that few libraries have resources coming in with http and others with https so is that the main issue here?

@nicolasnoble

[DeividVeloso]

I fix this using a Node version v10.13.0 after I delete node_modules and install with npm install again and works for me.

[mantielero]

It fails for me when I try to build from source in node v11:

[...]
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/ext/transport/chttp2/transport/chttp2_plugin.o
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.o
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/ext/transport/chttp2/transport/flow_control.o
../deps/grpc/src/core/ext/transport/chttp2/transport/flow_control.cc: In member function 'virtual uint32_t grpc_core::chttp2::TransportFlowControl::MaybeSendUpdate(bool)':
../deps/grpc/src/core/ext/transport/chttp2/transport/flow_control.cc:193:50: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
       static_cast<const uint32_t>(target_window());
                                                  ^
cc1plus: all warnings being treated as errors
make: *** [grpc.target.mk:470: Release/obj.target/grpc/deps/grpc/src/core/ext/transport/chttp2/transport/flow_control.o] Error 1
[...]

Is it my bad?

[nicolasnoble]

Which is a duplicate of https://github.com/grpc/grpc/issues/15576

Please stop trying to use grpc with node 11 until node-pre-gyp has node 11 support. Then we'll be able to properly assist.

[mantielero]

Sorry mate.

Found the rootcause. The culprit is mapbox/node-pre-gyp#426. A workaround for now with node 11 is to always add the --build-from-source option on npm install, which won't attempt an http download, and won't trigger the problem.

So this is not applicable then.

[swoodivyesh]

Sorry mate.

Found the rootcause. The culprit is mapbox/node-pre-gyp#426. A workaround for now with node 11 is to always add the --build-from-source option on npm install, which won't attempt an http download, and won't trigger the problem.

So this is not applicable then.

it is! @mantielero that's what resolved it for me. What I understood from this problem that node-pre-gyp has few resources who fetch in data from http requests and others from https request which causes the Mixed Content Erorr you can read more about it here : https://developers.google.com/web/fundamentals/security/prevent-mixed-content/what-is-mixed-content

This is totally my understanding so correct me if I'm wrong.

[nicolasnoble]

@mantielero your issue is that you can't compile from source because of https://github.com/grpc/grpc/issues/15576 since your compiler isn't agreeing with the code we publish. Once we can have node-pre-gyp's node 11 support, we'll be able to publish node 11 prebuilt binaries and you won't have to compile from source.

[JesusAlbornoz]

I have to downgrade to v10.13.0 of Node, delete my node_modules folder and do npm install... Works for me

[stupidly-logical]

The only solution I found was to downgrade Node to the current lts : 10.13.0

Steps:

1. sudo npm cache clean -f
2. sudo npm install -g n
3. sudo n 10.13.0
4. npm install firebase

[nicolasnoble]

I've published grpc 1.16.1, that now depends on node-pre-gyp 0.12.0. Through this update, this issue here should now be fixed, and also, node 11 support is in.

[youpiyoful]

Same problem, i fix him just with sudo... strange i think it was not a permission problem... don t understand

[nicolasnoble]

Your issue is most likely unrelated to what was going on here. Please open a new issue with the full report of the failure.