Node-rdkafka: npm install of node-rdkafka is failing

[Nagasai-Aytha]

Hi

I am getting below error when trying in my docker Linux machine

node util/configure

checking for OS or distribution... ok(B (Debian)
checking for C compiler from CC env... failed(B
checking for gcc (by command)... failed(B
checking for clang (by command)... failed(B
checking for cc (by command)... failed(B (fail)
checking for C++ compiler from CXX env... failed(B
checking for C++ compiler (g++)... failed(B
checking for C++ compiler (clang++)... failed(B
checking for C++ compiler (c++)... failed(B (fail)
checking executable ld... failed(B (disable)
checking executable nm... failed(B (disable)
checking executable objdump... failed(B (disable)
checking executable strip... failed(B (disable)
checking for pkgconfig (by command)... failed(B
checking for install (by command)... ok(B
checking for PIC (by compile)... failed(B (disable)
checking for GNU-compatible linker options... failed(B
checking for OSX linker options... failed(B
checking for GNU linker-script ld flag... failed(B
checking for Solaris linker-script ld flag... failed(B (ignore)
checking for __atomic_32 (by compile)... failed(B
checking for __atomic_32_lib (by compile)... failed(B
checking for __sync_32 (by compile)... failed(B (disable)
checking for __atomic_64 (by compile)... failed(B
checking for __atomic_64_lib (by compile)... failed(B
checking for __sync_64 (by compile)... failed(B (disable)
checking for socket (by compile)... failed(B
checking for socket_nsl (by compile)... failed(B (fail)
parsing version '0x000b01ff'... ok(B (0.11.1)
checking for libpthread (by compile)... failed(B (fail)
checking for zlib (by compile)... failed(B (disable)
checking for libcrypto (by compile)... failed(B (disable)
checking for liblz4 (by compile)... failed(B (disable)
checking for libssl (by compile)... failed(B (disable)
checking for libsasl2 (by compile)... failed(B (disable)
checking for libsasl (by compile)... failed(B (disable)
checking for crc32chw (by compile)... failed(B (disable)
checking for regex (by compile)... failed(B (disable)
checking for librt (by compile)... failed(B
checking for strndup (by compile)... failed(B (disable)
checking for strerror_r (by compile)... failed(B (disable)
checking for libdl (by compile)... failed(B (disable)
checking for nm (by env NM)... failed(B

###########################################################(B
### Configure failed ###(B
###########################################################(B
### Accumulated failures: ###(B
###########################################################(B
 cc (WITH_CC) (B
module: cc
action: fail
reason:
command 'cc --version' failed:
mklove/modules/configure.base: line 1184: cc: command not found

 cxx (WITH_CXX) C++ compiler (c++)(B
module: cc
action: fail
reason:
command 'c++ --version' failed:
mklove/modules/configure.base: line 1184: c++: command not found

 socket_nsl () (B
module: socket
action: fail
reason:
compile check failed:
CC: CC
flags:
-g -O2 -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wall -Werror _mkltmpXAXlsW.c -o _mkltmpXAXlsW.c.o -lsocket -lnsl :
mklove/modules/configure.base: line 960: -g: command not found
source:

include

include

include

void foo (void) {
int s = socket(0, 0, 0);
close(s);
}

 libpthread () (B
module: librdkafka
action: fail
reason:
compile check failed:
CC: CC
flags: -lpthread
-g -O2 -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wall -Werror _mkltmpbB77Jn.c -o _mkltmpbB77Jn.c.o -lpthread:
mklove/modules/configure.base: line 960: -g: command not found
source: #include

npm WARN prefer global [email protected] should be installed with -g

[email protected] install /xxx/xxx/node_modules/snappy
node-gyp rebuild

gyp ERR! build error
gyp ERR! stack Error: not found: make
gyp ERR! stack at getNotFoundError (/usr/lib/node_modules/npm/node_modules/which/which.js:13:12)
gyp ERR! stack at F (/usr/lib/node_modules/npm/node_modules/which/which.js:68:19)
gyp ERR! stack at E (/usr/lib/node_modules/npm/node_modules/which/which.js:80:29)
gyp ERR! stack at /usr/lib/node_modules/npm/node_modules/which/which.js:89:16
gyp ERR! stack at /usr/lib/node_modules/npm/node_modules/which/node_modules/isexe/index.js:44:5
gyp ERR! stack at /usr/lib/node_modules/npm/node_modules/which/node_modules/isexe/access.js:8:5
gyp ERR! stack at FSReqWrap.oncomplete (fs.js:114:15)
gyp ERR! System Linux 3.10.0-693.2.2.el7.x86_64
gyp ERR! command "/usr/bin/nodejs" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /xxx/xxx/node_modules/snappy
gyp ERR! node -v v7.10.1
gyp ERR! node-gyp -v v3.5.0
gyp ERR! not ok

npm ERR! Failed at the [email protected] install script 'node-gyp rebuild'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the node-rdkafka package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs node-rdkafka
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls node-rdkafka
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:

Could you please help to fix this issue?

Thanks
Sai

[analytik]

I'm having a similar problem, using Node.js v8.9.1 - but for some reason I won't get even to the part where it starts building:

root@loadtest-a066112bfea4f247a5c67e1d2569bb29-0s179:/project# npm i node-rdkafka@2
npm WARN lifecycle [email protected]~preinstall: cannot run in wd %s %s (wd=%s) [email protected] node util/configure /project/node_modules/.staging/node-rdkafka-b896f0b4

> [email protected] install /project/node_modules/node-rdkafka
> node-gyp rebuild

Please run configure before using this script
gyp: Call to 'node ../util/get-lib-dirs.js include' returned exit status 1 while in /project/node_modules/node-rdkafka/deps/librdkafka.gyp. while loading dependencies of binding.gyp while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:336:16)
gyp ERR! stack     at emitTwo (events.js:126:13)
gyp ERR! stack     at ChildProcess.emit (events.js:214:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Linux 4.9.24-coreos
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /project/node_modules/node-rdkafka
gyp ERR! node -v v8.9.1
gyp ERR! node-gyp -v v3.6.0
gyp ERR! not ok
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1

My Dockerfile includes apt-get update && apt-get install -y libssl-dev liblz4-dev libsasl2-dev libpthread-stubs0-dev build-essential pkg-config - has any of the dependencies changed since node-rdkafka 1.x?

EDIT: Builds of 2.2.1, 2.2.0, 2.1.1 fail. However, 2.1.1 has a slightly different error message which might be useful:

Please run configure before using this script
gyp: Call to 'node ../util/librdkafka-config-checker.js WITH_SNAPPY' returned exit status 1 while in /project/node_modules/node-rdkafka/deps/librdkafka.gyp. while loading dependencies of binding.gyp while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1

[email protected] builds successfully, with three openssl warnings - gist.

[lovedebug]

same install issue with node-rdkafka 2.2.1 and nodejs 8.9.1 on windows

[webmakersteve]

Can you post your dockerfiles here? And I'm pretty sure they are different issues.

@Nagasai-Aytha what OS are you using in your docker container? The output suggests that you don't have a fully featured c++ compiler.

[jianglinchun]

The right way to install node-rdkafka on centos:

under project root
……
cd node_module
wget https://github.com/Blizzard/node-rdkafka/archive/v2.2.1.tar.gz
tar xvf v2.2.1.tar.gz
mv node-rdkafka-2.2.1 node-rdkafka
cd node-rdkafka/
cd deps/
wget https://github.com/edenhill/librdkafka/archive/v0.11.3.tar.gz
tar xvf v0.11.3.tar.gz
rm -rf ./librdkafka
mv librdkafka-0.11.3 librdkafka
cd librdkafka/
./configure
cd ../../
node-gyp rebuild
……

This will resolve node ../util/get-lib-dirs.js include issuse
@analytik @webmakersteve

[analytik]

Dockerfile:

FROM debian:jessie
RUN apt-get update \
    && apt-get install -y curl \ 
    && apt-get clean

COPY npm-install-461.sh /tmp/node/npm-install.sh
RUN chmod a+x  /tmp/node/npm-install.sh

RUN curl -sL https://nodejs.org/download/release/v8.9.1/node-v8.9.1-linux-x64.tar.gz -o /tmp/node.tar.gz \
    && mkdir -p /tmp/node \
    && tar -xzf /tmp/node.tar.gz -C /tmp/node --strip-components=1 \
    && cp /tmp/node/bin/* /usr/bin/ \
    && /tmp/node/npm-install.sh \
    && rm /tmp/node.tar.gz \
    && rm -rf /tmp/node/ \
    && rm -rf /usr/lib/node_modules/npm/.nyc_output \
    && npm install -g nodemon \
    && npm cache clean --force \
    && mkdir -p /project

WORKDIR /project

ENV TERM xterm

RUN cd /project \
    && apt-get update \
    && apt-get install -y \
        libssl-dev liblz4-dev libsasl2-dev libpthread-stubs0-dev build-essential pkg-config

RUN npm install [email protected]

CMD ["true"]

npm-install-461.sh in gist

With these, I get this result:

Step 9/10 : RUN npm install [email protected]
 ---> Running in f2a52c7dc291
npm WARN lifecycle [email protected]~preinstall: cannot run in wd %s %s (wd=%s) [email protected] node util/configure /project/node_modules/.staging/node-rdkafka-b896f0b4

> [email protected] install /project/node_modules/node-rdkafka
> node-gyp rebuild

Please run configure before using this script
gyp: Call to 'node ../util/get-lib-dirs.js include' returned exit status 1 while in /project/node_modules/node-rdkafka/deps/librdkafka.gyp. while loading dependencies of binding.gyp while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:336:16)
gyp ERR! stack     at emitTwo (events.js:126:13)
gyp ERR! stack     at ChildProcess.emit (events.js:214:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Linux 4.14.4-coreos
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /project/node_modules/node-rdkafka
gyp ERR! node -v v8.9.1
gyp ERR! node-gyp -v v3.6.0
gyp ERR! not ok
npm WARN enoent ENOENT: no such file or directory, open '/project/package.json'
npm WARN project No description
npm WARN project No repository field.
npm WARN project No README data
npm WARN project No license field.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2017-12-21T12_29_22_045Z-debug.log
The command '/bin/sh -c npm install [email protected]' returned a non-zero code: 1

[webmakersteve]

I just updated the configure utility so it logs why it fails, when it fails. Can you try to use the code at master and see if it helps us any?

[localzjaz]

I've been working through that issue Call to 'node ../util/get-lib-dirs.js include' returned exit status 1 recently. For me it only happens on node6/7 (not node4/8/9) on my dockers.

You can work through it by using a non-root user for npm install or using the npm --unsafe-perm flag (which I'm not sure about yet). Possibly something to do with install scripts trying to write/read somewhere that npm install won't allow during an install started by a root user?

[analytik]

Hi, thanks to both of you for your responses.

I've created a gist for this - the first two are the npm log + console output from the broken build with plain old npm install, the third shows when I try localzjaz's suggestions and do npm install --unsafe-perm - if you don't find an easy fix for this, I can just use the --unsafe-perm workaround in our case.

[buzzvamsi]

@webmakersteve Any updates on this issue? I still see it fails when i do a npm install

[webmakersteve]

Sorry. I've been very busy lately and haven't had enough time to devote to library maintenance.

@buzzvamsi are you having the exact same issue or is it separate? npm install issues can deceivingly look similar even when they are very different.

In any event, when debugging issues with the utility scripts, i recommend doing a force install in NPM and then running each step manually so you can see if it works like that. If scripts don't get executed, problems will show up like the one above.

As far as unsafe perm, if you are going to run the script as root (like in a docker container) I think it may be par for the course. If anyone has any suggestions on what the library can do to fix the problem please let me know,

[4ng3l0]

I solved this problem by adding the following line to my Dockerfile (from ubuntu):

apt-get install -y build-essential

I hope this helps.

[Alexis01]

Hi,

I'm trying to build docker adding apt-get install -y build-essential but I have got the same error

> [email protected] preinstall /home/jenkins/workspace/my-project/node_modules/node-rdkafka
> node util/configure

checking for OS or distribution... ok (Linux)
checking for C compiler from CC env... failed
checking for gcc (by command)... failed
checking for clang (by command)... failed
checking for cc (by command)... failed (fail)
checking for C++ compiler from CXX env... failed
checking for C++ compiler (g++)... failed
checking for C++ compiler (clang++)... failed
checking for C++ compiler (c++)... failed (fail)
checking executable ld... failed (disable)
checking executable nm... failed (disable)
checking executable objdump... failed (disable)
checking executable strip... failed (disable)
checking for pkgconfig (by command)... failed
checking for install (by command)... ok
checking for PIC (by compile)... failed (disable)
checking for GNU-compatible linker options... failed
checking for OSX linker options... failed
checking for GNU linker-script ld flag... failed
checking for Solaris linker-script ld flag... failed (ignore)
checking for __atomic_32 (by compile)... failed
checking for __atomic_32_lib (by compile)... failed
checking for __sync_32 (by compile)... failed (disable)
checking for __atomic_64 (by compile)... failed
checking for __atomic_64_lib (by compile)... failed
checking for __sync_64 (by compile)... failed (disable)
checking for socket (by compile)... failed
checking for socket_nsl (by compile)... failed (fail)
parsing version '0x000b03ff'... ok (0.11.3)
checking for libpthread (by compile)... failed (fail)
checking for zlib (by compile)... failed (disable)
checking for libcrypto (by compile)... failed (disable)
checking for liblz4 (by compile)... failed (disable)
checking for libssl (by compile)... failed (disable)
checking for libsasl2 (by compile)... failed (disable)
checking for libsasl (by compile)... failed (disable)
checking for crc32chw (by compile)... failed (disable)
checking for regex (by compile)... failed (disable)
checking for librt (by compile)... failed
checking for strndup (by compile)... failed (disable)
checking for strerror_r (by compile)... failed (disable)
checking for libdl (by compile)... failed (disable)
checking for nm (by env NM)... failed



###########################################################
###                  Configure failed                   ###
###########################################################
### Accumulated failures:                               ###
###########################################################
 cc (WITH_CC) 
    module: cc
    action: fail
    reason:
command 'cc --version' failed:
mklove/modules/configure.base: line 1184: cc: command not found

 cxx (WITH_CXX) C++ compiler (c++)
    module: cc
    action: fail
    reason:
command 'c++ --version' failed:
mklove/modules/configure.base: line 1184: c++: command not found

 socket_nsl () 
    module: socket
    action: fail
    reason:
compile check failed:
CC: CC
flags: 
  -g -O2 -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wall -Werror _mkltmphBbLmu.c -o _mkltmphBbLmu.c.o -lsocket -lnsl -L/home/jenkins/workspace/my-project/node_modules/node-rdkafka/build/Release :
mklove/modules/configure.base: line 960: -g: command not found
source: 
#include <sys/types.h>
#include <sys/socket.h>
#include <unistd.h>
void foo (void) {
   int s = socket(0, 0, 0);
   close(s);
}

 libpthread () 
    module: librdkafka
    action: fail
    reason:
compile check failed:
CC: CC
flags: -lpthread
  -g -O2 -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wall -Werror _mkltmpQsFb94.c -o _mkltmpQsFb94.c.o  -L/home/jenkins/workspace/my-project/node_modules/node-rdkafka/build/Release -lpthread:
mklove/modules/configure.base: line 960: -g: command not found
source: #include <pthread.h>


> [email protected] install /home/jenkins/workspace/my-project/node_modules/node-rdkafka
> node-gyp rebuild

gyp ERR! configure error 
gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable.
gyp ERR! stack     at PythonFinder.failNoPython (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:483:19)
gyp ERR! stack     at PythonFinder.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:397:16)
gyp ERR! stack     at F (/usr/local/lib/node_modules/npm/node_modules/which/which.js:68:16)
gyp ERR! stack     at E (/usr/local/lib/node_modules/npm/node_modules/which/which.js:80:29)
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/which/which.js:89:16
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/which/node_modules/isexe/index.js:42:5
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/which/node_modules/isexe/mode.js:8:5
gyp ERR! stack     at FSReqWrap.oncomplete (fs.js:152:21)
gyp ERR! System Linux 4.4.111-k8s
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/jenkins/workspace/dev-coreplatform-trackid-manager-svc/node_modules/node-rdkafka
gyp ERR! node -v v8.10.0
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok 
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above

[webmakersteve]

@Alexis01: The logs show that you need python in your image, so I'm not sure it's the same issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.