Node-rdkafka: Error: librdkafka++.so.1: cannot open shared object file

[JaapRood]

Reposting of #394, as it was closed by the issue author without being fixed.

When trying to use run on Heroku (and I'm guessing other linuxed-powered server environments), the following error is triggered trying to require [email protected]:

Error: librdkafka++.so.1: cannot open shared object file: No such file or directory
    at Object.Module._extensions..node (module.js:681:18)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at bindings (/app/node_modules/bindings/bindings.js:81:44)
    at Object.<anonymous> (/app/node_modules/node-rdkafka/librdkafka.js:10:32)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)

The author of #394 suggested adding an Aptfile to your project, however, this is non-customary for npm packages. That said, I can't exactly suggest a solution myself, being really unfamiliar with how Node.js binds to C/C++, the build systems work, etc.

[JaapRood]

I was thinking about what on a conceptual level could be causing this. One important thing to note is that Heroku apps are built in a folder and then moved to a different one. I've ran into issues before where tools used in this process use absolute file paths rather than relative ones.

Could be totally unrelated, but given that it's about a file not being able to be found, I thought I'd mention it.

[alexander-alvarez]

I had this issue when building and moving assets from one docker image to another...
which we initially needed to do because we were using node-slim which doesn't have all of the libraries to build librdkafka. Building it on the regular node image, and not copying it over to a different container from where it was built fixed the issue for me.

[JaapRood]

Thanks for your input @alexander-alvarez, that puts some more weight behind this being a portability issue, most likely through the use of an absolute path rather than a relative one.

[webmakersteve]

Does Heroku not move the entire node_modules directory? I see people have been having problems binding dynamically to those libraries since the build change. I don't use Heroku so things aren't tested there, but if anyone has any insight into this issue would happily take a contribution to fix this issue for Heroku users

[battlecow]

@webmakersteve I run my app in docker and copy the entire node_modules directory over to the final image, I have had to go about deleting this lib directory and re-running npm install on image build.

[JaapRood]

@webmakersteve just like @battlecow does with his app in Docker, Heroku builds from project root and then copies the project root and all of it's contents to different locations. More specifically, it creates a compressed slug from the result of the build, so that it can easily provision new (but identical) servers with new instances, having to only build the project once.

I've seen a similar issue in another project a while back, where absolute symlinks were used rather than relative ones, causing them to break after moving the project root: https://github.com/lerna/lerna/issues/423

[sven-petersen]

Experiencing the same issue on cloud foundry. Interesting is that it worked library version 2.2.3 but not with 2.3.x. Unfortunately I cannot use the older version as I then face this issue https://github.com/edenhill/librdkafka/issues/1691

So I assume it must be caused by some change that was introduced between 2.2.3 and 2.3.0.

[sven-petersen]

Commit https://github.com/Blizzard/node-rdkafka/commit/3289f412649d20e8a15b3d4eb9c584fd3178bb24 broke it for me.

As the folder/file structure was discussed here maybe the following gist helps: https://gist.github.com/aktivalux/730f30469b3db33a3f2c885979147097
It shows the folder/file structure for the application at runtime. There are two librdkafka++.so.1 files in the following locations:

/home/vcap/app/node_modules/node-rdkafka/build/Release/librdkafka++.so.1
/home/vcap/app/node_modules/node-rdkafka/deps/librdkafka/src-cpp/librdkafka++.so.1

[bgedik]

Any update on this?

[linorsolomon]

+1

[rajan471]

I am also facing the same issue. Any resolution on this.

[bobrowadam]

@JaapRood did you find any solution for this issue?

[webmakersteve]

Sorry everyone. I'm back after a tough few weeks with little time to prioritize my open source contributions.

I upgraded the library to 2.3.3 which makes it so the configure step happens in the gypfile again. When I was seeing the issue you on this thread on a coworker's machine what I did to the gypfile fixed it. Can you please try again on the new build?

[JaapRood]

@webmakersteve all your work is greatly appreciated! Finding time and resources to work on open-source is hard, especially with every day work concerns tugging at us as well. Thanks for all the time you manage to put in, we benefit from it daily!

[JaapRood]

Regarding the fix: we had to resort to running on a custom build of 2.2.x with cherry picked commits (a situation I can't wait to get out of), which makes it a bit tricky for me to test this issue. Maybe one of @battlecow, @bobrowadam, @aktivalux or someone else can give it a go? Thanks!

[rajan471]

For me it worked after installing "build-essentials" on Ubuntu.
apt install build-essentials

[sven-petersen]

Hi @webmakersteve thanks for looking into it and trying to help us here.
Unfortunately using github:Blizzard/node-rdkafka#v2.3.3 did not fix it on cloud foundry for me. deploy/build logs can be found here: gist

[battlecow]

Just wanted to report back as I finally got around to setting up a reproducer. I have the same Jenkins build running npm install and creating docker images from 2.2.2 forward. No errors from the 2.2.x series but unfortunately this still affects me with the 2.3.3 version. As a side effect of the compile changes made to 2.3.3 I can no longer build successfully on my Mac.
In my Jenkinsfile I am using upstream Docker node:8.11 ( https://hub.docker.com/_/node/ ) and then building my image with Docker version 17.12.1 from CoreOS stable: https://coreos.com/releases/#1688.5.3 .
I was not however able to reproduce locally with separate Docker containers on my Mac with Docker for Mac version: 18.05.0-ce

[jjjmm]

Also facing the same issue on ubuntu with 2.3.x version. The 2.2.x seems to be working fine

[ctrom]

I encountered this when building our app on a node image and then transferring the built result to a different image. My testing indicates that @JaapRood's suggestion that this is due to a relative/absolute path issue is correct. If I ensure that the build path on the first image exactly matches the path on the runtime image, the librdkafka dependency resolves correctly. If the paths differ at all, the dependency is not found.

[ctrom]

I'd like to see this fixed in the code for this project, but the workaround that we're applying is to BUILD_LIBRDKAFKA=0 yarn install and provide the librdkafka dependency by doing an apk add librdkafka in the alpine image we are using. The BUILD_LIBRDKAFKA environment variable is used in the binding.gyp file to control whether the librdkafka dependency gets built as part of the npm/yarn install or should not be built and should be found in /usr.

Note that this only works with alpine:3.7 and higher (due to the version of librdkafka that is required) which means that you'll need to be using at least the node:10-alpine image as the node:9-alpine image uses an older version of alpine.

[wiesson]

@ctrom, I some steps to reproduce, should it work like this? Somehow node-gyp is still being executed :(

docker run -ti node:10-alpine sh
apk add --no-cache --virtual librdkafka
BUILD_LIBRDKAFKA=0 yarn add node-rdkafka

[ctrom]

@wiesson When doing the BUILD_LIBRDKAFKA=0 yarn install, some of the build steps still need to run. You will need to install the build dependencies on the node:alpine image so the build can complete, or use an image that already has those dependencies in place.

[saraht129]

Hi, I am experiencing the exact issue, and I would like to know whether it's possible to configure library search path through an environment variable or something?

It's quite a usual case where we build and deploy on different environments, and search path should be configurable instead of being set by the package script on the build phase.

For the time being is there a way to know where the search path is being configured so we can re-configure?

Edit: I temporarily run node-gyp rebuild in node_modules/node-rdkafka during deployment just so the paths can be corrected. And it works this way.

[bobrowadam]

We temporrarly solved this issue by supplying the following env var:
LD_LIBRARY_PATH=<application path>/node_modules/node-rdkafka/build/deps
Yes, it ain't pretty

[alexander-alvarez]

Could this be solved by removing all of the <(module_root_dir)>/ since the paths are relative by default?

File path names are relative to the directory in which the .gyp file lives.

https://gyp.gsrc.io/docs/UserDocumentation.md#add-a-source-file-that-builds-on-all-platforms

https://github.com/Blizzard/node-rdkafka/blob/master/binding.gyp#L79
would become build/deps/librdkafka.so, and so on?

[webmakersteve]

I'm open to changing the path to a relative path instead of using the variable if that fixes this issue. Does it fix the problem for the people who are experiencing it? Would definitely be open to a PR if so.

[Michael-Hoffmann]

I had the same Issue on Ubuntu 18.04. This worked for me:

• sudo apt install librdkafka-dev
• reinstall node-rdkafka

[pimvanderheijden]

@bobrowadam 's fix worked for me!

[yfinkelstein]

This problem showed up for me on Ubuntu 18.04 for the first time. It did not exist on Ubuntu 16.04 and earlier.

[SomiFrame]

@Michael-Hoffmann it works for me

[vgenev]

I did both - @Michael-Hoffmann and then @bobrowadam fixes on Ubuntu 18.04 to be able to run the app and connect a consumer to existing topic. After the reinstall, suggested by @Michael-Hoffmann, the app was starting but I got this error: Segmentation fault (core dumped)
Adding the env var suggested by @bobrowadam fixed that, as well.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.