Sharp: Cross-platform, includes Meteor: g++: error: ../lib/libvips-cpp.so: No such file or directory

[cspecter]

I'm trying to install sharp into a Docker instance running my Meteor app and I'm having a ton of problems installing sharp. It seems that it can not find the correct files for libvips to build.

Here is my dockerfile.

FROM ubuntu:16.04
MAINTAINER Chad Specter

ENV METEORD_DIR /opt/meteord
COPY scripts $METEORD_DIR

RUN bash $METEORD_DIR/init.sh

EXPOSE 80
ENTRYPOINT bash $METEORD_DIR/run_app.sh

//init.sh
set -e

bash $METEORD_DIR/lib/install_base.sh
bash $METEORD_DIR/lib/install_node.sh
bash $METEORD_DIR/lib/install_phantomjs.sh
bash $METEORD_DIR/lib/cleanup.sh

//install_base.sh
set -e
apt-get update -y
apt-get install -y curl bzip2 build-essential libssl-dev python git subversion aptitude libstdc++6 pkg-config libglib2.0-0 libvips-dev libavahi-compat-libdnssd-dev

ldconfig

//install_meteor.sh
set -e

curl -sL https://install.meteor.com | sed s/--progress-bar/-sL/g | /bin/sh

//install_node.sh
set -e

curl -sL https://deb.nodesource.com/setup_4.x | bash -
apt-get update -y
apt-get install -y nodejs
node -v
npm -v
npm -g install npm@latest
npm i -g node-gyp
node-gyp install 4.6.0
node-gyp -v
npm install --global node-pre-gyp
npm install --global nan got semver caw tar
export npm_config_loglevel=http; export npm_config_spin=false

//install_phantomjs.sh
set -e
apt-get -y install libfreetype6 libfreetype6-dev fontconfig
ARCH=`uname -m`
PHANTOMJS_VERSION=1.9.8
PHANTOMJS_TAR_FILE=phantomjs-${PHANTOMJS_VERSION}-linux-${ARCH}.tar.bz2

cd /usr/local/share/
curl -L -O https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-${PHANTOMJS_VERSION}-linux-${ARCH}.tar.bz2
tar xjf $PHANTOMJS_TAR_FILE
ln -s -f /usr/local/share/phantomjs-${PHANTOMJS_VERSION}-linux-${ARCH}/bin/phantomjs /usr/local/share/phantomjs
ln -s -f /usr/local/share/phantomjs-${PHANTOMJS_VERSION}-linux-${ARCH}/bin/phantomjs /usr/local/bin/phantomjs
ln -s -f /usr/local/share/phantomjs-${PHANTOMJS_VERSION}-linux-${ARCH}/bin/phantomjs /usr/bin/phantomjs

rm $PHANTOMJS_TAR_FILE

//cleanup.sh
set -e
# Autoremove any junk
apt-get autoremove -y

# Clean out docs
rm -rf /usr/share/doc /usr/share/doc-base /usr/share/man /usr/share/locale /usr/share/zoneinfo

# Clean out package management dirs
rm -rf /var/lib/cache /var/lib/log

# Clean out /tmp
rm -rf /tmp/*

# Clear npm cache
npm cache clear



md5-019ffcb662c52593adfc6ded31e5660b




> [email protected] install /bundle/bundle/programs/server
> node npm-rebuild.js


> [email protected] install /bundle/bundle/programs/server/npm/node_modules/bcrypt
> node-gyp rebuild

make: Entering directory '/bundle/bundle/programs/server/npm/node_modules/bcrypt/build'
  CXX(target) Release/obj.target/bcrypt_lib/src/blowfish.o
  CXX(target) Release/obj.target/bcrypt_lib/src/bcrypt.o
  CXX(target) Release/obj.target/bcrypt_lib/src/bcrypt_node.o
  SOLINK_MODULE(target) Release/obj.target/bcrypt_lib.node
  COPY Release/bcrypt_lib.node
make: Leaving directory '/bundle/bundle/programs/server/npm/node_modules/bcrypt/build'

> [email protected] install /bundle/bundle/programs/server/npm/node_modules/sharp
> node-gyp rebuild

make: Entering directory '/bundle/bundle/programs/server/npm/node_modules/sharp/build'
  TOUCH Release/obj.target/libvips-cpp.stamp
  CXX(target) Release/obj.target/sharp/src/common.o
  CXX(target) Release/obj.target/sharp/src/metadata.o
  CXX(target) Release/obj.target/sharp/src/operations.o
  CXX(target) Release/obj.target/sharp/src/pipeline.o
  CXX(target) Release/obj.target/sharp/src/sharp.o
  CXX(target) Release/obj.target/sharp/src/utilities.o
  SOLINK_MODULE(target) Release/obj.target/sharp.node
g++: error: ../lib/libvips-cpp.so: No such file or directory
g++: error: ../lib/libvips.so: No such file or directory
g++: error: ../lib/libglib-2.0.so: No such file or directory
g++: error: ../lib/libgobject-2.0.so: No such file or directory
g++: error: ../lib/libcairo.so: No such file or directory
g++: error: ../lib/libcroco-0.6.so: No such file or directory
g++: error: ../lib/libexif.so: No such file or directory
g++: error: ../lib/libffi.so: No such file or directory
g++: error: ../lib/libfontconfig.so: No such file or directory
g++: error: ../lib/libfreetype.so: No such file or directory
g++: error: ../lib/libgdk_pixbuf-2.0.so: No such file or directory
g++: error: ../lib/libgif.so: No such file or directory
g++: error: ../lib/libgio-2.0.so: No such file or directory
g++: error: ../lib/libgmodule-2.0.so: No such file or directory
g++: error: ../lib/libgsf-1.so: No such file or directory
g++: error: ../lib/libgthread-2.0.so: No such file or directory
g++: error: ../lib/libharfbuzz.so: No such file or directory
g++: error: ../lib/libjpeg.so: No such file or directory
g++: error: ../lib/liblcms2.so: No such file or directory
g++: error: ../lib/liborc-0.4.so: No such file or directory
g++: error: ../lib/libpango-1.0.so: No such file or directory
g++: error: ../lib/libpangocairo-1.0.so: No such file or directory
g++: error: ../lib/libpangoft2-1.0.so: No such file or directory
g++: error: ../lib/libpixman-1.so: No such file or directory
g++: error: ../lib/libpng.so: No such file or directory
g++: error: ../lib/librsvg-2.so: No such file or directory
g++: error: ../lib/libtiff.so: No such file or directory
g++: error: ../lib/libwebp.so: No such file or directory
g++: error: ../lib/libxml2.so: No such file or directory
g++: error: ../lib/libz.so: No such file or directory
sharp.target.mk:179: recipe for target 'Release/obj.target/sharp.node' failed
make: Leaving directory '/bundle/bundle/programs/server/npm/node_modules/sharp/build'
make: *** [Release/obj.target/sharp.node] Error 1
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:211:12)
gyp ERR! System Linux 4.4.0-36-generic
gyp ERR! command "/usr/bin/nodejs" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /bundle/bundle/programs/server/npm/node_modules/sharp
gyp ERR! node -v v4.6.2
gyp ERR! node-gyp -v v3.4.0
gyp ERR! not ok 

npm ERR! Linux 4.4.0-36-generic
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "rebuild" "--update-binary"
npm ERR! node v4.6.2
npm ERR! npm  v4.0.2
npm ERR! code ELIFECYCLE
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script 'node-gyp rebuild'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the sharp package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs sharp
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls sharp
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /bundle/bundle/programs/server/npm/npm-debug.log

npm WARN [email protected] No description
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.
npm ERR! Linux 4.4.0-36-generic
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "install" "--unsafe-perm"
npm ERR! node v4.6.2
npm ERR! npm  v4.0.2
npm ERR! code ELIFECYCLE
npm ERR! [email protected] install: `node npm-rebuild.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script 'node npm-rebuild.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the meteor-dev-bundle package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node npm-rebuild.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs meteor-dev-bundle
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls meteor-dev-bundle
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /bundle/bundle/programs/server/npm-debug.log

[lovell]

Hello, are you using sharp v0.16.2 as served from npm or your modified fork?

Ubuntu 16.04 provides libvips v8.2.2 but sharp v0.16.2 requires v8.3.3 so there's no need to install the libvips-dev package as sharp will download the most appropriate version of own binaries.

[cspecter]

Thanks for the quick reply. I'm using your v0.16.2 package, not my modified one. I've tried the Dockerfile with and without libvips-dev but I still get the same error. If I manually deploy my app without using Docker into a Ubuntu 16.04 instance everything installs fine, but I'd prefer to use Docker as it simplifies my deployment process. Something must be going on with Docker and node-gyp.

[lovell]

Have you run meteor npm install in /bundle/bundle/programs/server?

[cspecter]

Same result. Also, Meteor automatically runs npm rebuild on launch. I have a couple other binary libraries like bcrypt, which build fine, but any that try to download source like sharp fail.

[cspecter]

Ok. For now I'm just going to cut out Docker and deploy right to my ubuntu server. I like Docker because I can quickly scale across multiple servers, but it's not really needed for this app. Definitely something going on within Docker though that is preventing various NPMs from downloading additional source and building properly. I'm game to try out any fixes if others are having this issue.

[lovell]

I notice you're globally installing some modules so this might be a root vs non-root thing. The --unsafe-perm flag can help here, e.g. npm install --unsafe-perm.

[cspecter]

OK, I think this is a Meteor problem and not anything to do with your packages. Meteor runs it's own rebuild script when deploying and it is having problems either with the relative paths or downloading the source. I'm going to post over there and see if I can get any insight. I was able to manually run npm i sharp inside my Docker instance with no problems, but when Meteor starts up it thinks that it needs to rebuild the binary packages and fails.

[lovell]

@cspecter Thanks for the update. I'll close this for now but please feel free to re-open if you don't get the answers you need from the Meteor maintainers.

[cspecter]

@lovell Sounds good. Actually it looks like it is a node-gyp problem. You can recreated it by running npm install --save sharp on a dev system, like a MacOS, then move over the files to an Ubuntu system, then run node-gyp rebuild and it will fail.

[lovell]

Using npm run clean within node_modules/sharp (to remove the arch-specific vendor/ directory and friends) before node-gyp rebuild might work.

[cspecter]

As per a suggestion from @abernix, if I removed the /node_modules/sharp/lib directory (which contained macOS-compiled .dylib files) and ran node-gyp rebuild again it seemed to work, so I have a workaround, but it still needs a fix in node-gyp or it needs to be taken into account in Meteor. Thanks for the help!

[abernix]

@lovell When I was triaging this within Meteor, I believe I found that npm run clean didn't remove the lib directory (and I'm not saying that I think it should!) so the problem would still exist.

Ultimately outside of Meteor the same problem exists if you npm install sharp on OS X, move the app to Ubuntu and run node-gyp rebuild. This to me indicates a problem as that seems like the entire purpose of node-gyp rebuild. I'm not sure that there was anything sharp could do about this, but I wasn't familiar enough with node-gyp to know what it should/should-not do (slash be capable of).

[lovell]

@abernix Hello, to use the pre-compiled libvips and dependencies provided by sharp, you'll need to run npm install (or npm rebuild) on the runtime platform. The node_modules/sharp/vendor directory is where the platform-specific binaries are stored, so this is the one to remove when cross-compiling (this is what the clean script does).

Future versions of sharp may check the platform matches - see #186 for this.

[abernix]

Thanks for your response, @lovell. I fully understand the need to run npm install / npm rebuild on the runtime platform. As stated above, npm rebuild is being called (which calls node-gyp rebuild behind the scenes). Unfortunately, npm rebuild fails because of the vendor/ directory being there. Yes, I've discovered (through trial and error) that removing the vendor/ directory allows the npm rebuild to finish (which recreates the vendor/ directory), but this breaks the normal workflow of using npm rebuild.

If the vendor directory that sharp created was created inside the build directory, then it would be automatically cleaned up by the clean step (powered by node-gyp) of npm rebuild, rather than needing to rely on an external npm run clean script which node-gyp does not respect.

Do you not agree that the following should work without additional steps?:

Starting on macOS

$ mkdir temp-node
$ cd temp-node
$ npm init -y
$ npm install sharp --save
$ cd ..
$ # copy temp-node directory to Ubuntu system

Continuing on Ubuntu

$ cd temp-node # same directory, same untouched state as previously
$ npm rebuild # This should not fail!

[abernix]

Also, I see you edited this to include "Meteor" in the subject, but I'll re-clarify this has nothing to do with Meteor.

[lovell]

Created #861 to improve things.