Sharp: Lambda: can't install with static libvips

[kevinchn]

I need to compile libvips since the bundled one with sharp doesn't support bmp, and I want to run this inside aws lambda so I am compiling libvips as a static library. This is done in the amazonlinux docker container.

I ran into 2 problems.
1) During npm install

bash-4.2# PKG_CONFIG_PATH=~/vips/lib/pkgconfig/ npm install --runtime_link=static -prefix=~/sharp_test --unsafe-perm sharp
-
> [email protected] install /root/sharp_test/node_modules/sharp
> node-gyp rebuild

readelf: Error: '/root/vips/lib/libvips-cpp.so': No such file
gyp: Call to 'if readelf -Ws "$(PKG_CONFIG_PATH=":$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig:/usr/lib/pkgconfig" pkg-config --variable libdir vips-cpp)/libvips-cpp.so" | c++filt | grep -qF __cxx11;then echo "1";else echo "0";fi' returned exit status 0 while in binding.gyp. while trying to load binding.gyp

These are the flags I used to install vips
./configure --enable-static --disable-shared --prefix=$HOME/vips
I got around this by removing --disable-shared
Does sharp really need both shared and static libraries of libvips?

2) After that I got another error during npm install

bash-4.2# PKG_CONFIG_PATH=~/vips/lib/pkgconfig/ npm install --runtime_link=static -prefix=~/sharp_test --unsafe-perm sharp
-
> [email protected] install /root/sharp_test/node_modules/sharp
> node-gyp rebuild

make: Entering directory `/root/sharp_test/node_modules/sharp/build'
  TOUCH Release/obj.target/libvips-cpp.stamp
  CXX(target) Release/obj.target/sharp/src/common.o
  CXX(target) Release/obj.target/sharp/src/metadata.o
  CXX(target) Release/obj.target/sharp/src/operations.o
  CXX(target) Release/obj.target/sharp/src/pipeline.o
  CXX(target) Release/obj.target/sharp/src/sharp.o
  CXX(target) Release/obj.target/sharp/src/utilities.o
  SOLINK_MODULE(target) Release/obj.target/sharp.node
/usr/bin/ld: cannot find -ljbig
/usr/bin/ld: cannot find -lffi
collect2: error: ld returned 1 exit status
make: *** [Release/obj.target/sharp.node] Error 1
make: Leaving directory `/root/sharp_test/node_modules/sharp/build'

[kevinchn]

Actually I created a fresh docker container, did everything again, and that error didn't come up this time. However, I am now getting a run time error

Error: libvips-cpp.so.42: cannot open shared object file: No such file or directory
    at Error (native)
    at Object.Module._extensions..node (module.js:434:18)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/root/sharp_test/node_modules/sharp/lib/constructor.js:8:15)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)

Which seems to suggest sharp is still using the shared libvips library and not using the static library?
I used the same flags when installing as above
vips:
./configure --enable-static --prefix=$HOME/vips
sharp:
PKG_CONFIG_PATH=~/vips/lib/pkgconfig/ npm install --runtime_link=static -prefix=~/sharp_test --unsafe-perm sharp

[lovell]

As you've seen, the current setup is unlikely to work with static linking.

Is there something that prevents the use of shared libraries in Lambda? You should be able to set the LD_LIBRARY_PATH environment variable to their location.

http://docs.aws.amazon.com/lambda/latest/dg/env_variables.html

[kevinchn]

I was following this guide and was under the impression that I must use static libraries: https://aws.amazon.com/blogs/compute/nodejs-packages-in-lambda/
I'll try shared libraries today and report back.
Thanks!

[lovell]

@listonb Your AWS blog article may need a small update to reflect the recent(ish) ability to use environment variables with Lambda.

[kevinchn]

I got it working locally with shared libraries, but I'm getting an error when it runs on AWS lambda

module initialization error: Error
at Error (native)
at Object.Module._extensions..node (module.js:434:18)
at Module.load (module.js:343:32)
at Function.Module._load (module.js:300:12)
at Module.require (module.js:353:17)
at require (internal/module.js:12:17)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/constructor.js:8:15)
at Module._compile (module.js:409:26)
at Object.Module._extensions..js (module.js:416:10)
at Module.load (module.js:343:32)
at Function.Module._load (module.js:300:12)
at Module.require (module.js:353:17)
at require (internal/module.js:12:17)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/index.js:3:15)
at Module._compile (module.js:409:26)
at Object.Module._extensions..js (module.js:416:10)

Which looks like something inside build/Release/sharp.node
I'm not really sure what this means. Any ideas?

Nevermind, AWS didn't show the specific error for some reason. I had to use the "Test" button to see it. It seems like libwebp and libexif couldn't be found even though Amazon listed them in the list of packages (https://aws.amazon.com/amazon-linux-ami/2016.03-packages/). I added those inside my zip and it works now.
Thanks!

[lovell]

Great to hear you got it working, thanks for confirming.

[johnrjj]

@kevinchn -- Running into the same issue here w/ Lambda. (the same stacktrace of module initialization error: Error etc...)

If you don't mind answering, what was the process for packaging/linking the two dependencies in your zip payload?

[kevinchn]

@johnrjj
1) I built and installed vips normally (should see libvips shared libraries in /usr/local/lib/)
2) npm installed sharp with the environment variable PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
3) copied /usr/local/lib/libvips.so.42 and libvips-cpp.so.42 into the "lib" folder of the zip (Lambda searches this folder for libraries at runtime)

Hope that helps.

[tayfun]

Thanks @kevinchn for the guidance. I saw the exact same module initialization error on Cloudwatch but AWS was not verbose about what was causing it. After testing on Lambda console, I found some libraries were missing. For bmp support, I did the following:

1. Used the following docker file to build vips with ImageMagick so bmp is supported: https://gist.github.com/tayfun/5c72763b8b30bb73fdfeee3896cdc3ea
2. Used the following makefile for first building the docker image, installing Sharp on the docker container and copying related libraries and then packaging them to a zip file to be uploaded to AWS Lambda: https://gist.github.com/tayfun/4e6535e6627987769a45202c7c0b191a

Hope this helps someone.

[ismailqau]

Worked for me:

1. Build and installed normally (with libpoppler support)
2. copied libvips.so.42 and libvips-cpp.so.42 from /usr/local/lib/ to /usr/lib/