Sharp: Provide pre-built "sharp.node" binary for common platforms and Node versions

[mikemorris]

https://github.com/mapbox/node-pre-gyp

[lovell]

Hi Mike, node-pre-gyp looks interesting, thank you for the suggestion.

From scanning the mapnik examples, it looks like it statically links dependencies into a single pre-compiled .node binary for each combination of architecture, platform and node version.

Would we also need to build and provide statically compiled versions of libvips and its dependants first?

Is there any licensing impact to consider when static linking both LGPL and GPL2 dependencies with Apache 2.0 sources.

I'd be happy to accept a PR that integrates with node-pre-gyp.

The existing preinstall.sh script could be used to populate a VM/container environment in which to build a statically linked version of libvips.

I'm able to provide an S3 bucket to host pre-compiled files in, if that's the easiest way to achieve this.

There are plenty of people to help test this amongst the contributors to the "meaning of life" issue that is #42.

[lovell]

Handy guide to using various CI providers as the build infrastructure for compiling binaries:
http://cylonjs.com/blog/2014/11/19/creating-multiplatform-precompiled-binaries-for-node-modules/

[lovell]

NPM's "Roadmap area of focus: native modules" is relevant here.

[lovell]

Most of my comment https://github.com/lovell/sharp/issues/186#issuecomment-88019499 about static linking is no longer relevant as sharp ships with pre-built shared libraries that can be used to generate a sharp.node shared library.

Windows should probably be the priority/test case for this work.

[lovell]

I'm going to investigate the use of prebuild-ci and (a possibly-modified version of) prebuild-install.

The libvips-downloading logic in the binding.* files will need to be extracted as node-gyp build may no longer be run.

[mpseidel]

👍 - building sharp everytime in our docker builds has become a bottleneck

[jingsam]

Installing sharp is painful and error prone because of rebuilding from source. Providing precompiled binaries would alleviate this pain.

[jeremyfa]

Thanks for looking at this! _sharp_ is excellent and deserves to be as easy as possible to install 👍

[appjitsu]

+1 - id love to use this for our purposes, but not worth it because node-gyp is involved.

[ArmorDarks]

Worth to check out https://github.com/sass/node-sass and see how they solved similar issue. They shipping compiledlibsass for various platforms too, and, thankfully, on Windows and other popular platforms it doesn't require nor node-gyp, nor anything related to it (unless you want to build on your own machine).

For many Node projects the drop of node-gyp and Python will be a huge step forward, for sure.

[nanom1t]

Providing precompiled binaries would be awesome. Maybe there is a way to divide sharp into 2 parts: npm package and .deb/.rpm package in such way as graphicsmagick does:

sudo apt-get install graphicsmagick

[ArmorDarks]

npm package and .deb/.rpm package in such way as graphicsmagick does:

Unfortunately, it will solve issue only for unix users, but not for Windows ones.

[ghost]

I highly recommend looking into prebuild, it is the easiest way to provide prebuilt binaries of native node addons.

node-serialport's CI configs (appveyor.yml and .travis.yml) are great examples of how to set up prebuild in CI.

I also recently opened a PR to node-usb-detection (https://github.com/MadLittleMods/node-usb-detection/pull/47) to add prebuild support. This PR gives a nice clear outline of the whole process for getting prebuild up and running.

[lovell]

Quick update for this popular issue: I've successfully started using (and contributing patches to) prebuild with other native modules and agree it provides what we need. Before this can happen there is work to be done to separate the logic in the binding.gyp/binding.js files that deals with downloading libvips binaries from any compilation steps.

[lovell]

An organisation that uses sharp has offered to sponsor this task so expect some progress and requests to help test, especially on Windows, soon.

[lovell]

Work-in-progress branch at https://github.com/lovell/sharp/tree/prebuild

• [x] Move pkgconfig-based libvips detection logic from binding.gyp to JS
• [x] Move download logic from binding.gyp to JS
• [x] Move Windows-only DLL copy logic from binding.gyp to JS
• [x] Integrate with prebuild-ci
• [x] Integrate with prebuild-install
• [x] Document
• [x] Spend more time on Windows support
• [x] Test, test, test

[elijahbee]

Can't wait for this, building from source is rather problematic

Failed at the [email protected] install script 'node-gyp rebuild'.

[lovell]

Should anyone be able to help test this, especially on Windows, please try running the following in a newly-created temporary directory.

npm install lovell/sharp#prebuild

yarn add lovell/sharp#prebuild

This should download and use pre-compiled libvips and sharp binaries. Please note that git needs to be available as this method is installing via GitHub but won't be required when published via npm.

If a globally-installed copy of libvips is detected then it will intentionally fallback to building sharp from source.

If it works, please +1 this comment.

If it doesn't work, please add a comment with full details of the platform used, all relevant version numbers and a link to a Gist with the install log.

[lovell]

Success on Windows, finally.

Here's the log when installing on a fresh Windows machine without MSVC etc.

C:\Temp>node -v
v9.5.0

C:\Temp>npm install lovell/sharp#prebuild

> [email protected] install C:\Temp\node_modules\sharp
> (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)

info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.6.1/libvips-8.6.1-win32-x64.tar.gz
info sharp Creating C:\Temp\node_modules\sharp\build\Release
info sharp Copying DLLs from C:\Temp\node_modules\sharp\vendor\lib to C:\Temp\node_modules\sharp\build\Release
prebuild-install info begin Prebuild-install version 2.5.0
prebuild-install info looking for local prebuild @ prebuilds\sharp-v0.20.0-prebuild.test.1-node-v59-win32-x64.tar.gz
prebuild-install info looking for cached prebuild @ C:\...\AppData\Roaming\npm-cache\_prebuilds\https-github.com-lovell-sharp-releases-download-v0.20.0-prebuild.test.1-sharp-v0.20.0-prebuild.test.1-node-v59-win32-x64.tar.gz
prebuild-install info found cached prebuild
prebuild-install info unpacking @ C:\...\AppData\Roaming\npm-cache\_prebuilds\https-github.com-lovell-sharp-releases-download-v0.20.0-prebuild.test.1-sharp-v0.20.0-prebuild.test.1-node-v59-win32-x64.tar.gz
prebuild-install info unpack resolved to C:\Temp\node_modules\sharp\build\Release\sharp.node
prebuild-install info unpack required C:\Temp\node_modules\sharp\build\Release\sharp.node successfully
prebuild-install info install Successfully installed prebuilt binary!

+ [email protected]
added 70 packages in 43.907s

[xcb6]

win7, node -v (v8.9.3) => install ok
thanks

[MatousLeadPages]

win10, node -v (v8.9.4) => works great

[KyleAMathews]

Tested on mac and it worked perfectly (well... once I deleted the global libvips install I forgot I had :-))

[kolpav]

win10, v9.6.0, works perfectly

[Prozi]

Win10 finally without problems <3

[pronebird]

Does one still need to compile libvips for prebuilt to work?

[lovell]

@pronebird sharp has provided a prebuilt libvips since v0.15.0 for most platforms.

"If a globally-installed copy of libvips is detected then it will intentionally fallback to building sharp from source." https://github.com/lovell/sharp/issues/186#issuecomment-362691105

So if for whatever reason you are still compiling libvips and installing it globally then the prebuilt sharp has to be ignored and instead will be compiled against and linked with the global libvips version.

[pronebird]

@lovell I wasn't aware of this.

Not that long ago I started seeing this error on my heroku-16 instance:

Error: libvips-cpp.so.42: cannot open shared object file: No such file or directory

ldd emits:

ldd node_modules/sharp/build/Release/sharp.node  on ⬢ staging... up, run.7438 (Free)
    linux-vdso.so.1 =>  (0x00007ffd11f85000)
    libvips-cpp.so.42 => not found
    libvips.so.42 => not found
    libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007fc9484d1000)
    libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007fc94827e000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc947efc000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc947bf3000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc9479dd000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc947613000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fc9473a3000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc947186000)
    libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fc946f7e000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fc948a1b000)

I think what happens is that node_modules/sharp/vendor/lib is being cleaned up by the heroku-postbuild script that I have, such as

yarn install --production=false && yarn run build && yarn install --production=true

Since I do some compilation on server I install dev dependencies first then compile and finally clean them up with yarn install --production=true. But that doesn't seem to be the case when using npm instead of yarn. Also I can't reproduce this locally on my mac.

[lovell]

The prebuild branch has now been merged to the master branch and will shortly be released as v0.20.0.

npm install lovell/sharp

yarn add lovell/sharp

[pronebird]

@lovell does it change anything in terms of configuration or what's the benefit?

[lovell]

@pronebird The primary benefit of providing prebuilt sharp.node binaries is that the current Python and C++ compiler install-time dependencies will no longer be required for most users, making installation simpler and easier.

The npm install --build-from-source flag (a "standard" used by both node-pre-gyp and prebuild-install) will be respected if you still wish to compile from source.

[pronebird]

@lovell ah right so it was still compiling sharp even though vips binaries were prebuilt. 👍

[lovell]

v0.20.0 is now available, thanks everyone for feedback/testing, and a big thank you to the sponsor of this work (that wishes to remain anonymous for now).