Sharp: Packaging sharp with nexe as painlessly as possible

[jaubourg]

Hey @lovell :)

I'm currently using nexe to package a command line utility that has sharp as a dependency.

Since native modules cannot be integrated into the final executable, my build script copies sharp.node into a sub-directory next to the executable ("./sharp/"). It also generates an adapter that properly locates this copy of sharp.node from within the executable.

It works perfectly under windows. Of course, the build script has to copy the plethora of dlls that comes with sharp, but they're simply put into the same directory as the copy of sharp.node.

However, in OSX, the .dylib files are (strangely) not in the same directory as sharp.node when sharp is built (but in a "cousin"-directory). So when I copy those files into the new sub-directory next to the packaged executable, I logically get an error about @rpath/somelib.number.dylib not being found.

So, I'm wondering...

• why the inconsistency between windows and OSX ? Can't the dylibs be in the same directory as sharp.node, and be referenced from sharp.node as such, so that they could be copied/moved together more easily?
• should I expect the same issue with .dso files in linux?

I'd very much like not to have to recreate the whole node_modules hierarchy just to copy dynamic libraries if at all possible. Right now, the "copy the .node file together with dynamic libraries in the same directory" behavior is triggered by nexe itself while recursively exploring require statements (so I __know__ for a fact a native module found that way actually is a runtime dependency and not a dev dependency). Also, I don't have to worry about the "depth" of the dependency if I keep it flat (whether it's in a node_module within a node_module within a node_module or not makes no difference whatsoever). I'm sure you get the idea.

Any help, pointer, workaround would be deeply appreciated.

Take care and thanks again for the herculean work you pull on sharp on a daily basis,

Julian

[lovell]

Bonsoir, Windows doesn't support rpath, hence all the .dll files have to be copied to live alongside the sharp.node shared library.

Linux and OS X do support rpath, so sharp.node is built to look for shared libraries in the ../../vendor/lib location relative to it (e.g. node_modules/sharp/vendor/lib). You'll probably need to either ensure this location exists relative to sharp.node or modify the paths in binding.gyp before building.

[jaubourg]

Bonsoir @lovell :)

Got it. But my question really is: what is the advantage / interest of having sharp.node look for ../../vendor/lib? Is it standard for native modules under OS X and Linux or is it just arbitrary? How is it bad practice to put the dynamic libraries inside the same directory as the .node?

I mean, pointing to a sub-directory inside the relative grand-parent directory means someone moving the built lib around has to put the .node inside a directory inside a directory (both of whatever names, really) just so that they can create vendor/lib (name mandatory this time) to store referenced libs. I admit I fail to see the gain in flexibility @rpath is supposed to bring here ;)

If I understand correctly, one cannot dynamically change @rpath? So I'd need to fork sharp if I wanted to change the relative bindings in bindings.gyp, correct?

And thanks for the quick reply, always a pleasure interacting with you =)

[lovell]

The build/ directory gets deleted and recreated every time npm install (or similar) is run so we need to put the downloaded binaries somewhere else.

Having a vendor/ directory for third party dependencies is a relatively common pattern I've seen used elsewhere (e.g. golang).

Rather than have to copy ~7MB of files from vendor/ into build/ every time npm install is run, it makes more sense to link directly to them in their vendor/ location if possible, hence the use of rpath where available.

In terms of packaging for the nexe platform, perhaps look at tools like chrpath -r (Linux) and install_name_tool -rpath (OS X) to modify the rpath within the compiled sharp.node binary.

[jaubourg]

Thanks for pointing to these tools! I admit I'm a total newb when it comes to those platforms and their build tools.

I'll make some tests and I'll report back with my findings (could be useful to others).

[patrickhulce]

Hey there @lovell, love the project thanks for all your hard work! :)

I'm running into similar struggles bundling up sharp with pkg, specifically, the node_modules/sharp/vendor/lib directory doesn't exist on my machine for some reason and I can't figure out why. Everything works on travis and on a coworker's machine when he cloned my repo, but locally I can't package up the exe without that folder :/ any pointers on when this folder gets downloaded and when it doesn't or how to force a download?

[lovell]

@patrickhulce Hello, when npm install (and npm rebuild) are run, npm runs node-gyp with the binding.gyp config. It uses pkg-config to search for a globally-installed libvips. If not found, it will invoke the download_vips() function from binding.js to populate the vendor directory with the relevant libvips binaries for the detected platform.

[jaubourg]

FYI I ended up reproducing the hierarchy (keeping only .dll and .node files) than pruning the build location. Not ideal but I can live with it. A compact, ready to incorporate, version would be cool.

Anyway, sorry for the late feedback ;)