Pkg: Requiring sharp

[sebastianarena]

Hello great team at zeit!
I'm having a very difficult time trying to get the following module to work: https://github.com/lovell/sharp

It seems the library uses quite a lot of dynamic loading, and also compiles a .node.
I think I advanced a little putting scripts and assets into pkg config, but in the end, I get a runtime error:

RangeError: Index out of range at checkOffset (buffer.js:821:11) at Buffer.readInt32LE (buffer.js:974:5) at evalmachine.<anonymous>:0 at evalmachine.<anonymous>:0 at Object.Module._extensions..node (evalmachine.<anonymous>:0) at Module.load (module.js:487:32) at tryModuleLoad (module.js:446:12) at Function.Module._load (module.js:438:3) at Module.require (module.js:497:17) at Module.require (evalmachine.<anonymous>:0) at require (internal/module.js:20:19) at Object.console./snapshot/project/node_modules/sharp/lib/constructor.js.dev (evalmachine.<anonymous>:0)

Here's what I added into pkg:

scripts: [
            'node_modules/sharp/lib/**/*.js',
            'node_modules/sharp/binding.js'
]

assets: [
            'node_modules/sharp/lib/icc/**/*',
            'node_modules/sharp/vendor/**/*',
            'node_modules/sharp/binding.gyp'
]

Any help would be appreciated!

[sebastianarena]

After a bit of digging in the sharp library, it turns out the problem is generated by requiring the sharp.node itself.
I believe I'm on the right path since if I run the executable without the sharp.node in the same folder, it shows a missing module error.

I compiled a version commenting out the requiring of sharp.node inside sharp and it clearly moves forwards, still crashes obviously, but not with the checkOffset issue.

So there something wrong with the following line in node_modules/sharp/lib/constructor.js

const sharp = require('../build/Release/sharp.node');

As I understand it the .node might require external libraries that sharp provides (such as libvips), which I've added as assets in pkg, but it seems without luck.

[igorklopov]

This is your case: https://github.com/zeit/pkg/issues/104
Feel free to reopen if not.

[sebastianarena]

Hi.
Thanks for replying.
I've also reached out to the good people at sharp about this.And they referred me back to the same issue.

I'm only testing locally on MacOS, and it is not working.
I tried specifying the target to be macos, and keeping the test within macos.

I suspected of yarn, and moved back to npm to compile the .node, but nothing changed.
I'm suspecting of nvm now, which is what I use to switch node versions, but I don't think that is it.

I understand you are still working for proper .node support, so maybe you won't bother on this.
Use me to test whatever you need. It seems I reached a case that is not so easy to solve :(

Best from us!

[sebastianarena]

@igorklopov hi! I have an update on this!
I tried the new pkg 4.0.0, and when running it it shows a better error description, and now can see why the sharp.node object is not loaded.

Error: dlopen(/build/sharp.node, 1): Library not loaded: @rpath/libvips-cpp.42.dylib Referenced from: /build/sharp.node Reason: image not found

I tried putting the dylib in the same directory as the .node without luck.

Any ideas?

[sebastianarena]

@igorklopov also, I joined the Zeit slack, you can find me under @sarena if you want to reach out with questions or tests you want me to do, anything I can do to help!

[sebastianarena]

@igorklopov I just finally managed to make this work.
I'm leaving this comment for other people that might run into this issue.

My best solution for using sharp, was to make a folder named "modules" next to the executable that pkg makes. Inside that, copy the entire sharp that you find in your node_modules, which has a compiled sharp.node for the platform you are running (if you need to build on a platform, and run on another, I don't think its going to work as of [email protected]). From there, pkg requires that the .node files are located in the same folder as the final executable. But since sharp need a specific folder structure, I just did a symbolic link pointing to the proper location. The structure is left with something like this:

/build/your_executable_from_pkg
/build/sharp.node -> /build/modules/sharp/build/Release/sharp.node
/build/modules/sharp (entire folder that is left in node_modules after npm install)

This might be a suitable solution for other people that have .node files that depend on external libraries. Copying the .node file is just not enough.

[thisjeremiah]

Thanks @sebastianarena! Your solution worked for me on macos, and I'm guessing it works on linux as well. For those who are trying to achieve this on Windows, I was able to get something like this to work (without symlinks):

/build/your_executable_from_pkg
/build/sharp.node
/build/lib*.dll (~50 of these)

I grabbed sharp.node and all ~50 of the lib*.dll files exclusively from the node_modules/sharp/build/Release folder after installing sharp via npm on Windows, using the version of Node I was targeting.

Hope that helps someone else!

[soptimizer]

@igorklopov I just finally managed to make this work.
I'm leaving this comment for other people that might run into this issue.

My best solution for using sharp, was to make a folder named "modules" next to the executable that pkg makes. Inside that, copy the entire sharp that you find in your node_modules, which has a compiled sharp.node for the platform you are running (if you need to build on a platform, and run on another, I don't think its going to work as of [email protected]). From there, pkg requires that the .node files are located in the same folder as the final executable. But since sharp need a specific folder structure, I just did a symbolic link pointing to the proper location. The structure is left with something like this:

/build/your_executable_from_pkg
/build/sharp.node -> /build/modules/sharp/build/Release/sharp.node
/build/modules/sharp (entire folder that is left in node_modules after npm install)

This might be a suitable solution for other people that have .node files that depend on external libraries. Copying the .node file is just not enough.

Hi, thanks for your sharing. But, could you give us detailed information about symbolic link pointing with example?

[MichaelGatesDev]

Unfortunately, @thisjeremiah' solution does not work for me.

Something went wrong installing the "sharp" module

libvips-cpp.so.42: cannot open shared object file: No such file or directory

- Remove the "node_modules/sharp" directory, run "npm install" and look for errors
- Consult the installation documentation at https://sharp.pixelplumbing.com/en/stable/install/
- Search for this error at https://github.com/lovell/sharp/issues

Anyone have any ideas? I have to remain multiplatform, at least for both Windows and Linux.

[shyser]

This is my solution for egg server. Hope it helps.
build.js

const fs = require('fs');
const path = require('path');
const fse = require('fs-extra');
const child_process = require('child_process');

const DestDir = path.join(__dirname, 'pkg-dist');
const BEFORE = 'before';
const AFTER = 'after';

(async function main() {
  if (!fs.existsSync(DestDir)) fs.mkdirSync(DestDir);
  else fse.emptyDirSync(DestDir);
  await exec({cmd: 'npm', args: ['run', 'tsc']});
  handleModuleSharp(BEFORE);
  handlePackage(BEFORE);
  await exec({cmd: 'pkg', args: ['.', '--targets', 'node10-linux-x64', '--out-path', DestDir, '--debug']});
  handlePackage(AFTER);
  handleModuleSharp(AFTER);
  handleAssets();
})();

function handlePackage(arg) {
  const pkgName = path.join(__dirname, 'package.json');
  const pkgInfo = JSON.parse(fs.readFileSync(pkgName));
  pkgInfo.egg.typescript = arg === BEFORE ? undefined : true;
  fs.writeFileSync(pkgName, JSON.stringify(pkgInfo, null, 2) + '\n');
};

function handleAssets() {
  const srcFiles = [
    path.join(__dirname, 'bin'),
  ];
  srcFiles.forEach(filepath => {
    fse.copySync(filepath, path.join(DestDir, path.basename(filepath)), {recursive: true});
  });
};

function handleModuleSharp(arg) {
  const pkgName = path.join(__dirname, 'package.json');
  const pkgInfo = JSON.parse(fs.readFileSync(pkgName));
  if (!pkgInfo.dependencies.sharp) return;

  const moduleSharpDir = path.join(__dirname, 'node_modules/sharp');
  const keyStr = {
    raw: "require('../build/Release/sharp.node')",
    replaced: "require(require('path').join(process.cwd(), 'sharp/build/Release/sharp.node'))",
  };
  const targetDir = path.join(moduleSharpDir, 'lib');
  fs.readdirSync(targetDir).forEach(filename => {
    if (!filename.endsWith('.js')) return;
    const filepath = path.join(targetDir, filename);
    let content = fs.readFileSync(filepath, {encoding: 'utf8'});
    if (arg === BEFORE) content = content.replace(keyStr.raw, keyStr.replaced);
    else if (arg === AFTER) content = content.replace(keyStr.replaced, keyStr.raw);
    fs.writeFileSync(filepath, content);
  });
  if (arg === AFTER) {
    fse.copySync(path.join(moduleSharpDir, 'build'), path.join(DestDir, 'sharp/build'), {recursive: true});
    fse.copySync(path.join(moduleSharpDir, 'vendor/lib'), path.join(DestDir, 'sharp/vendor/lib'), {recursive: true});
  };
};

async function exec(params) {
  const cwd = params.cwd || process.cwd();
  const ok = await new Promise((r) => {
    const cmd = child_process.spawn(params.cmd, params.args, { cwd });
    cmd.stderr.pipe(process.stderr);
    cmd.stdout.pipe(process.stdout);
    cmd.on('close', (code) => {
      const ok = code === 0;
      if (!ok) console.log(`${JSON.stringify(params)}failed,code${code}`);
      r(ok);
    });
  });
  return ok;
};

[einsqing]

@StephenSeraph 实际测试不能使用了

[einsqing]

解决了