Pkg: Error spawn ENOENT when an imported module inside the application spawns an *.exe

Created on 2 Feb 2018 · 13Comments · Source: vercel/pkg

Hi @igorklopov ,

I am using desktop-screenshot in my application.
When I packaged my application, pkg couldn't include the desktop-screenshot js files as it has a non-literal argument in require. I have included it in my application package.json as

"pkg": { "assets": [ "node_modules/desktop-screenshot/capture/bin/scrot/scrot", "node_modules/desktop-screenshot/capture/bin/nircmd.exe" ], "scripts": [ "node_modules/desktop-screenshot/capture/*.js" ] }
From pkg build logs, I found that it included everything.

This line of code from desktop-screenshot(/capture/win32.js)

var nircmd = childProcess.spawn(path.join(__dirname, "bin", "nircmd.exe"), ["savescreenshot", options.output]);

fails in error

Error: spawn C:\snapshot\myapp\node_modules\desktop-screenshot\capture\bin\nircmd.exe ENOENT
at _errnoException (util.js:1024:11)
at Process.ChildProcess._handle.onexit (internal/child_process.js:190:19)
at onErrorNT (internal/child_process.js:372:16)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickDomainCallback (internal/process/next_tick.js:218:9)

I am facing the same issue with another dependency dialog-node as well. Looks like there is an issue in the way the paths are handled from inside the executable.

@bat-tomr is helping for a workaround on the same issue #1

Can you please help me out in solving this?

Source

rohit5ram

👍1

Most helpful comment

Hey guys, I just found a workaround here. In order to make syscalls(like spawn) work on pkg.js' snapshot filesystem, we can just copy our assets to disk, then access them outside of snapshot filesystem:

const source = path.join(__dirname, 'foo.exe');
const target = path.join('somewhere/foo.exe');

fs.copyFileSync(source, target); // ENOENT
child_process.spawn(target);

However, this will not work because fs.copyFileSync() made another syscall: copyfile, it cannot find source inside of snapshot filesystem.

But, instead of using copyFileSync, we can read assets into memory then write them into disk:

const fs = require('fs');
const utils = require('util');

const copyFile = utils.promisify(fs.copyFile);
const chmod = utils.promisify(fs.chmod);

async function copy(source, target) {
  if (process.pkg) {
    // use stream pipe to reduce memory usage
    // when loading a large file into memory.
    fs.createReadStream(source).pipe(fs.createWriteStream(target));
  } else {
    await copyFile(source, target);
  }
}

const source = path.join(__dirname, 'foo.exe');
const target = path.join('somewhere/foo.exe');

await copy(source, target); // this should work
await chmod(target, 0o765); // maybe need to grant execute permission

child_process.spawn(target);

micooz on 25 Feb 2018

👍8

All 13 comments

My two cents...

It looks like pkg.js does not allow OS tools to access assets within executables created by pkg.js (at least I did not find a way).

Example:

code snippet from index.js:

  var process = spawn('/bin/bash', ['-c', 'test.sh']

If index.js is called by 'nodejs index.js' it works, but if it is packaged using pkg.js (index.js and test.sh), this will fai. /bin/bash can't find test.sh because it is within pkg.js' snapshot filesystem and this filesystem is not accessible from outside the package, therefore bash can't access it.

In order to allow such a use case, pkg.js would need to be extended to allow for the deployment of some assets outside the generated executable. This would make pkg.js more of a packaging tool rather than a tool to generate executables from your nodejs projects.

My opinion: I think pkg.js is a great tool to create executables out of nodejs projects but it is not a packaging tool as the name suggests. For proper packaging other tools like msi, NSIS, dpkg-deb, fpm,... should probably be used. In my opinion, it is perfectly fine to use one tool to generate executables and another tool to package. My only critic would be that a different project name than pkg.js would be less misleading.

ccc-tomr on 4 Feb 2018

I just experienced something similar when trying to run ngrok. The dirty fix I'm using right now is to have a copy of ngrok.exe outside of the executable created by pkg, and then in ngrok's index.js instead of this:

ngrok = spawn(
        bin,
        start,
        {cwd: dir});

I'm doing this:

ngrok = spawn(
        path.join(process.cwd(), "ngrok.exe"),
        start);

curtgrimes on 18 Feb 2018

const source = path.join(__dirname, 'foo.exe');
const target = path.join('somewhere/foo.exe');

fs.copyFileSync(source, target); // ENOENT
child_process.spawn(target);

However, this will not work because fs.copyFileSync() made another syscall: copyfile, it cannot find source inside of snapshot filesystem.

But, instead of using copyFileSync, we can read assets into memory then write them into disk:

const fs = require('fs');
const utils = require('util');

const copyFile = utils.promisify(fs.copyFile);
const chmod = utils.promisify(fs.chmod);

async function copy(source, target) {
  if (process.pkg) {
    // use stream pipe to reduce memory usage
    // when loading a large file into memory.
    fs.createReadStream(source).pipe(fs.createWriteStream(target));
  } else {
    await copyFile(source, target);
  }
}

const source = path.join(__dirname, 'foo.exe');
const target = path.join('somewhere/foo.exe');

await copy(source, target); // this should work
await chmod(target, 0o765); // maybe need to grant execute permission

child_process.spawn(target);

micooz on 25 Feb 2018

👍8

I like Micooz approach, although it might not be practical when the required assets are large. To read required assets in memory and then write them to disk every time the pkg created binary is called would delay the start of the binary , especially if those required assets are large.

For now, I am going to stick with my current approach:

use pkg.js to create a binary
use a proper packaging tool (NSIS, dpkg-deb, fpm) to create an installer

bat-tomr on 25 Feb 2018

Personally, I find all proposed solutions not acceptable because... what if you're referring not simply to a single executable but that executable within its environment?
Take e.g. wmic in Windows. Its executable is in a directory along with a whole bunch of dlls.
I guess one could package all of that into an installer but... why? They presumably exist on every Windows installation.

And this doesn't seem like a novel idea - you can usually spawn whatever is globally available without having to distribute it along with your executable? (Not only talking about node here but also Python, C++, etc.)

s-h-a-d-o-w on 4 May 2018

Found a workflow for me based on the following observations. Feel free to correct me. (Might be worth dedicating a chapter in the README about how the spawning of processes can be handled?)

Executing anything outside actually works as expected as long as you either omit cwd or specify an absolute path outside the snapshot file system. One option mentioned above is process.cwd(). (No using of __dirname anywhere in the call to spawn!)
No spawning of anything within snapshot file system possible:

> [debug] The file was included as asset content
  D:\development\github\spotify-ad-blocker\node_modules\wmi-client\lib\cmd.exe

 [...]

Error: spawn D:\snapshot\spotify-ad-blocker\node_modules\wmi-client\lib\cmd.exe ENOENT

If the external executable needs assets from your "bundle", you will have to copy them to the regular file system (see https://github.com/zeit/pkg/issues/342#issuecomment-368303496). I think %APPDATA% is a decent candidate.

So what I've done myself is spawning wmic by using process.cwd() as cwd, essentially:

    spawn('wmic', args, {
        cwd: process.cwd(),
    });

And for testing purposes, I copied an asset required for wmic from the snapshot file system to %TEMP% using the method @micooz posted. Works just fine.

s-h-a-d-o-w on 6 May 2018

👍1

so, @s-h-a-d-o-w has provided a great fix for the typical use, but it does not work for my use case, where the spawned process relies on __dirname, not the cwd to run the executable. The only fix I can think of would be to create a webpack loader that dynamically replaces __dirname with the directory relative to the process.cwd(), but that seems less than ideal. Does anyone else have any other suggestions for when process.cwd() won't be reliable to create the path to the executable, for instance, when the executable is within a node_modules folder?

zwhitchcox on 29 Jul 2018

What you're describing is something I guess I'm doing this with one of the dependencies in one of my projects - search for "tray_windows_release.exe" here: https://github.com/s-h-a-d-o-w/spotify-ad-blocker/blob/v2.0.0/package.json
The following workflow is related to this:
https://github.com/zeit/pkg/issues/329#issuecomment-398151429

Essentially, whatever binary you want to spawn (whether native addon or regular executable), you have to extract it from the snapshot file system to the user's file system yourself.
If following the same directory structure is an option for you (meaning - you recreate the same path towards the executable as it exists in your dev environment), then using process.cwd() without modification should work.
(If you want to extract them somewhere else though, like maybe %TEMP%, see the workflow that I just linked to above)

s-h-a-d-o-w on 29 Jul 2018

Ok, so, I just ended up duplicating the original maintainer's project, because it's not that big, and I don't really want to have it outside the project. Thanks!

zwhitchcox on 30 Jul 2018

So, just out of interest, when I tried to create a directory outside of the project and then require, so,

fs.copy('node_modules/clipboardy', 'c:/Program Files/clipboardy'),

and then include it in my project, require('c:/Program Files/clipboardy'), I got an error, and even when I just copied it directly to the c: directory. Is this not possible? It seems like pkg bootstraps the require function, but does it provide the old require as well somewhere?

zwhitchcox on 30 Jul 2018

Hi there! I've the same issue with osrm native modules. So there is no simpler way to spawn process from assets now?