Pkg: [Request] Support bundling native addons inside the executable

[maxogden]

Hi, we are using pkg and loving it so far, but it would be much nicer if we could include our .node files inside the single executable at the end. I saw this mentioned in the readme as not resolved, but could not find and issue for it.

[Clement-TS]

@igorklopov is there anything planned to support this? Any direction / advice we could take to help the community?

[igorklopov]

When application is running it must have .node files on filesystem anyway. So the scenario is 1) take .node files as assets into executable at compilation time. 2) also require (as first line of application) some utility that scans /snapshot/ for .node files and extracts them to path.dirname(process.execPath). Both of the points can be done externally. No new features of pkg is needed for it.

[gurisko]

@igorklopov afaik .node files won't be included into assets at all since the build log states:

Warning Cannot include addon %1 into executable.

So a new feature is still needed to allow native files to be included in the build.

Would you mind giving an example of step 2? Thanks!

[Clement-TS]

@gurisko I'm stuck there as well.

[maxogden]

If the .node files must be extracted, then we will probably just distribute a pure-js version of our application using pkg, and tell people to use npm install to get the native addon features. I can't assume the user is running the executable in a folder that they have write access into, so having the application extract files is problematic.

I wonder if there is a way to tell node to use a custom path in e.g. /tmp for .node files, rather than in the ./ relative to the binary? If I could have my executable place its node files in a world writable location like /tmp then this scheme would work for me.

[gurisko]

I ended up bundling the app with webpack where I included this rule:

      {
        test: /\.node$/,
        include: /node_modules/,
        use: [
          {
            loader: 'awesome-node-loader',
            options: {
              name: 'libs/[hash].[ext]',
              rewritePath: '.',
            },
          },
        ],
      },

And now we ship our app with the build and the "libs" folder where .node files are located.

@maxogden Try using it with {rewritePath: '/tmp'}

[gurisko]

I updated awesome-node-loader plugin that if you have a relative rewritePath you can choose to use __dirname or path.dirname(process.execPath) simple by setting useDirname. So we are now using

      {
        test: /\.node$/,
        include: /node_modules/,
        use: [
          {
            loader: 'awesome-node-loader',
            options: {
              name: 'libs/[hash].[ext]',
              rewritePath: '.',
              useDirname: false,
            },
          },
        ],
      },

with our packaged app and we don't have to execute our app from the same directory where libs is located.

[s-h-a-d-o-w]

1) take .node files as assets into executable at compilation time.

But why doesn't pkg allow this? If I try to include processlist.node, I get The addon must be distributed with executable as %2 but if I rename it to processlist.whatever, it is fine?

So we can do e.g. this:

const path = require('path');
const fs = require('fs-extra');
fs.writeFileSync(
    path.join(path.dirname(process.execPath), 'processlist.node'),
    fs.readFileSync(path.join(__dirname, '../bin/processlist.renametonode'))
);

But not this:

const path = require('path');
const fs = require('fs-extra');
fs.writeFileSync(
    path.join(path.dirname(process.execPath), 'processlist.node'),
    fs.readFileSync(path.join(__dirname, '../bin/processlist.node'))
);

Seems like an arbitrary constraint to me...

[s-h-a-d-o-w]

Here's how I ended up dealing with this, maybe this will helps others in the future (and maybe inspiration for the pkg team? :smile: ):

PREPARATION

• As a beforebuild step, I use renamer to rename all .node files it finds in my repo to .foolpkg (and back to .node again in an afterbuild step) and add those I need as assets. (see my package.json)

START OF APP

• Right at the beginning of my app, I extract them to a temporary location.
• In order for pkg to find them at this temporary location, using patch-package, I patch pkg's bootstrap.js (there's a second patch in there described below for cleaning up on exit) to not look for things at the location where the app was first started but instead the current working directory. Which I change to that temporary location also in the bootstrap part of my app.

EXITING THE APP
Usually, you won't need to clean up after yourself, since... you know... it's tmp. Everyone leaves trash there anyway. :wink:
But if for whatever reason, like me, you do - here is how (it's not that straightfoward, since native addons can't be deleted until the process requiring them is dead):

• On process's exit event (can be found in here), I change the cwd back to the original one and the app spawns a detached version of itself (which is only possible by removing pkg's overwritten spawn() - I didn't invest the time in figuring out what the point of this is but I know that my app can do everything without it), supplying a CLI argument used for cleanup.
• Just like the code for extracting the native addon, code triggered by --cleanup is in the bootstrap part of my app. It prevents my other code from being executed and keeps trying to delete the directory until it eventually can (i.e. when the original process of the app is dead).

[dinigo]

A workarround but pretty elegant though @s-h-a-d-o-w. I'd find this very useful if it was officially implemented as you did. This _native modules_ thing brought me crazy yesterday. I'll give it a try. Thanks! and thanks pkg team.

[s-h-a-d-o-w]

@dinigo
Thanks! Coincidentally, just yesterday, I changed it a bit to be more elegant by using renamer (see my updated comment above - I also fixed the broken links in it). :smile:

[AHgPuK]

Hi @igorklopov !
I found an interesting approach for loading modules from memory
Please, have a look at https://x-c3ll.github.io/posts/fileless-memfd_create/

[nornagon]

FWIW, electron handles this by unpacking on demand, see https://github.com/electron/electron/blob/6f91af93433df4e9c0e7fb592e5b2dcb0b1c7888/docs/tutorial/application-packaging.md#extra-unpacking-on-some-apis