Electron-builder: Nested node_modules disappears after building

[bradydowling]

• Version: 20.26.0

• Target: macOS (.dmg)

Similar to https://github.com/electron-userland/electron-builder/issues/1539, I'm working with an app where I've cloned another git repo int and I have several node_modules folders. When I build, the node_modules folder in the nested git repo disappears. This happens whether asar is enabled or disabled. It also happens regardless of whether or not the inner git repo is officially designated as a git submodule or if I've just manually cloned it in.

Here's my folder structure:

main.js (starts app and forks process from the express app's index.js)
node_modules
package.json
cloned-express-app
   |
    --node_modules
      index.js
      package.json

Here's a Stack Overflow issue I just created that may explain it better (I didn't want to copy/paste but let me know if that's preferred) and here's the repo I'm working with.

It's also worth noting that I've tried building this for Windows and my friend said it works fine on there. I haven't tested that recently though so I can't give much info on that immediately. I also haven't inspected the Resources contents of the packaged app (or whatever the .exe counterpart would be) so I can't provide info on that but I do know that when I look in the mac app's Resources directory there is no node_modules folder within the nested git repo.

[fpsqdb]

Same issue here for me.

• version: 20.26.0

• target: windows

 "files": [
            "!**/**.map",
            "!**/**.ts",
            "!node_modules",
            "package.json",
            "node_modules/semver/**",
            "node_modules/fs-extra/**",
            "node_modules/graceful-fs/**",
            "node_modules/jsonfile/**",
            "node_modules/universalify/**"
        ],

When I build, there is no node_modules folder in the package. That's not my expected, it should have those folders and files in the package.

node_modules/semver,
node_modules/fs-extra,
node_modules/graceful-fs,
node_modules/jsonfile,
node_modules/universalify

But if I downgrade to 20.15.0, it works fine.

[bradydowling]

Interesting, I'll need to give this a try, thanks for the note.

[bradydowling]

It did indeed package node_modules with this old version (20.15.0) on macOS as well, thank you @fpsqdb.

[hn3000]

In version 20.16 the way node_modules includes work was changed: explicit includes are now ignored, instead all production dependencies are used. Unfortunately, excludes are still allowed to affect the node_modules folders and that means your !node_modules takes effect.

Looking at your example above:

    "files": [
        ...,
        "!node_modules",
        ...,
        "node_modules/semver/**",
        "node_modules/fs-extra/**",
        "node_modules/graceful-fs/**",
        "node_modules/jsonfile/**",
        "node_modules/universalify/**",
        ...
    ],

The first line will prevent inclusion of anything in the node_modules folder, the other lines are ignored since version 20.16, resulting in an app.asar that does not include any node_modules.

Remove the exclusion for node_modules and newer versions than 20.15 should work fine. In fact, you should be able to remove all lines mentioning the node_modules folder.

(Edit: tried to explain in more detail.)

[fpsqdb]

@hn3000 There's a condition that when use webpack to bundle packages, so you don't need all the dependencies. And if there are some packages exclude from webpack(such as set externals in webpack config) we should add them individually, that's my situations.

[hn3000]

Then maybe a second package.json can help? https://www.electron.build/tutorials/two-package-structure

[TanninOne]

And a change like this was made in a minor update (20.15 -> 20.16)? Intentionally?

[TanninOne]

Am I missing something here? How is this bug still not addressed?
Afaict all releases post 20.15.0 are unusable and should be considered extremely dangerous, how is this not top priority?
@develar ? I don't want to be annoying, but I've not been able to produce a working build with any electron-builder after 20.15.0 and I'm starting to panic as you don't even seem to acknowledge there is a problem here.
And no, hn3000s explanation does not fully explain the problem, I get this error even entirely without any files excludes.

My project depends on semver@^5.6.0
A dependency I include (semvish) depends on semver@^4.6.3
electron-builder packages semver@^4.6.3 twice, in semvish/node_modules and the top-level node_modules, causing bugs in my main application, breaking electron-updater among other things.
And I'll just assume this is one problem of many caused by this.

[grantcv1]

My application brings together 9 independent submodules (which have a life of their own in in other apps) and each has their own node_module. I've been using a much earlier version of electron-builder (19.55.3) for over a year without any issues -- all the node_modules were copied into my installer. After upgrading to the latest version, I've lost 3 days of development trying to understand this issue. It really need to be address. Seconding what TanninOne said, without fixing this issue, electron-builder is unusable.

[theproductiveprogrammer]

Is there any update regarding this issue? Or any workaround? I've spent 2 days fiddling with various setting and nothing is working. I can't proceed to release an installer at this rate. Anyone have any suggestions?

[bradydowling]

Doesn't it work if you fix the version?

On Mon, Jun 10, 2019, 11:06 PM Charles [e-mail] wrote:

Is there any update regarding this issue? Or any workaround? I've spent 2
days fiddling with various setting and nothing is working. I can't proceed
to release an installer at this rate. Anyone have any suggestions?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/electron-userland/electron-builder/issues/3185?email_source=notifications&email_token=ABS4AU2KOM6CAIW3NILP74TPZ4JCBA5CNFSM4FMURQ42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXLZX5I#issuecomment-500669429,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABS4AU2J2QFT3QYNZSANNKDPZ4JCBANCNFSM4FMURQ4Q
.

[theproductiveprogrammer]

I tried that but when I rolled back to 20.15.0 I got a new error:

[xmldom error]  invalid doc source
@#[line:0,col:undefined]
TypeError: Cannot read property 'documentElement' of undefined
    at parse
(ai-ico/package/elife-node/node_modules/plist/lib/parse.js:68:9)
    at
ai-ico/package/elife-node/node_modules/electron-builder-lib/src/electron/electronMac.ts:51:25
From previous event:
    at createMacApp
(ai-ico/package/elife-node/node_modules/electron-builder-lib/out/electron/electronMac.js:245:17)
    at
ai-ico/package/elife-node/node_modules/electron-builder-lib/src/electron/ElectronFramework.ts:56:11
    at Generator.next (<anonymous>)
From previous event:
    at beforeCopyExtraFiles
(ai-ico/package/elife-node/node_modules/electron-builder-lib/out/electron/ElectronFramework.js:167:17)
    at beforeCopyExtraFiles
(ai-ico/package/elife-node/node_modules/electron-builder-lib/src/electron/ElectronFramework.ts:131:14)
    at
ai-ico/package/elife-node/node_modules/electron-builder-lib/src/platformPackager.ts:223:13
    at Generator.next (<anonymous>)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
<snip>

So I'm sticking with the latest version. I've edited the electron-builder code for now and removed the node_modules filter but that's probably the wrong thing to do.

Do you know _why_ electron-builder is removing node_modules? I tried looking at the code but the commenting is very sparse and I can't figure it out.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[florisporro]

Has this been fixed? If not, it's definitely something that should be addressed, it's a big headache.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[TanninOne]

Not fixed yet

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[TanninOne]

Still relevant. Blows my mind that this very critical bug gets absolutely no attention.

[laysent]

I have create a PR request to the demo repo, hopefully it helps anyone. PR: https://github.com/bradydowling/electron-server-hello-world/pull/1
The root cause is packages/app-builder-lib/src/util/AppFileWalker.ts, where it exclude node_modules folder when putting files into app folder. Notice that later electron-builder has one step to decide which packages in node_modules should be included in final app folder, however it does not analyze other node_modules that has been ignored in first step.
The workaround for this would be to use workspaces, put every modified / private packages in workspace, so that yarn will help you link these packages in node_modules. This way, electron-builder can correctly pack things.
Few things to notice:

1. You will have to use at least 22.2.0 version, otherwise there is a bug that incorrectly handles nested node_modules (see this PR: https://github.com/electron-userland/electron-builder/pull/4371)
2. Don't forget to add packages in your package.json as depenedency, so that electron-builder will help you pack it (be sure to use the correct version number, so that yarn knows it should use your local version, instead of the package in npm registry)

[tessus]

@develar is this issue even on the radar? It's kind of serious that if you have a working app and package it with electron-builder, it creates a broken, not working app.

[laurent22]

We have this issue in Joplin - it has one submodule, which builds fine, expect that electron-builder doesn't package the node_modules folder.

If the bug is too complex to fix, perhaps there should be an option to skip this node_modules detection logic, and just package everything in the source dir. Not sure why the current logic is there, maybe to save space, but I'd rather have a bigger app than a broken one.

[tessus]

@theproductiveprogrammer

I've edited the electron-builder code for now and removed the node_modules filter

How did you do this? Do you have a patch somewhere?

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[tessus]

Is this still relevant?

Very much so. Not fixing this makes using sub modules in electron projects impossible wihout duplication. Idiotic really.

[TanninOne]

The big problem here isn't unnecessary duplication, that would be fairly harmless. The big problem is that the software, after bundling with this, doesn't fulfill the version requirements on dependencies from package.json any more.
If you're not aware of this problem your software may be affected by bugs or security holes in dependencies that you assumed are ruled out through the version ranges you specified.

[dmnsgn]

Having the same issue here.

My project is packaging a node cli tool alongside my main electron codebase. This node cli tool is in a subfolder and has a package.json with dependencies in a node_modules folder that I'd like to keep as is in my packaged app.

No matter what I try, two-package.json (doesn't fit the use case), adding node_modules in afterPack (see https://github.com/electron-userland/electron-builder/issues/3104#issuecomment-595847011, it won't work with asar) or any files filter combination, my node_modules never end where they are suppose to.

@develar it seems like the solution would be tweaking the app file walker (see https://github.com/electron-userland/electron-builder/issues/3185#issuecomment-571193649)? What would help speed things up to solve this longstanding issue? Are you waiting for contributions?

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[laurent22]

Is this still relevant?

Yes, electron-builder still breaks perfectly valid projects.

[bedney]

Yep, still seeing that here as well. Completely breaks my app. I have a workaround by setting asar: false and using extraResources to copy my node_modules directory into the proper place under the app directory, but this cost me a day trying to figure out why node_modules wasn't being copied even though I didn't see it in the excluded files list.

[onfire4g05]

This is definitely not a great solution, but it could be a temporary "fix". You can add this FileSet to your files section for each submodule:

{
    "from": "some/location/with/node_modules",
    "to": "some/location/with/node_modules",
    "filter": [
        "!**/{test,__tests__,tests,powered-test,example,examples,CHANGELOG.md,README.md,README,readme.md,readme}",
        "!**/*.d.ts",
        "!**/.bin",
        "!**/*.{iml,o,hprof,orig,pyc,pyo,rbc,swp,csproj,sln,xproj}",
        "!**/._*",
        "!**/{.DS_Store,.git,.hg,.svn,CVS,RCS,SCCS,.gitignore,.gitattributes}",
        "!**/{__pycache__,thumbs.db,.flowconfig,.idea,.vs,.nyc_output}",
        "!**/{appveyor.yml,.travis.yml,circle.yml}",
        "!**/{npm-debug.log,yarn.lock,.yarn-integrity,.yarn-metadata.json}"
    ]
}

Several problems with this though: it will obviously get super long if you have a lot of submodules, and dev dependencies will be included in the build. There may (probably?) are other issues, too.

The dev dep issue, I think, is the biggest issue, but is possibly something that could be worked/hacked around as well.

[nikitalita]

@onfire4g05 's solution gave me an idea. I have made an afterpack hook that unbundles the asar, hoists the node modules, and then rebuilds the asar with the same unpacked resources. It's not perfect; I only check for the "files" parameter when looking for files to exclude, and it doesn't do the "smartunpack" that asar does, but I tried to explain as much as I could so that people could easily modify this. If this bug is biting you, give it a whirl:

https://gist.github.com/nikitalita/80ec616a120366444491a1c671abae97

[karlsnyder0]

We are also seeing this issue in version 22.9.1! To replicate this issue have the following build directory format-

build/
 index.js
 node_modules/test
  index.js
 nm/test
  index.js

Electron-builder nukes the "node_modules" folder but leaves the "nm" folder alone.

I put together a patch for electron-builder version 22.9.1 that can be used with the patch-package npm package.

https://www.npmjs.com/package/patch-package

/patches/app-builder-lib+22.9.1.patch

diff --git a/node_modules/app-builder-lib/out/util/AppFileWalker.js b/node_modules/app-builder-lib/out/util/AppFileWalker.js
index 0f9cdad..108b75a 100644
--- a/node_modules/app-builder-lib/out/util/AppFileWalker.js
+++ b/node_modules/app-builder-lib/out/util/AppFileWalker.js
@@ -88,7 +88,7 @@ function createAppFilter(matcher, packager) {

   const filter = matcher.createFilter();
   return (file, fileStat) => {
-    if (!nodeModulesFilter(file, fileStat)) {
+    if (!['www/node_modules'].some(v => str.includes(v)) && !nodeModulesFilter(file, fileStat)) {
       return false;
     }

@@ -110,7 +110,7 @@ class AppFileWalker extends FileCopyHelper {
       // https://github.com/electron-userland/electron-builder/issues/1539
       // but do not filter if we inside node_modules dir
       // update: solution disabled, node module resolver should support such setup
-      if (file.endsWith(nodeModulesSystemDependentSuffix)) {
+      if (!['www/node_modules'].some(v => str.includes(v)) && file.endsWith(nodeModulesSystemDependentSuffix)) {
         return false;
       }
     } else {

To protect your node_module paths add a portion of the node_module path in the array as a string. For example, the /build/www/node_modules path is part of my build. Adding "www/node_modules" protects that path.

I would prefer to submit a PR for this however I have no experience with this repo and would prefer to not make the code worse.

[tessus]

I believe the developer actively ignores this issue. Nothing will happen. This has been open for more than 2 years, yet still no acknowledgement - nothing.

Unfortunately there's nothing else but electron-builder so we have to live with this garbage great tool.

[karlsnyder0]

I believe the developer actively ignores this issue. Nothing will happen. This has been open for more than 2 years, yet still no acknowledgement - nothing.

Unfortunately there's nothing else but electron-builder so we have to live with this ~garbage~ great tool.

I've updated my comment and included a patch that we will now use to work around this issue.