Electron-forge: Importing some native modules creates blank app, dev-tools disconnect
- [x] I have read the contribution documentation for this project.
- [x] I agree to follow the code of conduct that this project follows, as appropriate.
- [x] I have searched the issue tracker for an issue that matches the one I want to file, without success.
I am trying to build a react app that uses native modules, namely: better-sqlite3.
Unfortunately there is no example how to build a real js or react app. So I bootstraped an app using ideas from this comment: https://github.com/electron-userland/electron-forge/issues/1164#issuecomment-532621940. You can find it here: https://github.com/barbalex/kapla5test
I have done:
npx create-electron-app kapla5test --template=webpackyarn add react react-dom @babel/core babel-loader @babel/preset-env @babel/preset-react- Added babel-loader to webpack.renderer.config.js:
js rules.push({ test: /\.jsx?$/, exclude: /node_modules/, use: [{ loader: 'babel-loader' }] }) - added .babelrc
js { "presets": [ "@babel/preset-env", "@babel/preset-react" ] } rendered jsx inside renderer.js:
import React from 'react' import { render } from 'react-dom' render(<div>hello world</div>, document.getElementById('root'))altered index.html by adding
<div id="root"></div>:<!DOCTYPE html> <html> <head> <meta charset="UTF-8"> <title>Hello World!</title> </head> <body> <div id="root"></div> <h1>馃挅 Hello World!</h1> <p>Welcome to your Electron application.</p> </body> </html>- altered main.js from
js mainWindow = new BrowserWindow({ width: 800, height: 600 })
to
js mainWindow = new BrowserWindow({ width: 800, height: 600, webPreferences: { nodeIntegration: true, }, })
Result: yarn start works as expected: Electron starts up, showing the expected html.
So I try to use a native module: yarn add better-sqlite3
yarn start still works fine.
I now import better-sqlite3 in renderer.js:
import React from 'react'
import { render } from 'react-dom'
import betterSqlite from 'better-sqlite3'
render(<div>hello world</div>, document.getElementById('root'))
Now, when I run yarn start the following happens:
The main process logs:
Hash: 01fb7c0047224da542ea
Version: webpack 4.41.0
Time: 288ms
Built at: 2019-10-08 12:43:46
Asset Size Chunks Chunk Names
index.js 33 KiB main [emitted] main
index.js.map 38.1 KiB main [emitted] [dev] main
Entrypoint main = index.js index.js.map
[./node_modules/debug/src/browser.js] 5.69 KiB {main} [built]
[./node_modules/debug/src/common.js] 5.79 KiB {main} [built]
[./node_modules/debug/src/index.js] 314 bytes {main} [built]
[./node_modules/debug/src/node.js] 4.37 KiB {main} [built]
[./node_modules/electron-squirrel-startup/index.js] 1 KiB {main} [built]
[./node_modules/has-flag/index.js] 320 bytes {main} [built]
[./node_modules/ms/index.js] 2.95 KiB {main} [built]
[./node_modules/supports-color/index.js] 2.71 KiB {main} [optional] [built]
[./src/main.js] 1.83 KiB {main} [built]
[child_process] external "child_process" 42 bytes {main} [built]
[electron] external "electron" 42 bytes {main} [built]
[os] external "os" 42 bytes {main} [built]
[path] external "path" 42 bytes {main} [built]
[tty] external "tty" 42 bytes {main} [built]
[util] external "util" 42 bytes {main} [built]
...which looks fine.
The renderer logs:
wait until bundle finished: /main_window
Asset written to disk: .webpack\renderer\main_window\index.html
Asset written to disk: .webpack\renderer\native_modules\build\integer.node
Asset written to disk: .webpack\renderer\native_modules\build\better_sqlite3.node
Asset written to disk: .webpack\renderer\main_window\index.js
Hash: cd75c20b932270669b69
Version: webpack 4.41.0
Time: 1914ms
Built at: 2019-10-08 12:43:48
Asset Size Chunks Chunk Names
main_window/index.html 285 bytes [emitted]
main_window/index.js 2.99 MiB main_window [emitted] main_window
native_modules/build\better_sqlite3.node 2.06 MiB [emitted]
native_modules/build\integer.node 532 KiB [emitted]
Entrypoint main_window = main_window/index.js
[0] multi ./src/renderer.js webpack-hot-middleware/client 40 bytes {main_window} [built]
[./node_modules/ansi-html/index.js] 4.16 KiB {main_window} [built]
[./node_modules/better-sqlite3/lib/database.js] 2.67 KiB {main_window} [built]
[./node_modules/better-sqlite3/lib/index.js] 155 bytes {main_window} [built]
[./node_modules/better-sqlite3/lib/sqlite-error.js] 774 bytes {main_window} [built]
[./node_modules/html-entities/index.js] 231 bytes {main_window} [built]
[./node_modules/integer/lib/index.js] 1.44 KiB {main_window} [built]
[./node_modules/react-dom/index.js] 1.33 KiB {main_window} [built]
[./node_modules/react/index.js] 190 bytes {main_window} [built]
[./node_modules/webpack-hot-middleware/client-overlay.js] (webpack)-hot-middleware/client-overlay.js 2.17 KiB {main_window} [built]
[./node_modules/webpack-hot-middleware/client.js] (webpack)-hot-middleware/client.js 7.68 KiB {main_window} [built]
[./node_modules/webpack-hot-middleware/node_modules/strip-ansi/index.js] (webpack)-hot-middleware/node_modules/strip-ansi/index.js 161 bytes {main_window} [built]
[./node_modules/webpack-hot-middleware/process-update.js] (webpack)-hot-middleware/process-update.js 4.35 KiB {main_window} [built]
[./node_modules/webpack/buildin/module.js] (webpack)/buildin/module.js 497 bytes {main_window} [built]
[./src/renderer.js] 1.05 KiB {main_window} [built]
+ 24 hidden modules
Child html-webpack-plugin for "main_window\index.html":
Asset Size Chunks Chunk Names
main_window/index.html 534 KiB 0
Entrypoint undefined = main_window/index.html
[./node_modules/html-webpack-plugin/lib/loader.js!./src/index.html] 428 bytes {0} [built]
[./node_modules/lodash/lodash.js] 528 KiB {0} [built]
[./node_modules/webpack/buildin/module.js] (webpack)/buildin/module.js 497 bytes {0} [built]
Compiled successfully.
...which I _think_ looks fine too
But the electron window remains white and the dev-tools disconnect.
What am I doing wrong?
Console output when you run electron-forge with the environment variable DEBUG=electron-forge:*. (Instructions on how to do so here). Please include the stack trace if one exists.
Sorry, I did not manage to get this working.
Please provide either a failing minimal testcase (with a link to the code) or detailed steps to
reproduce your problem. Using electron-forge init is a good starting point, if that is not the
source of your problem.
- clone https://github.com/barbalex/kapla5test
- cd into it
- run
yarn start - try with and without importing better-sqlite3 in renderer.js
- watch console in electron
barbalex
All 22 comments
I tried this with a different native module: sqlite3. On windows I got a bunch of bad errors concerning versions of python and vscode. So I tried on MacOS.
There I get this error:
Uncaught Error: Cannot open /Users/alexandergabriel/kapla5test/node_modules/sqlite3/lib/binding/node-v72-darwin-x64/node_sqlite3.node: Error: The module '/Users/alexandergabriel/kapla5test/node_modules/sqlite3/lib/binding/node-v72-darwin-x64/node_sqlite3.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 72. This version of Node.js requires
NODE_MODULE_VERSION 73. Please try re-compiling or re-installing
the module (for instance, using `npm rebuild` or `npm install`).
at Object.<anonymous> (node_sqlite3.node:1)
at Object../node_modules/sqlite3/lib/binding/node-v72-darwin-x64/node_sqlite3.node (index.js:45925)
at __webpack_require__ (bootstrap:726)
at fn (bootstrap:100)
at Object../node_modules/sqlite3/lib/sqlite3.js (sqlite3.js:3)
at __webpack_require__ (bootstrap:726)
at fn (bootstrap:100)
at Module../src/renderer.js (renderer.js:1)
at __webpack_require__ (bootstrap:726)
at fn (bootstrap:100)
Also: attempting to load better-sqlite3 ends in a blank app and dev-tools disconnecting.
Which shows:
- the error also occurs on MacOS
- it may have to do with node versions?
barbalex
on 8 Oct 2019
Could be related to https://github.com/electron-userland/electron-forge/issues/1072
barbalex
on 8 Oct 2019
I installed nan-hello-world and imported that in renderer.js.
On Windows10 the app starts up happily.
On MacOS it shows a blank page and dev-tools disconnect.
EDIT: Nope, nan-hello-world also works on MacOS.
Don't know why the trouble exists with better-sqlite3 and sqlite3.
barbalex
on 8 Oct 2019
I tried it with node v10.16.3 instead of 12.9.0. Same results.
barbalex
on 8 Oct 2019
Hi @barbalex,
I had the same issue as you when adding sqlite3 to my electron-forge webpack template based app. After some deep dive into the electron-forge webpack-plugin and node-pre-gyp I found a rather simple solution. Some background information: The sqlite3 node package uses node-pre-gyp to handle native bindings. The main script of the package, lib/sqlite3.js, requires node-pre-gyp and uses its exposed find function to lookup the path of the native module bindings. The webpack-asset-relocator-loader recognizes this pattern and does its own path resolution for the native module bindings to replace the call to node-pre-gyp's find function with the _correct_ native module binding path. This happens at compile time. Even though the original call to find results in dead code, webpack-asset-relocator-loader dives into node-pre-gyp to resolve it completely - and finds some dynamic requires that it cannot handle.
To sum up, the problem is two-fold:
1) node-pre-gyp uses dynamic requires that lead to issues during compile time (Module './' cannot be resolved')
2) the compile runtime is used to determine native module bindings.
I am using the following workaround which works well for me so far:
1) Bypass node-pre-gyp during webpack compilation
2) Set process.versions.electron to make webpack-asset-relocator-loader believe the current runtime is elctron
For step 1) you can create a simple module tools/node-pre-gyp-bypass/index.js, say, with the following content:
module.exports = {
find: () => {}
};
then in your webpack main or renderer config, respectively, add:
resolve: {
modules: ['node_modules', 'tools'],
alias: {
'node-pre-gyp': 'node-pre-gyp-bypass'
}
}
For step 2) you can do the following. Make sure you're using a separate script as your forge config, forge.config.js, say. Then alter it in the following way:
process.versions.electron = process.env.npm_package_devDependencies_electron;
module.exports = {
...,
hooks: {
prePackage: async () => {
delete process.versions.electron;
}
}
};
The prePackage hook makes sure that tools used by electron-forge for packaging don't get fooled into assuming the wrong runtime. By the way, the Preparing native dependencies step in electron-forge already uses the correct target runtime.
I hope this is of any help to you!
@MarshallOfSound, to me it seems Step 2 should be standard, since we always want to x-compile in the context of electron-forge. Or am I missing something?
hundertmark
on 19 Oct 2019
@hundertmark
Thank you so much for your extremely valuable input.
Following your advice I manage to get sqlite3 to work. Nice! Especially considering that I understand near null concerning webpack which is why I could not really comprehend your explanation. Which made me expect that the project would blow up on me after making the changes, as I would surely mess up.
Unfortunately I am porting a mature project to electron-forge. And it uses better-sqlite3. When I import better-sqlite3 that still causes the electron window to remain white and the dev-tools to disconnect.
Maybe this could help?: https://github.com/JoshuaWise/better-sqlite3/issues/126#issuecomment-535459620
Updated testcase:
- clone https://github.com/barbalex/kapla5test
- cd into it
- run
yarn && yarn start - see that it works when sqlite3 is imported in
renderer.js鉂わ笍 - try with importing
better-sqlite3instead of sqlite3 in renderer.js 馃槥 - watch console in electron
barbalex
on 19 Oct 2019
I actually got better-sqlite3 to work in dev mode by following https://github.com/JoshuaWise/better-sqlite3/issues/126#issuecomment-535459620 in my test project: https://github.com/barbalex/kapla5test
Unfortunately after packaging the app can't find better-sqlite3 馃槥
Maybe there is a better way than declaring better-sqlite3 external?
Or there is a way to make electron find the copied files? (I probably botched up that part)
barbalex
on 19 Oct 2019
Hi again!
Not on y dev machine right now, but I had a brief look at your repo. If I remember it correctly, the packaging configuration ignores everything outside of .webpack/ which is the base directory for webpacks output in electron-forge standard configuration. So, that you're setup works in dev mode is because there you're running your app "from the root directory", which contains your dependencies anyway. You can verify that it still works without the copy task - the important part is that the "externals entry" tells webpack to leave your require in peace... Anyway, try to change the target of your copy task to "./.webpack/renderer/node_modules/better-sqlite3". The packager should include it from there.
Webpack is a brilliant tool, but it gets pushed to its boundaries here. The asset-relocator-loader I mentioned earlier is also a quite sophisticated piece of software, which 'tries' to predict all relevant "require scenarios". My recommendation would be to follow Joshua's approach. I will probably switch to that as well.
hundertmark
on 20 Oct 2019
@hundertmark
I have now set the copy task to copy to ./.webpack/renderer/node_modules/better-sqlite3 in webpack.main.config.js and webpack.renderer.config.js:
plugins: [
new CopyPlugin([
{
from: './node_modules/better-sqlite3/',
to: './.webpack/renderer/node_modules/better-sqlite3', // still under node_modules directory so it could find this module
},
]),
],
When I run yarn start better-sqlite3 works as you have explained.
When I run yarn package the dev tools do not crash any more but this error appears in the console:
Uncaught Error: Cannot find module 'better-sqlite3'
Require stack:
- C:\Users\alexa\kapla5test\out\kapla5test-win32-x64\resources\app\.webpack\renderer\main_window\index.html
at Module._resolveFilename (internal/modules/cjs/loader.js:627)
at Function.Module._resolveFilename (C:\Users\alexa\kapla5test\out\kapla5test-win32-x64\resources\electron.asar\common\reset-search-paths.js:41)
at Function.Module._load (internal/modules/cjs/loader.js:531)
at Module.require (internal/modules/cjs/loader.js:685)
at require (internal/modules/cjs/helpers.js:16)
at Object.<anonymous> (external "better-sqlite3":1)
at n (bootstrap:19)
at Module.<anonymous> (renderer.js:1)
at n (bootstrap:19)
at Object.<anonymous> (index.js:6)
The better-sqlite3 folder has been copied to .webpack\main\.webpack\renderer\node_modules and .webpack\renderer\.webpack\renderer\node_modules.
It can also be found in .\out\kapla5test-win32-x64\resources\app\.webpack\main\.webpack\renderer\node_modules and .out\kapla5test-win32-x64\resources\app\.webpack\renderer\.webpack\renderer\node_modules.
What surprises me is that there is a node_modules folder at .out\kapla5test-win32-x64\resources\app\node_modules. But this one is empty.
I tried to add this path to be copied in to also:
new CopyPlugin([
{
from: './node_modules/better-sqlite3/',
to: './.webpack/node_modules/better-sqlite3', // still under node_modules directory so it could find this module
},
]),
but that did not help.
I then installed lodash, imported it in renderer.js, ran yarn package and tried to find lodash in the .out folder. With the idea that this would show me where I needed to copy better-sqlite3 to. But I could not find lodash (aside from in the index.js file in .out\kapla5test-win32-x64\resources\app\.webpack\renderer\main_window).
As said and I guess meanwhile rather obvious: I am a complete webpack and electron-forge noob so I don't think I can get any further myself.
barbalex
on 20 Oct 2019
Ah, sorry, my fault... The path is relative to webpack base_dir which is "./.webpack" in this case. So just leave out the .webpack, that is, copy to "./node_modules/better-sqlite3" or "./renderer/node_modules". Module resolution, by default, traverses the parent chain up to the app root folder and looks into "node_modules" folders (if present).
But you will actually run into issues anyway, because now you only copy better-sqlite3, but not (most) of its dependencies - which get hoisted to the app's root node_modules folder. As a sanity check, you could do the following: in your forge-config.js alter the packerConfig as follows:
packagerConfig: {
ignore: file => {
return false;
}
},
The webpack plugin will complain in red during start or packaging, but for testing purposes just ignore that. Now the packager copies all dependencies. If your packaged app doesn't work then, you won't get any further with copying around stuff... I am working on smarter solution for my own project right now. But currently I am also considering to just drop the webpack plugin and use a plain electron-forge setup.
hundertmark
on 20 Oct 2019
@hundertmark
Yeah, webpack warns me for not minimizing file size. But it seems to work! In the test project it works 100%. In my ported mature project it works 99.8%. Only remaining problem is that when packaged some .wof files are not found:
GET file:///C:/Users/alexa/kapla4/out/Kapla-win32-x64/resources/app/.webpack/renderer/main_window/448c34a56d699c29117adc64c43affeb.woff2 net::ERR_FILE_NOT_FOUND
GET file:///C:/Users/alexa/kapla4/out/Kapla-win32-x64/resources/app/.webpack/renderer/main_window/fa2772327f55d8198301fdb8bcfc8158.woff net::ERR_FILE_NOT_FOUND
I guess I will have to check if I missed a loader.
You have just unblocked my project that was blocked for a few weeks due to not managing to build it any more.
If you ever come close to Thalwil, Switzerland (right next to Z眉rich), I'd love to invite you to Dinner or even just a beer! Thanks a lot!
barbalex
on 21 Oct 2019
I will leave this issue open due to the fact that the working solution is a bit of a hack and it seems that there should be ways to achieve working with native dependencies more elegantly in electron-forge.
barbalex
on 21 Oct 2019
For anyone using electron-forge & better-sqlite3 combo, I found a working solution by following the steps below:
Installation on Windows 10
- Install
windows-build-toolsglobally
$ npm install -g windows-build-tools
Install
python 2.7& add it to the pathGo to
C:\Users%USERNAME%.windows-build-toolsand runBuildTools_Full.exeorvs_BuildTools.exe. Then openBuildToolsand clickModify. Finally installVC++ 2015.3 v14.00(v140) toolset for desktop.Create a TypeScript template with Electron Forge
$ npx create-electron-app better-sqlite3-test --template=typescript-webpack
- Install
better-sqlite3&@types/better-sqlite3
$ npm install better-sqlite3
$ npm install -D @types/better-sqlite3
NOTE: Thanks to electron-forge, it handles compiling native modules automatically. See
@marshallofsound/webpack-asset-relocator-loaderinsidewebpack.rules.js
Add better-sqlite3 code to src/index.ts
import SQLite from 'better-sqlite3'
.
.
.
const db = SQLite('data.db')
.
.
.
const createWindow = () => {
.
.
.
mainWindow.webContents.once('dom-ready', () => {
const tableName = 'cats'
const createTable = db.prepare(
`CREATE TABLE IF NOT EXISTS ${tableName} (name CHAR(20), age INT)`
)
createTable.run()
const insert = db.prepare(
`INSERT INTO ${tableName} (name, age) VALUES (@name, @age)`
)
const insertMany = db.transaction((cats) => {
for (const cat of cats) insert.run(cat)
})
const selectAllCats = db.prepare(`SELECT * FROM ${tableName}`)
const rows = selectAllCats.all()
if (!rows.length)
insertMany([
{ name: 'Joey', age: 2 },
{ name: 'Sally', age: 4 },
{ name: 'Junior', age: 1 },
])
rows.forEach((row) => {
console.log(row)
})
})
.
.
.
}
.
.
.
Run start script
$ yarn start # runs `electron-forge start`
I'll add the installation steps for Mac soon but should be much simpler than Windows :)
deadcoder0904
on 31 Mar 2020
@barbalex See my above comment. better-sqlite3 released v7.0.0 which removes a dependency on another native module integer so only better-sqlite3 native module remains.
And as you can see above I don't think it requires a hack anymore. Pretty straightforward I think. I have only tested on Windows but I think Windows is the hardest to get right so should work in a much simpler way in Linux/Mac :)
deadcoder0904
on 26 Apr 2020
@deadcoder0904
Thanks a lot for helping.
Unfortunately I am using the webpack template (npx create-electron-app kapla5test --template=webpack). And this makes things different - probably a lot more complicated.
I keep getting this error:
which is because of webpack.
barbalex
on 26 Apr 2020
@barbalex I tried following the steps of your original comment. There was just 1 issue with it. You added better-sqlite3 in renderer.js.
It should be added in main.js. That's the only thing I changed & it worked on my machine without problems. No errors. No warnings. Works as expected. So try again by just including better-sqlite3 in main.js :)
deadcoder0904
on 26 Apr 2020
@deadcoder0904
Thanks for trying.
Maybe I am missing something basic here. But I need better-sqlite3 not in main.js but in the rendered app.
Maybe there is a way I could load it in main.js and the pass it's instance to the renderer?
barbalex
on 26 Apr 2020
@barbalex You can perform your query in main.js & then use IPC to pass the result in renderer.js. No need to pass an instance or anything.
Better yet, read https://jlongster.com/secret-of-good-electron-apps to get a better understanding of how to make Electron apps have a snappy UX
deadcoder0904
on 27 Apr 2020
Better yet, read https://jlongster.com/secret-of-good-electron-apps to get a better understanding of how to make Electron apps have a snappy UX
Wow
O.k., back to the drawing board. Time to power up 馃挭
Seems that what I was missing is not necessarily basic but it certainly _is_ fundamental.
Thanks a lot, @deadcoder0904
barbalex
on 27 Apr 2020
No problem @barbalex. Happens to the best of us 馃榿
deadcoder0904
on 27 Apr 2020
It seems that in electron-forge main.js itself is dragged through webpack. And the same happens to additional browser windows and threads created in main.js.
So after a few hours I have given up implementing the great ideas from https://jlongster.com/secret-of-good-electron-apps. There is simply no way to access/require a file like you could in node.js.
barbalex
on 27 Apr 2020
I have found a better solution: https://github.com/electron-userland/electron-forge/issues/1451#issuecomment-606039498
Using @hundertmark 's solution my production package was about 680MB. With this on it is reduced to 216MB.
Thanks to @hundertmark and @NaridaL for providing these great ideas.
barbalex
on 13 Jul 2020
Related issues
NaridaL
路
4Comments
Slapbox
路
4Comments
pkinnucan
路
4Comments
damianesteban
路
3Comments
mistermicheels
路
4Comments
Most helpful comment
Wow
O.k., back to the drawing board. Time to power up 馃挭
Seems that what I was missing is not necessarily basic but it certainly _is_ fundamental.
Thanks a lot, @deadcoder0904