Node-sqlite3: webpack can not pack sqlite3

[wotermelon]

cant find module sqlite3

webpack + sqlite3 not working

[tmcw]

Sorry, this is expected behavior. webpack does not work with native modules, and sqlite3 is one - this is a native binding to the SQLite C library. You might want to try a browser-native solution, like localStorage, or sqlite.js.

[MattJeanes]

Webpack can and does work with native modules using the webpack-node-loader. I've managed to import sharp using this and it works as expected with no node_modules folder present entirely

[pamtbaau]

@MattJeanes This sounds promising... Unfortunately, I cannot get it to work with sqlite3 using webpack. Do you perhaps have a working example of electron/webpack/sqlite3?

[MattJeanes]

@pamtbaau I spent probably about 6 hours trying to get node working under webpack, and I did get it working but there was a lot of hacks in the webpack config to make it do things correctly, and I gave up and just switched back to wiping the folder + doing a yarn install when I upgraded the app

If you're interested, here's my webpack config I made: https://gist.github.com/MattJeanes/1753027da4aca18a3b903fedef5d57e5

Webpack would work perfectly if everything in node was done statically, but in the world of binary modules and dynamic imports, it doesn't work correctly and in my opinion isn't worth the hacks required to make it work.

[pamtbaau]

@MattJeanes Thanks for your efforts!

6 hours? I've been trying for about a year on and off... :-) I switched over to sql.js a year ago and over to Lovefield on top of IndexedDB about 6 months ago. But sqlite keeps drawing my attention some how...

I will have a look at your solution later today.

[pamtbaau]

@MattJeanes Yikes, that's an ugly workaround...

Tried several configuration options, but still getting loads of issues thrown at me. Complaints of not being able to resolve node modules like Can't resolve 'fs'.

This morning I implemented a way of bypassing webpack. I open a sqlite database in plain .js file and load this file in the index.html of electron. In my typescript files I import { Database } from "sqlite3"; and declare var db: Database;. Now I can run db.each('select * from...') and have type checking. I don't like it, but it works...

[MattJeanes]

Generally with Electron apps I've seen the entire node modules packaged up and extracted along with the app code, I'm not sure I've ever seen a webpack'd one, most likely because of these issues. It works OK for small simple apps but when it starts to get bigger and require native modules and ones that use dynamic imports it just messes right up..

[MattJeanes]

Btw, target: "node", from my config should solve the Can't resolve 'fs' and related problems

[pamtbaau]

Where did the tightly packed vendor lock-in database and app tools go... They just worked. Now I need to stack package on package on package... waiting for it to fall apart.

Sigh, I'm disclosing my age I think... :-)

[pamtbaau]

target: 'node' didn't bring me much luck...

Thank you for all the effort and spare time you've put in... I appreciate it.

[MattJeanes]

@pamtbaau weird - that should have solved that particular error at least.. are you using webpack 1? I'm using webpack 3.. Well good luck on your endeavours, I've officially given up with it personally 😄

[pamtbaau]

@MattJeanes Finally.... import { Database } from 'sqlite3'; now working in Electron/sqlite3 webpack bundle.

In webpack.config.js add:

module.exports = {
    entry: {
        'app': './root/of/your/app.ts',
    },

    externals: (() => {
        var nodeModules = {};
        fs.readdirSync('node_modules')
            .filter(function (x) {
                return ['.bin'].indexOf(x) === -1;
            })
            .forEach(function (mod) {
                nodeModules[mod] = 'commonjs ' + mod;
            });
        return nodeModules;
    })(),

Fair is fair, I found the solution here: Backend Apps with Webpack (Part I) from James Long

[MattJeanes]

@pamtbaau all that seems to do is just exclude node_modules from being packed so it doesn't really solve any problems unfortunately..

[eis84]

Hi everyone, and thank you for having spoken in public about this issue.
I have the same problem: electron, sqlite3, webpak ... but in addition the project I'm working with uses webpackMerge.
In the webpack.config.js file
There is this code:
Module.exports = [
WebpackMerge (electronMainConf, devConfig),
WebpackMerge (electronRendererConf, devConfig)
]

Why load a main and a render of the page.

As I can add
... ...
Externals: (() => {
Var nodeModules = {};
fs.readdirSync ( 'node_modules')
.filter (function (x) {
Return ['.bin']. IndexOf (x) === -1;
})
.forEach (function (mod) {
NodeModules [mod] = 'commonjs' + mod;
});
Return nodeModules;
}) ()
... ...
??

[ruslanas]

I'm not sure if this is applicable for this project, but here's how I did it in my Electron app.
HTML

<script>
        const sqlite3 = require("sqlite3");
        require("./dist/main.js"); // 
</script>

webpack.config.js

module.exports {
    // ...
    output: {
        filename: "./dist/main.js",
    },
    externals: {
        sqlite3: "sqlite3",
    },
}

[pamtbaau]

@ruslanas, When using require('sqlite') inside the html page, you are loading it inside Electron's main process. That means the renderer process (your app) has no direct access to sqlite and needs to use ipcRenderer to call main to perform your queries. Main will then return the results using ipcMain.

In that case, your solution works...

I've tried that scenario, but didn't like it:

• I want all my code (ui logic, business logic and data logic) in one place: my app (= the renderer process)
• ipc introduces a performance overhead

To be able to import sqlite inside the renderer process commonjs is required:

externals: {
        sqlite3: 'commonjs sqlite3',
    },

Now you can import sqlite inside the renderer process using import { Database } from 'sqlite3';

As a side note:
When using webpack, you don't need to add require("./dist/main.js") to the html page. HtmlWebpackPlugin will do that for you.

[eis84]

                                                                                  Thanks for the answers.

I already tried those methods.
But nothing, I gave the client the program that starts with a .bat that recompiles the code every time.
I changed the working language now, javascript I leave it for the interactions of the web pages, the applications are back to them with Visual C ++ Invio eseguito dallo smartphone BlackBerry 10. Da: pamtbaauInviato: mercoledì 27 dicembre 2017 16:26A: mapbox/node-sqlite3Rispondi a: mapbox/node-sqlite3Cc: eis84; CommentOggetto: Re: [mapbox/node-sqlite3] webpack can not pack sqlite3 (#698)@ruslanas, When using require('sqlite') inside the html page, you are loading it inside Electron's main process. That means the renderer process (your app) has no direct access to sqlite and needs to use ipcRenderer to call main to perform your queries. Main will then return the results using ipcMain.
In that case, your solution works...
I've tried that scenario, but didn't like it:

I want all my code (ui logic, business logic and data logic) in one place: my app (= the renderer process)
ipc introduces a performance overhead

To be able to import sqlite inside the renderer process commonjs is required:
externals: {
sqlite3: 'commonjs sqlite3',
},

Now you can import sqlite inside the renderer process using import { Database } from 'sqlite3';
As a side note:
When using webpack, you don't need to add require("./dist/main.js") to the html page. HtmlWebpackPlugin will do that for you.

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/mapbox/node-sqlite3","title":"mapbox/node-sqlite3","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/mapbox/node-sqlite3"}},"updates":{"snippets":[{"icon":"PERSON","message":"@pamtbaau in #698: @ruslanas, When using require('sqlite') inside the html page, you are loading it inside Electron's main process. That means the renderer process (your app) has no direct access to sqlite and needs to use ipcRenderer to call main to perform your queries. Main will then return the results using ipcMain.\r\n\r\nIn that case, your solution works...\r\n\r\nI've tried that scenario, but didn't like it:\r\n- I want all my code (ui logic, business logic and data logic) in one place: my app (= the renderer process)\r\n- ipc introduces a performance overhead\r\n\r\nTo be able to import sqlite inside the renderer process commonjs is required:\r\n\r\n\r\nexternals: {\r\n sqlite3: 'commonjs sqlite3',\r\n },\r\n\r\n\r\nNow you can import sqlite inside the renderer process using import { Database } from 'sqlite3';\r\n\r\nAs a side note: \r\nWhen using webpack, you don't need to add require(\"./dist/main.js\") to the html page. HtmlWebpackPlugin will do that for you."}],"action":{"name":"View Issue","url":"https://github.com/mapbox/node-sqlite3/issues/698#issuecomment-354129269"}}}

[pamtbaau]

@eis84 I have created a repository with an app using Angular5, Bootstrap 4, sqlite3 and webpack. It compiles, runs and can be packaged.

Have a look at it here.

[searene]

I spent several hours on it today and finally solved this issue. The solution is pretty simple, just add externals in webpack.config.js. Here is a sample.

module.exports = {
  entry: 'src/Main.ts',
  ...
  // add the following lines
  externals: {
    sqlite3: 'commonjs sqlite3'
  }
}

The idea is to prevent webpack from parsing sqlite3. If you don't use electron, that's all you need to do. But if you run sqlite3 with electron, you have to make sure the dedicated sqlite3 binary for electron could be generated. Add the following lines in package.json.

"scripts": {
   "postinstall": "install-app-deps"
}

Then install dependencies and build binaries.

npm install --save-dev electron-builder
npm run postinstall

Now try rerunning webpack, it should work now.

[dpatra]

Thanks a ton @searene , it works like a charm.

[redplane]

@searene , I followed your solution above, but now I'm facing with an error:

Cannot find module node_sqlite3.node when I use require('sqlite3')

[wotermelon]

@redplane Use node-gyp to build the sqlite3 module,because sqlite3 is a native module. try?

[redplane]

@wotermelon , I tried and worked.
Forgot replying you :).

But when I pack my electron app with sqlite 3 (using require('sqlite3'), my app faces with the same error.

[cliffordfajardo]

I had the same issue. In addition to doing what @searene said, which was having webpack ignore sqlite3:

module.exports = {
  externals: {
    sqlite3: 'commonjs sqlite3'
  }
}

I later installed electron-rebuild as a dev dependency npm i --save-dev electron-rebuild then added it to my package.json's postinstall script.

//package.json

  "scripts": {
    ...,
    "postinstall": "electron-rebuild"
  },

What electron-rebuild will do is build all binaries for you after you.