Better-sqlite3: can't install under linux

[uestueab]

I get the following Error

Traceback (most recent call last): File "/usr/lib/node_modules/node-gyp/gyp/gyp_main.py", line 50, in <module> sys.exit(gyp.script_main()) File "/usr/lib/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 554, in script_main return main(sys.argv[1:]) File "/usr/lib/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 547, in main return gyp_main(args) File "/usr/lib/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 532, in gyp_main generator.GenerateOutput(flat_list, targets, data, params) File "/usr/lib/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 2215, in GenerateOutput part_of_all=qualified_target in needed_targets) File "/usr/lib/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 794, in Write extra_mac_bundle_resources, part_of_all) File "/usr/lib/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 978, in WriteActions part_of_all=part_of_all, command=name) File "/usr/lib/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 1724, in WriteDoCmd force = True) File "/usr/lib/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 1779, in WriteMakeRule cmddigest = hashlib.sha1(command if command else self.target).hexdigest() TypeError: Unicode-objects must be encoded before hashing gyp ERR! configure error gyp ERR! stack Error:gyp` failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/usr/lib/node_modules/node-gyp/lib/configure.js:351:16)
gyp ERR! stack at ChildProcess.emit (events.js:210:5)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)
gyp ERR! System Linux 5.3.6-arch1-1-ARCH
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/geeray/uni/win-this/WebAnwendungen2/Praktikum/WebAnwendung2BackendAPI/WebAnw2Backend/node_modules/better-sqlite3
gyp ERR! node -v v12.12.0
gyp ERR! node-gyp -v v6.0.0
gyp ERR! not ok
npm WARN [email protected] No repository field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/geeray/.npm/_logs/2019-10-15T15_49_09_447Z-debug.log
`

[uestueab]

Alright, i had:
node v12.12.0
node-gyp v6

i had to downgrade node-gyp to version v5.0.3

[cekvenich]

@flatusv how did you downgrade?

[uestueab]

@cekvenich I had those packages installed via package manager. So for arch linux in /var/cache/pacman/pkg I looked for the next node-gyp version and did a "sudo pacman -U node-gyp.."

[burningTyger]

beginning with node v.12.13 it will also fail to compile. Had to revert to v12.12.0 and node-gyp@5

[wait-what]

Same issue here. Downgrading to [email protected] and node v12.12.0 fixes it.

[cekvenich]

Node v12.13 is LTS and it works on linux and mac without hacking node-gyp:
https://github.com/JoshuaWise/integer/issues/22

The issue is that majority of people install node via nvm: and they have no problems. So I switched to nvm.
I plan to try node v13.x

[imtbl]

Can confirm this issue on Arch using Node.js 13.0.1 and node-gyp 6.0.0.

[Sparkenstein]

Having the same issue on Manjaro, on NVS (Node Version Switcher) on Node 12.x.
Solved by downgrading version to 10.x
Anyone who is having this issue, try downgrading your node version to LTS by

nvm install --lts  // for nvm
nvs add lts        // for nvs

[cekvenich]

nvm should work with any version including 13.

[kevingarman]

Same issue here...was using Arch node/npm packages. Switched to nvm and it's working now...had to use v10 LTS. Thanks @Sparkenstein

nvm install --lts=dubnium

[andykais]

So weirdly this has stopped being an issue with pnpm (see original bug report here https://github.com/pnpm/pnpm/issues/2075) but is still an issue with npm. I do not know where exactly the fix occurred in pnpm though. If you are able to switch to pnpm, you can be unblocked by the latest nodejs installed via a native package manager. Fair warning though, it is not a drop in replacement. You will need to consider a few implications. It works for me though!

[cekvenich]

So I think I have solution for all.
Notes are in the 'integer' package for the same developer.
Short story, use nvm to install node and use python v2.

We should consolidate all these note under wiki.

[flaschbier]

Please note that Python 2 will be out of maintenance all too soon...
https://www.python.org/dev/peps/pep-0373/#update

[cekvenich]

@flaschbier Clearly. But when I build on my machines under v3 (fedora default) I get a python error - fixed when I switch to v2.
(I did see node v 13.1 adding support for v3, but who knows what else in npm is needed to convert )

[mikeshultz]

Also seen on Arch Linux with node v10 LTS.

Switching system-level python versions is not a reasonable solution.

[cekvenich]

@mikeshultz but it may not be related to this project. this may have to do w/ node js itself. so if there are PR the should go there. Or we can wait for node js upstream to fix it.
If there is another solution: I'm all ears. As end user of this nice lib, python 2 works for me. It is a problem for any downstream users of my lib: since I'm sure they don't know how to configure linux.

[mikeshultz]

but it may not be related to this projec

This is the only package I'm having issues building, and better-sqlite3 is literally the only results for searching this error.

I'm glad python 2 works for you @cekvenich, but changing what /usr/bin/python points to will cripple our systems and is not a reasonable workaround.

[cekvenich]

@mikeshultz What other C based package works w/ python 3 that you are using?

Whatever it is reasonable, I would agree with you, maybe not reasonable, and I listed the problem it makes for me. I posted it, but I'm ok if it is seen as 'unreasonable'. I suspect that once Node supports python 3 (as per recent git commit) the issue may fix itself.

There is also a sqlite .js based package that is ~ 20% slower in my tests than a C based one.

[austinmccalley]

I still had an issue working with python3 when installing. Switched to python2.7 seemed to fix the issue. It seems to be a problem with the NPM package integer.

[cekvenich]

@austinmccalley Thx for verifying. Maybe not the integer, but node itself(when you use a C module). I'm looking at release notes for node v12.13.1 and it says they are now supporting python v3.
I do plan to test python v3 w/ node v12.13.1 when I get to it.

[theflyingape]

Python 2 has been replaced with Python 3. This should be addressed here as a package install dependency. In the interim, I found that this works for me (using Fedora 31):

[rhurst@rampage dankdomain]$ python -V
Python 3.7.5

[rhurst@rampage dankdomain]$ ll /usr/bin/python*
lrwxrwxrwx 1 root root     9 Oct 17 07:54 /usr/bin/python -> ./python3
lrwxrwxrwx 1 root root     9 Oct 21 09:13 /usr/bin/python2 -> python2.7
-rwxr-xr-x 1 root root 16072 Oct 21 09:13 /usr/bin/python2.7
lrwxrwxrwx 1 root root     9 Oct 17 08:21 /usr/bin/python3 -> python3.7
-rwxr-xr-x 2 root root 15440 Oct 17 08:21 /usr/bin/python3.7
-rwxr-xr-x 2 root root 15440 Oct 17 08:21 /usr/bin/python3.7m
-rwxr-xr-x 1 root root   396 Sep  3 14:03 /usr/bin/python3-mako-render
lrwxrwxrwx 1 root root     3 Oct  1 11:01 /usr/bin/python3-sip -> sip
-rwxr-xr-x 1 root root  2558 Jul 25 10:14 /usr/bin/python-argcomplete-check-easy-install-script
-rwxr-xr-x 1 root root   318 Jul 25 10:14 /usr/bin/python-argcomplete-tcsh
-rwxr-xr-x 1 root root  4119 Jul 27 04:33 /usr/bin/python_count

[rhurst@rampage dankdomain]$ sudo alternatives --install /usr/bin/python python /usr/bin/python2 2
[rhurst@rampage dankdomain]$ sudo alternatives --install /usr/bin/python python /usr/bin/python3 1
[rhurst@rampage dankdomain]$ python -V
Python 2.7.17

[cdbattags]

Any update on this now that Node v12 (Erbium) is the current LTS?

Stacktrace:

Traceback (most recent call last):
  File "/home/deploy_user/.nvm/versions/node/v12.13.1/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py", line 50, in <module>
    sys.exit(gyp.script_main())
  File "/home/deploy_user/.nvm/versions/node/v12.13.1/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 554, in script_main
    return main(sys.argv[1:])
  File "/home/deploy_user/.nvm/versions/node/v12.13.1/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 547, in main
    return gyp_main(args)
  File "/home/deploy_user/.nvm/versions/node/v12.13.1/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 532, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "/home/deploy_user/.nvm/versions/node/v12.13.1/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 2215, in GenerateOutput
    part_of_all=qualified_target in needed_targets)
  File "/home/deploy_user/.nvm/versions/node/v12.13.1/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 794, in Write
    extra_mac_bundle_resources, part_of_all)
  File "/home/deploy_user/.nvm/versions/node/v12.13.1/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 978, in WriteActions
    part_of_all=part_of_all, command=name)
  File "/home/deploy_user/.nvm/versions/node/v12.13.1/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 1724, in WriteDoCmd
    force = True)
  File "/home/deploy_user/.nvm/versions/node/v12.13.1/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 1779, in WriteMakeRule
    cmddigest = hashlib.sha1(command if command else self.target).hexdigest()
TypeError: Unicode-objects must be encoded before hashing
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/home/deploy_user/.nvm/versions/node/v12.13.1/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:351:16)
gyp ERR! stack     at ChildProcess.emit (events.js:210:5)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)
gyp ERR! System Linux 4.15.0-72-generic
gyp ERR! command "/home/deploy_user/.nvm/versions/node/v12.13.1/bin/node" "/home/deploy_user/.nvm/versions/node/v12.13.1/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /srv/cdbattaglia.com/node_modules/better-sqlite3

[cekvenich]

@cdbattags I'm just a user: works great with any node version w/ python v2.

[mcoffin]

I can also confirm that this is a problem with python3 compatibility with node-gyp. The issue is present in v10.x.x, but appears to have been fixed before v12.x.x.
For temporary workarounds (without fixing the upstream problem in node-gyp... this issue is sort-of mis-filed under the wrong repository), you have the following options.

1. Set PYTHON=<python 2 executable> in your environment while running npm
2. Side-grade python 3 -> python 2 globally (probably not smart).
3. Use some kind of alternatives manager (like update-alternatives on ubuntu as @theflyingape mentioned above)

[theflyingape]

@mcoffin altering this environment variable also works for me, i.e.,
$ env PYTHON=$(which python2) npm install
Thanks!

[cekvenich]

@mcoffin Where should it be filed? Can you file and post here so we can +1 please.

[mcoffin]

@cekvenich Well, it's already been fixed in later LTS versions of node-gyp, so I doubt that they will ever add python 3 compatibility to v10, as there are much newer LTS version available now, which they are likely focusing on. If you want to use v10, I'd say just stick to using python2 for node-gyp. If you really do want to file a bug though, it would belong under node-gyp I think.

[cekvenich]

@mcoffin I get it to work w/ python 2 on the latest versions of everything, including node v13.x. I can never get it to work w/ python 3. So if you have a python 3 reproducible case, please share.

Aside, I know it uses node-gyp, but I don't see node-gyp in the dependencies.

[aminpaks]

I get this error with both Python 2 and 3!

Node: v12.14.0
node-gyp: v5.0.7

Python: v2.6

> [email protected] install /Users/{user}/Documents/Development/sqljs/node_modules/integer
> node-gyp rebuild

Traceback (most recent call last):
  File "/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py", line 47, in <module>
    import gyp
  File "/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 11, in <module>
    import argparse
ImportError: No module named argparse
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:351:16)
gyp ERR! stack     at ChildProcess.emit (events.js:210:5)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)
gyp ERR! System Darwin 17.7.0
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/{user}/Documents/Development/sqljs/node_modules/integer
gyp ERR! node -v v12.14.0
gyp ERR! node-gyp -v v5.0.5
gyp ERR! not ok 
npm WARN [email protected] No repository field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/{user}/.npm/_logs/2019-12-31T13_22_55_439Z-debug.log

Python: v3.7.3:

> [email protected] install /Users/{user}/Documents/Development/sqljs/node_modules/integer
> node-gyp rebuild

Traceback (most recent call last):
  File "/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py", line 50, in <module>
    sys.exit(gyp.script_main())
  File "/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 554, in script_main
    return main(sys.argv[1:])
  File "/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 547, in main
    return gyp_main(args)
  File "/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 532, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 2215, in GenerateOutput
    part_of_all=qualified_target in needed_targets)
  File "/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 802, in Write
    self.WriteCopies(spec['copies'], extra_outputs, part_of_all)
  File "/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 1145, in WriteCopies
    env = self.GetSortedXcodeEnv()
  File "/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 1885, in GetSortedXcodeEnv
    additional_settings)
  File "/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/xcode_emulation.py", line 1616, in GetSortedXcodeEnv
    additional_settings)
  File "/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/xcode_emulation.py", line 1527, in _GetXcodeEnv
    if XcodeVersion() >= '0500' and not env.get('SDKROOT'):
TypeError: '>=' not supported between instances of 'tuple' and 'str'
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:351:16)
gyp ERR! stack     at ChildProcess.emit (events.js:210:5)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)
gyp ERR! System Darwin 17.7.0
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/{user}/Documents/Development/sqljs/node_modules/integer
gyp ERR! node -v v12.14.0
gyp ERR! node-gyp -v v5.0.5
gyp ERR! not ok 
npm WARN [email protected] No repository field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/{user}/.npm/_logs/2019-12-31T13_26_10_787Z-debug.log

Oops this issue is related to linux, I'm installing on macOS :(

[cekvenich]

@aminpaks Try 2.7

[aminpaks]

@cekvenich tried with 2.7 and this is what I get:

> [email protected] install /Users/{user}/Documents/Development/sqljs/node_modules/better-sqlite3
> node-gyp rebuild

  TOUCH b857c92884e9598d609f6be182a2595df7a8e00f.intermediate
  ACTION deps_sqlite3_gyp_locate_sqlite3_target_extract_sqlite3 b857c92884e9598d609f6be182a2595df7a8e00f.intermediate
  TOUCH Release/obj.target/deps/locate_sqlite3.stamp
  CC(target) Release/obj.target/sqlite3/gen/sqlite3/sqlite3.o
  LIBTOOL-STATIC Release/sqlite3.a
  File "./gyp-mac-tool", line 607
    return {k: self._ExpandVariables(data[k], substitutions) for k in data}
                                                               ^
SyntaxError: invalid syntax
make: *** [Release/sqlite3.a] Error 1
rm b857c92884e9598d609f6be182a2595df7a8e00f.intermediate
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:210:5)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)
gyp ERR! System Darwin 17.7.0
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/{user}/Documents/Development/sqljs/node_modules/better-sqlite3
gyp ERR! node -v v12.14.0
gyp ERR! node-gyp -v v5.0.5
gyp ERR! not ok 
npm WARN [email protected] No repository field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/{user}/.npm/_logs/2020-01-02T12_24_41_365Z-debug.log

Here is the full log.

[aminpaks]

Apparently there's a new version v6.0.1 of node-gyp that fixed the issue but I still get the same error above cuz npm is not using it!

The reason is npm is using the old version v5.0.7 internally.

Result of $ npm i better-sqlite3 --verbose

gyp info using [email protected]
gyp info using [email protected] | darwin | x64

[aminpaks]

It seems people are moving away from node-gyp to a native js solution but it will take time.

If you're interested about this story.

[cekvenich]

Thanks @aminpaks for the link to the source of all our problems and the context for a solution!

[cekvenich]

@aminpaks you get error: rm b857c92884e9598d609f6be182a2595df7a8e00f.intermediate

I suggest you clean all your .npm (and .yarn files) and remove node, npm, global node libs, etc.
Then using nvm install node again and try with 2.7. It works for me every time. (also I use yarn instad of npm)

[edrex]

I was stuck on this (Arch Linux) for awhile and was able to get past by deleting my .node_modules and global module dir, then running with env PYTHON=$(which python2) yarn start as suggested by @theflyingape

[cekvenich]

FYI there is a new version of node-gyp:
https://github.com/nodejs/node-gyp/blob/master/CHANGELOG.md

I think if you install it globally python 3 may work (until new versions of node are released with he new node-gyp, and that maybe allow us to close all these build bugs)

[cekvenich]

And node LTS just did a release touching gyp:
https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V12.md#12.14.1
Funny that latest note release does not have those PRs, only LTS.
I plan to test LTS!

[aminpaks]

I found this a while back too, but didn't work for me.

It talks about submodules of npm and how to update them:

$ [sudo] npm explore npm -g -- npm install node-gyp@latest

[thomotron]

I found this a while back too, but didn't work for me.

$ [sudo] npm explore npm -g -- npm install node-gyp@latest

This worked for me without needing to touch my Python install (thankfully).

I was having trouble with that solution initially as npm was spitting out npm ERR! It doesn't look like npm is installed.. I upgraded npm with npm (npm install -g npm) and the above command worked perfectly.

[cekvenich]

@thomotron Thank you for that information. I plan to try it, except I use yarn.

[cekvenich]

@thomotron your solution works! This issue should now be closed.

[aminpaks]

@thomotron what's your Python version?

[imtbl]

The following does also work for me on Arch without having to fiddle with subdependencies:

$ PYTHON=$(which python2) yarn install

The sad state of Node.js.

[cekvenich]

It's npm. @mserajnik Feel free to upvote this upstream https://github.com/npm/cli/issues/287

[JoshuaWise]

This should be fixed in the latest version of better-sqlite3 (v6.0.1), due to the use of prebuilt binaries.