Node-sqlite3: Please add binary for arm64

[creationix]

I'm trying to use this module on some arm64 devices and our build times are really long because there is no prebuilt binary for us.

node-pre-gyp ERR! Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v3.1.13/node-v57-linux-arm64.tar.gz 
node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v57 ABI) (falling back to source compile with node-gyp) 

I'm using the new LTS version of node.

You can reproduce this on a raspberry pi 3 using https://github.com/bamarni/pi64 which gives a 64-bit kernel. It does eventually compile from source, but every time uses about 15 minutes of time.

[creationix]

Hmm, actually, I'm having trouble build from source in some cases as well.

/srv/walnut/core/node_modules/sqlite3 # uname -m
aarch64
/srv/walnut/core/node_modules/sqlite3 # npm rebuild sqlite3
/srv/walnut/core/node_modules/sqlite3 # node -r sqlite3
module.js:538
    throw err;
    ^

Error: Cannot find module '/srv/walnut/core/node_modules/sqlite3/lib/binding/node-v57-linux-arm64/node_sqlite3.node'
    at Function.Module._resolveFilename (module.js:536:15)
    at Function.Module._load (module.js:466:25)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/srv/walnut/core/node_modules/sqlite3/lib/sqlite3.js:4:15)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)

I don't think it even tried to build from source when I initially ran npm install and npm rebuild doesn't seem to do anything.

Any ideas how I can debug node-prep-gyp issues here?

[creationix]

There is no ./lib/binding folder at all FWIW, which leads me to believe more it didn't try building the binary.

[springmeyer]

Please add binary for arm64

Yes, this would really help users. To make providing an arm64 binary sustainable for each release, we'd need an automated way of doing this. Do you have suggestions on how to script building for arm64 on travis (or similar free CI infra)?

There is no ./lib/binding folder at all FWIW, which leads me to believe more it didn't try building the binary.

Odd, @creationix can you try passing --build-from-source to your npm rebuild command? Without that node-pre-gyp will be attempting to install from a remote binary each time, by default. Clearly something is wrong with the fallback though that I can't tell from your logs.

[creationix]

Assuming your CI server has support for qemu-aarch64-static and docker, something like the following should work:

FROM arm64v8/node:carbon
COPY qemu-aarch64-static /usr/bin/qemu-aarch64-static
RUN apt-get update
RUN apt-get install -y git build-essential
RUN git clone --depth=1 https://github.com/mapbox/node-sqlite3.git
WORKDIR /node-sqlite3
RUN make

See this article for more details on using qemu for cross-compiling with docker.

[johnelliott]

Another raspberry pi 3 node-sqlite user here. Pre-built would be wonderful 👍

[SridharRamasami]

One more raspberry pi 3 user here.

[lucianodlf]

another user is added to the order

[rzr]

I guess this could be built on qemu , check this related change:
https://github.com/mapbox/node-sqlite3/pull/1028

[k1r0s]

yet another non amd64 user

[rzr]

Relate to: https://github.com/mapbox/node-sqlite3/pull/1028

[Slos]

and another ARM user here. I've been trying all sorts of tricks to get this to build on my raspi without success. Please add pre built binary to the repo... thanks a lot!

[derkomai]

And another one user. Couldn't make it work on a raspberry pi zero w.

[rzr]

have you tried to build in docker ? may I also request this package to land in debian too

https://tracker.debian.org/pkg/node-sqlite3

[songzhuang]

Hi,
I can volunteer to create the build for the Pi. If someone knows the steps, please point me to it. I guess it would be:
1) where to get the original files
2) how to compile
3) where and how to push the compiled files.

Regards,

[rzr]

Well I used this PR to build offline:

https://github.com/mapbox/node-sqlite3/pull/1028

@songzhuang you can try to see how this can be adapted for arm64 for @k1r0s

Good luck

[LudekStoklasa]

Hi, any idea where to download binary for arm64 ?
I have been trying to compile it myself on Synology NAS DS118 (ARMv8 64), but was unsuccessful :-/
Any tip where to download it or how to compile it?

[rzr]

I managed to build current version using docker, it tooks me less than 8min

I shared file at:
https://bintray.com/rzr/devel/sqlite3#files
0568e185fdeb1433d8d92694f2d35a25e48694c53a210eaf09ed522e1c0633c9 sqlite3/v4.0.6/node-v57-linux-arm64.tar.gz

Patch to come next

Relate-to: https://github.com/mapbox/node-sqlite3/issues/1138

[n1ru4l]

With docker buildx it is quite easy to build binaries for multiple platforms.

We are currently using it for building docker images for all platforms. We could incorporate it into the CI for easily generating pre-built binaries for SQLite.

I think it would be as easy as cloning this repository, running npm install --ignore-scripts and npx node-pre-gyp install --build-from-source and then upload the build artifacts.

A matrix could then cover all the platforms e.g. linux/arm/v6, linux/arm/v7, linux/arm64

[louisabraham]

Any update?

[n1ru4l]

@louisabraham I created this PR that adds binaries, feel free to review and test the binaries. https://github.com/mapbox/node-sqlite3/pull/1362

[louisabraham]

Thank you so much, they work!

[gpetrov]

Any progress in making the binaries officially available?
I'm getting:

#13 11.19 > [email protected] install /opt/node_app/node_modules/sqlite3
#13 11.19 > node-pre-gyp install --fallback-to-build
#13 11.19 
#13 11.35 node-pre-gyp WARN Using request for node-pre-gyp https download 
#13 11.81 node-pre-gyp WARN Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.0/napi-v3-linux-arm64.tar.gz 
#13 11.81 node-pre-gyp WARN Pre-built binaries not found for [email protected] and [email protected] (node-v83 ABI, glibc) (falling back to source compile with node-gyp)

Note linux/arm64 binaries are very important for the upcoming Docker Desktop for Apple Silicon M1 as those will run natively there.