Influxdb: [arm] Influxd in build 2.0.0-beta.12_linux_arm64 unable to start

[FixTestRepeat]

__Steps to reproduce:__

1. upgrade (`wget https://dl.influxdata.com/influxdb/releases/influxdb_2.0.0-beta.12_linux_arm64.tar.gz ; tar xvf influxdb_2.0.0-beta.12_linux_arm64.tar.gz ; sudo cp influxdb_2.0.0-beta.12_linux_arm64/influxd /usr/local/bin/influxd
2. stop/start: (sudo systemctl restart influxdb2.service)
3. check status: (sudo systemctl status influxdb2.service)

__Expected behavior:__
Influxd should be self contained / no dependencies

__Actual behavior:__
influxdb_2.0.0-beta.12 has been dynamically linked, requiring external libs.

influxdb2.service - InfluxDB 2.0 service file.
   Loaded: loaded (/lib/systemd/system/influxdb2.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2020-06-21 07:57:46 AEST; 1s ago
     Docs: https://v2.docs.influxdata.com/v2.0/get-started/
  Process: 24369 ExecStart=/usr/local/bin/influxd run --reporting-disabled --engine-path /mnt/Influx/.influxdbv2/engine --bolt-path /mnt/Influx/.influxdbv2/influxd.bolt
 Main PID: 24369 (code=exited, status=203/EXEC)

Jun 21 07:57:46 raspiM systemd[1]: Started InfluxDB 2.0 service file..
Jun 21 07:57:46 raspiM systemd[24369]: influxdb2.service: Failed to execute command: No such file or directory
Jun 21 07:57:46 raspiM systemd[24369]: influxdb2.service: Failed at step EXEC spawning /usr/local/bin/influxd: No such file or directory
Jun 21 07:57:46 raspiM systemd[1]: influxdb2.service: Main process exited, code=exited, status=203/EXEC
Jun 21 07:57:46 raspiM systemd[1]: influxdb2.service: Failed with result 'exit-code'.

__Environment info:__

• System info: Linux 4.19.118-v8+ aarch64
• InfluxDB version: influxdb_2.0.0-beta.12
• Other relevant environment details: n/a

__Config:__

[Unit]
Description=InfluxDB 2.0 service file.
Documentation=https://v2.docs.influxdata.com/v2.0/get-started/
After=network-online.target

[Service]
User=influxdb
Group=influxdb
ExecStart=/usr/local/bin/influxd run \
        --reporting-disabled \
#       --log-level debug \
        --engine-path "/mnt/Influx/.influxdbv2/engine" \
        --bolt-path "/mnt/Influx/.influxdbv2/influxd.bolt"
#Restart=on-failure

[Install]
WantedBy=multi-user.target

__File output for 2.0.0-beta.12:__

influxdb_2.0.0-beta.12_linux_arm64/influxd: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, Go BuildID=aHEnMbH3jnHbzTod6bt9/c7v578K9qlMX-P1uQU6F/T6FL3PGJEis7OEaYjwAw/CVpdbj1YgY9bLl5D0YTO, BuildID[sha1]=4bac3fd7d9409ed975bc29a4754638c43f69e24b, stripped

__File output for previous beta 2.0.0-beta.10:__

influxdb_2.0.0-beta.10_linux_arm64/influxd: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=3yWPtBaN8cmoTWpU2Zap/gnrg3SOyddixPc6D7TzU/0MazSKoIGhGWFEF5fAE5/9HvOGkG_BRrqvdK9d1bh, stripped

[drpuur]

Is still the same with beta.13:

[root@mb-dev influxdb_2.0.0-beta.13_linux_amd64]# ldd influx influxd
influx:
./influx: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by ./influx)
    linux-vdso.so.1 =>  (0x00007ffc297e1000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f71861b2000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7185f96000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f7185d80000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f71859b2000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f71863b6000)
influxd:
./influxd: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by ./influxd)
    linux-vdso.so.1 =>  (0x00007ffd76abe000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6170167000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f616ff63000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f616fd4d000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f616f97f000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6170383000)

[FixTestRepeat]

@russorat , is this the new norm? If not can you organize to get another build for beta13 ?

[russorat]

thanks for opening this. let me ask around internally and see what our options are.

[rickspencer3]

@nathanielc the next beta milestone is second week of July, seems pretty important that this works by then ;)

Could you please assign it appropriately and get in your next sprint, if possible?

[russorat]

@FixTestRepeat @drpuur @gonghito and others on this thread, looks like a recent change to a dependent library had a side effect of turning our binaries from static to dynamic. We didn't go out of our way to make static binaries originally, they just happened to be that way (See #17675).

We are working on build changes to add static builds to the next release due the second week of july. Until then, if you need static binaries, you will need to stay on Beta-10.

[wolffcm]

We've made this dependent on #17675, and moved that issue into our current sprint which started today. We are fairly certain that issues with the arm64 build will be resolved by having static linking. Once that issue is addressed, we will make sure arm64 builds are working right, and hopefully we can just close this issue.

[davidgs]

Same issue is present on linux amd64 build for beta13.
./influxd: /lib64/libc.so.6: versionGLIBC_2.18' not found (required by ./influxd)`

[russorat]

Just released a beta 14. Can someone check the binary to ensure it is statically compiled again? https://github.com/influxdata/influxdb/releases/tag/v2.0.0-beta.14

[FixTestRepeat]

@russorat , beta14 seems to be still dynamically linked:

file influxdb_2.0.0-beta.14_linux_arm64/influxd
influxdb_2.0.0-beta.14_linux_arm64/influxd: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, Go BuildID=N_XxBa8kFqmZ6SssmuJ1/UqYGjsR4enaFdh-e2A5I/neOu2BlGX4WsHFjqBVBd/U2hsrHoNAQZPF5tfH0xf, BuildID[sha1]=fcffa4c7005e82153d816aedc43393877dbf51cb, stripped

[jsternberg]

We're planning to try and fix the arm64 binaries at some point in the future, but we had already built the amd64 binaries statically for a different task so switching OSS to build the same way was only a small amount of work. Since we changed flux to use a C ABI (the Rust code), we haven't yet built the arm64 statically and we haven't had time to set up a developer with an arm64 machine to test the resulting binaries.

The instructions for building statically on ARM64 should be similar to AMD64, but figuring out the cross compiler toolchains we need and the build flags is work we haven't been able to do yet. If you have any insight that would greatly aid us in building the binary statically again.

For AMD64, the following is required:

1. A toolchain for Rust that builds the Rust code fully statically. There's a musl compiler toolchain for AMD64 that does this.
2. A musl-gcc compiler that compiles the cgo stubs statically. We use musl-gcc on AMD64 for this.

When I took an initial look at this, I couldn't find a musl compiler toolchain for ARM64 using rustup and I hadn't researched where to find a musl-gcc compiler that cross compiled for arm64 yet. We're going to try to investigate these for an upcoming beta but just haven't been able to yet.

[FixTestRepeat]

thanks @jsternberg - i'd like to help but can't get the build environment to work. Are the instructions on here still current?

edit: currently fails here :

$ go install ./...
# pkg-config --cflags  -- flux
Started pkg-config  {"arg0": "/home/bob/.gvm/pkgsets/go1.13/global/bin/pkg-config", "args": ["--cflags", "--", "flux"]}
Found pkg-config executable {"path": "/usr/bin/pkg-config"}
Determined module root  {"path": "/home/bob/.gvm/pkgsets/go1.13/global/pkg/mod/github.com/influxdata/[email protected]"}
Flux module is the main module  {"modroot": "/home/bob/.gvm/pkgsets/go1.13/global/pkg/mod/github.com/influxdata/[email protected]"}
Executing cargo build   {"dir": "/home/bob/.cache/go-build/pkgconfig/github.com/influxdata/[email protected]/libflux", "target": "x86_64-unknown-linux-gnu"}
Error installing library    {"name": "flux", "error": "exec: \"cargo\": executable file not found in $PATH"}
pkg-config: exit status 1

[jsternberg]

Yes they are. The error message you see there indicates that it can't find the cargo executable for compiling rust. You can either ensure cargo is on the path or you can set the CARGO environment variable to the path.

[FixTestRepeat]

hi @jsternberg , where is current investigation at? I noticed compiling using make influxd also produces dynamically linked binary for ordinary linux amd64 arch (before even looking at cross-compiling). What build command and flags are being used for the official beta builds currently? (Given that those are statically linked)

[bob@manjaro20 influxdb]$ make influxd
env GO111MODULE=on go build -tags '' -ldflags " -X main.commit=65d9b2c088" -o bin/linux/influxd ./cmd/influxd
[bob@manjaro20 influxdb]$ file bin/linux/influxd
bin/linux/influxd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=8958ce0cd87744256c9dff8c15fa2c0ae354417d, for GNU/Linux 3.2.0, not stripped

ed: amd64 official beta15:

influxdb_2.0.0-beta.15_linux_amd64/influxd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=4fmfdsBO2Iwog3tS5DgW/gfDhWhksjR3WhXqqmMJh/b51x3hkb5oHjzP_ifi9j/WPCUdJaVFnybsj9Lv_ye, stripped

[FixTestRepeat]

@russorat any news on this one?

I got another toolchain compiled, but now stuck on how to specify using it during make

[russorat]

@FixTestRepeat unfortunately, not much progress. Last time we looked, we couldn't find a rust toolchain for static compilation on arm. Do you know of any?

[davidgraeff]

There is https://rust-lang.github.io/rustup-components-history/aarch64-unknown-linux-musl.html and https://rust-lang.github.io/rustup-components-history/arm-unknown-linux-musleabi.html

[FixTestRepeat]

Anyone had any luck with these?
Russ ?

[davidgraeff]

Some of the rc's and the release doesn't even have arm64 builds available anymore. Is this not a supported architecture?

[danxmoran]

@FixTestRepeat @drpuur @davidgs @davidgraeff our 2.0.3 release includes an updated ARM64 build which we hope will address issues like this. You can download the binaries here: https://dl.influxdata.com/influxdb/releases/influxdb2-2.0.3_linux_arm64.tar.gz

Please let me know how it goes if you try it out!

[danxmoran]

I'm going to close this issue since I've seen reports in the community Slack that the new ARM binaries do (at least) start up, but please do open new issues if you try them out and hit problems!