Electrs: Electrs build fails on Rust version 1.38.0

[openoms]

Had two reports on multiple build error when attempting to install with the latest Rust version 1.38.0.
1.37.0 gives no errors, but new installs break. https://github.com/openoms/bitcoin-tutorials/issues/18

[richdbtc]

ACK @openoms issue, details below:

Rust version-architecture-operating system: 1.38.0-armv7-unknown-linux-gnueabihf

Execution of the following compile command:

cargo build --release

fails on compilation of support libs:

error: Could not compile lazy_static.

Caused by:
  process didn't exit successfully: rustc --crate-name lazy_static /home/electrs/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-1.3.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=b1f83d44d9cacfbb -C extra-filename=-b1f83d44d9cacfbb --out-dir /home/electrs/electrs/target/release/deps -L dependency=/home/electrs/electrs/target/release/deps --cap-lints allow (signal: 11, SIGSEGV: invalid memory reference)
warning: build failed, waiting for other jobs to finish...

Similar errors for scopeguard, libc, and arrayvec

Building with default target of debug does not result in failure.

Downgrading to Rust 1.3.7 in the following manner results in build success for release target:

rustup install 1.37.0
rustup override set 1.37.0

[romanz]

Should be fixed @ latest master.

[romanz]

Builds successfully on x86:

$ cargo --version && cargo build --release
cargo 1.38.0 (23ef9a4ef 2019-08-20)
   Compiling semver-parser v0.7.0
   Compiling arrayvec v0.4.11
   Compiling lazy_static v1.4.0
   Compiling nodrop v0.1.13
   Compiling libc v0.2.62
   Compiling cfg-if v0.1.10
   Compiling scopeguard v1.0.0
   Compiling rayon-core v1.6.0
   Compiling either v1.5.3
   Compiling proc-macro2 v0.4.30
   Compiling unicode-xid v0.1.0
   Compiling syn v0.15.44
   Compiling memchr v2.2.1
   Compiling failure_derive v0.1.5
   Compiling version_check v0.1.5
   Compiling log v0.4.8
   Compiling glob v0.3.0
   Compiling rustc-demangle v0.1.16
   Compiling byteorder v1.3.2
   Compiling proc-macro2 v1.0.4
   Compiling bitflags v1.2.0
   Compiling regex-syntax v0.6.12
   Compiling unicode-width v0.1.6
   Compiling quick-error v1.2.2
   Compiling unicode-xid v0.2.0
   Compiling ansi_term v0.11.0
   Compiling vec_map v0.8.1
   Compiling termcolor v1.0.5
   Compiling bindgen v0.49.2
   Compiling strsim v0.8.0
   Compiling autocfg v0.1.6
   Compiling serde v1.0.101
   Compiling syn v1.0.5
   Compiling peeking_take_while v0.1.2
   Compiling shlex v0.1.1
   Compiling smallvec v0.6.10
   Compiling configure_me_codegen v0.3.9
   Compiling roff v0.1.0
   Compiling matches v0.1.8
   Compiling winapi-build v0.1.1
   Compiling protobuf v2.8.1
   Compiling fmt2io v0.1.0
   Compiling ryu v1.0.0
   Compiling gcc v0.3.55
   Compiling unicode-segmentation v1.3.0
   Compiling percent-encoding v1.0.1
   Compiling arc-swap v0.4.3
   Compiling bech32 v0.6.0
   Compiling winapi v0.2.8
   Compiling prometheus v0.5.0
   Compiling chunked_transfer v0.3.1
   Compiling hex v0.3.2
   Compiling spin v0.4.10
   Compiling rustc-serialize v0.3.24
   Compiling itoa v0.4.4
   Compiling termcolor v0.3.6
   Compiling ascii v0.8.7
   Compiling fnv v1.0.6
   Compiling parse_arg v0.1.4
   Compiling hashbrown v0.5.0
   Compiling crossbeam-utils v0.6.6
   Compiling thread_local v0.3.6
   Compiling semver v0.9.0
   Compiling semver v0.6.0
   Compiling textwrap v0.11.0
   Compiling humantime v1.3.0
   Compiling nom v4.2.3
   Compiling error-chain v0.12.1
   Compiling man v0.1.1
   Compiling unicode-bidi v0.3.4
   Compiling unicode-normalization v0.1.8
   Compiling clang-sys v0.28.1
   Compiling kernel32-sys v0.2.2
   Compiling bitcoin-bech32 v0.9.0
   Compiling num-traits v0.2.8
   Compiling num-integer v0.1.41
   Compiling bincode v1.2.0
   Compiling crossbeam-queue v0.1.2
   Compiling crossbeam-channel v0.3.9
   Compiling rustc_version v0.2.3
   Compiling lru v0.1.17
   Compiling build-helper v0.1.1
   Compiling num_cpus v1.10.1
   Compiling atty v0.2.13
   Compiling time v0.1.42
   Compiling rand v0.4.6
   Compiling errno v0.2.4
   Compiling signal-hook-registry v1.1.1
   Compiling dirs v1.0.5
   Compiling page_size v0.4.1
   Compiling quote v0.6.13
   Compiling quote v1.0.2
   Compiling memoffset v0.5.1
   Compiling aho-corasick v0.7.6
   Compiling fxhash v0.2.1
   Compiling base64 v0.10.1
   Compiling clap v2.33.0
   Compiling rust-crypto v0.2.36
   Compiling signal-hook v0.1.10
   Compiling rand v0.3.23
   Compiling idna v0.1.5
   Compiling sysconf v0.3.4
   Compiling regex v1.3.1
   Compiling cexpr v0.3.5
   Compiling url v1.7.2
   Compiling env_logger v0.6.2
   Compiling chrono v0.4.9
   Compiling crossbeam-epoch v0.7.2
   Compiling synstructure v0.10.2
   Compiling toml v0.4.10
   Compiling bitcoin_hashes v0.3.2
   Compiling serde_test v1.0.101
   Compiling serde_json v1.0.40
   Compiling stderrlog v0.4.1
   Compiling tiny_http v0.6.2
   Compiling crossbeam-deque v0.7.1
   Compiling rayon v1.2.0
   Compiling cc v1.0.26
   Compiling serde_derive v1.0.101
   Compiling backtrace-sys v0.1.30
   Compiling libloading v0.5.2
   Compiling secp256k1 v0.12.2
   Compiling backtrace v0.3.38
   Compiling failure v0.1.5
   Compiling which v2.0.1
   Compiling bitcoin v0.18.2
   Compiling configure_me v0.3.3
   Compiling electrs v0.7.1 (/home/roman/Code/electrs)
   Compiling librocksdb-sys v6.1.2
   Compiling rocksdb v0.12.3
    Finished release [optimized] target(s) in 2m 07s

[romanz]

Please let me know if the issue persist on 1.38.0-armv7-unknown-linux-gnueabihf.

[openoms]

Please let me know if the issue persist on 1.38.0-armv7-unknown-linux-gnueabihf.

Sure will do, just did not have time to test yet. Will close this now and will let you know if there would be any issues when building with Rust 1.38.0-armv7.

[bavarianledger]

Please let me know if the issue persist on 1.38.0-armv7-unknown-linux-gnueabihf.

I'm still getting the same errors on 1.38.0-armv7-unknown-linux-gnueabihf using the lastest master.
@openoms Can you confirm this too?

[openoms]

Attempted to build on Odroid HC2 with
Rust stable-armv7-unknown-linux-gnueabihf version 1.38.0 (625451e37 2019-09-23)
Getting build errors after sudo -u electrs /home/electrs/.cargo/bin/cargo build --release:

   Compiling semver-parser v0.7.0
   Compiling arrayvec v0.4.11
   Compiling libc v0.2.62
   Compiling cfg-if v0.1.10
   Compiling lazy_static v1.4.0
   Compiling nodrop v0.1.13
   Compiling scopeguard v1.0.0
   Compiling rayon-core v1.6.0
   Compiling either v1.5.3
   Compiling proc-macro2 v0.4.30
error: Could not compile `scopeguard`.

Caused by:
  process didn't exit successfully: `rustc --crate-name scopeguard /home/electrs/.cargo/registry/src/github.com-1ecc6299db9ec823/scopeguard-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=6f24f6c835eddc79 -C extra-filename=-6f24f6c835eddc79 --out-dir /home/electrs/electrs/target/release/deps -L dependency=/home/electrs/electrs/target/release/deps --cap-lints allow` (signal: 11, SIGSEGV: invalid memory reference)
warning: build failed, waiting for other jobs to finish...
error: Could not compile `rayon-core`.

Caused by:
  process didn't exit successfully: `rustc --crate-name build_script_build /home/electrs/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/build.rs --color always --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=fa638daa4e32f715 -C extra-filename=-fa638daa4e32f715 --out-dir /home/electrs/electrs/target/release/build/rayon-core-fa638daa4e32f715 -L dependency=/home/electrs/electrs/target/release/deps --cap-lints allow` (signal: 11, SIGSEGV: invalid memory reference)
warning: build failed, waiting for other jobs to finish...
error: Could not compile `proc-macro2`.

Caused by:
  process didn't exit successfully: `rustc --crate-name build_script_build /home/electrs/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.30/build.rs --color always --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=cb3f0d8db81aeb7b -C extra-filename=-cb3f0d8db81aeb7b --out-dir /home/electrs/electrs/target/release/build/proc-macro2-cb3f0d8db81aeb7b -L dependency=/home/electrs/electrs/target/release/deps --cap-lints allow` (signal: 11, SIGSEGV: invalid memory reference)
warning: build failed, waiting for other jobs to finish...
free(): invalid next size (fast)
error: Could not compile `arrayvec`.

Caused by:
  process didn't exit successfully: `rustc --crate-name build_script_build /home/electrs/.cargo/registry/src/github.com-1ecc6299db9ec823/arrayvec-0.4.11/build.rs --color always --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=3d808923920ed907 -C extra-filename=-3d808923920ed907 --out-dir /home/electrs/electrs/target/release/build/arrayvec-3d808923920ed907 -L dependency=/home/electrs/electrs/target/release/deps --cap-lints allow` (signal: 6, SIGABRT: process abort signal)
warning: build failed, waiting for other jobs to finish...
thread '<unnamed>' panicked at 'assertion failed: (*tail).value.is_none()', src/libstd/sync/mpsc/mpsc_queue.rs:97:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
free(): invalid next size (fast)
error: Could not compile `libc`.

Caused by:
  process didn't exit successfully: `rustc --crate-name build_script_build /home/electrs/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.62/build.rs --color always --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=6443112b83d07c60 -C extra-filename=-6443112b83d07c60 --out-dir /home/electrs/electrs/target/release/build/libc-6443112b83d07c60 -L dependency=/home/electrs/electrs/target/release/deps --cap-lints allow` (signal: 6, SIGABRT: process abort signal)
warning: build failed, waiting for other jobs to finish...
error: Could not compile `semver-parser`.

Caused by:
  process didn't exit successfully: `rustc --crate-name semver_parser /home/electrs/.cargo/registry/src/github.com-1ecc6299db9ec823/semver-parser-0.7.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=cf7b948e30f829aa -C extra-filename=-cf7b948e30f829aa --out-dir /home/electrs/electrs/target/release/deps -L dependency=/home/electrs/electrs/target/release/deps --cap-lints allow` (signal: 11, SIGSEGV: invalid memory reference)

[openoms]

Problem confirmed on the Raspberry Pi 4 (RasiBlitz 1.3).
Rust 1.37.0 builds ok, but using

info: syncing channel updates for 'stable-armv7-unknown-linux-gnueabihf'
info: latest update on 2019-09-26, rust version 1.38.0 (625451e37 2019-09-23)

leads to:

   Compiling semver-parser v0.7.0
   Compiling arrayvec v0.4.11
   Compiling lazy_static v1.4.0
   Compiling libc v0.2.62
   Compiling nodrop v0.1.13
   Compiling cfg-if v0.1.10
   Compiling scopeguard v1.0.0
error: Could not compile `scopeguard`.

Caused by:
  process didn't exit successfully: `rustc --crate-name scopeguard /home/electrs/.cargo/registry/src/github.com-1ecc6299db9ec823/scopeguard-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=6f24f6c835eddc79 -C extra-filename=-6f24f6c835eddc79 --out-dir /home/electrs/electrs/target/release/deps -L dependency=/home/electrs/electrs/target/release/deps --cap-lints allow` (signal: 11, SIGSEGV: invalid memory reference)
warning: build failed, waiting for other jobs to finish...
error: Could not compile `libc`.

Caused by:
  process didn't exit successfully: `rustc --crate-name build_script_build /home/electrs/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.62/build.rs --color always --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=6443112b83d07c60 -C extra-filename=-6443112b83d07c60 --out-dir /home/electrs/electrs/target/release/build/libc-6443112b83d07c60 -L dependency=/home/electrs/electrs/target/release/deps --cap-lints allow` (signal: 11, SIGSEGV: invalid memory reference)
warning: build failed, waiting for other jobs to finish...
thread '<unnamed>' panicked at 'assertion failed: (*tail).value.is_none()', src/libstd/sync/mpsc/mpsc_queue.rs:97:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
free(): invalid pointer
error: Could not compile `arrayvec`.

Caused by:
  process didn't exit successfully: `rustc --crate-name build_script_build /home/electrs/.cargo/registry/src/github.com-1ecc6299db9ec823/arrayvec-0.4.11/build.rs --color always --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=3d808923920ed907 -C extra-filename=-3d808923920ed907 --out-dir /home/electrs/electrs/target/release/build/arrayvec-3d808923920ed907 -L dependency=/home/electrs/electrs/target/release/deps --cap-lints allow` (signal: 6, SIGABRT: process abort signal)
warning: build failed, waiting for other jobs to finish...
error: Could not compile `semver-parser`.

Caused by:
  process didn't exit successfully: `rustc --crate-name semver_parser /home/electrs/.cargo/registry/src/github.com-1ecc6299db9ec823/semver-parser-0.7.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=cf7b948e30f829aa -C extra-filename=-cf7b948e30f829aa --out-dir /home/electrs/electrs/target/release/deps -L dependency=/home/electrs/electrs/target/release/deps --cap-lints allow` (signal: 11, SIGSEGV: invalid memory reference)

[romanz]

IIUC, it seems that something got broken on the ARM Rust toolchain at 1.38 :(
Could you please try to reproduce the failure when building a single package, e.g. arrayvec on Raspberry Pi?

[romanz]

Please use https://github.com/romanz/test_arrayvec (which uses the same arrayvec version as electrs).
Thinking about it, maybe we can try updating the package versions to see if the problem persists when building on ARM (e.g. arrayvec can be bumped from 0.4.11 to 0.5.1)?

[openoms]

@romanz sorry, I am not very familiar with the Rust syntax. Can you please provide specific commands to build the single package?

I understand it is not a long term solution, but is there a problem if we keep using Rust 1.37.0 ?

[romanz]

No problem, please clone the repo (https://github.com/romanz/test_arrayvec) and run cargo build.

[romanz]

Please do keep using 1.37 while we fix the issue.

[romanz]

Could you try to reproduce this issue with Rust 1.39.0?

[openoms]

Could you try to reproduce this issue with Rust 1.39.0?

Builds without problems on armv7l (RPi4) on Rust 1.39.0 and the latest commit from this repo.
Issue solved.

[openoms]

Do you know a good method to fix the Rust installation to the last known to work (now 1.39.0) version?

For this issue we have used :

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y 
/home/electrs/.cargo/bin/rustup install 1.37.0 --force
/home/electrs/.cargo/bin/rustup override set 1.37.0

but this results to installing the latest and the selected version as well.
There must be an easier way to do this.

[romanz]

I am usually updating using:

$ rustup update stable
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'

  stable-x86_64-unknown-linux-gnu unchanged - rustc 1.39.0 (4560ea788 2019-11-04)

info: checking for self-updates

But I think that it can also work with specific versions (e.g. rustup update 1.39.0).

[openoms]

Rather than updating I would like to avoid people installing a newer Rust version with my script before I would have tested it to avoid issues like this one earlier.

So when for example 1.40 comes out how can I still force to still install the 1.39?
Could not find a way so far apart from installing the latest than downgrading.

[openoms]

ok, I think I got it for our case:

sudo -u electrs curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sudo -u electrs sh -s -- --default-toolchain 1.39.0 -y