Electrs: Core dump on startup [0.8.0]

[kallerosenbaum]

I just upgraded from 0.7.1 to 0.8.0. 0.7.1 worked fine but 0.8.0 prints this on the log:

$ cargo run --release
    Finished release [optimized] target(s) in 0.87s
     Running `target/release/electrs`
Config { log: StdErrLog { verbosity: Debug, quiet: false, timestamp: Off, modules: [], writer: "stderr", color_choice: Auto }, network_type: bitcoin, db_path: "/mnt/bitcoin/.electrs/db/mainnet", daemon_dir: "/mnt/bitcoin/.bitcoin", daemon_rpc_addr: V4(127.0.0.1:8332), cookie: Some("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"), electrum_rpc_addr: V4(192.168.1.3:50001), monitoring_addr: V4(127.0.0.1:4224), jsonrpc_import: false, index_batch_size: 100, bulk_index_threads: 4, tx_cache_size: 10485760, txid_limit: 100, server_banner: "Welcome to electrs 0.8.0 (Electrum Rust Server)!", blocktxids_cache_size: 10485760 }
DEBUG - Server listening on 127.0.0.1:4224
DEBUG - Running accept thread
INFO - NetworkInfo { version: 180100, subversion: "/Satoshi:0.18.1/" }
INFO - BlockchainInfo { chain: "main", blocks: 603796, headers: 603796, bestblockhash: "00000000000000000015fd5c7fc95b2040c3975f8d4ee86dd9b1995a387f6cde", pruned: false, initialblockdownload: false }
DEBUG - opening DB at "/mnt/bitcoin/.electrs/db/mainnet"
Illegal instruction (core dumped)

I upgraded by

1. Stop electrs 0.7.1
2. git fetch
3. git checkout Cargo.lock (I had to in order to checkout v0.8.0)
4. git checkout v0.8.0
5. cargo build --release

Here's my ~/.electrs/config.toml:

cookie = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
verbose = 3
db_dir = "/mnt/bitcoin/.electrs/db"
electrum_rpc_addr = "192.168.1.3:50001"

I also tried to move away the db dir to start from scratch, but it didn't help.

[kallerosenbaum]

$ rustc --version
rustc 1.38.0 (625451e37 2019-09-23)
$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
$ uname -a
Linux wilfrid 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[romanz]

Thanks for opening the issue!
Could you please update to Rust 1.39 to see if the problem reproduces?

[kallerosenbaum]

Same problem with rust 1.39:

$ git status
HEAD detached at v0.8.0
nothing to commit, working tree clean

$ rustc --version
rustc 1.39.0 (4560ea788 2019-11-04)

$ cargo build --release
   Compiling libc v0.2.65
   Compiling cfg-if v0.1.10
...
   Compiling librocksdb-sys v6.1.3
   Compiling rocksdb v0.12.4
    Finished release [optimized] target(s) in 12m 29s

$ cargo run --release
    Finished release [optimized] target(s) in 1.96s
     Running `target/release/electrs`
Config { log: StdErrLog { verbosity: Debug, quiet: false, timestamp: Off, modules: [], writer: "stderr", color_choice: Auto }, network_type: bitcoin, db_path: "/mnt/bitcoin/.electrs/db/mainnet", daemon_dir: "/mnt/bitcoin/.bitcoin", daemon_rpc_addr: V4(127.0.0.1:8332), cookie: Some("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"), electrum_rpc_addr: V4(192.168.1.3:50001), monitoring_addr: V4(127.0.0.1:4224), jsonrpc_import: false, index_batch_size: 100, bulk_index_threads: 4, tx_cache_size: 10485760, txid_limit: 100, server_banner: "Welcome to electrs 0.8.0 (Electrum Rust Server)!", blocktxids_cache_size: 10485760 }
DEBUG - Server listening on 127.0.0.1:4224
DEBUG - Running accept thread
INFO - NetworkInfo { version: 180100, subversion: "/Satoshi:0.18.1/" }
INFO - BlockchainInfo { chain: "main", blocks: 604459, headers: 604459, bestblockhash: "00000000000000000003e246dc64bd4bd94b62d88ae984ec66111d1441c91d09", pruned: false, initialblockdownload: false }
DEBUG - opening DB at "/mnt/bitcoin/.electrs/db/mainnet"
Illegal instruction (core dumped)

[romanz]

Not sure but maybe it's related to rust-rocksdb crate https://github.com/romanz/electrs/issues/159#issuecomment-524977082?

[romanz]

Could you please try https://github.com/romanz/electrs/commit/48622f4bfb1cf42818a3d7709e163aa3208728de?

[kallerosenbaum]

Still same issue

$ git status
HEAD detached at 48622f4
$ cargo run --release
    Finished release [optimized] target(s) in 2.61s
     Running `target/release/electrs`
Config { log: StdErrLog { verbosity: Debug, quiet: false, timestamp: Off, modules: [], writer: "stderr", color_choice: Auto }, network_type: bitcoin, db_path: "/mnt/bitcoin/.electrs/db/mainnet", daemon_dir: "/mnt/bitcoin/.bitcoin", daemon_rpc_addr: V4(127.0.0.1:8332), electrum_rpc_addr: V4(192.168.1.3:50001), monitoring_addr: V4(127.0.0.1:4224), jsonrpc_import: false, index_batch_size: 100, bulk_index_threads: 4, tx_cache_size: 10485760, txid_limit: 100, server_banner: "Welcome to electrs 0.8.0 (Electrum Rust Server)!", blocktxids_cache_size: 10485760 }
DEBUG - Server listening on 127.0.0.1:4224
DEBUG - Running accept thread
INFO - NetworkInfo { version: 180100, subversion: "/Satoshi:0.18.1/" }
INFO - BlockchainInfo { chain: "main", blocks: 604511, headers: 604511, bestblockhash: "00000000000000000003c243c467b34fbc3140d4c7a6782d871e77dbb12dd5dc", pruned: false, initialblockdownload: false }
DEBUG - opening DB at "/mnt/bitcoin/.electrs/db/mainnet"
Illegal instruction (core dumped)

[kallerosenbaum]

I also cloned the repo clean, checked out 0.8.0, built and run with the exact same result.

[romanz]

Could you please try to git bisect the history between 0.7.1 and 0.8.0?
I think that running cargo test should fail as well.
BTW, which OS & machine are you using?

[kallerosenbaum]

https://github.com/romanz/electrs/issues/193#issuecomment-554089461

[romanz]

Is it a cloud-based VM by chance?
If so, I can try to reproduce the issue...

[romanz]

Could you please try to git bisect the history between 0.7.1 and 0.8.0?

You can also try to downgrade just rocksdb crate version back to 0.12.1 since I think it was used at 0.7.1:

diff --git a/Cargo.toml b/Cargo.toml
index a73a731..671fc63 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "electrs"
-version = "0.7.1"
+version = "0.8.0"
 authors = ["Roman Zeyde <[email protected]>"]
 description = "An efficient re-implementation of Electrum Server in Rust"
 license = "MIT"
@@ -10,20 +10,23 @@ keywords = ["bitcoin", "electrum", "server", "index", "database"]
 documentation = "https://docs.rs/electrs/"
 readme = "README.md"
 edition = "2018"
+build = "build.rs"
+
+[package.metadata.configure_me]
+spec = "config_spec.toml"

 [profile.release]
 lto = true

 [features]
-latest_rust = []  # use latest Rust features (otherwise, support Rust 1.32)
+latest_rust = []  # use latest Rust features (otherwise, support Rust 1.34)

 [dependencies]
-arrayref = "0.3"
 base64 = "0.10"
 bincode = "1.0"
-bitcoin = "0.18"
-bitcoin_hashes = "0.3"
-clap = "2.31"
+bitcoin = { version = "0.21", features = ["use-serde"] }
+bitcoin_hashes = "0.7.1"
+configure_me = "0.3.3"
 crossbeam-channel = "0.3"
 dirs = "1.0"
 error-chain = "0.12"
@@ -35,7 +38,7 @@ lru = "0.1"
 num_cpus = "1.0"
 page_size = "0.4"
 prometheus = "0.5"
-rocksdb = "0.12.1"
+rocksdb = "0.12"
 rust-crypto = "0.2"
 serde = "1.0"
 serde_derive = "1.0"
@@ -45,3 +48,6 @@ stderrlog = "0.4.1"
 sysconf = ">=0.3.4"
 time = "0.1"
 tiny_http = "0.6"
+
+[build-dependencies]
+configure_me_codegen = "0.3.12"

[kallerosenbaum]

No, it's a physical machine in my basement. Nothing cloudy there, but a bit dusty.

[romanz]

Could you please build and run latest rust-rocksdb tests?

$ git clone https://github.com/rust-rocksdb/rust-rocksdb
$ git submodule update --init --recursive
$ cargo test

I am suspecting that https://github.com/rust-rocksdb/rust-rocksdb/blob/3c810163091fb8d343a3fa8373edd4b5685ed84d/librocksdb-sys/build.rs#L100 is causing the crash :(

[kallerosenbaum]

Right! Here's my output.

$ cargo test
    Updating crates.io index
  Downloaded cc v1.0.47
  Downloaded cexpr v0.3.6
  Downloaded num_cpus v1.11.1
  Downloaded jobserver v0.1.17
   Compiling libc v0.2.65
   Compiling log v0.4.8
...
   Compiling which v2.0.1
   Compiling librocksdb-sys v6.2.4 (/mnt/bitcoin/git/rust-rocksdb/librocksdb-sys)
warning: couldn't execute `llvm-config --prefix` (error: No such file or directory (os error 2))
warning: set the LLVM_CONFIG_PATH environment variable to a valid `llvm-config` executable
warning: snappy/snappy-sinksource.cc: In member function ‘virtual char* snappy::Sink::GetAppendBuffer(size_t, char*)’:
...
warning: snappy/snappy-sinksource.cc:39:36: warning: un
warning: bzip2/compress.c:256:11: note: ‘cost[2]’ was declared here
warning:     UInt16 cost[BZ_N_GROUPS];
warning:            ^~~~
   Compiling rocksdb v0.13.0 (/mnt/bitcoin/git/rust-rocksdb)
    Finished dev [unoptimized + debuginfo] target(s) in 8m 54s
     Running target/debug/deps/rocksdb-23fdbe4138dbea96
error: process didn't exit successfully: `/mnt/bitcoin/git/rust-rocksdb/target/debug/deps/rocksdb-23fdbe4138dbea96` (signal: 4, SIGILL: illegal instruction)

My CPU is pretty old:

$ cat /proc/cpuinfo
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 15
model name  : Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
stepping    : 11
microcode   : 0xba
cpu MHz     : 1622.013
cache size  : 4096 KB
physical id : 0
siblings    : 4
core id     : 0
cpu cores   : 4
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 10
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm pti tpr_shadow vnmi flexpriority dtherm
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips    : 4800.04
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

... And 3 more cores

[kallerosenbaum]

My CPU (Kentsfield) is from somewhere 2006-2009 which is older than Intel Nehalem (about 2010)

[kallerosenbaum]

My options seems to be

a) Use electrs 0.7.1
b) Use electrs 0.8.0 but use rocksdb 0.12.2 instead of 0.12.4 as git diff v0.7.1 v0.8.0 Cargo.lock suggests.

[kallerosenbaum]

cargo update -p rocksdb --precise 0.12.2 did the trick! It works now, but what something else will stop working?

[romanz]

I think that RocksDB is the only non-Rust crate, so the fix above should work well.

[romanz]

FTR, this is an open issue at https://github.com/rust-rocksdb/rust-rocksdb/issues/327.

[dmp1ce]

I cannot downgrade rocksdb to 0.12.2 with version 0.8.1.

error: failed to select a version for the requirement `rocksdb = "^0.13"`
  candidate versions found which didn't match: 0.12.2
  location searched: crates.io index
required by package `electrs v0.8.1 (/home/user)`

I guess I'll try again with 0.8.0 or 0.7.1.

[dmp1ce]

I have an old 2009 AMD, but I'm not sure if that is my issue with rocksdb. I get Illegal instruction (core dumped) right away when running cargo run --release.

Although, I'm running in docker without a toml configuration.

[dmp1ce]

cargo update -p rocksdb --precise 0.12.2 on 0.8.0 didn't help me. I still got Illegal instruction (core dumped).

0.7.1 wouldn't compile for me.

Updating crates.io index
error: failed to write /home/user/Cargo.lock

[dmp1ce]

Thank you for reopening, but in fact I did get 0.8.0 to work after running cargo update -p rocksdb --precise 0.12.2! I just needed to get my Dockerfile build working properly. So, it seems I am also hitting the rocksdb issue.

[kallerosenbaum]

Thank you for reopening, but in fact I did get 0.8.0 to work after running cargo update -p rocksdb --precise 0.12.2! I just needed to get my Dockerfile build working properly. So, it seems I am also hitting the rocksdb issue.

@dmp1ce Did you manage to downgrade rocksdb on 0.8.1 too? I haven't tried that yet.

[dmp1ce]

No, because 0.8.1 requires at least 0.13 of rocksdb. I don't know why.

https://github.com/romanz/electrs/blob/1498eed6bb07c029eef07bad43a4eb2c37e7d188/Cargo.toml#L41

I just used 0.8.0.

[romanz]

Let's downgrade to 0.12.2 until https://github.com/rust-rocksdb/rust-rocksdb/issues/327 is resolved.

[romanz]

0.8.2 should fix this issue.

[Kixunil]

Could this be bckported? https://github.com/Blockstream/electrs/commit/a03813d2816a43f4846f4335cbfe84848dcd4c84

I'm looking at compiling rocksdb with system library and having the dependency pinned makes it harder.

[Kixunil]

FYI, I managed to compile electrs with system libraries using this patch: https://github.com/rust-rocksdb/rust-rocksdb/pull/470

If it gets merged, then my issue will be resolved.