Electrs: Reorg may cause in electrs to crash during indexing

[romanz]

Symptoms

2018-10-04T07:53:58.985+00:00 - DEBUG - opening DB at "./db/testnet"
2018-10-04T07:53:58.996+00:00 - DEBUG - downloading new block headers (1414449 already indexed) from 00000000210004840364b52bc5e455d888f164e4264a4fec06a514b67e9d5722
2018-10-04T07:53:58.996+00:00 - TRACE - downloaded 0 block headers
thread 'main' panicked at 'assertion failed: (left == right)
left: 22579d7eb614a506ec4f4a26e464f188d855e4c52bb564038404002100000000,
right: `25c4970a821dd4e198561038ea2d9bb6944523786d45f1c92f5e000000000000`', src/index.rs:410:9
note: Run with RUST_BACKTRACE=1 for a backtrace.
2018-10-04T07:53:58.999+00:00 - TRACE - closing DB at "./db/testnet"

thread 'main' panicked at 'assertion failed: (left == right)
left: 22579d7eb614a506ec4f4a26e464f188d855e4c52bb564038404002100000000,
right: `25c4970a821dd4e198561038ea2d9bb6944523786d45f1c92f5e000000000000`', src/index.rs:410:9
stack backtrace:
    0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
            at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
    1: std::sys_common::backtrace::print
            at libstd/sys_common/backtrace.rs:71
            at libstd/sys_common/backtrace.rs:59
    2: std::panicking::default_hook::{{closure}}
            at libstd/panicking.rs:211
    3: std::panicking::default_hook
            at libstd/panicking.rs:227
    4: std::panicking::rust_panic_with_hook
            at libstd/panicking.rs:511
    5: std::panicking::continue_panic_fmt
            at libstd/panicking.rs:426
    6: std::panicking::begin_panic_fmt
            at libstd/panicking.rs:413
    7: electrs::index::Index::update
    8: electrs::app::App::update
    9: electrs::main
    10: std::rt::lang_start::{{closure}}
    11: std::panicking::try::do_call
            at libstd/rt.rs:59
            at libstd/panicking.rs:310
    12: __rust_maybe_catch_panic
            at libpanic_unwind/lib.rs:105
    13: std::rt::lang_start_internal
            at libstd/panicking.rs:289
            at libstd/panic.rs:392
            at libstd/rt.rs:58
    14: main
    15: __libc_start_main
    16: _start

00000000210004840364b52bc5e455d888f164e4264a4fec06a514b67e9d5722 @ 1414433
0000000000005e2fc9f1456d78234594b69b2dea38105698e1d41d820a97c425 @ 1414448

Analysis

Both blocks have difficulty = 1 and seem to be related to the CVE-2018-17144 exploit on testnet and its following re-orgs.

Note that the bestblockhash is ...5722 @ 1414433 (received from bitcoind).
This means that we already indexed ...c425 @ 1414449.
if no new blocks are found (according to the log), bestblockhash is included in the currently indexed chain, but but is behind the tip.

When applying the new headers, we have an empty new_headers list:
https://github.com/romanz/electrs/blob/fce7bd15b0d6203c96f6da28b165b52b15b568af/src/index.rs#L409

This causes HeaderList::apply() to return too early (keeping the in-memory block header list):
https://github.com/romanz/electrs/blob/fce7bd15b0d6203c96f6da28b165b52b15b568af/src/util.rs#L140

This results in a panic above:
https://github.com/romanz/electrs/blob/fce7bd15b0d6203c96f6da28b165b52b15b568af/src/index.rs#L410

[romanz]

This can be a great test-case to add -> #62

[dagurval]

Simply calling

bitcoin-cli invalidateblock `bitcoin-cli getbestblockhash`

will produce this crash

[romanz]

Should be resolved by #136