Lighthouse: Tokio panic when starting prior to genesis (blocked on tokio)

Created on 28 May 2020 · 2Comments · Source: sigp/lighthouse

Description

When starting a node from a known genesis state, but prior to the genesis time we observe a panic from tokio. The panic doesn't appear to have any noticeable effect, however we need to figure out what it is.

Version

master

Present Behaviour

The backtrace (below) was split across a few lines in my terminal, so I've pieced it back together. Hopefully I didn't miss anything.

thread 'tokio-runtime-workerhead_block' panicked at 'timer error: the timer is shutdown, must be called from the context of Tokio runtime:', /home/paul/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.21/src/time/delay.rs :960xc4fd…8b55:23,
head_state:stack backtrace
   0:     0x56411f78af24 - backtrace::backtrace::libunwind::trace::h5d52ba5f20882f09
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.44/src/backtrace/libunwind.rs:86
   1:     0x56411f78af24 - backtrace::backtrace::trace_unsynchronized::hceee092869668a74
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.44/src/backtrace/mod.rs:66
   2:     0x56411f78af24 - std::sys_common::backtrace::_print_fmt::ha312c2904605e4d5
                               at src/libstd/sys_common/backtrace.rs:78
   3:     0x56411f78af24 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h5b9981092140b727
                               at src/libstd/sys_common/backtrace.rs:59
   4:     0x56411f7b16dc - core::fmt::write::h5f6d7d8de88b4173
                               at src/libcore/fmt/mod.rs:1063
   5:     0x56411f782bf3 - std::io::Write::write_fmt::h893169117de3cc15
                               at src/libstd/io/mod.rs:1426
   6:     0x56411f78dd55 - std::sys_common::backtrace::_print::h8ab61d4120f7a335
                               at src/libstd/sys_common/backtrace.rs:62
   7:     0x56411f78dd55 - std::sys_common::backtrace::print::h8aae19fbb153bf2a
                               at src/libstd/sys_common/backtrace.rs:49
   8:     0x56411f78dd55 - std::panicking::default_hook::{{closure}}::h1ee5b7d8b6f83429
                               at src/libstd/panicking.rs:204
   9:     0x56411f78daa2 - std::panicking::default_hook::hd6c32c13403f9210
                               at src/libstd/panicking.rs:224
  10:     0x56411f78e3b2 - std::panicking::rust_panic_with_hook::h1f2449d529a25f22
                               at src/libstd/panicking.rs:470
  11:     0x56411f78df9b - rust_begin_unwind
                               at src/libstd/panicking.rs:378
  12:     0x56411f78df0b - std::panicking::begin_panic_fmt::h7c14cf40a8be97ee
                               at src/libstd/panicking.rs:332
  13:     0x56411f3a3ccc - <tokio::time::delay::Delay as core::future::future::Future>::poll::hc7e2438da39521ed
  14:     0x56411f39f17a - <tokio::time::interval::Interval as futures_core::stream::Stream>::poll_next::he5d695da55add43a
  15:     0x56411e89b532 - <std::future::GenFuture<T> as core::future::future::Future>::poll::h17ab7f32826db1c8
  16:     0x56411e9b9bfe - <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll::hd62fa1cc0424e0b2
  17:     0x56411e77dfd6 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h1d8313f12745e7ed
  18:     0x56411ea41171 - <futures_util::future::future::flatten::Flatten<Fut,<Fut as core::future::future::Future>::Output> as core::future::future::Future>::poll::h0f2683d1fb965e3f
  19:     0x56411e5a0cf0 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hdcf336ceb47cca60
  20:     0x56411e8302b9 - std::panicking::try::do_call::h8821fad60d35938a
  21:     0x56411f794827 - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:86
  22:     0x56411e717247 - tokio::runtime::task::raw::poll::h4a3196a4ccc6ca5c
  23:     0x56411f3a329c - std::thread::local::LocalKey<T>::with::hdac925c1110a5778
  24:     0x56411f3b0062 - tokio::runtime::thread_pool::worker::Context::run_task::h526911d23dfa9e9c
  25:     0x56411f3af347 - tokio::runtime::thread_pool::worker::Context::run::h13540360fdb453fe
  26:     0x56411f396ae3 - tokio::macros::scoped_tls::ScopedKey<T>::set::h0e994ac36aa55f95
  27:     0x56411f3af18b - tokio::runtime::thread_pool::worker::run::hd23953ac3de5780d
  28:     0x56411f3a6f79 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h9ebfe4038d870cdf
  29:     0x56411f3acdad - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h74715fb1442fc5c4
  30:     0x56411f39fdf9 - std::panicking::try::do_call::h536630edc46f8526
  31:     0x56411f794827 - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:86
  32:     0x56411f39a944 - tokio::runtime::task::harness::Harness<T,S>::poll::hf931ca277182c088
  33:     0x56411f398f77 - tokio::runtime::blocking::pool::Inner::run::hf02c030e1376faea
  34:     0x56411f3a2118 - tokio::runtime::context::enter::hbb70c3a21b07857c
  35:     0x56411f39c66f - std::sys_common::backtrace::__rust_begin_short_backtrace::hfd67d21939de4b3f
  36:     0x56411f39fee5 - std::panicking::try::do_call::hbc84766475ec0412
  37:     0x56411f794827 - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:86
  38:     0x56411f3a7c8a - core::ops::function::FnOnce::call_once{{vtable.shim}}::h47e958ba0118b652
  39:     0x56411f7795bf - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h36e46972b1b7f7b7
                               at /rustc/8d69840ab92ea7f4d323420088dd8c9775f180cd/src/liballoc/boxed.rs:1017
  40:     0x56411f79391d - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::hdde89c32ce6999de
                               at /rustc/8d69840ab92ea7f4d323420088dd8c9775f180cd/src/liballoc/boxed.rs:1017
  41:     0x56411f79391d - std::sys_common::thread::start_thread::h5ae3fdd4dad03dd5
                               at src/libstd/sys_common/thread.rs:13
  42:     0x56411f79391d - std::sys::unix::thread::Thread::new::thread_start::h76b2403b2434c4c0
                               at src/libstd/sys/unix/thread.rs:80
  43:     0x7fb1bc0cf422 - start_thread
  44:     0x7fb1bbfe2b83 - __GI___clone
  45:                0x0 - <unknown>

Steps to reproduce

git clone https://github.com/goerli/witti
mv witti/lighthouse ~/.lighthouse/future
lcli change-genesis-time ~/.lighthouse/future/genesis.ssz 1790616657
In ~/.lighthouse/future/config.yml change GENESIS_FORK_VERSION to something weird like 0x00000142.
lighthouse --debug-level trace bn --datadir /tmp/future --testnet-dir ~/.lighthouse/future