Jormungandr: Leader node crash during leader event processing on Windows 10

Created on 17 Oct 2019  路  5Comments  路  Source: input-output-hk/jormungandr

Build 0.6.1, latest master at the time.
Crashed once in the morning and once in the evening.

I had a quick gander at existing github issues but didn't see references to NonMonotonicDate, so apologies if this is a dupe of some existing known issue

Rust full backtrace as follows:

Oct 17 19:20:43.016 INFO leader event starting, date: 158.172, leader: 1, task: leadership
thread 'block1' panicked at 'called Result::unwrap() on an Err value: Error(Msg("cannot process leadership block"), State { next_error: Some(Error(CannotApplyBlock, State { next_error: Some(NonMonotonicDate { block_date: BlockDate { epoch: 158, slot_id: 172 }, chain_date: BlockDate { epoch: 158, slot_id: 172 } }), backtrace: InternalBacktrace { backtrace: Some(stack backtrace:
0: backtrace::backtrace::trace
1: backtrace::capture::Backtrace::new_unresolved
2: error_chain::backtrace::imp::InternalBacktrace::new
3: error_chain::State::new
4: jormungandr::blockchain::chain::Blockchain::apply_block
5: jormungandr::blockchain::chain::Blockchain::apply_and_store_block
6: futures::future::chain::Chain::poll
7: as futures::future::Future>::poll
8: as futures::future::Future>::poll
9: futures::task_impl::std::set
10: std::thread::local::LocalKey::with
11: futures::future::Future::wait
12: jormungandr::blockchain::process::handle_input
13: jormungandr::blockchain::process::handle_input
14: as futures::future::Future>::poll
15: std::panicking::try::do_call
16: panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
17: as futures::future::Future>::poll
18: as futures::future::Future>::poll
19: futures::task_impl::std::set
20: std::panicking::try::do_call
21: panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
22: tokio_threadpool::task::Task::run
23: tokio_threadpool::worker::Worker::run
24: tokio_threadpool::worker::Worker::run
25: std::thread::local::LocalKey::with
26: std::thread::local::LocalKey::with
27: core::slice:: 28: std::thread::local::LocalKey::with
29: std::thread::local::LocalKey::with
30: std::sys_common::backtrace::__rust_begin_short_backtrace
31: std::panicking::try::do_call
32: panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
33: std::thread::Builder::spawn
34: alloc::boxed::{{impl}}::call_once<(),FnOnce<()>>
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\src\liballocboxed.rs:787
35: alloc::boxed::{{impl}}::call_once
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\src\liballocboxed.rs:787
36: std::sys_common::thread::start_thread
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\sys_common\thread.rs:13
37: std::sys::windows::thread::{{impl}}::new::thread_start
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\sys\windows\thread.rs:47
38: BaseThreadInitThunk
39: RtlUserThreadStart
) } })), backtrace: InternalBacktrace { backtrace: Some(stack backtrace:
0: backtrace::backtrace::trace
1: backtrace::capture::Backtrace::new_unresolved
2: error_chain::backtrace::imp::InternalBacktrace::new
3: error_chain::State::new
4: jormungandr::blockchain::chain::Blockchain::apply_block
5: jormungandr::blockchain::chain::Blockchain::apply_and_store_block
6: futures::future::chain::Chain::poll
7: as futures::future::Future>::poll
8: as futures::future::Future>::poll
9: futures::task_impl::std::set
10: std::thread::local::LocalKey::with
11: futures::future::Future::wait
12: jormungandr::blockchain::process::handle_input
13: jormungandr::blockchain::process::handle_input
14: as futures::future::Future>::poll
15: std::panicking::try::do_call
16: panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
17: as futures::future::Future>::poll
18: as futures::future::Future>::poll
19: futures::task_impl::std::set
20: std::panicking::try::do_call
21: panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
22: tokio_threadpool::task::Task::run
23: tokio_threadpool::worker::Worker::run
24: tokio_threadpool::worker::Worker::run
25: std::thread::local::LocalKey::with
26: std::thread::local::LocalKey::with
27: core::slice:: 28: std::thread::local::LocalKey::with
29: std::thread::local::LocalKey::with
30: std::sys_common::backtrace::__rust_begin_short_backtrace
31: std::panicking::try::do_call
32: panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
33: std::thread::Builder::spawn
34: alloc::boxed::{{impl}}::call_once<(),FnOnce<()>>
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\src\liballocboxed.rs:787
35: alloc::boxed::{{impl}}::call_once
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\src\liballocboxed.rs:787
36: std::sys_common::thread::start_thread
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\sys_common\thread.rs:13
37: std::sys::windows::thread::{{impl}}::new::thread_start
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\sys\windows\thread.rs:47
38: BaseThreadInitThunk
39: RtlUserThreadStart
) } })', src\libcore\result.rs:1084:5
stack backtrace:
0: 0x7ff6b98dc127 - backtrace::backtrace::trace_unsynchronized
at C:\Users\VssAdministrator.cargo\registry\src\github.com-1ecc6299db9ec823backtrace-0.3.34\srcbacktrace\mod.rs:66
1: 0x7ff6b98dc127 - std::sys_common::backtrace::_print
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\sys_commonbacktrace.rs:47
2: 0x7ff6b98df60e - std::sys_common::backtrace::print
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\sys_commonbacktrace.rs:36
3: 0x7ff6b98df60e - std::panicking::default_hook::{{closure}}
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\panicking.rs:200
4: 0x7ff6b98df292 - std::panicking::default_hook
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\panicking.rs:214
5: 0x7ff6b98dfea0 - std::panicking::rust_panic_with_hook
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\panicking.rs:477
6: 0x7ff6b98df9d4 - std::panicking::continue_panic_fmt
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\panicking.rs:384
7: 0x7ff6b98df8b9 - std::panicking::rust_begin_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\panicking.rs:311
8: 0x7ff6b98f29bc - core::panicking::panic_fmt
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libcore\panicking.rs:85
9: 0x7ff6b98f2abf - core::result::unwrap_failed
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libcore\result.rs:1084
10: 0x7ff6b9541d0b - jormungandr::blockchain::process::handle_input::hb36abc5f63cf82da
11: 0x7ff6b953ca4f - jormungandr::blockchain::process::handle_input::hb36abc5f63cf82da
12: 0x7ff6b94a6cfe - as futures::future::Future>::poll::hdf44867b3ec08577
13: 0x7ff6b93c8d11 - std::panicking::try::do_call::h925a1252221b3de7
14: 0x7ff6b98e5132 - panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
15: 0x7ff6b91cbaa4 - as futures::future::Future>::poll::ha4c1a54803f1ce06
16: 0x7ff6b91c63a3 - as futures::future::Future>::poll::hdcf14b041dcb7853
17: 0x7ff6b98a7770 - futures::task_impl::std::set::hc2b6a41d30305b4e
18: 0x7ff6b98a7245 - std::panicking::try::do_call::haf5ecb2c18c6818b
19: 0x7ff6b98e5132 - panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
20: 0x7ff6b98aac35 - tokio_threadpool::task::Task::run::hf1caf5c58f819c12
21: 0x7ff6b98a6836 - tokio_threadpool::worker::Worker::run::he488ba52a0dd6c29
22: 0x7ff6b98a5f16 - tokio_threadpool::worker::Worker::run::he488ba52a0dd6c29
23: 0x7ff6b98859f9 - std::thread::local::LocalKey::with::h0cac0b507c9cfb0e
24: 0x7ff6b9885fb4 - std::thread::local::LocalKey::with::hd213661f91f4f02d
25: 0x7ff6b98812a0 - core::slice:: 26: 0x7ff6b98ab743 - std::thread::local::LocalKey::with::h7941d1d13feee31c
27: 0x7ff6b98ab007 - std::thread::local::LocalKey::with::h1d4c9ff7715f0050
28: 0x7ff6b98afe41 - std::sys_common::backtrace::__rust_begin_short_backtrace::h971c396e5e7492ff
29: 0x7ff6b98a719f - std::panicking::try::do_call::h443556b5e6ea267c
30: 0x7ff6b98e5132 - panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
31: 0x7ff6b98a7f70 - std::thread::Builder::spawn::h348c0e312cfaddd0
32: 0x7ff6b98cc0a7 - alloc::boxed::{{impl}}::call_once<(),FnOnce<()>>
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\src\liballocboxed.rs:787
33: 0x7ff6b98e3117 - alloc::boxed::{{impl}}::call_once
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\src\liballocboxed.rs:787
34: 0x7ff6b98e3117 - std::sys_common::thread::start_thread
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\sys_common\thread.rs:13
35: 0x7ff6b98e3117 - std::sys::windows::thread::{{impl}}::new::thread_start
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\sys\windows\thread.rs:47
36: 0x7ffb046e7bd4 - BaseThreadInitThunk
37: 0x7ffb04a2ced1 - RtlUserThreadStart
called Result::unwrap() on an Err value: Error(Msg("cannot process leadership block"), State { next_error: Some(Error(CannotApplyBlock, State { next_error: Some(NonMonotonicDate { block_date: BlockDate { epoch: 158, slot_id: 172 }, chain_date: BlockDate { epoch: 158, slot_id: 172 } }), backtrace: InternalBacktrace { backtrace: Some(stack backtrace:
0: backtrace::backtrace::trace
1: backtrace::capture::Backtrace::new_unresolved
2: error_chain::backtrace::imp::InternalBacktrace::new
3: error_chain::State::new
4: jormungandr::blockchain::chain::Blockchain::apply_block
5: jormungandr::blockchain::chain::Blockchain::apply_and_store_block
6: futures::future::chain::Chain::poll
7: as futures::future::Future>::poll
8: as futures::future::Future>::poll
9: futures::task_impl::std::set
10: std::thread::local::LocalKey::with
11: futures::future::Future::wait
12: jormungandr::blockchain::process::handle_input
13: jormungandr::blockchain::process::handle_input
14: as futures::future::Future>::poll
15: std::panicking::try::do_call
16: panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
17: as futures::future::Future>::poll
18: as futures::future::Future>::poll
19: futures::task_impl::std::set
20: std::panicking::try::do_call
21: panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
22: tokio_threadpool::task::Task::run
23: tokio_threadpool::worker::Worker::run
24: tokio_threadpool::worker::Worker::run
25: std::thread::local::LocalKey::with
26: std::thread::local::LocalKey::with
27: core::slice:: 28: std::thread::local::LocalKey::with
29: std::thread::local::LocalKey::with
30: std::sys_common::backtrace::__rust_begin_short_backtrace
31: std::panicking::try::do_call
32: panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
33: std::thread::Builder::spawn
34: alloc::boxed::{{impl}}::call_once<(),FnOnce<()>>
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\src\liballocboxed.rs:787
35: alloc::boxed::{{impl}}::call_once
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\src\liballocboxed.rs:787
36: std::sys_common::thread::start_thread
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\sys_common\thread.rs:13
37: std::sys::windows::thread::{{impl}}::new::thread_start
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\sys\windows\thread.rs:47
38: BaseThreadInitThunk
39: RtlUserThreadStart
) } })), backtrace: InternalBacktrace { backtrace: Some(stack backtrace:
0: backtrace::backtrace::trace
1: backtrace::capture::Backtrace::new_unresolved
2: error_chain::backtrace::imp::InternalBacktrace::new
3: error_chain::State::new
4: jormungandr::blockchain::chain::Blockchain::apply_block
5: jormungandr::blockchain::chain::Blockchain::apply_and_store_block
6: futures::future::chain::Chain::poll
7: as futures::future::Future>::poll
8: as futures::future::Future>::poll
9: futures::task_impl::std::set
10: std::thread::local::LocalKey::with
11: futures::future::Future::wait
12: jormungandr::blockchain::process::handle_input
13: jormungandr::blockchain::process::handle_input
14: as futures::future::Future>::poll
15: std::panicking::try::do_call
16: panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
17: as futuresthread '::future::Future>::poll
18: as futures::future::Future>::poll
19: futures::task_impl::std::set
20: std::panicking::try::do_call
21: panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
22: tokio_threadpool::task::Task::run
23: tokio_threadpool::worker::Worker::run
24: tokio_threadpool::worker::Worker::run
25: std::thread::local::LocalKey::with
26: std::thread::local::LocalKey::with
27: core::slice:: 28: std::thread::local::LocalKey::with
29: std::thread::local::LocalKey::with
30: std::sys_common::backtrace::__rust_begin_short_backtrace
31: std::panicking::try::do_call
32: panic_unwind::__rust_maybe_catch_panic
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libpanic_unwind\lib.rs:80
33: std::thread::Builder::spawn
34: alloc::boxed::{{impl}}::call_once<(),FnOnce<()>>
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\src\liballocboxed.rs:787
35: alloc::boxed::{{impl}}::call_once
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\src\liballocboxed.rs:787
36: std::sys_common::thread::start_thread
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\sys_common\thread.rs:13
37: std::sys::windows::thread::{{impl}}::new::thread_start
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\/src\libstd\sys\windows\thread.rs:47
38: BaseThreadInitThunk
39: RtlUserThreadStart
) } })leadership0
' panicked at 'Expect the event to not close', src\libcore\option.rsthread '

picture hodlonaut
👀1

Most helpful comment

NonMonotonicDate error seems to be originating at this bit of code:

if metadata.block_date <= new_ledger.date {
return Err(Error::NonMonotonicDate {
block_date: metadata.block_date,
chain_date: new_ledger.date,
});
}

as far as we can tell the nodes on the networking are relatively in sync time-wise, maybe a second difference here and there.

picture hodlonaut on 24 Oct 2019
👍2

All 5 comments

Worth noting that restarting with a clean storage: folder did not appear to help, it crashed again eventually.

hodlonaut picture hodlonaut on 17 Oct 2019

Still happening in v0.6.5 from time to time, always as a part of start of the leadership event.

hodlonaut picture hodlonaut on 19 Oct 2019

$ jormungandr --full-version
jormungandr 0.7.0-alpha.dev.1 (HEAD-3af63d09, release, linux [x86_64]) - [rustc 1.38.0 (625451e37 2019-09-23)]

Running in private testnet as per https://github.com/rdlrt/Alternate-Jormungandr-Testnet/blob/master/README.md

Oct 23 22:40:51.002 INFO leader event starting, date: 48.10, leader: 1, task: leadership
thread 'block0' panicked at 'called `Result::unwrap()` on an `Err` value: Error(Msg("cannot process leadership block"), State { next_error: Some(Error(CannotApplyBlock, S
tate { next_error: Some(NonMonotonicDate { block_date: BlockDate { epoch: 48, slot_id: 10 }, chain_date: BlockDate { epoch: 48, slot_id: 10 } }), backtrace: InternalBackt
race { backtrace: Some(stack backtrace:
   0: error_chain::backtrace::imp::InternalBacktrace::new
   1: error_chain::State::new
   2: <core::result::Result<T,E> as jormungandr::blockchain::chain::ResultExt<T>>::chain_err
   3: jormungandr::blockchain::chain::Blockchain::apply_block
   4: jormungandr::blockchain::chain::Blockchain::apply_and_store_block
   5: futures::future::chain::Chain<A,B,C>::poll
   6: <futures::future::map_err::MapErr<A,F> as futures::future::Future>::poll
   7: <futures::future::map::Map<A,F> as futures::future::Future>::poll
   8: futures::task_impl::std::set
   9: std::thread::local::LocalKey<T>::with
  10: futures::future::Future::wait
  11: jormungandr::blockchain::process::run_handle_input
  12: jormungandr::blockchain::process::handle_input
  13: <futures::stream::for_each::ForEach<S,F,U> as futures::future::Future>::poll
  14: std::panicking::try::do_call
  15: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  16: <futures::future::catch_unwind::CatchUnwind<F> as futures::future::Future>::poll
  17: <futures::future::map_err::MapErr<A,F> as futures::future::Future>::poll
  18: futures::task_impl::std::set
  19: std::panicking::try::do_call
  20: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  21: tokio_threadpool::task::Task::run
  22: tokio_threadpool::worker::Worker::run_task
  23: tokio_threadpool::worker::Worker::run
  24: std::thread::local::LocalKey<T>::with
  25: std::thread::local::LocalKey<T>::with
  26: std::thread::local::LocalKey<T>::with
  27: tokio::runtime::threadpool::builder::Builder::build::{{closure}}
  28: std::thread::local::LocalKey<T>::with
  29: std::thread::local::LocalKey<T>::with
  30: std::sys_common::backtrace::__rust_begin_short_backtrace
  31: std::panicking::try::do_call
  32: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  33: core::ops::function::FnOnce::call_once{{vtable.shim}}
  34: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/liballoc/boxed.rs:787
  35: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/liballoc/boxed.rs:787
      std::sys_common::thread::start_thread
             at src/libstd/sys_common/thread.rs:13
      std::sys::unix::thread::Thread::new::thread_start
             at src/libstd/sys/unix/thread.rs:79
  36: start_thread
  37: __clone
) } })), backtrace: InternalBacktrace { backtrace: Some(stack backtrace:
   0: error_chain::backtrace::imp::InternalBacktrace::new
   1: error_chain::State::new
   2: <core::result::Result<T,E> as jormungandr::blockchain::chain::ResultExt<T>>::chain_err
   3: jormungandr::blockchain::chain::Blockchain::apply_block
   4: jormungandr::blockchain::chain::Blockchain::apply_and_store_block
   5: futures::future::chain::Chain<A,B,C>::poll
   6: <futures::future::map_err::MapErr<A,F> as futures::future::Future>::poll
   7: <futures::future::map::Map<A,F> as futures::future::Future>::poll
   8: futures::task_impl::std::set
   9: std::thread::local::LocalKey<T>::with
  10: futures::future::Future::wait
  11: jormungandr::blockchain::process::run_handle_input
  12: jormungandr::blockchain::process::handle_input
  13: <futures::stream::for_each::ForEach<S,F,U> as futures::future::Future>::poll
  14: std::panicking::try::do_call
  15: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  16: <futures::future::catch_unwind::CatchUnwind<F> as futures::future::Future>::poll
  17: <futures::future::map_err::MapErr<A,F> as futures::future::Future>::poll
  18: futures::task_impl::std::set
  19: std::panicking::try::do_call
  20: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  21: tokio_threadpool::task::Task::run
  22: tokio_threadpool::worker::Worker::run_task
  23: tokio_threadpool::worker::Worker::run
  24: std::thread::local::LocalKey<T>::with
  25: std::thread::local::LocalKey<T>::with
  26: std::thread::local::LocalKey<T>::with
  27: tokio::runtime::threadpool::builder::Builder::build::{{closure}}
  28: std::thread::local::LocalKey<T>::with
  29: std::thread::local::LocalKey<T>::with
  30: std::sys_common::backtrace::__rust_begin_short_backtrace
  31: std::panicking::try::do_call
  32: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  33: core::ops::function::FnOnce::call_once{{vtable.shim}}
  34: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/liballoc/boxed.rs:787
  35: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/liballoc/boxed.rs:787
      std::sys_common::thread::start_thread
             at src/libstd/sys_common/thread.rs:13
      std::sys::unix::thread::Thread::new::thread_start
             at src/libstd/sys/unix/thread.rs:79
  36: start_thread
  37: __clone
) } })), backtrace: InternalBacktrace { backtrace: Some(stack backtrace:
   0: error_chain::backtrace::imp::InternalBacktrace::new
   1: error_chain::State::new
   2: <core::result::Result<T,E> as jormungandr::blockchain::chain::ResultExt<T>>::chain_err
   3: jormungandr::blockchain::chain::Blockchain::apply_block
   4: jormungandr::blockchain::chain::Blockchain::apply_and_store_block
   5: futures::future::chain::Chain<A,B,C>::poll
   6: <futures::future::map_err::MapErr<A,F> as futures::future::Future>::poll
   7: <futures::future::map::Map<A,F> as futures::future::Future>::poll
   8: futures::task_impl::std::set
   9: std::thread::local::LocalKey<T>::with
  10: futures::future::Future::wait
  11: jormungandr::blockchain::process::run_handle_input
  12: jormungandr::blockchain::process::handle_input
  13: <futures::stream::for_each::ForEach<S,F,U> as futures::future::Future>::poll
  14: std::panicking::try::do_call
  15: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  16: <futures::future::catch_unwind::CatchUnwind<F> as futures::future::Future>::poll
  17: <futures::future::map_err::MapErr<A,F> as futures::future::Future>::poll
  18: futures::task_impl::std::set
  19: std::panicking::try::do_call
  20: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  21: tokio_threadpool::task::Task::run
  22: tokio_threadpool::worker::Worker::run_task
  23: tokio_threadpool::worker::Worker::run
  24: std::thread::local::LocalKey<T>::with
  25: std::thread::local::LocalKey<T>::with
  26: std::thread::local::LocalKey<T>::with
  27: tokio::runtime::threadpool::builder::Builder::build::{{closure}}
  28: std::thread::local::LocalKey<T>::with
  29: std::thread::local::LocalKey<T>::with
  30: std::sys_common::backtrace::__rust_begin_short_backtrace
  31: std::panicking::try::do_call
  32: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
 33:     0x55c7966336c0 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h78576ee3568958b8
  34:     0x55c79664f9bf - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h2877b2aefb0fd627
                               at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/liballoc/boxed.rs:787
  35:     0x55c796662a80 - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h13f473dbcdaa0fef
                               at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/liballoc/boxed.rs:787
  36:     0x55c796662a80 - std::sys_common::thread::start_thread::h562a7c4953cb1d31
                               at src/libstd/sys_common/thread.rs:13
  37:     0x55c796662a80 - std::sys::unix::thread::Thread::new::thread_start::h382c6ae14181d163
                               at src/libstd/sys/unix/thread.rs:79
  38:     0x7f5d98f186db - start_thread
  39:     0x7f5d98a2988f - __clone
  40:                0x0 - <unknown>
thread 'leadership1' panicked at 'Expect the event to not close', src/libcore/option.rs:1166:5
stack backtrace:
   0:     0x55c79665eb0b - backtrace::backtrace::libunwind::trace::h89fcc71e59e3bc5b
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.34/src/backtrace/libunwind.rs:88
   1:     0x55c79665eb0b - backtrace::backtrace::trace_unsynchronized::h0bad9be1379e729a
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.34/src/backtrace/mod.rs:66
   2:     0x55c79665eb0b - std::sys_common::backtrace::_print::hd3382a1f33c473da
                               at src/libstd/sys_common/backtrace.rs:47
   3:     0x55c79665eb0b - std::sys_common::backtrace::print::h0ec6f03cfb8e76a6
                               at src/libstd/sys_common/backtrace.rs:36
   4:     0x55c79665eb0b - std::panicking::default_hook::{{closure}}::h96cbf7b454e3f557
                               at src/libstd/panicking.rs:200
   5:     0x55c79665e7e6 - std::panicking::default_hook::h95a8f00337383d83
                               at src/libstd/panicking.rs:214
   6:     0x55c79665f27d - std::panicking::rust_panic_with_hook::h92f98b46e22f14ed
                               at src/libstd/panicking.rs:477
   7:     0x55c79665ee02 - std::panicking::continue_panic_fmt::h25abfbb4e5b7043a
                               at src/libstd/panicking.rs:384
   8:     0x55c79665ece6 - rust_begin_unwind
                               at src/libstd/panicking.rs:311
   9:     0x55c79667c04d - core::panicking::panic_fmt::h7e9f94035af782b3
                               at src/libcore/panicking.rs:85
  10:     0x55c79667c0b7 - core::option::expect_failed::h292a6217b603c24f
                               at src/libcore/option.rs:1166
  11:     0x55c796103fac - futures::future::chain::Chain<A,B,C>::poll::h6ee4ddc7f96f5f15
  12:     0x55c79614ff25 - <futures::future::loop_fn::LoopFn<A,F> as futures::future::Future>::poll::hf84865cdce95420d
  13:     0x55c795eaa617 - std::panicking::try::do_call::h77824a5f673ec51b
  14:     0x55c79666373a - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:80
  15:     0x55c795ec86c2 - <futures::future::catch_unwind::CatchUnwind<F> as futures::future::Future>::poll::h0254daa532b196fb
  16:     0x55c795d43315 - <futures::future::map_err::MapErr<A,F> as futures::future::Future>::poll::h7077ef9e715adb37
  17:     0x55c79663ba92 - futures::task_impl::std::set::hb3d655f66f67803b
  18:     0x55c79663b592 - std::panicking::try::do_call::h9b3704b9d480004d
  19:     0x55c79666373a - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:80
  20:     0x55c79663b1b9 - tokio_threadpool::task::Task::run::h4b187d36deb72587
 21:     0x55c796635e36 - tokio_threadpool::worker::Worker::run_task::hc9f259dcdfd81f60
  22:     0x55c7966352d0 - tokio_threadpool::worker::Worker::run::h5271fdb5f65c80e1
  23:     0x55c796615c90 - std::thread::local::LocalKey<T>::with::h5e25bb4dbe76ab7f
  24:     0x55c796615b68 - std::thread::local::LocalKey<T>::with::h3a37773ff6611592
  25:     0x55c7966159ff - std::thread::local::LocalKey<T>::with::h11a52b305f93bec5
  26:     0x55c79661793e - tokio::runtime::threadpool::builder::Builder::build::{{closure}}::h1ee2c8640a2d3dca
  27:     0x55c7966386fa - std::thread::local::LocalKey<T>::with::h455e1461d116054d
  28:     0x55c7966384e9 - std::thread::local::LocalKey<T>::with::h10fc4bc733c32fc5
  29:     0x55c796632468 - std::sys_common::backtrace::__rust_begin_short_backtrace::hcfd1bc8d00bae548
  30:     0x55c79663b5dc - std::panicking::try::do_call::he64551aa5a88568b
  31:     0x55c79666373a - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:80
  32:     0x55c7966336c0 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h78576ee3568958b8
  33:     0x55c79664f9bf - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h2877b2aefb0fd627
                               at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/liballoc/boxed.rs:787
  34:     0x55c796662a80 - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h13f473dbcdaa0fef
                               at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/liballoc/boxed.rs:787
  35:     0x55c796662a80 - std::sys_common::thread::start_thread::h562a7c4953cb1d31
                               at src/libstd/sys_common/thread.rs:13
  36:     0x55c796662a80 - std::sys::unix::thread::Thread::new::thread_start::h382c6ae14181d163
                               at src/libstd/sys/unix/thread.rs:79
  37:     0x7f5d98f186db - start_thread
  38:     0x7f5d98a2988f - __clone
  39:                0x0 - <unknown>
Expect the event to not close
oldcryptogeek picture oldcryptogeek on 24 Oct 2019
👍1

NonMonotonicDate error seems to be originating at this bit of code:

if metadata.block_date <= new_ledger.date {
return Err(Error::NonMonotonicDate {
block_date: metadata.block_date,
chain_date: new_ledger.date,
});
}

as far as we can tell the nodes on the networking are relatively in sync time-wise, maybe a second difference here and there.

hodlonaut picture hodlonaut on 24 Oct 2019
👍2

we have a better way to reproduce the error in #1065 I will close this issue as this is better explained in the other one.

NicolasDP picture NicolasDP on 3 Nov 2019
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

gufmar picture gufmar  路  4Comments
dorin100 picture dorin100  路  3Comments
michaeljfazio picture michaeljfazio  路  4Comments
ghost picture ghost  路  4Comments
dorin100 picture dorin100  路  4Comments