Substrate: Panic: Trie lookup error: Database missing expected key

Created on 8 Feb 2019 · 17Comments · Source: paritytech/substrate

3 validator nodes, running mostly fine (one of them keep falling behind for some reason). 3 more full nodes, all 3 full nodes crashed after running for a while
restart gives following error

2019-02-07 21:58:52 Roles: FULL
2019-02-07 21:58:53 Best block: #8003
2019-02-07 21:58:53 Local node address is: /ip4/0.0.0.0/tcp/30333/p2p/QmPsnPYNiz3VMpJXvxwGBWcdRVKHKbTvtFnnrxc2NL9Xm6
2019-02-07 21:58:53 Kademlia random query has yielded empty results
2019-02-07 21:58:53 Listening for new connections on 0.0.0.0:9944.
2019-02-07 21:58:53 Idle (0 peers), best: #8003 (0xd0b8…1828), finalized #1593 (0x24e0…8a93), ⬇ 0 ⬆ 0
====================
stack backtrace:
   0: substrate_panic_handler::set::{{closure}}::h8c6bfb324807e0af (0x560aca78d64f)
   1: rust_panic_with_hook (0x560acab6249d)
             at src/libstd/panicking.rs:482
   2: continue_panic_fmt (0x560acab61f51)
             at src/libstd/panicking.rs:385
   3: rust_begin_unwind (0x560acab61e35)
   4: panic_fmt (0x560acab7c86c)
             at src/libcore/panicking.rs:85
   5: core::result::unwrap_failed::h34204b9443806977 (0x560ac9bd7e4d)
   6: <substrate_state_machine::ext::Ext<'a, H, B, T> as substrate_state_machine::Externalities<H>>::storage::h8503c7d781a2f5d4 (0x560ac9c22f2b)
   7: <substrate_executor::wasm_executor::FunctionExecutor<'e, E> as wasmi::host::Externals>::invoke_index::he001b56b6957f29b (0x560ac9d48907)
   8: wasmi::func::FuncInstance::invoke::hc19666a2fa542c8f (0x560ac9ee13c6)
   9: wasmi::runner::Interpreter::start_execution::h27ec5098d427db1a (0x560ac9f429d1)
  10: wasmi::func::FuncInstance::invoke::hc19666a2fa542c8f (0x560ac9ee1452)
  11: wasmi::module::ModuleInstance::invoke_export::he19b32b168d86dfe (0x560aca07c3c8)
  12: substrate_executor::wasm_executor::WasmExecutor::call_in_wasm_module::h04f37870b784637d (0x560ac9d5708b)
  13: <std::thread::local::LocalKey<T>>::with::h1a03f7dfd1bfaa0f (0x560ac9dbfbb7)
  14: substrate_state_machine::execute_using_consensus_failure_handler::h79dbff58735b2520 (0x560ac9ea9594)
  15: <substrate_client::call_executor::LocalCallExecutor<B, E> as substrate_client::call_executor::CallExecutor<Block, substrate_primitives::hasher::blake2::Blake2Hasher>>::call_at_state::h5d81654bfd37cfee (0x560aca016a4b)
  16: <substrate_client::client::Client<B, E, Block, RA>>::lock_import_and_run::{{closure}}::h0284170f83cb78a7 (0x560ac9c1021a)
  17: <substrate_client::client::Client<B, E, Block, RA> as substrate_consensus_common::block_import::BlockImport<Block>>::import_block::h71ce15e2f718af39 (0x560ac9c008a6)
  18: <substrate_finality_grandpa::import::GrandpaBlockImport<B, E, Block, RA, PRA> as substrate_consensus_common::block_import::BlockImport<Block>>::import_block::hba40f7ac4511cdeb (0x560ac9dfbed8)
  19: substrate_consensus_common::import_queue::import_many_blocks::h9a4df069f50ed464 (0x560ac9f3d5df)
  20: substrate_consensus_common::import_queue::import_thread::h1aef8acd6e75bb34 (0x560ac9f3be16)
  21: std::sys_common::backtrace::__rust_begin_short_backtrace::h26ef1f51493200e6 (0x560ac9cdc40d)
  22: std::panicking::try::do_call::h2ef0793bbad9113a (0x560ac9edec6f)
  23: __rust_maybe_catch_panic (0x560acab68319)
             at src/libpanic_unwind/lib.rs:92
  24: <F as alloc::boxed::FnBox<A>>::call_box::h6da0ec6574d3aeb7 (0x560aca0c5849)
  25: call_once<(),()> (0x560acab676fd)
             at /rustc/4b1e39b7b36d677803e40130ea29ee6d300abf6e/src/liballoc/boxed.rs:744
      start_thread
             at src/libstd/sys_common/thread.rs:14
      thread_start
             at src/libstd/sys/unix/thread.rs:81
  26: start_thread (0x7f7bb18d6493)
  27: __clone (0x7f7bb1401ace)
  28: <unknown> (0x0)
Thread 'ImportQueue' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0x370bd3c95ab9b31078178fb9c2d6d2806ddf1a9361851ec1190befffdd636422"', src/libcore/result.rs:997

with -l sync=trace

https://gist.github.com/xlc/f79510fe7361090bb752cb5336851eee

I2-bug 🐜

Source

xlc

👍1

Most helpful comment

The issue here is caused by two different branches of the trie having the same leaf node.

Since unlike ethereum we don't hash trie keys, it is possible for the same trie node to be inserted twice into two separate branches of the the same trie. When one branch gets deleted the node is marked as dead, though it is still used in the other branch.

I'll see if I can fix that by changing node hashing scheme to include partial keys.

arkpar on 19 Feb 2019

👍2

All 17 comments

Which commit hash? Substrate or Polkadot?

gavofyork on 8 Feb 2019

using this version 11a2418c9c0972dbbd82e2f39cdb9469a73a85f1
purge chain fixes it, so must be some race condition

xlc on 8 Feb 2019

Has this been running an older version with the same database? Could be database corruption caused by one of the issues fixed earlier.

arkpar on 8 Feb 2019

Everything was reset and a new genesis was generated before we upgrade version. It is at block 8000 and we have 6s block time so only been running for ~13h

xlc on 8 Feb 2019

I think I figured out the cause of this issue, it is relates to the generic asset module we are developing. But I still think it is a substrate bug.

The trouble block contains two transfer of the generic asset.

I had this issue before but a purge chain fixes it so I didn't look into it much. But this time it seems always broken when processing this trouble block.

This is part of the stack trace:

stack backtrace:
   0: substrate_panic_handler::panic_hook::h634713bd606dda3a (0x10a5a1f25)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/panic-handler/src/lib.rs:82
   1: substrate_panic_handler::set::{{closure}}::hd3db95c58b92aaa3 (0x10a5a1b26)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/panic-handler/src/lib.rs:31
   2: std::panicking::rust_panic_with_hook::h13c13167711ed60b (0x10b24f5f1)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libstd/panicking.rs:482
   3: std::panicking::continue_panic_fmt::hb731752a380cf78a (0x10b24f03c)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libstd/panicking.rs:385
   4: _rust_begin_unwind (0x10b24ef28)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libstd/panicking.rs:312
   5: core::panicking::panic_fmt::h33b4e6fb35ab9be8 (0x10b2686b1)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libcore/panicking.rs:85
   6: core::result::unwrap_failed::h1fc4f21bbf65edc3 (0x108d811af)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libcore/macros.rs:16
   7: <core::result::Result<T, E>>::expect::hfd7a0d5c47a9a47f (0x1083de139)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libcore/result.rs:825
   8: <substrate_state_machine::ext::Ext<'a, H, B, T> as substrate_state_machine::Externalities<H>>::storage::{{closure}}::h1d4a18315d58e554 (0x108303543)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/state-machine/src/ext.rs:186
   9: <core::option::Option<T>>::unwrap_or_else::h8679f1bfb086b486 (0x10807b31c)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libcore/option.rs:386
  10: <substrate_state_machine::ext::Ext<'a, H, B, T> as substrate_state_machine::Externalities<H>>::storage::ha819eb2e1f24241e (0x108303313)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/state-machine/src/ext.rs:185
  11: sr_io::read_storage::{{closure}}::h8dca7e65eca9e5fb (0x10a53e27d)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/sr-io/src/../with_std.rs:64
  12: sr_io::ext::with::{{closure}}::h313b15d6cce34842 (0x10a53e962)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/sr-io/<::environmental::environmental macros>:24
  13: environmental::with::{{closure}}::h1007dd51f0fd6b28 (0x10a53faa0)
             at /Users/bryanchen/.cargo/registry/src/github.com-1ecc6299db9ec823/environmental-1.0.0/src/lib.rs:105
  14: <std::thread::local::LocalKey<T>>::try_with::ha573de82de8eddff (0x10a53d60a)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libstd/thread/local.rs:299
  15: <std::thread::local::LocalKey<T>>::with::h124fc5d7126353d4 (0x10a53ce5f)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libstd/thread/local.rs:245
  16: environmental::with::h893c2efb66907e22 (0x10a53f96f)
             at /Users/bryanchen/.cargo/registry/src/github.com-1ecc6299db9ec823/environmental-1.0.0/src/lib.rs:99
  17: sr_io::ext::with::ha85b45d26b8280d3 (0x10a53e831)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/sr-io/<::environmental::environmental macros>:24
  18: sr_io::read_storage::hc0132a9bee046e29 (0x10a53e07c)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/sr-io/src/../with_std.rs:64
  19: srml_support::storage::unhashed::get::h84c739c895812c0b (0x108c92377)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/srml/support/src/storage/mod.rs:404
  20: srml_support::double_map::StorageDoubleMap::get::h3f592d82a07da93f (0x108c710cd)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/srml/support/src/double_map.rs:54
  21: <srml_generic_asset::Module<T>>::free_balance::hab4934d2c6498ae1 (0x108c72d88)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/srml/generic-asset/src/lib.rs:186
  22: <srml_generic_asset::Module<T>>::transfer::h8cf71620443f9cc9 (0x108c74380)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/srml/generic-asset/src/lib.rs:85
  23: <srml_generic_asset::Call<T> as srml_support::dispatch::Dispatchable>::dispatch::hdde289c7fff25f08 (0x108c75ab4)

This is my attempt to translate the stack trace translated to source code
Error starts here: It returns an error if key is missing
https://github.com/paritytech/substrate/blob/e8b558c05ba13a0ee63c7c90e8e4156be38237fd/core/state-machine/src/trie_backend_essence.rs#L72
https://github.com/paritytech/substrate/blob/e8b558c05ba13a0ee63c7c90e8e4156be38237fd/core/state-machine/src/ext.rs#L186
https://github.com/paritytech/substrate/blob/e8b558c05ba13a0ee63c7c90e8e4156be38237fd/core/sr-io/with_std.rs#L49
https://github.com/paritytech/substrate/blob/e8b558c05ba13a0ee63c7c90e8e4156be38237fd/srml/support/src/storage/mod.rs#L404
https://github.com/paritytech/substrate/blob/e8b558c05ba13a0ee63c7c90e8e4156be38237fd/srml/support/src/double_map.rs#L54

xlc on 11 Feb 2019

The database should never miss an expected key. If it's easily reproducible, please provide details - it would make things vastly easier to debug.

gavofyork on 11 Feb 2019

I am not sure how should I provide the reproduce steps as it requires connect to my internal testnet which is not public accessible.

I tried purge chain and starts with --pruning=archive which seems fixes this issue so I suspect it relates to db pruning.
I will keep digging it for a while and try to create some minimal reproducible steps.

xlc on 11 Feb 2019

Logging with -lstate-db=trace would be useful as well.

arkpar on 12 Feb 2019

This is the log. File too large for gist
https://drive.google.com/file/d/1K2u_BC8WDBmP4lUaKUiVj4M4eJism7uc/view?usp=sharing

xlc on 13 Feb 2019

The log looks OK. Could you maybe share the spec file and the archive db somewhere?

arkpar on 13 Feb 2019

Reproduced with the DB provided by @xlc.
Tracing shows that the key was inserted at genesis and removed at block 60. It has not been re-inserted and still somehow got accessed at block 48952. Might be a trie issue.

arkpar on 15 Feb 2019

maybe trie node encoding is messed up somehow

rphmeier on 18 Feb 2019

The issue here is caused by two different branches of the trie having the same leaf node.

I'll see if I can fix that by changing node hashing scheme to include partial keys.

arkpar on 19 Feb 2019

👍2

Hi, we are ChainX team
we meet the same error.
I think it's a critical bug

let me show more detail for this problem
our environment like this:
our project base on the substrate which commit is "e8b558c05ba13a0ee63c7c90e8e4156be38237fd" (2019/2/10)
we run 4 authority node to produce block, and all use --pruning archive mode, thus, it would contain all state.
we run many full node to sync block, and some nodes do not use --pruning archive mode, thus, some node would remove old state.
and when the full node sync blocks,

the node which use --pruning archive mode, normally sync block.
the node which use default config (no archive, retain 256 block state), all panic at the block number 27043

notice this block height is more than 256

the log is

2019-02-19 15:27:03 Failed to write to trie: Database missing expected key: 0xa7a5129d4b695ce4b72a059eec5bf44f70d5a1109e5e63744597a26f63b1b71a
Hash: given=454954446f6f379655e9eea0a214c16d849c461ee2b89c1742237c00750ac9d4, expected=568678ab87053959fb7da15bfedd6183946085048d0ec1c165b2140672e8e38d
2019-02-19 15:27:04 Syncing  4.6 bps, target=#35560 (1 peers), best: #27023 (0x974a…8580), finalized #27000 (0x026f…1949), ⬇ 0.3kiB/s ⬆ 0

====================

stack backtrace:
   0: substrate_panic_handler::panic_hook::h4c80c61cd518b75a (0x55eec241cd3d)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/panic-handler/src/lib.rs:82
   1: substrate_panic_handler::set::{{closure}}::hbb0ee31e0e60fe13 (0x55eec241c8d6)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/panic-handler/src/lib.rs:31
   2: rust_panic_with_hook (0x55eec27a38bd)
             at src/libstd/panicking.rs:482
   3: std::panicking::begin_panic::ha1e86c9c64bec910 (0x55eec26f67a7)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panicking.rs:412
   4: <chainx_runtime::xexecutive::Executive<System, Block, Context, Payment, Finalisation>>::final_checks::hf22c79d951fbe62c (0x55eec1f2268b)
             at runtime/src/xexecutive.rs:209
   5: <chainx_runtime::xexecutive::Executive<System, Block, Context, Payment, Finalisation>>::execute_block::hd5fd909b1e69198b (0x55eec1f20d9a)
             at runtime/src/xexecutive.rs:105
   6: <chainx_runtime::Runtime as substrate_client::runtime_api::runtime_decl_for_Core::Core<sr_primitives::generic::block::Block<sr_primitives::generic::header::Header<u64, sr_primitives::traits::BlakeTwo256, chainx_runtime::Log>, xr_primitives::generic::unchecked_mortal_compact_extrinsic::UncheckedMortalCompactExtrinsic<<srml_indices::Module<chainx_runtime::Runtime> as sr_primitives::traits::StaticLookup>::Source, u64, chainx_runtime::Call, sr_primitives::Ed25519Signature, u32>>>>::execute_block::hb3260379a48311cc (0x55eec1ed9e33)
             at runtime/src/lib.rs:362
   7: chainx_runtime::api::dispatch::h55b603af0cfa2f54 (0x55eec1f2ccda)
             at runtime/src/lib.rs:351
   8: <chainx_executor::Executor as substrate_executor::native_executor::NativeExecutionDispatch>::dispatch::{{closure}}::hddb5cc6fc3329f30 (0x55eec0b43c80)
             at /home/king/workspace/chainpool/ChainX/<::substrate_executor::native_executor::native_executor_instance macros>:14
   9: std::panicking::try::do_call::h3710619e202612ef (0x55eec0b439c3)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panicking.rs:297
  10: __rust_maybe_catch_panic (0x55eec27a9739)
             at src/libpanic_unwind/lib.rs:90
  11: std::panicking::try::h1f32ca7c344d329e (0x55eec0b43852)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panicking.rs:276
  12: std::panic::catch_unwind::h375a35de590fe767 (0x55eec0b43428)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panic.rs:388
  13: substrate_executor::native_executor::safe_call::h74c288476abfb813 (0x55eec0b42a09)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/executor/src/native_executor.rs:103
  14: substrate_executor::native_executor::with_native_environment::{{closure}}::h3df228a404f3a843 (0x55eec0b42978)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/executor/src/native_executor.rs:112
  15: environmental::using::{{closure}}::h9250eb6207a176df (0x55eec0b43bc1)
             at /home/king/.cargo/registry/src/github.com-1ecc6299db9ec823/environmental-1.0.0/src/lib.rs:90
  16: <std::thread::local::LocalKey<T>>::try_with::h23d632e7182b96e5 (0x55eec0b44139)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/thread/local.rs:296
  17: <std::thread::local::LocalKey<T>>::with::hef3d00b5cc2bfbd9 (0x55eec0b43f22)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/thread/local.rs:242
  18: environmental::using::hb0607093eeaaaf53 (0x55eec0b43a76)
             at /home/king/.cargo/registry/src/github.com-1ecc6299db9ec823/environmental-1.0.0/src/lib.rs:67
  19: sr_io::ext::using::h1d464473344411d2 (0x55eec0b4356a)
             at /home/king/workspace/chainpool/ChainX/<::environmental::environmental macros>:21
  20: sr_io::with_externalities::h719101822e3fb9e9 (0x55eec0b4348c)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/sr-io/src/../with_std.rs:208
  21: substrate_executor::native_executor::with_native_environment::h0fbf58e04fa0307e (0x55eec0b4292c)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/executor/src/native_executor.rs:112
  22: <chainx_executor::Executor as substrate_executor::native_executor::NativeExecutionDispatch>::dispatch::h20e69cc2f0c8a80e (0x55eec0b42630)
             at /home/king/workspace/chainpool/ChainX/<::substrate_executor::native_executor::native_executor_instance macros>:13
  23: <substrate_executor::native_executor::NativeExecutor<D> as substrate_state_machine::CodeExecutor<substrate_primitives::hasher::blake2::Blake2Hasher>>::call::{{closure}}::h8ab54b3d3a93cea7 (0x55eec015f35e)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/executor/src/native_executor.rs:261
  24: <std::thread::local::LocalKey<T>>::try_with::h497e4d82a56066f0 (0x55eec0708a6f)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/thread/local.rs:296
  25: <std::thread::local::LocalKey<T>>::with::h3752aa39db068ae5 (0x55eec06fe032)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/thread/local.rs:242
  26: <substrate_executor::native_executor::NativeExecutor<D> as substrate_state_machine::CodeExecutor<substrate_primitives::hasher::blake2::Blake2Hasher>>::call::ha3517c98fb5047e4 (0x55eec013e1f5)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/executor/src/native_executor.rs:201
  27: substrate_state_machine::execute_using_consensus_failure_handler::h5855500d8941d00a (0x55eec030a62d)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/state-machine/src/lib.rs:338
  28: <substrate_client::call_executor::LocalCallExecutor<B, E> as substrate_client::call_executor::CallExecutor<Block, substrate_primitives::hasher::blake2::Blake2Hasher>>::call_at_state::h2436532f601166fb (0x55eec0448676)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/call_executor.rs:266
  29: <substrate_client::client::Client<B, E, Block, RA>>::block_execution::h9c79ae1700bd1d30 (0x55eebf9b543b)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:806
  30: <substrate_client::client::Client<B, E, Block, RA>>::execute_and_import_block::he1a6f8e4bef63c48 (0x55eebf9c6d1d)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:739
  31: <substrate_client::client::Client<B, E, Block, RA>>::apply_block::he17c4d54d4b2e2c1 (0x55eebf9ace7c)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:674
  32: <substrate_client::client::Client<B, E, Block, RA> as substrate_consensus_common::block_import::BlockImport<Block>>::import_block::{{closure}}::hba9d7af3939c4a98 (0x55eebf9aa76c)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:1301
  33: <substrate_client::client::Client<B, E, Block, RA>>::lock_import_and_run::{{closure}}::hcdc3c6fed815d1c8 (0x55eebf9c1e46)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:608
  34: <substrate_client::client::Client<B, E, Block, RA>>::lock_import_and_run::he5c39686a449a393 (0x55eebf9bc21b)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:621
  35: <substrate_client::client::Client<B, E, Block, RA> as substrate_consensus_common::block_import::BlockImport<Block>>::import_block::h1d681f15493a7b65 (0x55eebf9aa515)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:1300
  36: <substrate_finality_grandpa::import::GrandpaBlockImport<B, E, Block, RA, PRA> as substrate_consensus_common::block_import::BlockImport<Block>>::import_block::hdd89b149fc03e551 (0x55eec03972e4)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/finality-grandpa/src/import.rs:196
  37: substrate_consensus_common::import_queue::import_single_block::h7890131ffeb8db37 (0x55eec01bdd63)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/consensus/common/src/import_queue.rs:459
  38: substrate_consensus_common::import_queue::import_many_blocks::h0518398b3f4a9ded (0x55eec01bb231)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/consensus/common/src/import_queue.rs:363
  39: substrate_consensus_common::import_queue::import_thread::h41c99576b7aad8e0 (0x55eec01ba2e0)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/consensus/common/src/import_queue.rs:276
  40: <substrate_consensus_common::import_queue::BasicQueue<B, V> as substrate_consensus_common::import_queue::ImportQueue<B>>::start::{{closure}}::hb1817216f0d24873 (0x55eec01b8ced)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/consensus/common/src/import_queue.rs:176
  41: std::sys_common::backtrace::__rust_begin_short_backtrace::h9e9cacc9cea5202f (0x55eec04bfbe2)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/sys_common/backtrace.rs:135
  42: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::h3aa320fb6f55333d (0x55eec03b8c60)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/thread/mod.rs:469
  43: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hd6ccaeef3b02be4d (0x55eec04e4bf0)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panic.rs:309
  44: std::panicking::try::do_call::h87893e184140a9f2 (0x55eec05319a9)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panicking.rs:297
  45: __rust_maybe_catch_panic (0x55eec27a9739)
             at src/libpanic_unwind/lib.rs:90
  46: std::panicking::try::hcf58bda4b4007f7a (0x55eec052f34f)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panicking.rs:276
  47: std::panic::catch_unwind::ha4d988f020d4ad09 (0x55eec04e54d2)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panic.rs:388
  48: std::thread::Builder::spawn_unchecked::{{closure}}::he82301628f336f6e (0x55eec03b8a05)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/thread/mod.rs:468
  49: <F as alloc::boxed::FnBox<A>>::call_box::h9ec9f86765a03527 (0x55eec03b91ca)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/liballoc/boxed.rs:734
  50: call_once<(),()> (0x55eec27a8b1d)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/liballoc/boxed.rs:744
      start_thread
             at src/libstd/sys_common/thread.rs:14
      thread_start
             at src/libstd/sys/unix/thread.rs:81
  51: start_thread (0x7f74c0b395a9)
  52: __clone (0x7f74c0656cbe)
  53: <unknown> (0x0)

Thread 'ImportQueue' panicked at 'Storage root must match that calculated.', runtime/src/xexecutive.rs:209

we have tried many times, make sure the problem could reproduced every times.
we think this problem is caused by the pruning.

I'm familiar with the MPT structure, I think this problem causes by refactor the MPT nodes, maybe the pruning mode cause unexpect remove for nodes

@gavofyork @arkpar
we could provide our source code and bootsnode to reproduce this problem, if you need code and our bootsnode, please provide the email, I would send those thing for you in private.

think you very much