Parity-ethereum: panic after updating from 2.6.8 to 2.7.0 when `Tracing` is enabled

[cipig]

Jan 29 18:28:55 Debian-94-stretch-64-minimal systemd[1]: Started Parity Ethereum Daemon.
Jan 29 18:28:56 Debian-94-stretch-64-minimal parity[8412]: 2020-01-29 18:28:56 UTC Starting Parity-Ethereum/v2.7.0-stable-aa0a703-20200128/x86_64-unknown-linux-gnu/rustc1.40.0
Jan 29 18:28:56 Debian-94-stretch-64-minimal parity[8412]: 2020-01-29 18:28:56 UTC Keys path /home/electrum/.local/share/io.parity.ethereum/keys/ethereum
Jan 29 18:28:56 Debian-94-stretch-64-minimal parity[8412]: 2020-01-29 18:28:56 UTC DB path /home/electrum/.local/share/io.parity.ethereum/chains/ethereum/db/906a34e69aec8c0d
Jan 29 18:28:56 Debian-94-stretch-64-minimal parity[8412]: 2020-01-29 18:28:56 UTC State DB configuration: fast +Trace
Jan 29 18:28:56 Debian-94-stretch-64-minimal parity[8412]: 2020-01-29 18:28:56 UTC Operating mode: active
Jan 29 18:28:56 Debian-94-stretch-64-minimal parity[8412]: 2020-01-29 18:28:56 UTC Warning: Warp Sync is disabled because tracing is turned on.
Jan 29 18:28:56 Debian-94-stretch-64-minimal parity[8412]: 2020-01-29 18:28:56 UTC Configured for Ethereum using Ethash engine
Jan 29 18:28:56 Debian-94-stretch-64-minimal parity[8412]: 2020-01-29 18:28:56 UTC Running without a persistent transaction queue.
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]: 2020-01-29 18:28:59 UTC Panic occurred, see stderr for details
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]: ====================
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]: stack backtrace:
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:    0:     0x56191d2c32ed - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:    1:     0x56191dac9558 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:    2:     0x56191dd4f90a - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:    3:     0x56191e58aa53 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:    4:     0x56191e58a7dd - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:    5:     0x56191e5a5f55 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:    6:     0x56191d37bd3c - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:    7:     0x56191d3820f5 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:    8:     0x56191d4f6baa - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:    9:     0x56191d4d2fde - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   10:     0x56191d53480c - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   11:     0x56191dba0c87 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   12:     0x56191de148d5 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   13:     0x56191db4e874 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   14:     0x56191dd43ec0 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   15:     0x56191dd29a72 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   16:     0x56191d227d92 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   17:     0x56191de25a6f - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   18:     0x56191de27803 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   19:     0x56191de23f92 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   20:     0x56191dec383f - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   21:     0x56191db3f754 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   22:     0x56191db57664 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   23:     0x56191e5ceb38 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   24:     0x56191e5ca445 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   25:     0x56191e5c880f - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   26:     0x56191e5b49c6 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   27:     0x56191e5ac6ea - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   28:     0x56191e5c68bd - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   29:     0x56191e5c7327 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   30:     0x56191e5a4b1e - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   31:     0x56191e5a7b4f - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   32:     0x7f97f1417fa2 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   33:     0x7f97f131f4ce - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]:   34:                0x0 - <no info>
Jan 29 18:28:59 Debian-94-stretch-64-minimal parity[8412]: Thread 'http.worker70' panicked at 'decode db value failed: RlpExpectedToBeList', src/libcore/result.rs:1165

start params are /usr/local/bin/parity --refuse-service-transactions --min-gas-price=1000000000 --no-serve-light --jsonrpc-apis=safe --jsonrpc-interface=all --jsonrpc-server-threads=8 --jsonrpc-port=8555 --no-persistent-txqueue --no-ancient-blocks --mode=active --tracing=on --pruning=fast --db-compaction=ssd --cache-size=1024 --min-peers=16 --max-peers=24

on 3 nodes it panicked right after start, on a 4-th node after couple of minutes

[uniibu]

We also got the same issue after updating v2.5.13-stable to v2.7.0-stable

[dvdplm]

@cipig @uniibu Are the nodes receiving RPC traffic? It would be very useful if you could run with -lrpc=trace and post the resulting log at the time of the panic.

[uniibu]

@cipig @uniibu Are the nodes receiving RPC traffic? It would be very useful if you could run with -lrpc=trace and post the resulting log at the time of the panic.

At first it was, then it is just stuck there on ... please wait.

[cipig]

restarted with -lrpc=trace and this is the output (it panicked right after start)

Jan 29 18:50:12 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:12 UTC main INFO parity_ethereum::run  Starting Parity-Ethereum/v2.7.0-stable-aa0a703-20200128/x86_64-unknown-linux-gnu/rustc1.40.0
Jan 29 18:50:12 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:12 UTC main INFO parity_ethereum::run  Keys path /home/electrum/.local/share/io.parity.ethereum/keys/ethereum
Jan 29 18:50:12 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:12 UTC main INFO parity_ethereum::run  DB path /home/electrum/.local/share/io.parity.ethereum/chains/ethereum/db/906a34e69aec8c0d
Jan 29 18:50:12 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:12 UTC main INFO parity_ethereum::run  State DB configuration: fast +Trace
Jan 29 18:50:12 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:12 UTC main INFO parity_ethereum::run  Operating mode: active
Jan 29 18:50:12 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:12 UTC main WARN parity_ethereum::run  Warning: Warp Sync is disabled because tracing is turned on.
Jan 29 18:50:12 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:12 UTC main INFO ethcore_service::service  Configured for Ethereum using Ethash engine
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC main INFO parity_ethereum::run  Running without a persistent transaction queue.
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC event.loop0 WARN jsonrpc_ipc_server::server  Removed existing file '/home/electrum/.local/share/io.parity.ethereum/jsonrpc.ipc'.
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker70 TRACE rpc  Request: [{"id":0,"jsonrpc":"2.0","method":"eth_gasPrice"},{"id":1,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x01484318172a8cafe57f91ec5965451f11235d54","latest"]},{"id":2,"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x01484318172a8cafe57f91ec5965451f11235d54","latest"]},{"id":3,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x119f4555b9af7cc54dc36024429194bfbb133b33","latest"]},{"id":4,"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x119f4555b9af7cc54dc36024429194bfbb133b33","latest"]},{"id":5,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x7c0c05d7f30e18d9d4061ef8408a51e55b45becc","latest"]},{"id":6,"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x7c0c05d7f30e18d9d4061ef8408a51e55b45becc","latest"]}].
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker70 DEBUG rpc  [None] Took 25ms
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker70 DEBUG rpc  Response: [{"jsonrpc":"2.0","result":"0x9502f900","id":0},{"jsonrpc":"2.0","result":"0x0","id":1},{"jsonrpc":"2.0","result":"0x0","id":2},{"jsonrpc":"2.0","result":"0x0","id":3},{"jsonrpc":"2.0","result":"0x0","id":4},{"jsonrpc":"2.0","result":"0x0","id":5},{"jsonrpc":"2.0","result":"0x5","id":6}].
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker40 TRACE rpc  Request: {"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":108}.
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker40 DEBUG rpc  Response: {"jsonrpc":"2.0","result":"0x8f1bf2","id":108}.
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker30 TRACE rpc  Request: [{"id":0,"jsonrpc":"2.0","method":"eth_gasPrice"},{"id":1,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x8be0ebfeb7dd2e547977d147f23b7cd3cd7c9e69","latest"]},{"id":2,"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x8be0ebfeb7dd2e547977d147f23b7cd3cd7c9e69","latest"]}].
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker60 TRACE rpc  Request: {"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1026}.
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker60 DEBUG rpc  Response: {"jsonrpc":"2.0","result":"0x8f1bf2","id":1026}.
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker30 DEBUG rpc  Response: [{"jsonrpc":"2.0","result":"0x9502f900","id":0},{"jsonrpc":"2.0","result":"0x0","id":1},{"jsonrpc":"2.0","result":"0x0","id":2}].
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker20 TRACE rpc  Request: [{"id":0,"jsonrpc":"2.0","method":"eth_gasPrice"},{"id":1,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x1ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","latest"]},{"id":2,"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x1ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","latest"]},{"id":3,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0x0f8c45b896784a1e408526b9300519ef8660209c"},"latest"]},{"id":4,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0x11016adcb85b65a0da8d8ddc3ade69dbad6bbca4"},"latest"]},{"id":5,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0x17aa18a4b64a55abed7fa543f2ba4e91f2dce482"},"latest"]},{"id":6,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0x1e4e36b3f011d862fd70006804da8fcefe89d3d8"},"latest"]},{"id":7,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0x4d2c05109a1309c6de0d3b7f06f397c9c41b8fae"},"latest"]},{"id":8,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0x516e5436bafdc11083654de7bb9b95382d08d5de"},"latest"]},{"id":9,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0x5c8118fc0237697422ced89a448dce2c8e34b4ef"},"latest"]},{"id":10,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0x626a14ec8bb0d92692ef704f19968c20727dede1"},"latest"]},{"id":11,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0x89885fc1f76c3f4cc719640e33c315227da7003a"},"latest"]},{"id":12,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0x8a3e08353e3c64d9fa5683bb5e2fbbf8aef7e7e9"},"latest"]},{"id":13,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0x91cdb5bb5969bfed2373e97378354052bbc606f2"},"latest"]},{"id":14,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0x943ed852dadb5c3938ecdc6883718df8142de4c8"},"latest"]},{"id":15,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0x98da3b3da368a23693eb609d5ce2d779896a31b5"},"latest"]},{"id":16,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0xb6f43025b29196af2dddd69b0a58afba079cd600"},"latest"]},{"id":17,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0xbc647aad10114b89564c0a7aabe542bd0cf2c5af"},"latest"]},{"id":18,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0xbc8deee89f1cf4b661514185aa1ab780336c4c4a"},"latest"]},{"id":19,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0xe6bc60a00b81c7f3cbc8f4ef3b0a6805b6851753"},"latest"]},{"id":20,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0xf8c595d070d104377f58715ce2e6c93e49a87f3c"},"latest"]},{"id":21,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000001ac505f02e6A6Aa7ABb1B8B99c7c43BC53DBa2dE","to":"0xfb8bf095ebcdad57d2e37573a505e7d3bafdd3cc"},"latest"]},{"id":22,"jsonrpc":"2.0","method":"eth_getBalance","params":["0xb161119D490A1d2AedFA78f54ec31f72B1408896","latest"]},{"id":23,"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0xb161119D490A1d2AedFA78f54ec31f72B1408896","latest"]},{"id":24,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0x0f8c45b896784a1e408526b9300519ef8660209c"},"latest"]},{"id":25,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0x11016adcb85b65a0da8d8ddc3ade69dbad6bbca4"},"latest"]},{"id":26,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0x17aa18a4b64a55abed7fa543f2ba4e91f2dce482"},"latest"]},{"id":27,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0x1e4e36b3f011d862fd70006804da8fcefe89d3d8"},"latest"]},{"id":28,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0x4d2c05109a1309c6de0d3b7f06f397c9c41b8fae"},"latest"]},{"id":29,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0x516e5436bafdc11083654de7bb9b95382d08d5de"},"latest"]},{"id":30,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0x5c8118fc0237697422ced89a448dce2c8e34b4ef"},"latest"]},{"id":31,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0x626a14ec8bb0d92692ef704f19968c20727dede1"},"latest"]},{"id":32,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0x89885fc1f76c3f4cc719640e33c315227da7003a"},"latest"]},{"id":33,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0x8a3e08353e3c64d9fa5683bb5e2fbbf8aef7e7e9"},"latest"]},{"id":34,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0x91cdb5bb5969bfed2373e97378354052bbc606f2"},"latest"]},{"id":35,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0x943ed852dadb5c3938ecdc6883718df8142de4c8"},"latest"]},{"id":36,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0x98da3b3da368a23693eb609d5ce2d779896a31b5"},"latest"]},{"id":37,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0xb6f43025b29196af2dddd69b0a58afba079cd600"},"latest"]},{"id":38,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0xbc647aad10114b89564c0a7aabe542bd0cf2c5af"},"latest"]},{"id":39,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0xbc8deee89f1cf4b661514185aa1ab780336c4c4a"},"latest"]},{"id":40,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0xe6bc60a00b81c7f3cbc8f4ef3b0a6805b6851753"},"latest"]},{"id":41,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0xf8c595d070d104377f58715ce2e6c93e49a87f3c"},"latest"]},{"id":42,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a08231000000000000000000000000b161119D490A1d2AedFA78f54ec31f72B1408896","to":"0xfb8bf095ebcdad57d2e37573a505e7d3bafdd3cc"},"latest"]},{"id":43,"jsonrpc":"2.0","method":"eth_getBalance","params":["0x5a211c7E5AEDb31c2195d32C477D1F835103e927","latest"]},{"id":44,"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x5a211c7E5AEDb31c2195d32C477D1F835103e927","latest"]},{"id":45,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0x0f8c45b896784a1e408526b9300519ef8660209c"},"latest"]},{"id":46,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0x11016adcb85b65a0da8d8ddc3ade69dbad6bbca4"},"latest"]},{"id":47,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0x17aa18a4b64a55abed7fa543f2ba4e91f2dce482"},"latest"]},{"id":48,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0x1e4e36b3f011d862fd70006804da8fcefe89d3d8"},"latest"]},{"id":49,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0x4d2c05109a1309c6de0d3b7f06f397c9c41b8fae"},"latest"]},{"id":50,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0x516e5436bafdc11083654de7bb9b95382d08d5de"},"latest"]},{"id":51,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0x5c8118fc0237697422ced89a448dce2c8e34b4ef"},"latest"]},{"id":52,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0x626a14ec8bb0d92692ef704f19968c20727dede1"},"latest"]},{"id":53,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0x89885fc1f76c3f4cc719640e33c315227da7003a"},"latest"]},{"id":54,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0x8a3e08353e3c64d9fa5683bb5e2fbbf8aef7e7e9"},"latest"]},{"id":55,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0x91cdb5bb5969bfed2373e97378354052bbc606f2"},"latest"]},{"id":56,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0x943ed852dadb5c3938ecdc6883718df8142de4c8"},"latest"]},{"id":57,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0x98da3b3da368a23693eb609d5ce2d779896a31b5"},"latest"]},{"id":58,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0xb6f43025b29196af2dddd69b0a58afba079cd600"},"latest"]},{"id":59,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0xbc647aad10114b89564c0a7aabe542bd0cf2c5af"},"latest"]},{"id":60,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0xbc8deee89f1cf4b661514185aa1ab780336c4c4a"},"latest"]},{"id":61,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0xe6bc60a00b81c7f3cbc8f4ef3b0a6805b6851753"},"latest"]},{"id":62,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0xf8c595d070d104377f58715ce2e6c93e49a87f3c"},"latest"]},{"id":63,"jsonrpc":"2.0","method":"eth_call","params":[{"data":"0x70a082310000000000000000000000005a211c7E5AEDb31c2195d32C477D1F835103e927","to":"0xfb8bf095ebcdad57d2e37573a505e7d3bafdd3cc"},"latest"]}].
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker20 DEBUG rpc  [None] Took 129ms
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker20 DEBUG rpc  Response: [{"jsonrpc":"2.0","result":"0x9502f900","id":0},{"jsonrpc":"2.0","result":"0x0","id":1},{"jsonrpc":"2.0","result":"0x42","id":2},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000017d78400","id":3},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":4},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":5},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000001bc16d674ec80000","id":6},{"jsonrpc":"2.0","result":"0x000000000000000000000000000000000000000000000001a055690d9db80000","id":7},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000005f5e100","id":8},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000005f5e100","id":9},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000e043da617250000","id":10},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":11},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":12},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000e725ffc0b2e8000","id":13},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":14},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":15},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":16},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":17},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000001bc16d674ec80000","id":18},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":19},{"jsonrpc":"2.0","result":"0x00000000000000000000000000000000000000000000000000000000000f4240","id":20},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000005f5e100","id":21},{"jsonrpc":"2.0","result":"0x0","id":22},{"jsonrpc":"2.0","result":"0x56","id":23},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000017d78400","id":24},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":25},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":26},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000001bc16d674ec80000","id":27},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":28},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":29},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000005f5e100","id":30},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000e043da617250000","id":31},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":32},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":33},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000e7d0877f9820000","id":34},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":35},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":36},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":37},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":38},{"jsonrpc":"2.0","result":"0x00000000000000000000000000000000000000000000000029a2241af62c0000","id":39},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":40},{"jsonrpc":"2.0","result":"0x00000000000000000000000000000000000000000000000000000000000f4240","id":41},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":42},{"jsonrpc":"2.0","result":"0x82fe1adbf6b9","id":43},{"jsonrpc":"2.0","result":"0x27","id":44},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":45},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":46},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":47},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000001bc16d674ec80000","id":48},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":49},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":50},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000005f5e100","id":51},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000e043da617250000","id":52},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":53},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":54},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000e4b4b8af6a70000","id":55},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":56},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":57},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":58},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":59},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":60},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000de0b6b3a7640000","id":61},{"jsonrpc":"2.0","result":"0x00000000000000000000000000000000000000000000000000000000000f4240","id":62},{"jsonrpc":"2.0","result":"0x0000000000000000000000000000000000000000000000000000000000000000","id":63}].
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker60 TRACE rpc  Request: {"jsonrpc":"2.0","method":"trace_filter","params":[{"fromAddress":["0xba1cd315b859498d839fed103e38e98f5d758564"],"fromBlock":"0x8f1a9a","toAddress":[],"toBlock":"0x8f1bf2"}],"id":109}.
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC http.worker60 WARN parity  Panic occurred, see stderr for details
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: ====================
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: stack backtrace:
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:    0:     0x55f752c0b2ed - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:    1:     0x55f753411558 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:    2:     0x55f75369790a - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:    3:     0x55f753ed2a53 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:    4:     0x55f753ed27dd - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:    5:     0x55f753eedf55 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:    6:     0x55f752cc3d3c - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:    7:     0x55f752cca0f5 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:    8:     0x55f752e3ebaa - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:    9:     0x55f752e1afde - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   10:     0x55f752e7c80c - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   11:     0x55f7534e8c87 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   12:     0x55f75375c8d5 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   13:     0x55f753496874 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   14:     0x55f75368bec0 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   15:     0x55f753671a72 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   16:     0x55f752b6fd92 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   17:     0x55f75376da6f - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   18:     0x55f75376f803 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   19:     0x55f75376bf92 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   20:     0x55f75380b83f - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   21:     0x55f753487754 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   22:     0x55f75349f664 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   23:     0x55f753f16b38 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   24:     0x55f753f12445 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   25:     0x55f753f1080f - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   26:     0x55f753efc9c6 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   27:     0x55f753ef46ea - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   28:     0x55f753f0e8bd - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   29:     0x55f753f0f327 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   30:     0x55f753eecb1e - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   31:     0x55f753eefb4f - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   32:     0x7f494ec86fa2 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   33:     0x7f494eb8e4ce - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:   34:                0x0 - <no info>
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: Thread 'http.worker60' panicked at 'decode db value failed: RlpExpectedToBeList', src/libcore/result.rs:1165
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: This is a bug. Please report it at:
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]:     https://github.com/paritytech/parity-ethereum/issues/new
Jan 29 18:50:13 Debian-94-stretch-64-minimal parity[20665]: 2020-01-29 18:50:13 UTC main INFO parity_ethereum::run  Finishing work, please wait...

looks like the method trace_filter is responsible

[dvdplm]

@cipig Ok, thank you. Can you try to run only trace_filter please? I.e. Stop incoming RPC traffic, then start the node, then issue a single {"jsonrpc":"2.0","method":"trace_filter","params":[{"fromAddress":["0xba1cd315b859498d839fed103e38e98f5d758564"],"fromBlock":"0x8f1a9a","toAddress":[],"toBlock":"0x8f1bf2"}],"id":109} e.g. with curl? Is that enough to trigger a panic?

[uniibu]

I got another instance which i also updated from v2.5.13-stable to v2.7.0-stable, however, this instance didnt have tracing enabled and it worked fine.
Could be similar to https://github.com/paritytech/parity-ethereum/issues/1704 because of the update on json rpc?(not sure)

[cipig]

i can't stop incoming RPC traffic and issue an RPC call at the same time... without doing dirty firewall tricks... but pretty sure the panick occurs in trace_filter because it happens on the other nodes too... same call, same panick

[dvdplm]

this instance didnt have tracing enabled and it worked fine.

Ok, that's good info.

[dvdplm]

but pretty sure the panick occurs in trace_filter because it happens on the other nodes too... same call, same panick

We'll try to reproduce on our end.

[uniibu]

Just to add, this also happened to us because of trace_filter.

  ...
  $return = $this->wallet->trace_filter($block_from, $block_to);
} catch (Exception $e) {
...

[uniibu]

Do we have any updates on this?

[niklasad1]

@uniibu

No, I don't have fully synced node with tracing enabled. Syncing now, will take a while.

Meanwhile, if you could build https://github.com/paritytech/parity-ethereum/tree/na-print-raw-kvdb-val and run it? It will print out the raw bytes that caused the decoding failure then please share your output.

[androzd]

Same issue appeared again for me. Log trace:

2020-01-30 10:55:22  main INFO parity_ethereum::run  Starting Parity-Ethereum/v2.7.0-stable-aa0a703-20200128/x86_64-unknown-linux-gnu/rustc1.40.0
2020-01-30 10:55:22  main INFO parity_ethereum::run  Keys path /home/eth/parity/keys/ethereum
2020-01-30 10:55:22  main INFO parity_ethereum::run  DB path /home/eth/parity/chains/ethereum/db/906a34e69aec8c0d
2020-01-30 10:55:22  main INFO parity_ethereum::run  State DB configuration: fast +Fat +Trace
2020-01-30 10:55:22  main INFO parity_ethereum::run  Operating mode: active
2020-01-30 10:55:22  main WARN parity_ethereum::run  Warning: Warp Sync is disabled because Fat DB is turned on.
2020-01-30 10:55:44  main INFO ethcore_service::service  Configured for Ethereum using Ethash engine
2020-01-30 10:55:44  main INFO parity_ethereum::run  Running without a persistent transaction queue.
2020-01-30 10:55:44  http.worker30 TRACE rpc  Request: {"jsonrpc":"2.0","method":"eth_getTransactionByHash","params":["0x66994c12392d131e2200d02709cfea9461bd0f662e1230e3a2e3d1e27246757d"],"id":1580378144}.
2020-01-30 10:55:44  http.worker30 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"blockHash":"0x02e57d101c7f620c6b3a9ee54369c8d92970ac75c7bb994e69ade05cbc7e3fff","blockNumber":"0x8f10f2","chainId":null,"condition":null,"creates":null,"from":"0x8babf0ba311aab914c00e8fda7e8558a8b66de5d","gas":"0xfde8","gasPrice":"0x826299e00","hash":"0x66994c12392d131e2200d02709cfea9461bd0f662e1230e3a2e3d1e27246757d","input":"0xa9059cbb0000000000000000000000009368ff2f1e0dbc552fbe7534e60d4c777df9d4fe000000000000000000000000000000000000000000000000000000138bb0d300","nonce":"0x4f9fd","publicKey":"0xa4de49d6e3a0b28d67cf8244c722ece6e1d1f4cdac5f3856337099d759b920fa99ab18f600437fca4eb8c0ec0a5e1608d4bad77300a7425941392043e3a4e358","r":"0x26bc607124af1bb6a022560a9abe401213fc71f8ceef5f9ba6fbd8a71608c343","raw":"0xf8ac8304f9fd850826299e0082fde89468e54af74b22acaccffa04ccaad13be16ed14eac80b844a9059cbb0000000000000000000000009368ff2f1e0dbc552fbe7534e60d4c777df9d4fe000000000000000000000000000000000000000000000000000000138bb0d3001ca026bc607124af1bb6a022560a9abe401213fc71f8ceef5f9ba6fbd8a71608c343a0042bc9ca390306dab6248c9fdbdf6ce653710ac17016f0bb1186eb74e7d4164f","s":"0x42bc9ca390306dab6248c9fdbdf6ce653710ac17016f0bb1186eb74e7d4164f","standardV":"0x1","to":"0x68e54af74b22acaccffa04ccaad13be16ed14eac","transactionIndex":"0x1","v":"0x1c","value":"0x0"},"id":1580378144}.
2020-01-30 10:55:44  http.worker10 TRACE rpc  Request: {"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1580378144}.
2020-01-30 10:55:44  http.worker10 DEBUG rpc  Response: {"jsonrpc":"2.0","result":"0x8f206b","id":1580378144}.
2020-01-30 10:55:44  http.worker00 TRACE rpc  Request: {"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params":["0x66994c12392d131e2200d02709cfea9461bd0f662e1230e3a2e3d1e27246757d"],"id":1580378144}.
2020-01-30 10:55:44  http.worker00 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"blockHash":"0x02e57d101c7f620c6b3a9ee54369c8d92970ac75c7bb994e69ade05cbc7e3fff","blockNumber":"0x8f10f2","contractAddress":null,"cumulativeGasUsed":"0x1958e","from":"0x8babf0ba311aab914c00e8fda7e8558a8b66de5d","gasUsed":"0xcac7","logs":[{"address":"0x68e54af74b22acaccffa04ccaad13be16ed14eac","blockHash":"0x02e57d101c7f620c6b3a9ee54369c8d92970ac75c7bb994e69ade05cbc7e3fff","blockNumber":"0x8f10f2","data":"0x000000000000000000000000000000000000000000000000000000138bb0d300","logIndex":"0x1","removed":false,"topics":["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x0000000000000000000000008babf0ba311aab914c00e8fda7e8558a8b66de5d","0x0000000000000000000000009368ff2f1e0dbc552fbe7534e60d4c777df9d4fe"],"transactionHash":"0x66994c12392d131e2200d02709cfea9461bd0f662e1230e3a2e3d1e27246757d","transactionIndex":"0x1","transactionLogIndex":"0x0","type":"mined"}],"logsBloom":"0x00000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000002000200000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000400000010000000000000000000000008000000800000000000000008000000","status":"0x1","to":"0x68e54af74b22acaccffa04ccaad13be16ed14eac","transactionHash":"0x66994c12392d131e2200d02709cfea9461bd0f662e1230e3a2e3d1e27246757d","transactionIndex":"0x1"},"id":1580378144}.
2020-01-30 10:55:44  http.worker30 TRACE rpc  Request: {"jsonrpc":"2.0","method":"trace_transaction","params":["0x66994c12392d131e2200d02709cfea9461bd0f662e1230e3a2e3d1e27246757d"],"id":1580378144}.
2020-01-30 10:55:44  http.worker30 WARN parity  Panic occurred, see stderr for details


====================

stack backtrace:
   0:     0x5627c66df2ed - <no info>
   1:     0x5627c6ee5558 - <no info>
   2:     0x5627c716b90a - <no info>
   3:     0x5627c79a6a53 - <no info>
   4:     0x5627c79a67dd - <no info>
   5:     0x5627c79c1f55 - <no info>
   6:     0x5627c6797d3c - <no info>
   7:     0x5627c679e0f5 - <no info>
   8:     0x5627c6912baa - <no info>
   9:     0x5627c6953298 - <no info>
  10:     0x5627c6fd807a - <no info>
  11:     0x5627c72308d5 - <no info>
  12:     0x5627c6f6a874 - <no info>
  13:     0x5627c715fec0 - <no info>
  14:     0x5627c7145a72 - <no info>
  15:     0x5627c6643d92 - <no info>
  16:     0x5627c7241a6f - <no info>
  17:     0x5627c7243803 - <no info>
  18:     0x5627c7243803 - <no info>
  19:     0x5627c723ff92 - <no info>
  20:     0x5627c72df83f - <no info>
  21:     0x5627c6f5b754 - <no info>
  22:     0x5627c6f73664 - <no info>
  23:     0x5627c79eab38 - <no info>
  24:     0x5627c79e6445 - <no info>
  25:     0x5627c79e480f - <no info>
  26:     0x5627c79d09c6 - <no info>
  27:     0x5627c79c86ea - <no info>
  28:     0x5627c79e28bd - <no info>
  29:     0x5627c79e3327 - <no info>
  30:     0x5627c79c0b1e - <no info>
  31:     0x5627c79c3b4f - <no info>
  32:     0x7f3584a506da - <no info>
  33:     0x7f358415588e - <no info>
  34:                0x0 - <no info>

[dvdplm]

So trace_transaction is also affected, thank you for the report.

[uniibu]

Is there anything we could do to have our wallet running? Obviously, we couldn't downgrade anymore since this update also upgrades the DB and is not backward compatible. We also couldn't simply export our addresses since it literally consists of thousands of addresses. We process several hundreds of ETH transactions every day(0xe9b1a2164368c00fc93e0e749d9b3cafa1bc6ee2) and that has since been on hold for over 18 hours already. Is there anything we could do to maybe downgrade and still keep all the addresses?

[androzd]

Sometimes multiple restarts helps me. But not for a long time (

[dvdplm]

We have identified the problem – that was the hard part – and are working on a fix. With some luck we'll have 2.7.1 out later today. I'm so sorry for the hassle this has caused you all. :/

[s3krit]

@androzd @uniibu Hi, first of all, apologies for this issue. As dvdplm mentioned, we identified the problem and believe we have fixed it. Could you please download and test the latest v2.7.1 (prerelease) build and verify whether this fixes the issue for you?

Linux: https://gitlab.parity.io/parity/parity-ethereum/-/jobs/368828/artifacts/download
Mac: https://gitlab.parity.io/parity/parity-ethereum/-/jobs/368832/artifacts/download
Windows: https://gitlab.parity.io/parity/parity-ethereum/-/jobs/368833/artifacts/download

[androzd]

2020-01-30 18:17:06  Starting Parity-Ethereum/v2.7.1-stable-6885be0-20200130/x86_64-unknown-linux-gnu/rustc1.40.0
2020-01-30 18:17:06  Keys path /home/eth/parity/keys/ethereum
2020-01-30 18:17:06  DB path /home/eth/parity/chains/ethereum/db/906a34e69aec8c0d
2020-01-30 18:17:06  State DB configuration: fast +Fat +Trace
2020-01-30 18:17:06  Operating mode: active
2020-01-30 18:17:06  Warning: Warp Sync is disabled because Fat DB is turned on.
2020-01-30 18:17:28  Configured for Ethereum using Ethash engine
2020-01-30 18:17:28  Running without a persistent transaction queue.
2020-01-30 18:17:29  Updated conversion rate to Ξ1 = US$178.55 (26669866 wei/gas)
2020-01-30 18:17:33  Panic occurred, see stderr for details


====================

stack backtrace:
   0:     0x5652a113151d - <no info>
   1:     0x5652a1934a08 - <no info>
   2:     0x5652a1bb681a - <no info>
   3:     0x5652a23ee533 - <no info>
   4:     0x5652a23ee2bd - <no info>
   5:     0x5652a2409bb5 - <no info>
   6:     0x5652a11e9c3c - <no info>
   7:     0x5652a11efff5 - <no info>
   8:     0x5652a1364b6a - <no info>
   9:     0x5652a13a5298 - <no info>
  10:     0x5652a1a73d1a - <no info>
  11:     0x5652a1c7cc75 - <no info>
  12:     0x5652a19be214 - <no info>
  13:     0x5652a1b97be0 - <no info>
  14:     0x5652a1bad1b2 - <no info>
  15:     0x5652a1c8e274 - <no info>
  16:     0x5652a1c8fe61 - <no info>
  17:     0x5652a1c8fe61 - <no info>
  18:     0x5652a1c8c372 - <no info>
  19:     0x5652a1d3465f - <no info>
  20:     0x5652a19aaff4 - <no info>
  21:     0x5652a19c6484 - <no info>
  22:     0x5652a2432748 - <no info>
  23:     0x5652a242e075 - <no info>
  24:     0x5652a242c43f - <no info>
  25:     0x5652a2418586 - <no info>
  26:     0x5652a241034a - <no info>
  27:     0x5652a242a4ed - <no info>
  28:     0x5652a242af57 - <no info>
  29:     0x5652a240877e - <no info>
  30:     0x5652a240b7af - <no info>
  31:     0x7f91fafac6da - <no info>
  32:     0x7f91fa6b188e - <no info>
  33:                0x0 - <no info>

Thread 'http.worker00' panicked at 'decode db value failed: RlpExpectedToBeData', src/libcore/result.rs:1165

This is a bug. Please report it at:

    https://github.com/paritytech/parity-ethereum/issues/new


2020-01-30 18:17:33  Finishing work, please wait...

[androzd]

@s3krit I got same issue after upgrading to v2.7.1

[dvdplm]

Ouch, that's not good. @androzd can you give us the RPC logs please? :/

[dvdplm]

@androzd I can confirm on my end that even with 2.7.1, querying the segment of the database that was written with 2.7.0 will likely cause a panic. Depending on your situation this might be pretty bad. You might be able to reset the db back to a block before the upgrade to 2.7.0 and the run 2.7.1 from there.
The command is parity db reset NUM (where NUM is a block from before the upgrade). This should work for people running archive nodes, but for other nodes it could be that there is not enough history available to reset. For those cases restoring from backup might be a better option.

:/

[androzd]

@dvdplm panic is not reproduced after 6 restarts

[uniibu]

Using build 2.7.1 seems to work for me on multiple trace_filter and trace_block. No panic has occured

[dvdplm]

@dvdplm panic is not reproduced after 6 restarts

That's good to hear. The root cause here is that code went in to the 2.7.0 release that changed the way some traces were read&written to the database. This meant that data from 2.6 (or earlier DBs) caused panics when read (i.e. when queried through an RPC call).
In version 2.7.1 we have removed the faulty code, but databases that at some point wrote data in the 2.7.0-format is now unreadable to v2.7.1 nodes. In sort of the opposite problem we had before.

All of that to say that the 2.7.1 nodes will work fine just as long as no RPC call comes in with a query that involves data that was written with v2.7.0. Restarting does not fix anything per se, but if a node does not receive RPC calls or otherwise try to read the data from the segment written by 2.7.0, then it will be fine.
It is possible that the RPC calls hitting your node all involve recent data and once the chain. progresses beyond the bad segment, all is good. Another way to fix this is as mentioned above resetting the DB or restoring it from backup to a point before 2.7.0 ran.

Do keep us updated with any new panics you might encounter, and we're especially interested in the RPC logs surrounding such events to be able to confirm that the above reasoning is indeed correct.

[uniibu]

We are currently running the 2.7.1 prerelease on production now, and it seems to be working and syncing fine.

[gituser]

@dvdplm why this release was marked as stable if it wasn't tested at all?
And this v2.7.0 release causes node operators to waste time on re-syncing their nodes, disk, network resourcers.

Just mark appropriately as rc1 release next time before moving into stable as all other cryptocurrencies do.

For those who want to build themselfs v2.7.1 pre-release - checkout stable branch and compile yourself - https://github.com/paritytech/parity-ethereum/tree/stable

[dvdplm]

why this release was marked as stable if it wasn't tested at all?

It was tested plenty, and yet this one slipped through. In hindsight it's pretty obvious we need to take a deep look at the QA procedures in place.

2.7 marks a change in the release strategy where we switch from a stable/beta/nightly to just having stable/nightly which means that the upgrade from 2.5 to 2.7 is a pretty big step. That made the QA effort a bigger than usual task which might be part of the story here.

Very sorry for the hassle this has caused to everyone involved.

[uniibu]

@dvdplm Jusy fyi, our wallet crashed again with the same panic error. Trying to restart it now with rpc logging. Will update when i get the logs.

update:

2020-01-30 20:00:40 UTC IO Worker #3 INFO import  Syncing #9384016 0x3b34…4033     7.80 blk/s  714.2 tx/s   67.2 Mgas/s      0+  478 Qed  #9384504   16/25 peers      3 MiB chain   94 MiB db   42 MiB queue   51 MiB sync  RPC:  0 conn,    0 req/s,  252 µs
2020-01-30 20:00:45 UTC IO Worker #0 INFO import  Syncing #9384059 0xf5fe…7b5d     8.60 blk/s  854.0 tx/s   68.8 Mgas/s      0+  442 Qed  #9384504   16/25 peers      4 MiB chain   96 MiB db   38 MiB queue   68 MiB sync  RPC:  0 conn,    0 req/s,  252 µs
2020-01-30 20:00:50 UTC IO Worker #1 INFO import  Syncing #9384099 0xe5cf…7f90     8.00 blk/s  814.6 tx/s   68.5 Mgas/s      0+  394 Qed  #9384504   16/25 peers      5 MiB chain   97 MiB db   34 MiB queue   68 MiB sync  RPC:  0 conn,    0 req/s,  252 µs
2020-01-30 20:00:55 UTC IO Worker #1 INFO import  Syncing #9384134 0xf893…5fb1     7.00 blk/s  585.4 tx/s   51.7 Mgas/s      0+  358 Qed  #9384504   18/25 peers      5 MiB chain   95 MiB db   32 MiB queue   68 MiB sync  RPC:  0 conn,    0 req/s,  252 µs
2020-01-30 20:01:00 UTC IO Worker #3 INFO import  Syncing #9384168 0x3670…02b6     6.80 blk/s  554.4 tx/s   48.4 Mgas/s      0+ 1420 Qed  #9385590   20/25 peers     15 MiB chain   90 MiB db  114 MiB queue    2 MiB sync  RPC:  0 conn,    0 req/s,  252 µs
2020-01-30 20:01:01 UTC http.worker30 TRACE rpc  Request: {"jsonrpc":"2.0","id":0,"method":"eth_blockNumber","params":[]}.
2020-01-30 20:01:01 UTC http.worker30 DEBUG rpc  Response: {"jsonrpc":"2.0","result":"0x8f30f2","id":0}.
2020-01-30 20:01:01 UTC http.worker30 TRACE rpc  Request: {"jsonrpc":"2.0","id":1,"method":"trace_filter","params":[{"fromBlock":"0x8f1ace","toBlock":"0x8f1aeb"}]}.
2020-01-30 20:01:01 UTC http.worker30 WARN parity  Panic occurred, see stderr for details
====================
stack backtrace:
   0:     0x557a2868351d - <no info>
   1:     0x557a28e86a08 - <no info>
   2:     0x557a2910881a - <no info>
   3:     0x557a29940533 - <no info>
   4:     0x557a299402bd - <no info>
   5:     0x557a2995bbb5 - <no info>
   6:     0x557a2873bc3c - <no info>
   7:     0x557a28741ff5 - <no info>
   8:     0x557a288b6b6a - <no info>
   9:     0x557a2889310e - <no info>
  10:     0x557a288f4f62 - <no info>
  11:     0x557a28fc45a7 - <no info>
  12:     0x557a291cec75 - <no info>
  13:     0x557a28f10214 - <no info>
  14:     0x557a290e9be0 - <no info>
  15:     0x557a290ff1b2 - <no info>
  16:     0x557a291e0274 - <no info>
  17:     0x557a291e1e61 - <no info>
  18:     0x557a291e1e61 - <no info>
  19:     0x557a291de372 - <no info>
  20:     0x557a2928665f - <no info>
  21:     0x557a28efcff4 - <no info>
  22:     0x557a28f18484 - <no info>
  23:     0x557a29984748 - <no info>
  24:     0x557a29980075 - <no info>
  25:     0x557a2997e43f - <no info>
  26:     0x557a2996a586 - <no info>
  27:     0x557a2996234a - <no info>
  28:     0x557a2997c4ed - <no info>
  29:     0x557a2997cf57 - <no info>
  30:     0x557a2995a77e - <no info>
  31:     0x557a2995d7af - <no info>
  32:     0x7f8cbb8b16da - <no info>
  33:     0x7f8cbafb688e - <no info>
  34:                0x0 - <no info>
Thread 'http.worker30' panicked at 'decode db value failed: RlpExpectedToBeData', src/libcore/result.rs:1165
This is a bug. Please report it at:
    https://github.com/paritytech/parity-ethereum/issues/new
2020-01-30 20:01:01 UTC main INFO parity_ethereum::run  Finishing work, please wait..

And this is 2.7.1

2020-01-30 19:51:44 UTC main INFO parity_ethereum::run  Starting Parity-Ethereum/v2.7.1-stable-6885be0-20200130/x86_64-unknown-linux-gnu/rustc1.40.0

[uniibu]

It seems like the panic error still occurs when you do trace_filter for blocks that were not written on 2.7.x-format DB. We manually skipped several hundreds of blocks and started the trace_filter query from the block where the 2.7.1 started to sync again, and it seems stable for now.

[gituser]

@uniibu yes, you can check this workaround - https://github.com/paritytech/parity-ethereum/issues/11422#issuecomment-580382210 you can rollback your DB backwards to the number of blocks specified in --pruning_history parameter otherwise you need to resync or don't query these blocks.

e.g. if you have --pruning_history 1024 you can rollback no more than 1024 blocks back from the current block.
./parity db reset 1024 means you will rollback your DB backwards current_block - 1024.

[uniibu]

@gituser Thanks. we are running in archive so that means we could reset on whatever block right?

[gituser]

@gituser Thanks. we are running in archive so that means we could reset on whatever block right?

Yes, for archive node you keep states for all blocks, so you can reset to any block.

[mpirrongelli]

I have the same issue, however, mine is a little different, when we request the balances using RPC the node stops syncing, I have updated to 2.7.1 but still the issue.

[dvdplm]

I have the same issue,

You mean your tracing node panics? Or something else? If your issue is about something else, please add a separate issue for it (with as much info as possible) or the discussion here risks becoming very confusing. Thanks.