Parity-ethereum: Stage 3 block verification failed (with --light option)

[petko10]

I'm running:
Operating system: Manjaro (Linux, Arch based)
Parity version: 1.7.0-1 , installed from the repos

I ran parity with the --light option. Several times I had the sync process stop advancing (while still giving updates in the cli, just with hdr/s = 0 and the block being the same). I restarted parity whenever that happened (ctrl-c, and re-run, like 3-4 times).
Now I got the error:

Stage 3 block verification failed for #3100000 (55a0…4876)
Error: Block(InvalidDifficulty(Mismatch { expected: 13828535640678, found: 13828267205222 }))

and I'm stuck with it. I guess I could try to clear the database/settings and re-run parity, but I thought there might be some additional debugging info that could be extracted from the situation. It's a pretty clean install, I only added the following as a config file:

[parity]
db_path = "/all/parity_chains"

[footprint]
db_compaction = "hdd"

[5chdn]

the 0 hdr/s issue is logged here #6319 and the block verification issue might be related to this issue. you could try resetting the database and sync again.

note that light client support is still highly experimental and you should not use it in production.

[carver]

I just wanted to note that I had exactly the same error on the same block, with the light client:

2017-09-27 14:27:53 Starting Parity/v1.7.2-beta-9f47909-20170918/x86_64-linux-gnu/rustc1.19.0
...
2017-09-27 14:27:53 Running in experimental Light Client mode.
...
2017-09-27 14:28:20 Stage 3 block verification failed for #3100000 (55a0…4876)
Error: Block(InvalidDifficulty(Mismatch { expected: 13828535640678, found: 13828267205222 }))

I'm using --db-compaction ssd, so that seems unrelated.

[HaleTom]

Me also, with exactly the same numbers. I don't have any compaction setting.

2017-09-28 15:52:25  Starting Parity/v1.7.2-beta-9f47909ed-20170918/x86_64-linux-gnu/rustc1.20.0
2017-09-28 15:52:25  Keys path /home/ravi/.local/share/io.parity.ethereum/keys/Foundation
2017-09-28 15:52:25  DB path /home/ravi/.local/share/io.parity.ethereum/chains/ethereum/db/906a34e69aec8c0d
2017-09-28 15:52:25  Path to dapps /home/ravi/.local/share/io.parity.ethereum/dapps
2017-09-28 15:52:25  Running in experimental Light Client mode.
2017-09-28 15:52:25  NAT mapped to external address 192.168.1.7:54318
2017-09-28 15:52:25  Public node URL: enode://47dc8a983dabc72badc48595fd69b64f22ca7c3d737ac4fb210e37335c1d3da568ea2b9282d3b39138cdc5e1fb558114e534e307fb446e2da47a3b4b82ece5b5@192.168.1.7:54318+50570
2017-09-28 15:52:29  Stage 3 block verification failed for #3100000 (55a0…4876)
Error: Block(InvalidDifficulty(Mismatch { expected: 13828535640678, found: 13828267205222 }))
2017-09-28 15:52:30  Syncing #3099999 3608…58d7     0 hdr/s   5206+    0 Qed  #3099999   16/50 peers   664 bytes cache 4 MiB queue  RPC:  0 conn,  0 req/s,   0 µs
...

Given 3 people with exactly matching numbers, would you consider reopening, @5chdn ?

Also regarding your work-around try resetting the database and sync again, I've searched the FAQ and can't find anything on that.

Do you mean rm -r ~/.local/share/io.parity.ethereum/chains/ethereum/db?

[5chdn]

@rphmeier ^

Can not reproduce this with latest master 1.8.x

[HaleTom]

Any tips on my question about the database reset would be appreciated. Doing my rm caused the block number count to start at 0 again.

[5chdn]

Yes resetting the database means you start at 0, a good way is to run parity --light db kill. But I think this is a confirmed issue and wont resolve this.

[timsomers]

I can confirm I've just had the same issue twice in a row, resetting the db in between. Twice on the same block, there must be something in it that light mode doesn't like.

2017-10-07 10:35:43  Syncing #3099441 271a…581d   955 hdr/s  25338+    0 Qed  #3099441   25/50 peers   10 MiB cache 21 MiB queue  RPC:  0 conn,  0 req/s,   0 µs
2017-10-07 10:35:44  Stage 3 block verification failed for #3100000 (55a0…4876)
Error: Block(InvalidDifficulty(Mismatch { expected: 13828535640678, found: 13828267205222 }))
2017-10-07 10:35:48  Syncing #3099999 3608…58d7   111 hdr/s  28522+    0 Qed  #3099999   25/50 peers   10 MiB cache 24 MiB queue  RPC:  0 conn,  0 req/s,   0 µs

I'm running parity 1.7.2 downloaded from https://github.com/paritytech/parity/releases, running on debian testing.

[mrbrdo]

I have the same issue on parity 1.7.3 on Mac OS X High Sierra.

2017-10-11 04:57:48  Syncing #3099722 e2ef�1c6c   737 hdr/s  16607+    0 Qed  #3099722    7/150 peers   10 MiB cache 14 MiB queue  RPC:  0 conn,  0 req/s,   0 µs
2017-10-11 04:57:48  Stage 3 block verification failed for #3100000 (55a0�4876)
Error: Block(InvalidDifficulty(Mismatch { expected: 13828535640678, found: 13828267205222 }))
2017-10-11 04:57:53  Syncing #3099999 3608�58d7    55 hdr/s  29999+    0 Qed  #3099999    9/150 peers   10 MiB cache 25 MiB queue  RPC:  0 conn,  0 req/s,   0 µs

Did db kill resolve the issue for anyone?

[MatejLach]

Can confirm this issue persists even on 1.8.0-nightly, it always gets stuck somewhere, but not always on the exact same block for me, db kill doesn't help.

[jchaager]

This issue is also present for me in the 1.8.1-beta downloaded from the parity release page.

[deonis78]

I am having the "Stage 3 block verification failed" error and synchronization stuck at block # 4370000 without --light option. In fact I ran parity without any arguments.

OS: Windows 10 x64. Parity version: Parity/v1.7.0-beta-5f2cabd-20170727/x86_64-windows-msvc/rustc1.18.0

2017-11-03 16:13:46  Syncing #4370000 57bf…c979     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #4369999    7/25 peers      2 MiB chain   61 MiB db  0 bytes queue   10 KiB sync  RPC:  2 conn,  1 req/s, 105 µs
2017-11-03 16:13:51  Syncing #4370000 57bf…c979     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #4369999    7/25 peers      2 MiB chain   61 MiB db  0 bytes queue   10 KiB sync  RPC:  2 conn,  2 req/s,  77 µs
2017-11-03 16:13:56  Syncing #4370000 57bf…c979     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #4369999    7/25 peers      2 MiB chain   61 MiB db  0 bytes queue   10 KiB sync  RPC:  2 conn,  2 req/s,  92 µs
2017-11-03 16:14:06  Syncing #4370000 57bf…c979     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #4369999    7/25 peers      2 MiB chain   61 MiB db  0 bytes queue   10 KiB sync  RPC:  2 conn,  1 req/s, 101 µs
2017-11-03 16:14:14  Stage 3 block verification failed for #4370000 (b1fc…785e)
Error: Block(InvalidDifficulty(Mismatch { expected: 2998009606615050, found: 2994347070619309 }))
2017-11-03 16:14:16  Syncing #4370000 57bf…c979     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #4370062    7/25 peers      2 MiB chain   61 MiB db  0 bytes queue  131 KiB sync  RPC:  2 conn,  2 req/s, 101 µs
2017-11-03 16:14:51     6/25 peers      2 MiB chain   61 MiB db  0 bytes queue   19 KiB sync  RPC:  2 conn,  1 req/s, 528 µs
2017-11-03 16:15:01  Syncing #4370000 57bf…c979     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #4369985    8/25 peers      2 MiB chain   61 MiB db  0 bytes queue   19 KiB sync  RPC:  2 conn,  1 req/s, 1093 µs
2017-11-03 16:15:26  Syncing #4370000 57bf…c979     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #4369984    8/25 peers      2 MiB chain   61 MiB db  0 bytes queue   19 KiB sync  RPC:  2 conn,  1 req/s, 444 µs
2017-11-03 16:16:01     7/25 peers      2 MiB chain   61 MiB db  0 bytes queue   19 KiB sync  RPC:  2 conn,  4 req/s,  48 µs
2017-11-03 16:16:36     7/25 peers      2 MiB chain   61 MiB db  0 bytes queue   19 KiB sync  RPC:  2 conn,  3 req/s, 679 µs
2017-11-03 16:17:11     6/25 peers      2 MiB chain   61 MiB db  0 bytes queue   19 KiB sync  RPC:  2 conn,  2 req/s, 230 µs
2017-11-03 16:17:46     6/25 peers      2 MiB chain   61 MiB db  0 bytes queue   19 KiB sync  RPC:  2 conn,  2 req/s, 135 µs
2017-11-03 16:18:21     7/25 peers      2 MiB chain   61 MiB db  0 bytes queue   19 KiB sync  RPC:  2 conn,  1 req/s, 157 µs
2017-11-03 16:18:56     7/25 peers      2 MiB chain   61 MiB db  0 bytes queue   19 KiB sync  RPC:  2 conn,  3 req/s,  85 µs
2017-11-03 16:19:31     8/25 peers      3 MiB chain   61 MiB db  0 bytes queue   19 KiB sync  RPC:  2 conn,  1 req/s, 162 µs
2017-11-03 16:19:41  Syncing #4370000 57bf…c979     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #4369953    9/25 peers      3 MiB chain   61 MiB db  0 bytes queue   56 KiB sync  RPC:  2 conn,  1 req/s, 603 µs
2017-11-03 16:19:51  Syncing #4370000 57bf…c979     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #4369951    9/25 peers      3 MiB chain   61 MiB db  0 bytes queue   33 KiB sync  RPC:  2 conn,  1 req/s,  93 µs
2017-11-03 16:20:26  Syncing #4370000 57bf…c979     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #4369947    9/25 peers      3 MiB chain   61 MiB db  0 bytes queue   33 KiB sync  RPC:  2 conn,  1 req/s,  77 µs

[5chdn]

@deonis78 there was a hard-fork on block 43700000 - please upgrade to 1.8.2 :)

[deonis78]

@5chdn Thank you very much for the explanation! Upgrading to 1.8.2 has helped.

[J-A-M-E-5]

Can confirm just got this with latest version of Parity.

Running full parity node with archiving and fat-db, having to sync on a SSD RAID0 array to try and make progress.

Starting Parity/v1.10.0-nightly-c060d95-20180209/x86_64-linux-gnu/rustc1.23.0

2018-02-12 13:50:41  Stage 3 block verification failed for #4370000 (57bf…c979)
Error: Block(InvalidDifficulty(Mismatch { expected: 2994347070619309, found: 2996546093872813 }))

However it still keeps syncing blocks so hasn't frozen at this block. So is this a warning or a error?

[5chdn]

If it keeps syncing it just received a block from another chain and discarded it, syncing from other light peers. In that cause it should be considered a warning not an error.

It's notable that this appears only on even blocks, e.g., 3100000, 4370000. So closing this as non-issue.

[5chdn]

Still happens with 1.11.6

[Tbaut]

closing as hardcoded headers should help. Please reopen with logs if it still happens.