Go-ethereum: In private PoA (clique) network with 2 instances of 1 sealer, mining produces blocks of equal hash, ERROR messages "Impossible reorg".

Created on 2 May 2018 · 7Comments · Source: ethereum/go-ethereum

In private PoA (clique) network with 2 instances of 1 sealer, mining produces blocks of equal hash, ERROR messages " Impossible reorg".

"2 instances of 1 sealer" means chain was created with 1 sealer. For better network stability one more instance of the sealer is run.

System information

Geth version: geth version

E:\Geth>geth version
Geth
Version: 1.8.2-stable
Git Commit: b8b9f7f4476a30a0aaf6077daade6ae77f969960
Architecture: amd64

OS & Version: Windows/Linux/OSX
Windows 7

Expected behaviour

Different block hashes on different nodes.
Works without warnings.

Actual behaviour

Equal hashes mined on different nodes.
Many messages like

Impossible reorg, please file an issue oldnum=78 oldhash=d7de80…a2c90c newnum=78 newhash=d7de80…a2c90c
but not after each import (node synch'ing)

Steps to reproduce the behaviour

generate new PoA (clique) network with puppeth
init node1
run node1
init node2
run node2
./geth.exe --datadir .bc5201node2/ --networkid 5201 --syncmode "full" --port 35202 --nodiscover --nousb --rpc --rpcaddr "localhost" --rpcport 8502 --rpcapi "admin,personal,db,eth,net,web3,txpool,miner" --mine --gasprice 0 --etherbase 0x31a5e0e68b4ca721a2005144f6c32f6d1c1c4b99 --unlock 0x31a5e0e68b4ca721a2005144f6c32f6d1c1c4b99 --password password31.txt
add static-nodes.json in node2 poiting to node1 (so node2 always connects to node1)

Backtrace

INFO [05-02|10:13:08] 🔗 block reached canonical chain INFO [05-02|10:13:08] 🔨 mined potential block INFO [05-02|10:13:08] Commit new mining work INFO [05-02|10:14:08] Successfully sealed new block INFO [05-02|10:14:08] Imported new chain segment ERROR[05-02|10:14:08] Impossible reorg, please file an issue INFO [05-02|10:14:08] Commit new mining work INFO [05-02|10:14:08] 🔗 block reached canonical chain INFO [05-02|10:14:08] 🔨 mined potential block INFO [05-02|10:14:08] Commit new mining work INFO [05-02|10:15:08] Successfully sealed new block INFO [05-02|10:15:08] Imported new chain segment INFO [05-02|10:15:08] 🔗 block reached canonical chain INFO [05-02|10:15:08] Commit new mining work INFO [05-02|10:15:08] 🔨 mined potential block INFO [05-02|10:16:08] Successfully sealed new block ERROR[05-02|10:16:08] Impossible reorg, please file an issue INFO [05-02|10:16:08] Imported new chain segment INFO [05-02|10:16:08] 🔗 block reached canonical chain INFO [05-02|10:16:08] Commit new mining work INFO [05-02|10:16:08] 🔨 mined potential block INFO [05-02|10:16:08] Commit new mining work INFO [05-02|10:17:08] Successfully sealed new block INFO [05-02|10:17:08] Imported new chain segment ERROR[05-02|10:17:08] Impossible reorg, please file an issue INFO [05-02|10:17:08] Commit new mining work number=63 hash=8f2139…cacc39 number=68 hash=7672e6…b93d02 number=69 txs=0 uncles=0 elapsed=0s number=69 hash=663e7e…738881 blocks=1 txs=0 mgas=0.000 elapsed=1ms mgasps=0.000 number=69 hash=663e7e…738881 cache=0.00B oldnum=69 oldhash=663e7e…738881 newnum=69 newhash=663e7e…738881 number=70 txs=0 uncles=0 elapsed=0s number=64 hash=6e83a4…661dce number=69 hash=663e7e…738881 number=70 txs=0 uncles=0 elapsed=0s number=70 hash=dde192…9a04cc blocks=1 txs=0 mgas=0.000 elapsed=0s mgasps=NaN number=70 hash=dde192…9a04cc cache=0.00B number=65 hash=630c89…40e2a8 number=71 txs=0 uncles=0 elapsed=0s number=70 hash=dde192…9a04cc number=71 hash=3402a4…791032 oldnum=71 oldhash=3402a4…791032 newnum=71 newhash=3402a4…791032 blocks=1 txs=0 mgas=0.000 elapsed=0s mgasps=NaN number=71 hash=3402a4…791032 cache=0.00B number=66 hash=d1f60c…949513 number=72 txs=0 uncles=0 elapsed=0s number=71 hash=3402a4…791032 number=72 txs=0 uncles=0 elapsed=0s number=72 hash=94b0d5…17af79 blocks=1 txs=0 mgas=0.000 elapsed=1ms mgasps=0.000 number=72 hash=94b0d5…17af79 cache=0.00B oldnum=72 oldhash=94b0d5…17af79 newnum=72 newhash=94b0d5…17af79 number=73 txs=0 uncles=0 elapsed=0s

on node2 no importing (that is a bit strange)

INFO [05-02|10:12:08] Commit new mining work                   number=68 txs=0 uncles=0 elapsed=7ms
INFO [05-02|10:13:08] Successfully sealed new block            number=68 hash=7672e6…b93d02
INFO [05-02|10:13:08] 🔗 block reached canonical chain          number=63 hash=8f2139…cacc39
INFO [05-02|10:13:08] 🔨 mined potential block                  number=68 hash=7672e6…b93d02
INFO [05-02|10:13:08] Commit new mining work                   number=69 txs=0 uncles=0 elapsed=1ms
INFO [05-02|10:14:08] Successfully sealed new block            number=69 hash=663e7e…738881
INFO [05-02|10:14:08] 🔗 block reached canonical chain          number=64 hash=6e83a4…661dce
INFO [05-02|10:14:08] 🔨 mined potential block                  number=69 hash=663e7e…738881
INFO [05-02|10:14:08] Commit new mining work                   number=70 txs=0 uncles=0 elapsed=2ms
INFO [05-02|10:15:08] Successfully sealed new block            number=70 hash=dde192…9a04cc
INFO [05-02|10:15:08] 🔗 block reached canonical chain          number=65 hash=630c89…40e2a8
INFO [05-02|10:15:08] 🔨 mined potential block                  number=70 hash=dde192…9a04cc
INFO [05-02|10:15:08] Commit new mining work                   number=71 txs=0 uncles=0 elapsed=1ms
INFO [05-02|10:16:08] Successfully sealed new block            number=71 hash=3402a4…791032
INFO [05-02|10:16:08] 🔗 block reached canonical chain          number=66 hash=d1f60c…949513
INFO [05-02|10:16:08] 🔨 mined potential block                  number=71 hash=3402a4…791032
INFO [05-02|10:16:08] Commit new mining work                   number=72 txs=0 uncles=0 elapsed=1ms
INFO [05-02|10:17:08] Successfully sealed new block            number=72 hash=94b0d5…17af79
INFO [05-02|10:17:08] 🔗 block reached canonical chain          number=67 hash=3099d6…3c06be
INFO [05-02|10:17:08] 🔨 mined potential block                  number=72 hash=94b0d5…17af79
INFO [05-02|10:17:08] Commit new mining work                   number=73 txs=0 uncles=0 elapsed=1ms
INFO [05-02|10:18:08] Successfully sealed new block            number=73 hash=f17e04…9b58b6
INFO [05-02|10:18:08] 🔗 block reached canonical chain          number=68 hash=7672e6…b93d02
INFO [05-02|10:18:08] 🔨 mined potential block                  number=73 hash=f17e04…9b58b6
INFO [05-02|10:18:08] Commit new mining work                   number=74 txs=0 uncles=0 elapsed=1ms
INFO [05-02|10:19:08] Successfully sealed new block            number=74 hash=e87947…4222da

The code in question is
https://github.com/ethereum/go-ethereum/blob/master/core/blockchain.go#L1322

// Ensure the user sees large reorgs
    if len(oldChain) > 0 && len(newChain) > 0 {
        logFn := log.Debug
        if len(oldChain) > 63 {
            logFn = log.Warn
        }
        logFn("Chain split detected", "number", commonBlock.Number(), "hash", commonBlock.Hash(),
            "drop", len(oldChain), "dropfrom", oldChain[0].Hash(), "add", len(newChain), "addfrom", newChain[0].Hash())
    } else {
        log.Error("Impossible reorg, please file an issue", "oldnum", oldBlock.Number(), "oldhash", oldBlock.Hash(), "newnum", newBlock.Number(), "newhash", newBlock.Hash())
}

So when blocks in 2 nodes are generated with identical hash, then synch'ing results in that block deltas name here oldChain and newChain are actually both of length == 0, so logs error

Impossible reorg, please file an issue

Either clique mining should not produces blocks with equal hashes (e.g. adding timestamp to hash function),
or this situation should be considered OK, and error message should not be displayed (e.g. code logic like if engine==clique then). But I think it is proper not to have equal hashes generated.

Please insure that clique mining (sealing) does not produce equal hashes.

triage

Source

paulvi

Most helpful comment

Looking forward to the fix. This is becoming a persistent issue in our POA network also.

brianmcmichael on 1 Apr 2019

👍4

All 7 comments

Please insure that clique mining (sealing) does not produce equal hashes.

The entire block's content is contained within the hash, and secp256k1 signatures are deterministic. I.e. for the same input (timestamp included), the same output will be generated.

I can try to take a look at how to handle duplicate blocks more gracefully. I guess we never really expected a freshly mined block to already exist in the chain.

karalabe on 2 May 2018

👍1

OK, sounds good.

Because in PoW setup every new block come in exact the same interval (e.g. every 60 seconds), so it is likely the case that timestamp (in seconds) is exactly the same, thus producing the same block hashes.

Is it by design, that if any node produces absolutely the same block?

OK, I tried setting up --identity option, the result hash was the same.
Then using --extradata, e.g. --extradata node2 resulted in different hash produced.

Yes, the should be better log message for a case when coming block is exactly the same (and with the same hash).

BTW, can you point me (within code) what exact parameter are passed to hashing algorithm?

paulvi on 2 May 2018

The header's content is the one being hashed. The node identity is just a networking layer info, the extra-data is embedded into the header, so that results in a modified hash.

karalabe on 2 May 2018

I can try to take a look at how to handle duplicate blocks more gracefully. I guess we never really expected a freshly mined block to already exist in the chain.

I'll take a shot at this.