Eos: Persist Accepted but Reversible Blocks

[bytemaster]

When a node shuts down the most recent 240 blocks (or more) are lost. This normally isn't a problem because the node will sync those blocks from other peers when it reboots; however, there exists one kind of crash where all nodes deterministically crash at the same time due to a bug in block production. This may or may not take down all speculative validators as well.

In this event the producers would be unable to restart the chain without generating "byzantine proofs".

To resolve this, a new shared memory file (chainbase) will be created that will store all applied but reversible blocks.

Shutdown Modes:

1. Graceful - fork database is logged to disk and restored on load
2. Ungraceful - fork database is not logged to disk leaving the forkdb and shared-memory state inconsistent and forcing a "replay". In theory all undo history could be popped until things are consistent.

Startup Modes

1. Replay - fork database is destroyed and block must resync the most recent 240 blocks from network
2. Resync - fork db & shared memory are gone and everything is downloaded fresh
3. Inconsistent - fork db is inconsistent with shared memory file

Getting around bugs

Sometimes a "bad block" introduces a bug and a node would like to startup without that block. This block could exist in either the fork database or the pending block cache. In some extreme cases (with byzantine faults) the block might exist in irreversible block log.

On startup nodes should be able to indicate "which block" they want to start on.

[trigx300]

I just updated to the latest and got this:
3490925ms thread-0 controller.cpp:261 ~controller_impl ] db.revision(): 1 head->block_num: 3 blog.read_head()->block_num(): 2
3490928ms thread-0 main.cpp:113 main ] 10 assert_exception: Assert Exception
end && end->block_num() == head->block_num: fork database exists but reversible block database does not, replay blockchain
{"blog_head":2,"head":3}
thread-0 controller.cpp:239 init

{}
thread-0  chain_plugin.cpp:417 plugin_startup

[forfrt]

@trigx300
Have you fixed it? I got the same error by upgrading and run the first three command of EOS wiki:

sudo ./eosio_build.sh
sudo make install
./nodeos -e -p eosio --plugin eosio::chain_api_plugin --plugin eosio::history_api_plugin

And the message is:

1729673ms thread-0   chain_plugin.cpp:202          plugin_initialize    ] initializing chain plugin
1729695ms thread-0   block_log.cpp:123             open                 ] Log is nonempty
1729696ms thread-0   block_log.cpp:136             open                 ] my->head->block_num(): 1
1729696ms thread-0   block_log.cpp:142             open                 ] Index is nonempty
1729720ms thread-0   http_plugin.cpp:285           plugin_initialize    ] configured http to listen on 127.0.0.1:8888
1729720ms thread-0   net_plugin.cpp:2827           plugin_initialize    ] Initialize net plugin
1729720ms thread-0   net_plugin.cpp:2848           plugin_initialize    ] host: 0.0.0.0 port: 9876
1729720ms thread-0   net_plugin.cpp:2920           plugin_initialize    ] my node_id is 8035193e4a3548ddfa10e6435896809db2c76e204fd1708678998c59cb185767
1729720ms thread-0   main.cpp:104                  main                 ] nodeos version 379cb1a9
1729720ms thread-0   main.cpp:105                  main                 ] eosio root is /home/forfrt/.local/share
1729721ms thread-0   fork_database.cpp:93          close                ] num_blocks_in_fork_db: 1
1729721ms thread-0   controller.cpp:261            ~controller_impl     ] db.revision(): 0 head->block_num: 1 blog.read_head()->block_num(): 1
1729724ms thread-0   main.cpp:113                  main                 ] 10 assert_exception: Assert Exception
db.revision() >= head->block_num: fork database is inconsistent with shared memory
    {"db":0,"head":1}
    thread-0  controller.cpp:243 init

    {}
    thread-0  chain_plugin.cpp:428 plugin_startup

I also tried to boot without eosio::chain_api_plugin and it gives this:

./nodeos -e -p eosio --plugin eosio::history_api_plugin

1747693ms thread-0   chain_plugin.cpp:202          plugin_initialize    ] initializing chain plugin
1747718ms thread-0   block_log.cpp:123             open                 ] Log is nonempty
1747718ms thread-0   block_log.cpp:136             open                 ] my->head->block_num(): 1
1747718ms thread-0   block_log.cpp:142             open                 ] Index is nonempty
1747742ms thread-0   http_plugin.cpp:285           plugin_initialize    ] configured http to listen on 127.0.0.1:8888
1747742ms thread-0   net_plugin.cpp:2827           plugin_initialize    ] Initialize net plugin
1747742ms thread-0   net_plugin.cpp:2848           plugin_initialize    ] host: 0.0.0.0 port: 9876
1747742ms thread-0   net_plugin.cpp:2920           plugin_initialize    ] my node_id is 1a21bccfa7d98f52deb91279243c4a899a1ae944bcef8a7069c5954bb56dc82d
1747744ms thread-0   main.cpp:104                  main                 ] nodeos version 379cb1a9
1747744ms thread-0   main.cpp:105                  main                 ] eosio root is /home/forfrt/.local/share
1747746ms thread-0   fork_database.cpp:93          close                ] num_blocks_in_fork_db: 1
1747746ms thread-0   controller.cpp:261            ~controller_impl     ] db.revision(): 0 head->block_num: 1 blog.read_head()->block_num(): 1
1747752ms thread-0   main.cpp:113                  main                 ] 10 assert_exception: Assert Exception
db.revision() >= head->block_num: fork database is inconsistent with shared memory
    {"db":0,"head":1}
    thread-0  controller.cpp:243 init

    {}
    thread-0  chain_plugin.cpp:428 plugin_startup

I have excluded all environmental problem by reinstalling the image.

Any advices?

[DustOneHub]

@forfrt
try nodeos --replay-blockchain

[venediktov]

on Mac you can clean up this path
rm -rf /Users/${your-username}/Library/Application\ Support/eosio/nodeos/data/*

To find out where your data is stored look at the the log of nodeos

1080508ms thread-0   main.cpp:105                  main                 ] eosio root is /Users/your-name/Library/Application Support

[Drug0j]

hi! 2018-10-11T09:22:45.332 thread-0 main.cpp:125 main ] 3020000 fork_database_exception: Fork database exception
reversible block database is inconsistent with fork database, replay blockchain
{"head":574302,"unconfimed":358546}
thread-0 controller.cpp:303 init

{}
thread-0  chain_plugin.cpp:655 plugin_startup

[Drug0j]

{}
thread-0  chain_plugin.cpp:655 plugin_startup

any ideas

[Drug0j]

okey!!!! ./nodeos -e -p eosio --hard-replay-blockchain --delete-all-blocks --plugin eosio::chain_api_plugin --plugin eosio::history_api_plugin

[Drug0j]

lol))