Steem: `steemd` failure can cause `block_log` unusable + endlessly filling index

[Gandalf-the-Grey]

On one of my test nodes, after low mem steemd failed, block_log seems to be corrupted for some reason.
(I suppose it was due to out of disk space on a device)
While trying to start steemd (v0.18.2) again with --replay it goes crazy at:

1679828ms th_a       block_log.cpp:118             open                 ] Log is nonempty
1679828ms th_a       block_log.cpp:149             open                 ] Index is empty
1679828ms th_a       block_log.cpp:256             construct_index      ] Reconstructing Block Log Index...

creating tens of gigabytes of block_log.index until I interrupt it or disk space ends
When used with steemd version v0.18.3 or v0.18.4 there's an assertion there:

1727363ms th_a       block_log.cpp:130             open                 ] Log is nonempty
1727363ms th_a       block_log.cpp:139             open                 ] Index is nonempty
1727363ms th_a       block_log.cpp:155             open                 ] Index is incomplete
1727363ms th_a       block_log.cpp:282             construct_index      ] Reconstructing Block Log Index...
1727383ms th_a       database.cpp:158              reindex              ] Replaying blocks...
1727384ms th_a       database.cpp:2617             _apply_block         ] 10 assert_exception: Assert Exception
head_block_time() < next_block.timestamp: 
    {"head_block_time":"2016-03-24T16:00:00","next":"1970-01-01T00:00:00","blocknum":1}
    th_a  database.cpp:2827 validate_block_header

    {}
    th_a  database.cpp:2845 validate_block_header
1727384ms th_a       database.cpp:2617             _apply_block         ] next_block.block_num(): 1 
1727384ms th_a       application.cpp:387           startup              ] 10 assert_exception: Assert Exception
head_block_time() < next_block.timestamp: 
    {"head_block_time":"2016-03-24T16:00:00","next":"1970-01-01T00:00:00","blocknum":1}
    th_a  database.cpp:2827 validate_block_header

    {}
    th_a  database.cpp:2845 validate_block_header
rethrow
    {"next_block.block_num()":1}
    th_a  database.cpp:2617 _apply_block

    {"next_block":{"previous":"0000000000000000000000000000000000000000","timestamp":"1970-01-01T00:00:00","witness":"","transaction_merkle_
root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000000000000000","transactions":[]}}
    th_a  database.cpp:2503 apply_block

    {"data_dir":"/home/steem/seednode/blockchain","shared_mem_dir":"/run/seednode"}
    th_a  database.cpp:198 reindex
1727384ms th_a       application.cpp:1008          startup              ] 10 assert_exception: Assert Exception
head_block_time() < next_block.timestamp: 
    {"head_block_time":"2016-03-24T16:00:00","next":"1970-01-01T00:00:00","blocknum":1}
    th_a  database.cpp:2827 validate_block_header

    {}
    th_a  database.cpp:2845 validate_block_header
rethrow
    {"next_block.block_num()":1}
    th_a  database.cpp:2617 _apply_block

    {"next_block":{"previous":"0000000000000000000000000000000000000000","timestamp":"1970-01-01T00:00:00","witness":"","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","transactions":[]}}
    th_a  database.cpp:2503 apply_block

    {"data_dir":"/home/steem/seednode/blockchain","shared_mem_dir":"/run/seednode"}
    th_a  database.cpp:198 reindex
rethrow
    {}
    th_a  application.cpp:387 startup
1727384ms th_a       main.cpp:213                  main                 ] Exiting with error:
10 assert_exception: Assert Exception
head_block_time() < next_block.timestamp: 
    {"head_block_time":"2016-03-24T16:00:00","next":"1970-01-01T00:00:00","blocknum":1}
    th_a  database.cpp:2827 validate_block_header

    {}
    th_a  database.cpp:2845 validate_block_header
rethrow
    {"next_block.block_num()":1}
    th_a  database.cpp:2617 _apply_block

    {"next_block":{"previous":"0000000000000000000000000000000000000000","timestamp":"1970-01-01T00:00:00","witness":"","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","transactions":[]}}
    th_a  database.cpp:2503 apply_block

    {"data_dir":"/home/steem/seednode/blockchain","shared_mem_dir":"/run/seednode"}
    th_a  database.cpp:198 reindex
rethrow
    {}
    th_a  application.cpp:387 startup

[mvandeberg]

Running out of disk space could certainly cause that. If a block cannot be written in its entirety, then the back reference after the block would not be written and the last 32 bytes would not be a valid reference. Opening the block log and trying to read it would result in garbage data.

[Gandalf-the-Grey]

FYI, I was able to use current steemd version (i.e. one with assert, so >= v0.18.3)
to bruteforce block_log by using:
truncate --size=-1 blockchain/block_log in a loop cutting garbage at the end until assert is gone.

[mvandeberg]

Duplicate #843 #844 #845