Solana: Rebuilding state on restart

[sagar-solana]

Problem

Replaying the ledger doesn't guarantee consistent voting behavior. i.e since votes are not persisted a node that is restarted might vote differently and end up getting slashed and building its state differently. Leaving it in a different state from the rest of the cluster.

Proposed Solution

Persist votes so that vote behavior can be controlled to rebuild state on restart.

Tag: @carllin @aeyakovenko

[sagar-solana]

@carllin sound ok ?

[aeyakovenko]

The voting proxy should handle this. Any state saved in the fullnode address space is insecure.

[sagar-solana]

Agreed.

[carllin]

@CriesofCarrots, fyi this is the issue we were tracking

One additional change I think we will also need in that change is that PoH should start from the last fork you voted on, rather than an arbitrary fork as it currently does here:

https://github.com/carllin/solana/blob/FixLeaderSchedule/core/src/validator.rs#L125, where the bank is set: https://github.com/carllin/solana/blob/FixLeaderSchedule/core/src/validator.rs#L114

[carllin]

@CriesofCarrots Hehe tacking on here. Another small issue, although I don't think it's too severe. It's possible that a node votes which creates a new root, but before replay stage can mark that slot as a root in blocktree here: https://github.com/solana-labs/solana/blob/master/core/src/replay_stage.rs#L426, the node is shut down.

Now on restart, that means blocktree_processor won't mark this slot as root when processing the ledger here: https://github.com/carllin/solana/blob/FixLeaderSchedule/core/src/blocktree_processor.rs#L228.

Once locktower is serialized, we will always track the latest root, so on restart blocktree_processor, we just need to always call blocktree.set_root() on this latest root just in case it wasn't recorded on node shutdown.

[carllin]

Also should change the working bank here to use the fork we last voted on: https://github.com/carllin/solana/blob/FixLeaderSchedule/core/src/bank_forks.rs#L131

[mvines]

We need to pull this into the v0.22.x line

[mvines]

https://github.com/solana-labs/solana/pull/9902 mostly fixes this issue.

What's remaining is correctly (?) setting the working bank on a restart. Currently the working bank is set to the highest bank, instead of following the the fork that the node last voted on.