Tendermint: Clearing sole validator causes panic

[cwgoes]

Situation: Tendermint running in-process (in gaia) with a single validator, that validator is revoked for downtime and unbonded.

Expected behavior: Tendermint halts.

Observed behavior: Tendermint crashes.

[05-30|21:55:30.336] Absent validator 0C31B449F7882A79BDF1E46287AA4D338696FEC9 at height 101 module=x/slashing 
I[05-30|21:55:30.336] Validator 0C31B449F7882A79BDF1E46287AA4D338696FEC9 past min height of 100 and below signed blocks threshold of 50 module=x/slashing 
I[05-30|21:55:30.337] Validator 0C31B449F7882A79BDF1E46287AA4D338696FEC9 slashed by fraction 1/100, removed 1/1 shares and burned 1 tokens module=x/stake 
I[05-30|21:55:30.338] Validator 0C31B449F7882A79BDF1E46287AA4D338696FEC9 revoked module=x/stake 
validator updates: [{[22 36 222 98 32 112 157 108 25 118 35 89 239 234 233 129 237 254 171 178 211 180 53 172 41 246 73 56 14 189 199 174 226 79 188 202 93] 0}]
I[05-30|21:55:30.339] Executed block                               module=state height=101 validTxs=0 invalidTxs=0
I[05-30|21:55:30.339] Updates to validators                        module=state updates="[{\"pub_key\":\"FiTeYiBwnWwZdiNZ7+rpge3+q7LTtDWsKfZJOA69x67iT7zKXQ==\",\"power\":0}]"
E[05-30|21:55:30.344] CONSENSUS FAILURE!!!                         module=consensus err="interface conversion: interface {} is nil, not *types.Validator" stack="goroutine 33 [running]:
runtime/debug.Stack(0xc4227747b8, 0xc0bf20, 0xc423012880)
    /home/cwgoes/.gvm/gos/go1.10.1/src/runtime/debug/stack.go:24 +0xa7
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func1(0xc4200e9600)
    /home/cwgoes/.go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:538 +0x57
panic(0xc0bf20, 0xc423012880)
    /home/cwgoes/.gvm/gos/go1.10.1/src/runtime/panic.go:502 +0x229
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/types.(*ValidatorSet).IncrementAccum(0xc423a32ff0, 0x1)
    /home/cwgoes/.go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/types/validator_set.go:61 +0x3a3
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.updateState(0xc42097e9e0, 0x11, 0x64, 0x0, 0xc423c10be0, 0x14, 0x20, 0x1, 0xc42269d900, 0x14, ...)
    /home/cwgoes/.go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:307 +0xcb
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc420088a20, 0xc42097e9e0, 0x11, 0x64, 0x0, 0xc423c10be0, 0x14, 0x20, 0x1, 0xc42269d900, ...)
    /home/cwgoes/.go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:90 +0x240
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc4200e9600, 0x65)
    /home/cwgoes/.go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1267 +0xbb2
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc4200e9600, 0x65)
    /home/cwgoes/.go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1198 +0x468
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit.func1(0xc4200e9600, 0x0, 0x65)
    /home/cwgoes/.go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1146 +0x98
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit(0xc4200e9600, 0x65, 0x0)
    /home/cwgoes/.go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1175 +0x802
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).addVote(0xc4200e9600, 0xc4231805a0, 0x0, 0x0, 0xc4200867d8, 0xc4200867e0, 0xc4229c1b08)
    /home/cwgoes/.go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1532 +0xbb4
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryAddVote(0xc4200e9600, 0xc4231805a0, 0x0, 0x0, 0xc4229c1b58, 0xab2799)
    /home/cwgoes/.go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1390 +0x56
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc4200e9600, 0xbee260, 0xc4221677f0, 0x0, 0x0)
    /home/cwgoes/.go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:607 +0x643
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc4200e9600, 0x0)
    /home/cwgoes/.go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:564 +0x5a5
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart
    /home/cwgoes/.go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:272 +0x140
"
I[05-30|21:55:33.258] Timed out                                    module=consensus dur=3s height=101 round=0 step=RoundStepPropose

[ebuchman]

Tendermint doesn't support not having a validator so this is worthy of a panic.

the validator heap expects to find validators:

https://github.com/tendermint/tendermint/blob/master/types/validator_set.go#L61

i dont think we can legitmately test removing validators using only one validator because you cant remove it

maybe we could be more graceful and not crash but really theres no other choice but to panic anyways …