Cosmos-sdk: Unable to unrevoke a validator

[7768]

I created a validator on this node, and it shows up as Unbonded right away:

~$ gaiacli stake validator $STAKEADDR
Validator
Owner: cosmosaccaddr1srk0wld7d2gvuvgzqfau3hgkuud7265mhupunv
Validator: cosmosvalpub1zcjduc3qlemmacc8250p2mr57e850crfux5cjhupv3dqyxezanjw9p3z628sy8z59c
Shares: Status Unbonded,  Amount: -8925015719954587479/0
Delegator Shares: 6629593233042021117/-8814037051973304320
Description: {7768xB2   }
Bond Height: 620090
...

So I try to unrevoke, and it's giving me a really weird error message That address is not associated with any known validator

~$ gaiacli stake unrevoke   --chain-id=gaia-6002   --name=stakingkey   $STAKEADDR
Defaulting to account number: 308
Defaulting to next sequence number: 1
Password to sign with 'stakingkey':
ERROR: CheckTx failed: (655561) === ABCI Log ===
Codespace: 10
Code:      201
ABCICode:  655561
Error:     --= Error =--
Message: That address is not associated with any known validator
Cause: <nil>
T: 0xc9
Msg Traces:
--= /Error =--

=== /ABCI Log ===

And for sanity check, I tried creating the validator again, and it's giving me this error. Validator already exist, cannot re-create validator

~$ gaiacli stake create-validator   --amount=1steak   --pubkey=$(gaiad tendermint show_validator)   --address-validator=$STAKEADDR   --moniker="7768xB2"   --chain-id=gaia-6002   --name=stakingkey
Defaulting to account number: 308
Defaulting to next sequence number: 1
Password to sign with 'stakingkey':
ERROR: CheckTx failed: (262345) === ABCI Log ===
Codespace: 4
Code:      201
ABCICode:  262345
Error:     --= Error =--
Message: Validator already exist, cannot re-create validator
Cause: <nil>
T: 0xc9
Msg Traces:
--= /Error =--

=== /ABCI Log ===

Trying it with the gas option, same error message.

~$ gaiacli stake unrevoke   --chain-id=gaia-6002   --name=stakingkey   $STAKEADDR  --gas=3000000
Defaulting to account number: 308
Defaulting to next sequence number: 1
Password to sign with 'stakingkey':
ERROR: CheckTx failed: (655561) === ABCI Log ===
Codespace: 10
Code:      201
ABCICode:  655561
Error:     --= Error =--
Message: That address is not associated with any known validator
Cause: <nil>
T: 0xc9
Msg Traces:
--= /Error =--

=== /ABCI Log ===

So it's "That address is not associated with any known validator" versus "Validator already exist, cannot re-create validator". Which one to trust? Also I simply can't unrevoke at all at the moment...

Explorer in the mean time shows the node has 1 voting power. Adding to the confusion.

[7768]

After some more testing, I can reliably reproduce this error now.

With the current state of gaia-6002 (block height 719771) you cannot unrevoke a validator if the validator has less than 3steak. The validator becomes unbonded regardless of how many unrevoke commands you issue.
It works if you have 3 or more steaks! Could it be we have lots of revoked validators and you need >=3 to get to top 100?

[7768]

Correction: it seems you need at least 4 steak now for a chance to unrevoke a validator.
The stakes are getting higher and higher!

[7768]

Okay I'm on to something.
So I bonded a validator with 3 steak at height 720512. It's unbonded right away.
Then I bonded a validator with 4 steak at height 720597. And it worked.

Right before block 720597 I ran this command to check the number of validators:

~$ gaiacli advanced tendermint validator-set | jq ".validators | length"
29

After block 720597, I ran the same command again, and it shows exactly the same number of validators.

~$ gaiacli advanced tendermint validator-set | jq ".validators | length"
29

The number of validators should have gone up by one, but this is not the case at all!

It turned out there's this validator with 3steaks that got unbonded right after when I bonded the 4steaks.

Look for bean-validator BF6C58D882B9D4BE677093D9F6EAE9CEBA1641FE with voting power 3.

Before height 720597, BF6C58D882B9 is voting happily. The last vote this 3steak validator casted was at around block 720598, after that it disappeared from any and all voting, presumably got revoked, though there are no error messages.

[7768]

I think someone should review this method in detail:
https://github.com/cosmos/cosmos-sdk/blob/develop/x/stake/keeper/validator.go#L201

@rigelrozanski @cwgoes

[cwgoes]

Indeed, this is confusing!

Understand that revoked and unbonded are two different states. At any given block, there are any number of validator candidates. The top hundred candidates are bonded - they must sign blocks and have value at stake for protocol violations - and the rest of the candidates are unbonded. When a validator violates protocol - currently by double-signing a block or by missing too many blocks in a row - they are revoked, which prevents them from being bonded. After a jail period, they can unrevoke themselves, which puts them back into consideration for one of the hundred bonded validator slots.

[cwgoes]

Unrevoking a validator isn't possible if that validator has never signed any blocks - this is intentional. Likely contributing to the confusion is that your validator should have been bonded - but wasn't, due to a bug in the cliff validator logic (https://github.com/cosmos/cosmos-sdk/issues/1484#issuecomment-402332659).

Closing in favor of https://github.com/cosmos/cosmos-sdk/issues/1519.

[alexanderbez]

That was a very helpful explanation -- thanks @cwgoes!