Cosmos-sdk: Precommit of a validator with voting power 0 is included in block

[kwunyeung]

When I'm checking the consistency of the voting power of my validator node for #1457 in different ways. I accidentally see this from the validator list of a block height.

http://35.237.213.249:46657/validators?height=622138

{
"address": "747A72E570DD41F95687D3C2DD40128A9DBF798A",
"pub_key": {
"type": "AC26791624DE60",
"value": "SahjvPu+E2ZeIJ+VZOeFaUUWou4Qtdl84lQTFDEBBPE="
},
"voting_power": 0,
"accum": 0
}

The prevote and precommit are always in the /consensus_state. I don't see any other information of that validator but its uptime is mostly 100%.

https://figment.network/cosmos/hubble/gaia-6002/validators/747A72E570DD41F95687D3C2DD40128A9DBF798A

[7768]

1496

[ebuchman]

Wow this should definitely have not happened. Seems like it could be a bug on the Tendermint side!

[ebuchman]

So it looks like if the app returns a validator with 0 power that doesn't already exist in the validator set, Tendermint will add it to the validator set with 0 power: https://github.com/tendermint/tendermint/blob/master/state/execution.go#L285

So sounds like that could be what happened here.

This is something Tendermint should check for (it shouldn't add validators with 0 power) but it also sounds like it might be a bug in Gaia trying to remove an already removed validator ...

[ebuchman]

Did a bit more searching.

The address 747A72E570DD41F95687D3C2DD40128A9DBF798A is a validator in block 571446 but not 571445. So I looked at block 571445: http://gaia-6002.coinculture.net:46657/block?height=571445

It contains a single transaction, which appears to be an unrevoke (required a bit of hacking on gaiadebug to get this information). It's from owner 631E2B3AF1F9FE4873D21B7B74CEF568E834F77D which corresponds to cosmosaccaddr1vv0zkwh3l8lysu7jrdahfnh4dr5rfama8v2xa9 but with the nodes currently frozen I can't do an easy lookup on that to check that it matches the validator 747A72E570DD41F95687D3C2DD40128A9DBF798A

In any case, looks like it might be a bug around unrevoking a validator with no power left.

[7768]

Not sure if it's related to unusual slashing like this one: https://github.com/cosmos/cosmos-sdk/issues/1433 .

I noticed 747A72E570DD41F95687D3C2DD40128A9DBF798A was getting slashed 100x of times, every once in a while. Is that the intended behaviour? I thought getting unrevoked = getting slashed once, or does the slashing algorithm keeps on going until there's nothing left?

[cwgoes]

The validator with Tendermint public key address 747A72E570DD41F95687D3C2DD40128A9DBF798A is owned by 768D3858BA531ACC90EC7E2B89A9CA75FB54729E (cosmosaccaddr1w6xnsk962vdvey8v0c4cn2w2wha4gu575temsy) in block 571446.

There definitely is a bug causing zero-power validators - https://github.com/cosmos/cosmos-sdk/issues/1513 - and at least one related to revocation - https://github.com/cosmos/cosmos-sdk/issues/1510 - but I'm not yet 100% certain either is the cause here.

Debugging via https://github.com/cosmos/cosmos-sdk/tree/cwgoes/debug-gaia-6002:

Validator with public key 747A72E570DD41F95687D3C2DD40128A9DBF798A, owned by 768D3858BA531ACC90EC7E2B89A9CA75FB54729E, height 571445 and 571444:

checking height 571445 true {768D3858BA531ACC90EC7E2B89A9CA75FB54729E PubKeyEd25519{49A863BCFBBE13665E209F9564E785694516A2EE10B5D97CE2541314310104F1} false {0 1487790809578620867442773511875277509751261941203231144487596781320040539686113117714668112521357604310343780546520269265996759493063201653249156305872275864048249727052032378023473949013111550606290672648172701268670765946719190624860625968578105388199289069712410685731033588346958368756103813189620530280493819435100371450595142463300137356784050633635975330054876369516895969667146527223963093819590422254597299594164052646569167964062475648882622055195827077451524152318754671799236676770202415881847519502341280622439778665539737997966806073765587332217080385529141246421360632599490723310166494812051243881595225289610582814699264336738188173756719518663169623413227319790128046904910307500261838865968743753510932331290893150774972402863329294206328278707963191962367552482299473584920752921887103961837072195805459243090932130814484962059934063823317210051183166437465903018913486238185187094679690553579885014594530660902041204193444518515897925357388174426148720633351123457960023136029669327359767239608638332122559531361130984854087494020352573897612790056467367616525542956129623185676049238928361529706753028057885791974922214955420230939282077756380131554355653041512437946189643229289327720075361120108035231540753795651071391637514279191910205472763595796745473615867158734765511801467826480734261963963971797734007520138816433482274452790596606640182365421464694134777127681789898566976009155550362754010304019427164052503837871208080091358790464881483543428735621534612181/495533780137919852499019041205611387826427682951771753357000089067152053547371347214074553478515791510914355855204775166129389549212308162117977364307605852260787936746469114571682199170598337091127631475621958474813894450457327209089507659980946751209594362681936157011903665432816592332431095346076162656806790741794531322573125560080153001351134793862977566649014578932208449212248186969950382414829470429703462529978875588685558388702128231520619759763143754370921855813782951707131865838516759684671012322442083418320420767206844403517131773561814488161547421662904896798641258440373733030748579430400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000} 3/1 {7768xA 795B40B1FF0B3726F9A0E5BC00435E38E698D43E https://cosmos.network Love!} 390028 0  0/1 0/1 0/1 0/1 0/1}
Validator found: owner 768D3858BA531ACC90EC7E2B89A9CA75FB54729E, power 0/1, status 0
checking height 571444 true {768D3858BA531ACC90EC7E2B89A9CA75FB54729E PubKeyEd25519{49A863BCFBBE13665E209F9564E785694516A2EE10B5D97CE2541314310104F1} false {0 950666904201463283374596458177075654812450710028690699109432182287126176704497986758494515742678889311130729653792336029236787155903033053362010936525654382104839779203962134425894109424290735401563568007869109854837602096655312448073299922478949201558983618916821579223900102789316645360608255628786563804313759935500852765108156536360316005086465417687221222984991840328552600392594652225887516607796705005662787587857964193119836623197044024516863062991546363403000598155138417560176629967996037623833549189232709380268527964856895697671835457818310644708602632196920356705654390334961673219261579002727961194849364063942140608285179448558215392731255716346376814090353797249506423045516961005053724572950913310383778539839345113313629110037866919511139974609456302730217180393157935064195706256338878607375111108474703698941216678180283207754547412293112928562904331264805461602525408955897127935305360294518561737830692091885609518303059450773928834704612956117934062863536163438217197722278482854687443552072575557062823117257469291031474569276646039038202491752393880933389531727538592654026257359129840029435811758094273904845313363928184695950021185478748755168572923177039881384756521388005587850388196326335947446786136036241352977778448235847997931151941930196400899858830797451859916194262963875369798819113380099251827143181588398720756022564020812367169170112357426698873409274181354818298274625095364548694063397180023152729952202859373291992153120109355021426904820227584063/316632328111764726709184081208477754240135311844007942674229140510352270206476761146464607739215469050554633252558503988517300193760253590662566882983828707048827580094779034124559256678340686839038566609706903648757515049712988392661937379508812487836141965956170565755540198186606211632739539344215676484280288970300178772640097473138821667705449546437446262329256278727637938191704483253396153299656153675932893795796473820064832644159036266013598287239281035706012276298453576333003425726386370269993555837430017823917303201698894185834484232237823069143489673419821187054572665577372272100049567744000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000} 3/1 {7768xA 795B40B1FF0B3726F9A0E5BC00435E38E698D43E https://cosmos.network Love!} 390028 0  0/1 0/1 0/1 0/1 0/1}
Validator found: owner 768D3858BA531ACC90EC7E2B89A9CA75FB54729E, power 0/1, status 0

Status 0 - unbonded in both blocks - as it should be. Not sure about the pool shares difference.

Validator owned by 631E2B3AF1F9FE4873D21B7B74CEF568E834F77D, who sent the unrevoke transaction, heights 571445 and 571444:

checking height 571445 true {631E2B3AF1F9FE4873D21B7B74CEF568E834F77D PubKeyEd25519{9DC57080F9CCF1C5295C4D1E6FE28799A0D61D6C886C352166D863125E94E110} false {2 5258032615117651899852706158537572432725268565512785385072012377303933170887168104829509036287015888720134760385871360403810075041379293460266594840298367505080371129149278150416919522463110676701193/1051608998529036450467906352727026537688811368602440940477667393827376565185238896160786263977849289303973568473316755440884770003053430626969815545682601041099483260785442003039250226914263040000000} 5627224759730706279878475669845428053719640148564877230502937484198544482798422642840741031173770435412135886426385783006766469841980121287411046635075702234693417053988907933/1125464314441973801906632791225269496270687545419998221004097722441548618384291777837010423149578330535688003196659985350381276390171076814050686073044019059716259840000000000 {ansonhk   } 571445 0  0/1 0/1 0/1 0/1 0/1}
Validator found: owner 631E2B3AF1F9FE4873D21B7B74CEF568E834F77D, pka 5DE0434D4E5ACA408BB0823B7243B79B1B251717, power 5258032615117651899852706158537572432725268565512785385072012377303933170887168104829509036287015888720134760385871360403810075041379293460266594840298367505080371129149278150416919522463110676701193/1051608998529036450467906352727026537688811368602440940477667393827376565185238896160786263977849289303973568473316755440884770003053430626969815545682601041099483260785442003039250226914263040000000, status 2
checking height 571444 true {631E2B3AF1F9FE4873D21B7B74CEF568E834F77D PubKeyEd25519{9DC57080F9CCF1C5295C4D1E6FE28799A0D61D6C886C352166D863125E94E110} true {0 2367143791286637901513533131345309247824285966774679793049279894880084636653955725076121784978966405717330500206771741210229701187781105495242157932970554618041732376472820086649894409250945465678677580591983728181440078844220943172027746748103185201669196816528544722905618832129069483544961494257577287574799381474450512343187228665223390798149163251987543137418783320652520015294996447790235855323091526504612323514259978100686551051243386453591625919150592248358265759749556614132180305886634238373266866381290104057767060334870362569342779384613700332437403253003315586806500244627144172481413510978947232611563775918450907442122844655398002893462576956689617993809711491235556520695741751456896518712940717020806547614577938686452696650404183149874830651802437587651248372821545367133271452703264567459747488524657190778832649091155134049130905209252894544199591695178727007855039233609373513709846492212768478739848059582249056361942808997520217660959167036286055089902809943873109082039299397500908878036062740807635390992608707531963231270663485217115523317200071460473935582357674560897715816186083794661374664300382729308998293346719237959702031353350912802846423441568607290938494971387185400876564328855244517123217511462936460464819197719435857142926272657454625571764853319114769686203054679505399593184693530788793284072900167819297315122971330829829146031201264742488646782807799228263038843277974040462557478896952841468767886498256075140666742726457741824119226578698129/477824531450773363814770117240286164094378764539341363005711519475109119185468189566793763650514723844263419483050578860984587322236547144141005834123109940987952521133687800268018842694513928644641713577578090578636078314094098737747833824316430125139162103503970467127045420491479961326402093985420184637524691412161466022805257126505091749644957095664352871064716277394592214819436830462983399056039036906709308519358213136508181259822633193249437738040375935764891345052370584519500881453621685226401351963911415086459294838018039437084835629792378954913117592924851179027993239878099863446421504000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000} 5627224759730706279878475669845428053719640148564877230502937484198544482798422642840741031173770435412135886426385783006766469841980121287411046635075702234693417053988907933/1125464314441973801906632791225269496270687545419998221004097722441548618384291777837010423149578330535688003196659985350381276390171076814050686073044019059716259840000000000 {ansonhk   } 570872 1  0/1 0/1 0/1 0/1 0/1}
Validator found: owner 631E2B3AF1F9FE4873D21B7B74CEF568E834F77D, pka 5DE0434D4E5ACA408BB0823B7243B79B1B251717, power 0/1, status 0

Unbonded in the earlier block and bonded in the latter, after an unrevoke, as expected.

[cwgoes]

Validator updates for block 571445: http://gaia-6002.coinculture.net:46657/block_results?height=571445 - both validators added.

Some of the confusion here might be caused by validator "toggling" between the SDK and Tendermint. The current Tendermint update logic - https://github.com/tendermint/tendermint/blob/dfa9a9a30a666132425b29454e90a472aa579a48/state/execution.go#L283 - means that passing a validator with zero power to Tendermint will have a different effect based on the current state - if a validator is passed to Tendermint with zero power that wasn't previously in the Tendermint validator set, Tendermint will add it to the validator set with zero power.

Possibly validator 747A72E570DD41F95687D3C2DD40128A9DBF798A was the cliff validator in block 571444 and was bonded then unbonded, which would cause Tendermint to add it with zero power. That's consistent with the store state above.

[cwgoes]

Hypothesis confirmed; loading the cliff validator at blocks 571445 & 571444 (https://github.com/cosmos/cosmos-sdk/tree/81f2606c491a842d6165dfe153b107f9c896b5f8), we find:

got cliff validator at height 571445: v8XºS̐ì~+©ÊuûTr
checking height 571445 true {768D3858BA531ACC90EC7E2B89A9CA75FB54729E PubKeyEd25519{49A863BCFBBE13665E209F9564E785694516A2EE10B5D97CE2541314310104F1} false {0 1487790809578620867442773511875277509751261941203231144487596781320040539686113117714668112521357604310343780546520269265996759493063201653249156305872275864048249727052032378023473949013111550606290672648172701268670765946719190624860625968578105388199289069712410685731033588346958368756103813189620530280493819435100371450595142463300137356784050633635975330054876369516895969667146527223963093819590422254597299594164052646569167964062475648882622055195827077451524152318754671799236676770202415881847519502341280622439778665539737997966806073765587332217080385529141246421360632599490723310166494812051243881595225289610582814699264336738188173756719518663169623413227319790128046904910307500261838865968743753510932331290893150774972402863329294206328278707963191962367552482299473584920752921887103961837072195805459243090932130814484962059934063823317210051183166437465903018913486238185187094679690553579885014594530660902041204193444518515897925357388174426148720633351123457960023136029669327359767239608638332122559531361130984854087494020352573897612790056467367616525542956129623185676049238928361529706753028057885791974922214955420230939282077756380131554355653041512437946189643229289327720075361120108035231540753795651071391637514279191910205472763595796745473615867158734765511801467826480734261963963971797734007520138816433482274452790596606640182365421464694134777127681789898566976009155550362754010304019427164052503837871208080091358790464881483543428735621534612181/495533780137919852499019041205611387826427682951771753357000089067152053547371347214074553478515791510914355855204775166129389549212308162117977364307605852260787936746469114571682199170598337091127631475621958474813894450457327209089507659980946751209594362681936157011903665432816592332431095346076162656806790741794531322573125560080153001351134793862977566649014578932208449212248186969950382414829470429703462529978875588685558388702128231520619759763143754370921855813782951707131865838516759684671012322442083418320420767206844403517131773561814488161547421662904896798641258440373733030748579430400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000} 3/1 {7768xA 795B40B1FF0B3726F9A0E5BC00435E38E698D43E https://cosmos.network Love!} 390028 0  0/1 0/1 0/1 0/1 0/1}
Validator found: owner 768D3858BA531ACC90EC7E2B89A9CA75FB54729E, pka 747A72E570DD41F95687D3C2DD40128A9DBF798A, power 0/1, status 0
got cliff validator at height 571444: v8XºS̐ì~+©ÊuûTr
checking height 571444 true {768D3858BA531ACC90EC7E2B89A9CA75FB54729E PubKeyEd25519{49A863BCFBBE13665E209F9564E785694516A2EE10B5D97CE2541314310104F1} false {0 950666904201463283374596458177075654812450710028690699109432182287126176704497986758494515742678889311130729653792336029236787155903033053362010936525654382104839779203962134425894109424290735401563568007869109854837602096655312448073299922478949201558983618916821579223900102789316645360608255628786563804313759935500852765108156536360316005086465417687221222984991840328552600392594652225887516607796705005662787587857964193119836623197044024516863062991546363403000598155138417560176629967996037623833549189232709380268527964856895697671835457818310644708602632196920356705654390334961673219261579002727961194849364063942140608285179448558215392731255716346376814090353797249506423045516961005053724572950913310383778539839345113313629110037866919511139974609456302730217180393157935064195706256338878607375111108474703698941216678180283207754547412293112928562904331264805461602525408955897127935305360294518561737830692091885609518303059450773928834704612956117934062863536163438217197722278482854687443552072575557062823117257469291031474569276646039038202491752393880933389531727538592654026257359129840029435811758094273904845313363928184695950021185478748755168572923177039881384756521388005587850388196326335947446786136036241352977778448235847997931151941930196400899858830797451859916194262963875369798819113380099251827143181588398720756022564020812367169170112357426698873409274181354818298274625095364548694063397180023152729952202859373291992153120109355021426904820227584063/316632328111764726709184081208477754240135311844007942674229140510352270206476761146464607739215469050554633252558503988517300193760253590662566882983828707048827580094779034124559256678340686839038566609706903648757515049712988392661937379508812487836141965956170565755540198186606211632739539344215676484280288970300178772640097473138821667705449546437446262329256278727637938191704483253396153299656153675932893795796473820064832644159036266013598287239281035706012276298453576333003425726386370269993555837430017823917303201698894185834484232237823069143489673419821187054572665577372272100049567744000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000} 3/1 {7768xA 795B40B1FF0B3726F9A0E5BC00435E38E698D43E https://cosmos.network Love!} 390028 0  0/1 0/1 0/1 0/1 0/1}
Validator found: owner 768D3858BA531ACC90EC7E2B89A9CA75FB54729E, pka 747A72E570DD41F95687D3C2DD40128A9DBF798A, power 0/1, status 0

[7768]

768D3858BA531ACC90EC7E2B89A9CA75FB54729E is 7768xA according to https://github.com/cosmos/cosmos-sdk/issues/1433

This was one of the first validators I setup online with 1steak (I believe?). It got automatically slashed (likely due to the raising minimum steak requirement) - and I was unable to unrevoke it. So I left it running for a while trying out various other tests.

After internalizing the 3/4steak minimum requirement, I destroyed this node along with a few other ones. Consensus failure happened shortly after.

[ebuchman]

This should be fixed on v0.20.0. Thanks!