Cosmos-sdk: genesis.json#validators cannot upgrade to v0.37.0

[michaelfig]

Summary of Bug

I'm trying to upgrade from v0.35.0 where I had the validators specified in the genesis.json as:

[...]
  "validators": [
    {
      "address": "8E271B09235CDE62BB0F05F9FD3BA097C6046BC6",
      "pub_key": {
        "type": "tendermint/PubKeyEd25519",
        "value": "BofC9dDJ+EoLDo1yC1qjo19opdQbDCeJywQ45AlqQMk="
      },
      "power": "10",
      "name": ""
    },
    {
      "address": "A5FC2FF9F4C6D1CA5767FE31ABED8F3EBA94600E",
      "pub_key": {
        "type": "tendermint/PubKeyEd25519",
        "value": "clrBTJtC3QjrotSn3/JEG7RX3KSRrhQyLj+QQL18jk8="
      },
      "power": "10",
      "name": ""
    }
}

I cannot yet use gentx, as the lotion-connect library does not support it (it requires the validators key in genesis.json).

When I did the following in app.go:InitChainer:

func (app *swingSetApp) InitChainer(ctx sdk.Context, req abci.RequestInitChain) abci.ResponseInitChain {
    var genesisState GenesisState

    err := app.cdc.UnmarshalJSON(req.AppStateBytes, &genesisState)
    if err != nil {
        panic(err)
    }

    res := app.mm.InitGenesis(ctx, genesisState)
    if len(res.Validators) == 0 {
        // Use the genesis.json validators
        res.Validators = req.Validators
    }
    return res
}

With a similar genesis.json, then I get the following:

Sep 10 21:40:06 451d95b1e0fc ag-chain-cosmos[18441]: I[2019-09-10|21:40:06.946] Committed state                              module=state height=1 txs=0 appHash=D248BCA6663D2EAD192F72FADE45985CEE5311C382261CC6D127AB79EC6B2ADD
Sep 10 21:40:12 451d95b1e0fc ag-chain-cosmos[18441]: E[2019-09-10|21:40:12.151] WARNING: Attempt to allocate proposer rewards to unknown proposer cosmosvalcons150gckg0tu4xxzgrfx7428v950hgc6ul4qf7x6d. This should happen only if the proposer unbonded completely within a single block, which generally should not happen except in exceptional circumstances (or fuzz testing). We recommend you investigate immediately. module=x/distribution
Sep 10 21:40:12 451d95b1e0fc ag-chain-cosmos[18441]: E[2019-09-10|21:40:12.152] CONSENSUS FAILURE!!!                         module=consensus err="runtime error: invalid memory address or nil pointer dereference" stack="goroutine 133 [running]:\nruntime/debug.Stack(0xc0022d60a8, 0x7fe7df5a8300, 0x7fe7dff817f0)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9f\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func2(0xc00113d880, 0x7fe7df6f2498)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:602 +0x59\npanic(0x7fe7df5a8300, 0x7fe7dff817f0)\n\t/usr/local/go/src/runtime/panic.go:522 +0x1b9\ngithub.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.AllocateTokensToValidator(0x7fe7df721820, 0xc000cfcce0, 0xc0000a24d0, 0xc0000a24d0, 0x7fe7df721820, 0xc000cfcd00, 0x7fe7df721860, 0xc000cfcd20, 0xc000ce3020, 0xc, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/distribution/keeper/allocation.go:105 +0x3c\ngithub.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.AllocateTokens(0x7fe7df721820, 0xc000cfcce0, 0xc0000a24d0, 0xc0000a24d0, 0x7fe7df721820, 0xc000cfcd00, 0x7fe7df721860, 0xc000cfcd20, 0xc000ce3020, 0xc, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/distribution/keeper/allocation.go:93 +0x115d\ngithub.com/cosmos/cosmos-sdk/x/distribution.BeginBlocker(0x7fe7df7301e0, 0xc0000b4040, 0x7fe7df73e040, 0xc0022c4240, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/distribution/abci.go:26 +0x2eb\ngithub.com/cosmos/cosmos-sdk/x/distribution.AppModule.BeginBlock(...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/distribution/module.go:130\ngithub.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(0xc0000a3730, 0x7fe7df7301e0, 0xc0000b4040, 0x7fe7df73e040, 0xc0022c4240, 0xa, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/types/module/module.go:290 +0x1cf\ngithub.com/Agoric/cosmic-swingset.(*swingSetApp).BeginBlocker(...)\n\t/usr/src/app/app.go:276\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(0xc000cae1
Sep 10 21:40:12 451d95b1e0fc ag-chain-cosmos[18441]: 20, 0xc0022c65c0, 0x20, 0x20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/baseapp.go:667 +0x470\ngithub.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(0xc00034ce40, 0xc0022c65c0, 0x20, 0x20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/abci/client/local_client.go:231 +0xe8\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(0xc000341780, 0xc0022c65c0, 0x20, 0x20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/proxy/app_conn.go:69 +0x6d\ngithub.com/tendermint/tendermint/state.execBlockOnProxyApp(0x7fe7df730ca0, 0xc00113ae60, 0x7fe7df738ba0, 0xc000341780, 0xc002276000, 0x7fe7df73fdc0, 0xc0000b6300, 0x6, 0xc000ce3880, 0x12)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:274 +0x3df\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc00011b180, 0xa, 0x0, 0xc000cedf30, 0x6, 0xc000ce3880, 0x12, 0x1, 0x0, 0xc000ba9280, ...)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:124 +0x181\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc00113d880, 0x2)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1325 +0x93e\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc00113d880, 0x2)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1256 +0x3e2\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit.func1(0xc00113d880, 0x0, 0x2)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1201 +0x93\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit(0xc00113d880, 0x2, 0x0)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1233 +0x617\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).addVote(0xc00113d880, 0xc00227e640, 0xc00025e090, 0x28, 0x103, 0x103, 0x103)\n\t/go/pkg/mod/gith
Sep 10 21:40:12 451d95b1e0fc ag-chain-cosmos[18441]: ub.com/tendermint/[email protected]/consensus/state.go:1658 +0xb01\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).tryAddVote(0xc00113d880, 0xc00227e640, 0xc00025e090, 0x28, 0xc002297410, 0xc000ca7c28, 0x7fe7de96077f)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1504 +0x5e\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc00113d880, 0x7fe7df713c40, 0xc000b700c8, 0xc00025e090, 0x28)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:688 +0x262\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc00113d880, 0x0)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:633 +0x66f\ncreated by github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:334 +0x13c\n"

The crash is due to a missing nil check in x/distribution/keeper/allocation.go circa line 89, where the validator is nil, but the truly troubling thing is the warning above and the fact that all the validators are unrecognized.

I also wonder why I need to change InitChainer to update res? Without that change, I get a failed assertion that len(req.Validators) != len(res.Validators) in baseapp/baseapp.go, where I would have expected it to be a seamless upgrade from v0.35.0.

Version

v0.37.0

Steps to Reproduce

Create a v0.37.0-based validator without gentx and instead try using the genesis.json validators key.

---

For Admin Use

• [ ] Not duplicate issue
• [ ] Appropriate labels applied
• [ ] Appropriate contributors tagged
• [ ] Contributor assigned/self-assigned

[ethanfrey]

I think you cannot use staking, distribution or slashing modules if you don't have the gentx staked in genesis.

I think it would be good to have a tutorial demonstrating how to run a poa testnet, eg. Static validators, which is much easier for dev testing

[alexanderbez]

@marbar3778 don't you have a POA module somewhere? Care to share any insights here? As far as the issue is concerned, I believe @ethanfrey is correct. What do you wish to be done here as an actional item?

[marbar3778]

heres the start of it https://github.com/cosmos/cosmos-sdk/pull/4971, have to complete a few things and write the spec then will be ready, will be spending some time on it in the coming days.

[marbar3778]

created a quick spec for the POA here: https://github.com/cosmos/cosmos-sdk/pull/5043, leave any comments or ideas youd like. it takes from the staking module quite a bit at this stage but this can change

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.