Cosmos-sdk: State export and genesis migration fails from v0.39.1 to v0.40.0-rc1

Created on 2 Nov 2020 · 8Comments · Source: cosmos/cosmos-sdk

Summary of Bug

Chain crashes with the following error after exporting state and migrating genesis from v0.39.1 to v0.40.0-rc1

I[2020-11-02|09:58:25.888] starting ABCI with Tendermint                module=main
panic: proto: Int64Value: illegal tag 0 (wire type 1)

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/store/rootmulti.getLatestVersion(0x202cb60, 0xc000e5a570, 0x2015f40)
        github.com/cosmos/[email protected]/store/rootmulti/store.go:903 +0x10b
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).LoadLatestVersion(0xc000f274a0, 0x2015f40, 0x2ddd480)
        github.com/cosmos/[email protected]/store/rootmulti/store.go:158 +0x37
github.com/cosmos/cosmos-sdk/baseapp.DefaultStoreLoader(0x2033ec0, 0xc000f274a0, 0xc00011f830, 0xc0004b97e0)
        github.com/cosmos/[email protected]/baseapp/baseapp.go:258 +0x31
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).LoadLatestVersion(0xc000129520, 0xc000ffd7f0, 0x7fb864542620)
        github.com/cosmos/[email protected]/baseapp/baseapp.go:248 +0x42
github.com/ovrclk/akash/app.NewApp(0x2015780, 0xc0004fefa0, 0x202cb60, 0xc000e5a570, 0x0, 0x0, 0x0, 0xc000135140, 0xc000e475b0, 0xd, ...)
        github.com/ovrclk/akash/app/app.go:407 +0x6aba
github.com/ovrclk/akash/cmd/akashd/cmd.newApp(0x2015780, 0xc0004fefa0, 0x202cb60, 0xc000e5a570, 0x0, 0x0, 0x1fd70c0, 0xc000f78000, 0x30ffffffff, 0x4)  
        github.com/ovrclk/akash/cmd/akashd/cmd/root.go:140 +0x93d
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc000e817e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2015300, 0xc000e4b7c0, ...)
        github.com/cosmos/[email protected]/server/start.go:208 +0x168
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc000f24840, 0x2e0dbd8, 0x0, 0x0, 0x0, 0x0)
        github.com/cosmos/[email protected]/server/start.go:117 +0x154
github.com/spf13/cobra.(*Command).execute(0xc000f24840, 0x2e0dbd8, 0x0, 0x0, 0xc000f24840, 0x2e0dbd8)
        github.com/spf13/[email protected]/command.go:850 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000d39600, 0x1bb47f4, 0x5, 0xc000e475b0)
        github.com/spf13/[email protected]/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/[email protected]/command.go:895
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        github.com/spf13/[email protected]/command.go:888
github.com/ovrclk/akash/cmd/akashd/cmd.Execute(0xc000d39600, 0x20214a0, 0xc000e60880)
        github.com/ovrclk/akash/cmd/akashd/cmd/root.go:86 +0x165
main.main()
        github.com/ovrclk/akash/cmd/akashd/main.go:12 +0x2a

Version

State export version:
Cosmos SDK version: v0.39.1
Akash version: v0.8.1

Genesis migration version:
Cosmos SDK version: v0.40.0-rc1
Branch name: bigbang

Steps to Reproduce

Exporting state from v0.8.1 binary:

akashd export --for-zero-height  > bigbang_genesis_export.json

Building the bigbang version of akashd and executing the following:

akashd migrate v0.40 bigbang_genesis_export.json > new_v40_genesis.json

Copying the new genesis to config directory and starting the chain:

cp new_v40_genesis.json ~/.akashd/config/genesis.json

akashd start

Folder with original genesis, exported state and migrated genesis: issue.tar.gz

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

bug needs-more-info

Source

kaustubhkapatral

All 8 comments

Note: It seams that the same commands worked on Gaia. I was able to load exported gaia snapshot. AFAIK @zmanian did the whole sequence (export -> migrate -> init -> start) successful (though it took lot of time because of https://github.com/cosmos/cosmos-sdk/issues/7682)

robert-zaremba on 2 Nov 2020

Also, we have rc2 over the corner - I would suggest debugging and redoing using rc2. What do you think?
/cc @clevinson

robert-zaremba on 2 Nov 2020

👍1

You need to provide an implementation of the tendermint conversion.

This is done here.
https://github.com/cosmos/gaia/blob/jack/gaiav3.0/app/migrate.go#L198-L229

zmanian on 2 Nov 2020

rc2 is now tagged if you'd like to test again against that.

Zaki is correct in that you do need to provide a manual migration for the tendermint consesnsus params. That needs to happen either as part of an application migrate script, or externally via jq manually (see Step #7 here). I'm unsure if the error reported here is corresponding to tm consensus params, but they need to be manually migrated either way, so make sure that is part of your migration steps.

PS- @anilCSE let's make sure we include the requirement about migrating tendermint consensus params in the migration guide docs you're working on

clevinson on 3 Nov 2020

👍1

In the attached issue.tar.gz, the genesis file looks good (tendermint conversion has already been done correctly).

However, I couldn't reproduce the panic with the attached new_v40_genesis.json, the chain ran fine. Would like more info.

amaurymartiny on 3 Nov 2020

You need to provide an implementation of the tendermint conversion.

This is done here.
https://github.com/cosmos/gaia/blob/jack/gaiav3.0/app/migrate.go#L198-L229

Thanks @zmanian , looks like these changes are covered in migrate command already.

@amaurymartiny for bigbang-1 testnet upgrade failure, looks like it is an issue with upgrade-handler, it didn't handle the state migrations (still debugging on this). For migrate + upgrade, it was just an issue with config files. We needed to update the config files and it's working!

Planning an internal testnet tomorrow to cross check this upgrade path and will close this issue once after the testing.