Cosmos-sdk: Gaia-9002 Crash log

Created on 6 Dec 2018 · 6Comments · Source: cosmos/cosmos-sdk

invariant broken: loose token invariance:
    pool.LooseTokens: 4409145988819987660.4767998960
    sum of account tokens: 4409145988819987660.4768000000" stack="goroutine 1201 [running]:
runtime/debug.Stack(0xc00a433ac0, 0xf38d60, 0xc003f91540)
    /usr/lib/golang/src/runtime/debug/stack.go:24 +0xa7
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func2(0xc000497c00, 0x1217900)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:576 +0x57
panic(0xf38d60, 0xc003f91540)
    /usr/lib/golang/src/runtime/panic.go:513 +0x1b9
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).assertRuntimeInvariants(0xc000af1000)
    src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/invariants.go:30 +0x453
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker(0xc000af1000, 0x1315a80, 0xc00f0eeb10, 0xc00c567ac0, 0xd, 0x17b09, 0x0, 0x0, 0x0, 0x0, ...)
    src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:218 +0x241
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker-fm(0x1315a80, 0xc00f0eeb10, 0xc00c567ac0, 0xd, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:165 +0xc8
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc00018a700, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:794 +0x105
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc0001b4cc0, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:239 +0xea
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc000978d40, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:77 +0x51
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x1316900, 0xc00086a500, 0x131ba80, 0xc000978d40, 0xc002e3cd20, 0xc00f0ee570, 0x1320540, 0xc0001c46b0, 0x24ca7, 0xc009c7e560, ...)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:265 +0x701
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc000112700, 0x7, 0x0, 0xc000b2ca20, 0x6, 0xc000b2ca30, 0x9, 0x17b08, 0x24ca7, 0xc009c7e560, ...)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:96 +0x15f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc000497c00, 0x17b09)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1290 +0xa8c
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc000497c00, 0x17b09)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1221 +0x451
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).addProposalBlockPart(0xc000497c00, 0xc00016dd40, 0xc00417e0f0, 0x28, 0xc00c3b3000, 0x2d9d, 0x2d9d)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1460 +0x4d1
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc000497c00, 0x13084a0, 0xc00016dd40, 0xc00417e0f0, 0x28)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:638 +0x7a0
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc000497c00, 0x0)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:607 +0x670
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:300 +0x132
invariant broken: loose token invariance:
    pool.LooseTokens: 4409145988819987660.4767998960
    sum of account tokens: 4409145988819987660.4768000000" stack="goroutine 1201 [running]:
runtime/debug.Stack(0xc00a433ac0, 0xf38d60, 0xc003f91540)
    /usr/lib/golang/src/runtime/debug/stack.go:24 +0xa7
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func2(0xc000497c00, 0x1217900)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:576 +0x57
panic(0xf38d60, 0xc003f91540)
    /usr/lib/golang/src/runtime/panic.go:513 +0x1b9
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).assertRuntimeInvariants(0xc000af1000)
    src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/invariants.go:30 +0x453
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker(0xc000af1000, 0x1315a80, 0xc00f0eeb10, 0xc00c567ac0, 0xd, 0x17b09, 0x0, 0x0, 0x0, 0x0, ...)
    src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:218 +0x241
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker-fm(0x1315a80, 0xc00f0eeb10, 0xc00c567ac0, 0xd, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:165 +0xc8
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc00018a700, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:794 +0x105
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc0001b4cc0, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:239 +0xea
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc000978d40, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:77 +0x51
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x1316900, 0xc00086a500, 0x131ba80, 0xc000978d40, 0xc002e3cd20, 0xc00f0ee570, 0x1320540, 0xc0001c46b0, 0x24ca7, 0xc009c7e560, ...)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:265 +0x701
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc000112700, 0x7, 0x0, 0xc000b2ca20, 0x6, 0xc000b2ca30, 0x9, 0x17b08, 0x24ca7, 0xc009c7e560, ...)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:96 +0x15f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc000497c00, 0x17b09)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1290 +0xa8c
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc000497c00, 0x17b09)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1221 +0x451
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).addProposalBlockPart(0xc000497c00, 0xc00016dd40, 0xc00417e0f0, 0x28, 0xc00c3b3000, 0x2d9d, 0x2d9d)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1460 +0x4d1
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc000497c00, 0x13084a0, 0xc00016dd40, 0xc00417e0f0, 0x28)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:638 +0x7a0
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc000497c00, 0x0)
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:607 +0x670
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart
    src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:300 +0x132

For Admin Use

[x] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

Source

hendrikhofstadt

Most helpful comment

{
        "Height": 97032,
        "Hash": "3FF40CAD512D1379EE70E2FCD274C7D14F531B5CD62BADDBF391AC8F1EB863C0",
        "Fee": {
            "gas": 200000,
            "amount": [
                {
                    "denom": "",
                    "amount": "0"
                }
            ]
        },
        "Memo": "",
        "IsOK": true,
        "ValidFormat": true,
        "Result": {
            "log": "Msg 0: ",
            "tags": [
                {
                    "key": "action",
                    "value": "withdraw_validator_rewards_all"
                },
                {
                    "key": "source-validator",
                    "value": "cosmosvaloper1shuqhpl273t96yg6nnqvyfeewj3ew3md6uy7uj"
                }
            ],
            "gas_used": 8082,
            "gas_wanted": 200000
        },
        "Messages": [
            {
                "data": {
                    "validator_addr": "cosmosvaloper1shuqhpl273t96yg6nnqvyfeewj3ew3md6uy7uj"
                },
                "type": "withdraw_validator_rewards_all"
            }
        ],
        "Signatures": [
            {
                "address": "85f80b87eaf4565d111a9cc0c2273974a397476d",
                "sequence": 3107
            }
        ]
    },
    {
        "Height": 97032,
        "Hash": "3F04C45EAD604CFF00BD4746ED015A64F617E6F2AEB3D11E03553D1F2E2CE7D0",
        "Fee": {
            "gas": 200000,
            "amount": [
                {
                    "denom": "",
                    "amount": "0"
                }
            ]
        },
        "Memo": "",
        "IsOK": true,
        "ValidFormat": true,
        "Result": {
            "log": "Msg 0: ",
            "tags": [
                {
                    "key": "action",
                    "value": "withdraw_delegation_rewards_all"
                },
                {
                    "key": "delegator",
                    "value": "cosmos1vkggdmhuacs2306uasqw3au8a8lmcmqhqngtj4"
                }
            ],
            "gas_used": 7498,
            "gas_wanted": 200000
        },
        "Messages": [
            {
                "data": {
                    "delegator_addr": "cosmos1vkggdmhuacs2306uasqw3au8a8lmcmqhqngtj4"
                },
                "type": "withdraw_delegation_rewards_all"
            }
        ],
        "Signatures": [
            {
                "address": "659086eefcee20a8bf5cec00e8f787e9ffbc6c17",
                "sequence": 3678
            }
        ]
    },
    {
        "Height": 97032,
        "Hash": "4610B78843D9AD031F3A72745E372C8AA868CB2A408095D30A7C2D3676448EB5",
        "Fee": {
            "gas": 200000,
            "amount": [
                {
                    "denom": "",
                    "amount": "0"
                }
            ]
        },
        "Memo": "",
        "IsOK": true,
        "ValidFormat": true,
        "Result": {
            "log": "Msg 0: ",
            "tags": [
                {
                    "key": "action",
                    "value": "withdraw_validator_rewards_all"
                },
                {
                    "key": "source-validator",
                    "value": "cosmosvaloper10505nl7yftsme9jk2glhjhta7w0475uvl4k8ju"
                }
            ],
            "gas_used": 12682,
            "gas_wanted": 200000
        },
        "Messages": [
            {
                "data": {
                    "validator_addr": "cosmosvaloper10505nl7yftsme9jk2glhjhta7w0475uvl4k8ju"
                },
                "type": "withdraw_validator_rewards_all"
            }
        ],
        "Signatures": [
            {
                "address": "7d1f49ffc44ae1bc9656523f795d7df39f5f538c",
                "sequence": 12164
            }
        ]
    },

These are the transactions of the block

hendrikhofstadt on 6 Dec 2018

👍4

All 6 comments

Runtime invariants are asserted at the end of each block, so this discrepancy was caused by a transaction (or begin block / end block operation) in the block we were attempting to execute.

cwgoes on 6 Dec 2018

{
        "Height": 97032,
        "Hash": "3FF40CAD512D1379EE70E2FCD274C7D14F531B5CD62BADDBF391AC8F1EB863C0",
        "Fee": {
            "gas": 200000,
            "amount": [
                {
                    "denom": "",
                    "amount": "0"
                }
            ]
        },
        "Memo": "",
        "IsOK": true,
        "ValidFormat": true,
        "Result": {
            "log": "Msg 0: ",
            "tags": [
                {
                    "key": "action",
                    "value": "withdraw_validator_rewards_all"
                },
                {
                    "key": "source-validator",
                    "value": "cosmosvaloper1shuqhpl273t96yg6nnqvyfeewj3ew3md6uy7uj"
                }
            ],
            "gas_used": 8082,
            "gas_wanted": 200000
        },
        "Messages": [
            {
                "data": {
                    "validator_addr": "cosmosvaloper1shuqhpl273t96yg6nnqvyfeewj3ew3md6uy7uj"
                },
                "type": "withdraw_validator_rewards_all"
            }
        ],
        "Signatures": [
            {
                "address": "85f80b87eaf4565d111a9cc0c2273974a397476d",
                "sequence": 3107
            }
        ]
    },
    {
        "Height": 97032,
        "Hash": "3F04C45EAD604CFF00BD4746ED015A64F617E6F2AEB3D11E03553D1F2E2CE7D0",
        "Fee": {
            "gas": 200000,
            "amount": [
                {
                    "denom": "",
                    "amount": "0"
                }
            ]
        },
        "Memo": "",
        "IsOK": true,
        "ValidFormat": true,
        "Result": {
            "log": "Msg 0: ",
            "tags": [
                {
                    "key": "action",
                    "value": "withdraw_delegation_rewards_all"
                },
                {
                    "key": "delegator",
                    "value": "cosmos1vkggdmhuacs2306uasqw3au8a8lmcmqhqngtj4"
                }
            ],
            "gas_used": 7498,
            "gas_wanted": 200000
        },
        "Messages": [
            {
                "data": {
                    "delegator_addr": "cosmos1vkggdmhuacs2306uasqw3au8a8lmcmqhqngtj4"
                },
                "type": "withdraw_delegation_rewards_all"
            }
        ],
        "Signatures": [
            {
                "address": "659086eefcee20a8bf5cec00e8f787e9ffbc6c17",
                "sequence": 3678
            }
        ]
    },
    {
        "Height": 97032,
        "Hash": "4610B78843D9AD031F3A72745E372C8AA868CB2A408095D30A7C2D3676448EB5",
        "Fee": {
            "gas": 200000,
            "amount": [
                {
                    "denom": "",
                    "amount": "0"
                }
            ]
        },
        "Memo": "",
        "IsOK": true,
        "ValidFormat": true,
        "Result": {
            "log": "Msg 0: ",
            "tags": [
                {
                    "key": "action",
                    "value": "withdraw_validator_rewards_all"
                },
                {
                    "key": "source-validator",
                    "value": "cosmosvaloper10505nl7yftsme9jk2glhjhta7w0475uvl4k8ju"
                }
            ],
            "gas_used": 12682,
            "gas_wanted": 200000
        },
        "Messages": [
            {
                "data": {
                    "validator_addr": "cosmosvaloper10505nl7yftsme9jk2glhjhta7w0475uvl4k8ju"
                },
                "type": "withdraw_validator_rewards_all"
            }
        ],
        "Signatures": [
            {
                "address": "7d1f49ffc44ae1bc9656523f795d7df39f5f538c",
                "sequence": 12164
            }
        ]
    },

These are the transactions of the block

hendrikhofstadt on 6 Dec 2018

👍4

I bet that https://github.com/cosmos/cosmos-sdk/pull/2958 resolved this issue... wish there was a way to simulate the blockchain easily to test if this is true of not

rigelrozanski on 7 Dec 2018

relevant now merged in (but wasn't for this testnet) https://github.com/cosmos/cosmos-sdk/pull/2982#issuecomment-443748942

rigelrozanski on 7 Dec 2018

relevant now merged in (but wasn't for this testnet) #2982 (comment)

Hmm, maybe - #2982 only changed the logic in state export. Maybe we're leaving rewards in a dist info before deleting it here, though, and that's causing the problem (which we would need to fix separately).

cwgoes on 7 Dec 2018

Pretty sure this can be closed.