Cosmos-sdk: Bug causing the chain to halt

[dbriggsie]

Summary of Bug

I think I have found a bug on a testnet version. Maybe Bounty worthy..
Firstly I passed a param-change proposal (see below)

{
  "title": "Rwd Change",
  "description": "This is tansaction to test a parameter-change for rewards.",
  "changes": [
    {
      "subspace": "distribution",
      "key": "baseproposerreward",
      "value": "0.999000000000000000"
    }
  ],
  "deposit": [
    {
      "denom": "uscrt",
      "amount": "10000000"
    }
  ]
}

Then within 5000-8000 blocks, the entire chain stopped..

ERROR LOG:

Mar 16 08:36:03 li1960-49 enigmad[1409]: I[2020-03-16|08:36:03.208] starting ABCI with Tendermint                module=main
Mar 16 08:36:03 li1960-49 enigmad[1409]: panic: negative coin amount
Mar 16 08:36:03 li1960-49 enigmad[1409]: goroutine 1 [running]:
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/cosmos/cosmos-sdk/types.DecCoins.Sub(...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/types/dec_coin.go:307
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.AllocateTokens(0x169cf60, 0xc000ad0200, 0xc000aba460, 0xc000aba460, 0x169cf60, 0xc000ad0230, 0x169cfa0, 0xc000ad0280, 0xc000ab1500, 0xc, ...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/x/distribution/keeper/allocation.go:66 +0x1589
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/cosmos/cosmos-sdk/x/distribution.BeginBlocker(0x16ad9a0, 0xc000034068, 0x16c2100, 0xc001df8880, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/x/distribution/abci.go:26 +0x2e6
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/cosmos/cosmos-sdk/x/distribution.AppModule.BeginBlock(...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/x/distribution/module.go:147
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(0xc000abb030, 0x16ad9a0, 0xc000034068, 0x16c2100, 0xc001df8880, 0xa, 0x0, 0x0, 0x0, 0x0, ...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/types/module/module.go:297 +0x1ca
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/enigmampc/EnigmaBlockchain.(*EnigmaChainApp).BeginBlocker(...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/enigmachain/app.go:391
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(0xc000d23e00, 0xc00202a6a0, 0x20, 0x20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:136 +0x469
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(0xc000d209c0, 0xc00202a6a0, 0x20, 0x20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/abci/client/local_client.go:231 +0x101
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(0xc000e2a450, 0xc00202a6a0, 0x20, 0x20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/proxy/app_conn.go:69 +0x6b
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x16ae3a0, 0xc000de8580, 0x16bb520, 0xc000e2a450, 0xc001ef9500, 0x16c4080, 0xc000b9e068, 0x6, 0xc000cf5570, 0xe)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:280 +0x3e1
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc000e8fc00, 0xa, 0x0, 0xc000cf5550, 0x6, 0xc000cf5570, 0xe, 0x29c8a, 0xc000cde380, 0x20, ...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:131 +0x17a
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(0xc000b310b0, 0xa, 0x0, 0xc000cf5550, 0x6, 0xc000cf5570, 0xe, 0x29c8a, 0xc000cde380, 0x20, ...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/replay.go:475 +0x233
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlocks(0xc000b310b0, 0xa, 0x0, 0xc000cf5550, 0x6, 0xc000cf5570, 0xe, 0x29c8a, 0xc000cde380, 0x20, ...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/replay.go:455 +0x3f3
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(0xc000c530b0, 0xa, 0x0, 0xc000cf5550, 0x6, 0xc000cf5570, 0xe, 0x29c8a, 0xc000cde380, 0x20, ...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/replay.go:386 +0xeef
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc000b310b0, 0x16c4d80, 0xc000108bd0, 0x7f83d63ddd63, 0x115f208)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/replay.go:269 +0x485
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/tendermint/tendermint/node.doHandshake(0x16c4080, 0xc000b9e068, 0xa, 0x0, 0xc000cf5550, 0x6, 0xc000cf5570, 0xe, 0x29c8a, 0xc000cde380, ...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/node/node.go:281 +0x19a
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/tendermint/tendermint/node.NewNode(0xc000d232c0, 0x16a7ce0, 0xc000b546e0, 0xc000b88360, 0x168b7c0, 0xc000b9bdc0, 0xc000b88590, 0x149b570, 0xc000b885a0, 0x16ae3a0, ...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/node/node.go:597 +0x343
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/cosmos/cosmos-sdk/server.startInProcess(0xc000d28460, 0x149bd98, 0x1d, 0x0, 0x0)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:157 +0x4c1
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/cosmos/cosmos-sdk/server.StartCmd.func1(0xc000d42000, 0x1fde8a0, 0x0, 0x0, 0x0, 0x0)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:67 +0xb4
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/spf13/cobra.(*Command).execute(0xc000d42000, 0x1fde8a0, 0x0, 0x0, 0xc000d42000, 0x1fde8a0)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/spf13/[email protected]/command.go:826 +0x460
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/spf13/cobra.(*Command).ExecuteC(0xc0000a4c80, 0x13366cb, 0xc00043fe90, 0x591822)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/spf13/cobra.(*Command).Execute(...)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
Mar 16 08:36:03 li1960-49 enigmad[1409]: github.com/tendermint/tendermint/libs/cli.Executor.Execute(0xc0000a4c80, 0x149c1d0, 0x13198a6, 0x10)
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/libs/cli/setup.go:89 +0x3c
Mar 16 08:36:03 li1960-49 enigmad[1409]: main.main()
Mar 16 08:36:03 li1960-49 enigmad[1409]:         /home/assafmo/workspace/enigmachain/cmd/enigmad/main.go:69 +0x7ab
Mar 16 08:36:03 li1960-49 systemd[1]: enigma-node.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 16 08:36:03 li1960-49 systemd[1]: enigma-node.service: Failed with result 'exit-code'.

Version

0.1.0-3-gff4c7f8

Steps to Reproduce

(as above with proposal)

---

For Admin Use

• [ ] Not duplicate issue
• [ ] Appropriate labels applied
• [ ] Appropriate contributors tagged
• [ ] Contributor assigned/self-assigned

[toml01]

A few more notes on top of @dbriggsie 's:

logs:

Mar 16 05:02:52 ip-172-31-44-28 enigmad[20612]: I[2020-03-16|05:02:52.767] Executed block                               module=state height=171146 validTxs=0 invalidTxs=0
Mar 16 05:02:52 ip-172-31-44-28 enigmad[20612]: I[2020-03-16|05:02:52.780] Committed state                              module=state height=171146 txs=0 appHash=FB4739B6F0D4FED77D431922E340B95B8144BF37483D3C1225431311A5BB229D
Mar 16 05:02:58 ip-172-31-44-28 enigmad[20612]: E[2020-03-16|05:02:58.075] CONSENSUS FAILURE!!!                         module=consensus err="negative coin amount" stack="goroutine 1419 [running]:\nruntime/debug.Stack(0xc0012c2ce0, 0x1115f40, 0x16671b0)\n\t/usr/lib/go-1.13/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2(0xc002741c00, 0x149b478)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:613 +0x57\npanic(0x1115f40, 0x16671b0)\n\t/usr/lib/go-1.13/src/runtime/panic.go:679 +0x1b2\ngithub.com/cosmos/cosmos-sdk/types.DecCoins.Sub(...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/types/dec_coin.go:307\ngithub.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.AllocateTokens(0x169cf60, 0xc000eab5b0, 0xc000ab8460, 0xc000ab8460, 0x169cf60, 0xc000eab5e0, 0x169cfa0, 0xc000eab630, 0xc000e9fbe0, 0xc, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/x/distribution/keeper/allocation.go:66 +0x1589\ngithub.com/cosmos/cosmos-sdk/x/distribution.BeginBlocker(0x16ad9a0, 0xc000034048, 0x16c2100, 0xc005382800, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/x/distribution/abci.go:26 +0x2e6\ngithub.com/cosmos/cosmos-sdk/x/distribution.AppModule.BeginBlock(...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/x/distribution/module.go:147\ngithub.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(0xc000ab9030, 0x16ad9a0, 0xc000034048, 0x16c2100, 0xc005382800, 0xa, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/types/module/module.go:297 +0x1ca\ngithub.com/enigmampc/EnigmaBlockchain.(*EnigmaChainApp).BeginBlocker(...)\n\t/home/assafmo/workspace/enigmachain/app.go:391\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(0xc000d197c0, 0xc004071c00, 0x20, 0x20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:136 +0x469\ngithub.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(0xc000e7cd20, 0xc004071c00, 0x20, 0x20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/abci/client/local_client.go:231 +0x101\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(0xc000eab1b0, 0xc004071c00, 0x20, 0x20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/proxy/app_conn.go:69 +0x6b\ngithub.com/tendermint/tendermint/state.execBlockOnProxyApp(0x16ae3a0, 0xc0027fa9a0, 0x16bb520, 0xc000eab1b0, 0xc003dc3a40, 0x16c4080, 0xc000011138, 0x6, 0xc00272a9a0, 0xe)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:280 +0x3e1\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc0000fce00, 0xa, 0x0, 0xc00272a980, 0x6, 0xc00272a9a0, 0xe, 0x29c8a, 0xc0038a1660, 0x20, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:131 +0x17a\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc002741c00, 0x29c8b)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1431 +0x8f5\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc002741c00, 0x29c8b)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1350 +0x383\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit.func1(0xc002741c00, 0x0, 0x29c8b)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1285 +0x90\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit(0xc002741c00, 0x29c8b, 0x0)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1322 +0x61a\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc002741c00, 0xc005087c20, 0xc003fd0b70, 0x28, 0xc0012c9a38, 0xd31f92, 0x0)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1819 +0xa39\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc002741c00, 0xc005087c20, 0xc003fd0b70, 0x28, 0xf136b9f2d600ff82, 0x108, 0x100)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1642 +0x59\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc002741c00, 0x168b640, 0xc0028f4538, 0xc003fd0b70, 0x28)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:709 +0x252\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc002741c00, 0x0)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:644 +0x6eb\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:335 +0x13a\n"
Mar 16 05:36:48 ip-172-31-44-28 enigmad[20612]: E[2020-03-16|05:36:48.395] Connection failed @ sendRoutine              module=p2p [email protected]:60440 conn=MConn{149.248.55.89:60440} err="pong timeout"
Mar 16 05:36:48 ip-172-31-44-28 enigmad[20612]: E[2020-03-16|05:36:48.402] Stopping peer for error                      module=p2p peer="Peer{MConn{149.248.55.89:60440} 842822b88cb5762ba99cee50a37156c6d0a6c452 in}" err="pong timeout"

The above panic occurs at roughly the same time of the mentioned proposal voting_end_time:

{
  "content": {
    "type": "cosmos-sdk/ParameterChangeProposal",
    "value": {
      "title": "Rwd Change",
      "description": "This is tansaction to test a parameter-change for rewards.",
      "changes": [
        {
          "subspace": "distribution",
          "key": "baseproposerreward",
          "value": "\"0.999000000000000000\""
        }
      ]
    }
  },
  "id": "4",
  "proposal_status": "Passed",
  "final_tally_result": {
    "yes": "1118157000000",
    "abstain": "0",
    "no": "0",
    "no_with_veto": "0"
  },
  "submit_time": "2020-03-15T21:02:46.31257786Z",
  "deposit_end_time": "2020-03-16T05:02:46.31257786Z",
  "total_deposit": [
    {
      "denom": "uscrt",
      "amount": "10000000"
    }
  ],
  "voting_start_time": "2020-03-15T21:02:46.31257786Z",
  "voting_end_time": "2020-03-16T05:02:46.31257786Z"
}

[okwme]

Hi @dbriggsie I don't recognize the format of your version. Is this a reference to the version of cosmos-sdk (currently 0.38.1-165e134 is the latest release) or is it gaia or something else?

[toml01]

Hi @okwme This is a testnet we ran, based on https://github.com/cosmos/cosmos-sdk/releases/tag/v0.38.1

I'm guessing it has something to do with these calculations:
https://github.com/cosmos/cosmos-sdk/blob/56c5868975250d52e17c6da5ffd9a64737db7c45/x/distribution/keeper/allocation.go#L44-L54

When the base_proposer_reward is as high as 0.999, probably causes some problems?

[alexanderbez]

What version of the SDK are you using? We've added parameter validation recently, specifically for this case: https://github.com/cosmos/cosmos-sdk/blob/master/x/distribution/types/params.go#L72

[toml01]

Thanks @alexanderbez , that's probably what went wrong.
We are running v0.38.1, does it not include the update yet?

[alexanderbez]

Interestingly, yes 0.38.1 does include the validation checks. What's the bonus reward value?

[toml01]

These are the distribution module params:

community_tax: "0.020000000000000000"
base_proposer_reward: "0.999000000000000000"
bonus_proposer_reward: "0.040000000000000000"
withdraw_addr_enabled: true

[alexanderbez]

Yes, this should've caused the change to error and as a result, the proposal should not have passed.

[dbriggsie]

I got a prompt when submitting a proposal that was over 1.0 that it was too large.. so the transaction failed..
But when I set it less, it went through fine. With no issues.

And the bonus_reward never passed before chain stopped.
I’m guessing because it’s 0.999+ 0.04 that it broke?

[alexanderbez]

I’m guessing because it’s 0.999+ 0.04 that it broke?

Yes.

But I see the issue. The validation functions are on the individual parameters:

func (p *Params) ParamSetPairs() paramtypes.ParamSetPairs {
    return paramtypes.ParamSetPairs{
        paramtypes.NewParamSetPair(ParamStoreKeyCommunityTax, &p.CommunityTax, validateCommunityTax),
        paramtypes.NewParamSetPair(ParamStoreKeyBaseProposerReward, &p.BaseProposerReward, validateBaseProposerReward),
        paramtypes.NewParamSetPair(ParamStoreKeyBonusProposerReward, &p.BonusProposerReward, validateBonusProposerReward),
        paramtypes.NewParamSetPair(ParamStoreKeyWithdrawAddrEnabled, &p.WithdrawAddrEnabled, validateWithdrawAddrEnabled),
    }
}

Hence, when you update baseproposerreward, it only calls validateBaseProposerReward which doesn't check the entire set.

Thanks for raising this issue. We need to think on how to best address this (cross-param validation dependencies).

[alexanderbez]

Supporting this feature I think will not be trivial, although I can think of one clean way to do it, it would require having an invariant that would break the current design. I'm open to suggestions and ideas though!

I think the best course of action for now, is to combat this via sufficient documentation so that proposals are properly vetted before being voted upon.

I will update the x/distribution params doc to address this issue in the meantime => https://github.com/cosmos/cosmos-sdk/pull/5813

[alexanderbez]

I think whatever issue you're facing is orthogonal, no? This issue is in regard to invalid parameter updates and cross-parameter validation dependencies.

We've done exports/major upgrades in the mainnet I believe about 3-4 times now without issue.

[toml01]

You are probably right. Deleted that comment, it's not related to this issue.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.