Prysm: Beaconnode: runtime panic: Index out of bounds

Created on 30 Sep 2020 · 5Comments · Source: prysmaticlabs/prysm

🐞 Bug Report

Description

Running the beacon node with --spadina fails for me with an index out of bounds.
What I did was run it an older version normally with no flags. This synced up the chain properly, then I switched to the newest version and ran ./prysm.sh beacon-chain --http-web3provider=$HOME/.ethereum/goerli/geth.ipc --spadina which produced the following error: [2020-09-30 08:27:10] ERROR main: database contract is 0x07b39f4fde4a38bace212b546dac87c58dfe3fdc but tried to run with 0x48b597f4b53c21b48ad95c7256b49d1779bd5890
I then tried to delete my beaconchaindata directory. After this I could sync until it failed with an out of bounds.

Has this worked before in a previous version?

Yep it worked on the previous release

🔬 Minimal Reproduction

> ./prysm.sh beacon-chain --http-web3provider=$HOME/.ethereum/goerli/geth.ipc
ctrl+c
> rm -rf $Home/.eth2/beaconchaindata
> ./prysm.sh beacon-chain --http-web3provider=$HOME/.ethereum/goerli/geth.ipc --spadina

## 🔥 Error


[2020-09-30 08:29:17]  INFO powchain: Minimum number of validators reached for beacon-chain to start ChainStartTime=2020-09-29 14:00:13 +0200 CEST
[2020-09-30 08:29:17]  INFO blockchain: Initialized beacon chain genesis state
panic: runtime error: index out of range [2338] with length 0

goroutine 228 [running]:
github.com/prysmaticlabs/prysm/proto/beacon/db.(*ChainStartData).MarshalToSizedBuffer(0xc00010f080, 0xc00c2b6000, 0x3c8143, 0x870778, 0x54686, 0x3c8144, 0x0)
    bazel-out/k8-opt/bin/proto/beacon/db/linux_amd64_stripped/db_go_proto%/github.com/prysmaticlabs/prysm/proto/beacon/db/powchain.pb.go:659 +0x40b
github.com/prysmaticlabs/prysm/proto/beacon/db.(*ETH1ChainData).MarshalToSizedBuffer(0xc0026dd8c0, 0xc00c2b6000, 0x870778, 0x870778, 0x101, 0xc002c59358, 0xc002c593d0)
    bazel-out/k8-opt/bin/proto/beacon/db/linux_amd64_stripped/db_go_proto%/github.com/prysmaticlabs/prysm/proto/beacon/db/powchain.pb.go:558 +0x353
github.com/prysmaticlabs/prysm/proto/beacon/db.(*ETH1ChainData).XXX_Marshal(0xc0026dd8c0, 0xc00c2b6000, 0x0, 0x870778, 0xc0026dd800, 0x1, 0x1, 0x1ffffffff, 0x10, 0x10)
    bazel-out/k8-opt/bin/proto/beacon/db/linux_amd64_stripped/db_go_proto%/github.com/prysmaticlabs/prysm/proto/beacon/db/powchain.pb.go:52 +0x5b
github.com/gogo/protobuf/proto.Marshal(0x1cc0f40, 0xc0026dd8c0, 0x8, 0x8, 0xc005518680, 0xc002c594d0, 0x10245e8)
    external/com_github_gogo_protobuf/proto/table_marshal.go:2940 +0x2d6
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.(*Store).SavePowchainData.func1(0xc0001d4380, 0x1, 0xc0001d4380)
    beacon-chain/db/kv/powchain.go:19 +0x86
go.etcd.io/bbolt.(*DB).Update(0xc0008ae000, 0xc002c595a0, 0x0, 0x0)
    external/io_etcd_go_bbolt/db.go:694 +0x8e
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.(*Store).SavePowchainData(0xc0008f8480, 0x1cca840, 0xc0004d2700, 0xc0026dd8c0, 0x0, 0x0)
    beacon-chain/db/kv/powchain.go:17 +0xd4
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).ProcessLog(0xc000465b00, 0x1cca840, 0xc0004d2700, 0xb4213cb5f497b548, 0x179db456725cd98a, 0x9058bd79, 0xc00715f500, 0x1, 0x4, 0xc005ba9200, ...)
    beacon-chain/powchain/log_processing.go:91 +0x53c
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).processPastLogs(0xc000465b00, 0x1cca840, 0xc0004d2700, 0xaf3ac7a77df82f3b, 0xbd3ee82449f92b9)
    beacon-chain/powchain/log_processing.go:328 +0x307
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).initPOWService(0xc000465b00)
    beacon-chain/powchain/service.go:658 +0x27e
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).run(0xc000465b00, 0xc0001425a0)
    beacon-chain/powchain/service.go:673 +0x89
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).Start.func1(0xc000465b00)
    beacon-chain/powchain/service.go:244 +0x50
created by github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).Start
    beacon-chain/powchain/service.go:242 +0x60

## 🌍 Your Environment **Operating System:**

  
linux

What version of Prysm are you running? (Which release)

  
newest: v1.0.0-alpha.27

Anything else relevant (validator index / public key)?

Bug

Source

MariusVanDerWijden

Most helpful comment

Purging the Eth2 dir solved the problem
rm -rf $HOME/.eth2
But I think there should be some check that this gracefully exits

MariusVanDerWijden on 30 Sep 2020

👍2

All 5 comments

Purging the Eth2 dir solved the problem
rm -rf $HOME/.eth2
But I think there should be some check that this gracefully exits

MariusVanDerWijden on 30 Sep 2020

👍2

@prestonvanloon any confirmations if this has been fixed? @MariusVanDerWijden have you encountered the issue again?

0xKiwi on 15 Oct 2020

I haven't tried, but the repro is pretty easy: init the node with one genesis, kill the node, run with another genesis

MariusVanDerWijden on 15 Oct 2020

No longer possible given we return err if saving a nil eth1 data

rauljordan on 22 Oct 2020

Got same issue. I did what was suggested by @MariusVanDerWijden and later when I was trying to move to Pyrmont testnet /srv/prysm$ ./prysm.sh beacon-chain --pyrmont --http-web3provider=/home/jeffprestes/.ethereum/goerli/geth.ipc
I got this error: ERROR main: database contract is 0x00000000219ab540356cbb839cbe05303d7705fa but tried to run with 0x8c5fecdc472e27bc447696f431e425d02dd46a8c