Prysm: Sync skips blocks

[stefa2k]

Looks like the sync process skipped a block:

[2019-09-30 09:43:46]  INFO initial-sync: Processing block 15951/15956. Estimated 2s remaining. blocks per second=2.2 peers=5/5
[2019-09-30 09:43:46]  INFO forkchoice: Executing state transition on block slot=15951
[2019-09-30 09:43:46]  INFO blockchain: Finished state transition and updated fork choice store for block attestations=1 deposits=0 root=e0b9002c53f28d3c791f3631400d168746f5610119ee79e0a35dbc791171e750 slot=15951
[2019-09-30 09:43:46]  INFO initial-sync: Processing block 15952/15956. Estimated 1s remaining. blocks per second=2.2 peers=5/5
[2019-09-30 09:43:46]  INFO forkchoice: Executing state transition on block slot=15952
[2019-09-30 09:43:46]  INFO forkchoice: Starting next epoch depositIndex=150 epoch=1994 finalizedEpoch=1992 justifiedEpoch=1993 numValidators=149 previousJustifiedEpoch=1992
[2019-09-30 09:43:46]  INFO blockchain: Finished state transition and updated fork choice store for block attestations=5 deposits=0 root=e9e7fc231dbe1b1e4a4d919dcd4a3ed1e0feedcafefba58b0223dc41da46ffcf slot=15952
[2019-09-30 09:43:46]  INFO initial-sync: Processing block 15953/15956. Estimated 1s remaining. blocks per second=2.3 peers=5/5
[2019-09-30 09:43:46]  INFO forkchoice: Executing state transition on block slot=15953
[2019-09-30 09:43:46]  INFO blockchain: Finished state transition and updated fork choice store for block attestations=8 deposits=0 root=a31a76245b5e61e7f3562d58ab1918cd096504d18edcaea62de68458b29f4ca1 slot=15953
[2019-09-30 09:43:46]  INFO initial-sync: Processing block 15955/15956. Estimated 0s remaining. blocks per second=2.4 peers=5/5
panic: could not process block from fork choice service: pre state of slot 15955 does not exist

goroutine 166 [running]:
github.com/prysmaticlabs/prysm/beacon-chain/sync/initial-sync.(*InitialSync).Start(0xc00020a000)
        beacon-chain/sync/initial-sync/service.go:108 +0x98d
created by github.com/prysmaticlabs/prysm/shared.(*ServiceRegistry).StartAll
        shared/service_registry.go:44 +0x23e

Take a close look to the block number, 15954 is missing.

[santiagorp]

I am getting the same error. I am running a beacon node already for a couple of days (docker) but I tried to start a new one from scratch using bazel and I cannot sync even one block:

goroutine 213 [running]:[2019-10-03 12:40:26]  WARN node: Deleting beaconchain.db from data directory
[2019-10-03 12:40:26]  INFO node: Checking DB database-path=/home/santi/prysm-data/beaconchaindata
[2019-10-03 12:40:27]  INFO node: Fetching testnet cluster address endpoint=https://prylabs.net/contract
[2019-10-03 12:40:27]  INFO node: Starting beacon node version=Git commit: {STABLE_GIT_COMMIT}. Built at: {DATE}
[2019-10-03 12:40:27]  INFO registry: Starting 8 services: [*p2p.Service *powchain.Service *operations.Service *blockchain.Service *initialsync.InitialSync *sync.RegularSync *rpc.Service *prometheus.Service]
[2019-10-03 12:40:27]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=wss://goerli.prylabs.net/websocket
[2019-10-03 12:40:27]  INFO rpc: RPC-API listening on port port=:4000
[2019-10-03 12:40:27]  WARN rpc: You are using an insecure gRPC connection! Provide a certificate and key to connect securely
[2019-10-03 12:40:27]  INFO prometheus: Collecting metrics at endpoint endpoint=:8080
[2019-10-03 12:40:27]  INFO blockchain: Waiting to reach the validator deposit threshold to start the beacon chain...
[2019-10-03 12:40:28]  INFO p2p: Node started p2p server multiAddr=/ip4/192.168.1.136/tcp/13000/p2p/16Uiu2HAkwJzj6TeN6Fae4pULurxM6EcLNHzbHDALrUQsRiLbyEyc
[2019-10-03 12:40:35]  INFO powchain: Minimum number of validators reached for beacon-chain to start ChainStartTime=2019-10-01 00:00:00 +0000 UTC
[2019-10-03 12:40:35]  INFO blockchain: Genesis time reached, starting the beacon chain
[2019-10-03 12:40:36]  INFO initial-sync: Starting initial chain sync...
[2019-10-03 12:40:36]  INFO initial-sync: Waiting for enough peer handshakes before syncing handshakes=0/1
[2019-10-03 12:40:41]  INFO initial-sync: Waiting for enough peer handshakes before syncing handshakes=0/1
[2019-10-03 12:40:46]  INFO initial-sync: Waiting for enough peer handshakes before syncing handshakes=0/1
[2019-10-03 12:40:51]  INFO initial-sync: Waiting for enough peer handshakes before syncing handshakes=0/1
[2019-10-03 12:40:56]  INFO initial-sync: Waiting for enough peer handshakes before syncing handshakes=0/1
[2019-10-03 12:41:01]  INFO initial-sync: Waiting for enough peer handshakes before syncing handshakes=0/1
[2019-10-03 12:41:06]  INFO initial-sync: Waiting for enough peer handshakes before syncing handshakes=0/1
[2019-10-03 12:41:11]  INFO initial-sync: Waiting for enough peer handshakes before syncing handshakes=0/1
[2019-10-03 12:41:14]  INFO p2p: New peer connected peer=16Uiu2HAmHNTSqL3FaCyQvHfbBKbFk1AFWxbYxQZoyqog4VM1pxCb
[2019-10-03 12:41:14]  INFO p2p: New peer connected peer=16Uiu2HAm2sMFM6s5kt33FtovENpxaf3ENKyjvPkHcB7zZDuNtvQu
[2019-10-03 12:41:14]  INFO p2p: New peer connected peer=16Uiu2HAmTWDtrRezvgPg1jhCFaxMTeEYVCSxtTfKmVP4GsqSZBA3
[2019-10-03 12:41:14]  INFO p2p: New peer connected peer=16Uiu2HAmMSHjTKWhtPm23qp3b86nEw4C6Qv4pfzMqrrP6iGGSf5m
[2019-10-03 12:41:14]  INFO p2p: New peer connected peer=16Uiu2HAmCixeWr8MyXx1FuA5dawZTQFQbF517hfNArM6rpEsxqPu
[2019-10-03 12:41:16]  INFO initial-sync: Waiting for enough peer handshakes before syncing handshakes=5/1
[2019-10-03 12:41:17]  INFO initial-sync: Processing block 2/36412 - estimated time remaining 202h16m40s blocksPerSecond=0.1 peers=5/5
panic: could not process block from fork choice service: pre state of slot 2 does not exist

goroutine 213 [running]:
github.com/prysmaticlabs/prysm/beacon-chain/sync/initial-sync.(*InitialSync).Start(0xc00060fdd0)
    beacon-chain/sync/initial-sync/service.go:107 +0x98d
created by github.com/prysmaticlabs/prysm/shared.(*ServiceRegistry).StartAll
    shared/service_registry.go:44 +0x23e

github.com/prysmaticlabs/prysm/beacon-chain/sync/initial-sync.(*InitialSync).Start(0xc00060fdd0)
    beacon-chain/sync/initial-sync/service.go:107 +0x98d
created by github.com/prysmaticlabs/prysm/shared.(*ServiceRegistry).StartAll
    shared/service_registry.go:44 +0x23e

I have tried several times to check if I was lucky with different peers without luck so far...

[jamesthesken]

I also have the same issue @santiagorp after a fresh clone and build using:

• Ubuntu 16.04
• Bazel v1.0.0
• Golang 1.11.1 linux/amd64

[Skretch]

Have this same issue:

panic: could not process block from fork choice service: pre state of slot 97393 does not exist

goroutine 185 [running]:
github.com/prysmaticlabs/prysm/beacon-chain/sync/initial-sync.(*InitialSync).Start(0xc000af1d40)
        beacon-chain/sync/initial-sync/service.go:106 +0x97d
created by github.com/prysmaticlabs/prysm/shared.(*ServiceRegistry).StartAll
        shared/service_registry.go:44 +0x23e

panic: could not process block from fork choice service: pre state of slot 97444 does not exist

goroutine 183 [running]:
github.com/prysmaticlabs/prysm/beacon-chain/sync/initial-sync.(*InitialSync).Start(0xc000d24780)
        beacon-chain/sync/initial-sync/service.go:106 +0x97d
created by github.com/prysmaticlabs/prysm/shared.(*ServiceRegistry).StartAll
        shared/service_registry.go:44 +0x23e



md5-53831b1cbab828dcac99eade77d50766



panic: could not process block from fork choice service: pre state of slot 97493 does not exist

goroutine 190 [running]:
github.com/prysmaticlabs/prysm/beacon-chain/sync/initial-sync.(*InitialSync).Start(0xc000ce7b00)
        beacon-chain/sync/initial-sync/service.go:106 +0x97d
created by github.com/prysmaticlabs/prysm/shared.(*ServiceRegistry).StartAll
        shared/service_registry.go:44 +0x23e

It gets a little further every time a try to sync

[pjindal]

Having this issue as well

[prestonvanloon]

I believe this was indeed resolved by #3862. There are still a few pre-state issues that will hopefully be resolved by #3873.