Prysm: Beacon Node Stuck in Sync

[rauljordan]

Our beacon nodes frequently get stuck when syncing with the testnet from blocks 217 - 256. The node endlessly requests that range of blocks and never seems to move past it, while CPU usage is close to 100%. @nisdas hypothesized there is some issue with bolt being resource constrained while in a docker environment due to how it does MMAP system calls and how it allocates data. Changing boltDB.AllocSize = 8 * 1024 * 1024 and clearing a node DB fixed the issue for me locally and was able to sync. We believe this is related to the root cause of what triggered #4476.

[prestonvanloon]

This is still reproducible after #4485 :(

[prestonvanloon]

Example logs

[2020-01-10 12:14:54]  INFO initial-sync: Processing block 504/3674 - estimated time remaining 3s blocksPerSecond=880.7 peers=8/12
[2020-01-10 12:14:54]  INFO initial-sync: Processing block 505/3674 - estimated time remaining 3s blocksPerSecond=880.8 peers=8/12
[2020-01-10 12:14:54]  INFO initial-sync: Processing block 506/3674 - estimated time remaining 3s blocksPerSecond=880.8 peers=8/12
[2020-01-10 12:14:54]  INFO initial-sync: Processing block 507/3674 - estimated time remaining 3s blocksPerSecond=880.9 peers=8/12
[2020-01-10 12:14:54]  INFO initial-sync: Processing block 508/3674 - estimated time remaining 3s blocksPerSecond=880.9 peers=8/12
[2020-01-10 12:14:54]  INFO initial-sync: Processing block 509/3674 - estimated time remaining 3s blocksPerSecond=881.0 peers=8/12
[2020-01-10 12:14:54]  INFO initial-sync: Processing block 510/3674 - estimated time remaining 3s blocksPerSecond=881.0 peers=8/12
[2020-01-10 12:14:54]  INFO initial-sync: Processing block 511/3674 - estimated time remaining 3s blocksPerSecond=881.0 peers=8/12
[2020-01-10 12:14:54]  INFO initial-sync: Processing block 512/3674 - estimated time remaining 3s blocksPerSecond=881.1 peers=8/12

[prestonvanloon]

What can't be seen in the logs above are at DEBUG level. If a block can't be processed because it's parent is not in the DB, that block is skipped and we move onto the next one. If we failed to process all of the blocks, we query peers again and start over.

One of our current theories right now are that Prysm is somehow generating a genesis block with the wrong state root. From what we noticed, the genesis state in all nodes is correct, but blocks fail to process due to an invalid genesis block. In fact, this is what stalled the start of the testnet yesterday.

We have also observed that this incident is less likely to occur in a less constrained environment, i.e. not running in docker. If you can, run from bazel. Although we run docker containers for our own nodes, we have not experienced this issue since we wiped databases yesterday after testnet genesis.

[torfbolt]

Can confirm, whenever I start the client it gets stuck trying to sync the same ~100 blocks over and over.

Rarely it succeeds with a block range and then gets immediately stuck again with the next couple of blocks. So for me this bug makes it impossible to run the beacon client.

[Domintek]

I can also confirm that the client gets stuck trying to sync the same blocks between 217 and 256. I am running in docker and unable to use bazel. I have repeated this scenario multiple times over the last three days with no change. Suggestions?