Lighthouse: Out of sync after BlockProcessingFailure on Pyrmont

[plamarque]

Description

Beacon node fell out of sync on pyrmont and does not seem to catch up.

Version

lighthouse --version Lighthouse v0.3.5-7e4ee58 BLS Library: blst
running on ARM64 Ubuntu 20.04

Present Behaviour

Logs indicate :

WARN BlockProcessingFailure outcome: MissingBeaconBlock(0xebe21a8eae3592c98ea9f7d77b2888c88db9372c507037f2d93e7f349e7e14bf), msg: unexpected condition in processing block.

After a restart this error is shown in the logs :
WARN BlockProcessingFailure outcome: DBError(DBError { message: "Error { message: "Corruption: bad block type" }" }), msg: unexpected condition in processing block.

Expected Behaviour

node would remain in sync

Steps to resolve

not found. resyncing from scratch.

[paulhauner]

DBError(DBError { message: "Error { message: "Corruption: bad block type" }" })

This looks to me like the database on-disk is corrupt. Has this only happened once @plamarque?

I've never seen this before and if it has only happened once I would be tempted to say its a hardware/OS error that is not caused by LH and is not something that LH can feasibly tolerate. As such, I'd be tempted to close this as a "not related to lighthouse" issue.

However, first I'd be keen to hear if it's happened multiple times and perhaps see what @michaelsproul thinks.

[michaelsproul]

This is an obscure LevelDB error. I can't find much info online except:

• https://github.com/google/leveldb/issues/611
• https://github.com/Nexus-Mods/Vortex/issues/2531

The LevelDB devs seem to think it's most likely caused by file-system corruption, putting it out of scope not just for Lighthouse but also LevelDB itself :sweat_smile:

It might be interesting to run a SMART self-test on your disk to check that it's not failing. A fsck of your filesystem might also reveal something. I guess you're using ext4?

[plamarque]

It happened just once during my first attempt to sync with pyrmont. Nothing bad happened to the server (no hard reboot for example). It was just syncing happily. I reported here as I was initially suspecting this might be a rare bug related to the new DB compaction features. To work around, I renamed the DB folder and tried another full sync and it seems to work well now.

I'm using ext 4, on an Ubuntu VM. I'll try to run fsck and report back here.

[lightclient]

I also encountered some db corruption today. I deleted the db and am syncing from scratch. I was running 5828ff12, and upgraded to v1.0 to be safe. The corruption appears to have come from regular crashes due to tokio-runtime segfaults:

tokio-runtime-w[3460]: segfault at 470ba01feea ip 00005641f1424f1b sp 00007f59e8ee21f8 error 6 in lighthouse[5641ef75e000+1d9e000]