Lighthouse: Restart after power outage throws validator into a boot-up loop due to existing lock file.

[judasbenhur]

Description

I just had a short power outage. After power was back the beacon node runs normally while validator client gets stuck in a init loop with the following log:

Nov 30 08:26:37 ethnode-135d9bfbd lighthouse[4671]: Nov 30 08:26:37.445 INFO Lighthouse started                      version: Lighthouse/v1.0.0-c6baa0e
Nov 30 08:26:37 ethnode-135d9bfbd lighthouse[4671]: Nov 30 08:26:37.445 INFO Configured for testnet                  name: mainnet
Nov 30 08:26:37 ethnode-135d9bfbd lighthouse[4671]: Nov 30 08:26:37.445 WARN The mainnet specification is being used. This not recommended (yet).
Nov 30 08:26:37 ethnode-135d9bfbd lighthouse[4671]: Nov 30 08:26:37.449 INFO Starting validator client               validator_dir: "/var/lib/lighthouse/validators", beacon_node: http://localhost:5052/
Nov 30 08:26:37 ethnode-135d9bfbd lighthouse[4671]: Nov 30 08:26:37.452 INFO Completed validator discovery           new_validators: 0
Nov 30 08:26:39 ethnode-135d9bfbd lighthouse[4671]: Nov 30 08:26:39.170 ERRO Failed to initialize validator          validator: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, error: LockfileExists("/var/lib/lighthouse/validators/xxxxxxxxxxxxxxxxxxxxxxx/keystore-m_12381_3600_0_0_0-1606216636.json.lock")
Nov 30 08:26:39 ethnode-135d9bfbd lighthouse[4671]: Nov 30 08:26:39.170 CRIT Failed to start validator client        reason: Unable to initialize validators: LockfileExists("/var/lib/lighthouse/validators/xxxxxxxxxxxxxxxxxxxxxx/keystore-m_12381_3600_0_0_0-1606216636.json.lock")
Nov 30 08:26:39 ethnode-135d9bfbd lighthouse[4671]: Nov 30 08:26:39.170 INFO Internal shutdown received              reason: Failed to start validator client
Nov 30 08:26:39 ethnode-135d9bfbd lighthouse[4671]: Nov 30 08:26:39.170 INFO Shutting down..
Nov 30 08:26:39 ethnode-135d9bfbd systemd[1]: lighthouse-validator.service: Succeeded.
Nov 30 08:26:44 ethnode-135d9bfbd systemd[1]: lighthouse-validator.service: Scheduled restart job, restart counter is at 139.

Version

v1.0.0-c6baa0e

Present Behaviour

Validator client gets stuck in an infinite loop trying to initialize.

Expected Behaviour

There should be an automatic way handle lockfiles when the system didn't shut down gracefully. From power outage or other issues.

Steps to resolve

Manually removing the lockfile?

[AgeManning]

Use the command lighthouse vc --delete-lockfiles to resolve this

[judasbenhur]

thanks for the fast response. Is this something I can keep in the boot config for better unattended stability or is this not recommended?

[AgeManning]

It's not recommended. The lockfiles exist to prevent multiple validators running at once which can cause slashable offenses.

We have these here to protect your validators from being slashed. You can add it to your start up scripts but you may slash your validators if you accidentally run the application twice or have two instances running.

[michaelsproul]

Upgrading to v1.0.2 will fix this entirely. Lockfiles are (safely) no longer respected across power outages or sudden crashes

[judasbenhur]

ok got it. So treat it as a manual process until future fix. Thanks everyone.

[michaelsproul]

v1.0.2 is already released, you can download it here: https://github.com/sigp/lighthouse/releases/tag/v1.0.2

The lock file thing was fixed in v1.0.1 too

[judasbenhur]

Thank you Michael!