Lighthouse: If beacon is restarted after genesis state has been generated but before genesis event, it will fail.

[yorickdowne]

Description

• LH 0.2.12 on spadina
• infura goerli source

LH beacon was running when genesis state was reached. LH beacon was then restarted, before genesis event. This has LH beacon count up to genesis state again and try to generate genesis state again, which fails with error that pubkey_cache.ssz already exists.

beacon_1            | Sep 28 17:09:14.222 INFO Genesis ceremony complete               genesis_time: 1601380813, genesis_validators: 2856, service: beacon
beacon_1            | Sep 28 17:09:14.281 INFO Block production enabled                method: json rpc via http, endpoint: https://goerli.infura.io/v3/USERID
beacon_1            | Sep 28 17:09:14.300 CRIT Error connecting to eth1 node. Please ensure that you have an eth1 http server running locally on http://localhost:8545 or pass an external endpoint using `--eth1-endpoint <SERVER-ADDRESS>`. Also ensure that `eth` and `net` apis are enabled on the eth1 http server, warning: BLOCK PROPOSALS WILL FAIL WITHOUT VALID ETH1 CONNECTION, service: beacon
beacon_1            | Failed to start beacon node: Failed to build beacon chain: Unable to init validator pubkey cache: ValidatorPubkeyCacheFileError("Persistence file already exists: \"/var/lib/lighthouse/beacon/pubkey_cache.ssz\"")
beacon_1            | 2020/09/28 17:09:14 Command exited with error: exit status 1

[AgeManning]

Woops, wrong issue.

[michaelsproul]

Will be fixed for-real by #1686, see that PR for root cause analysis

[yorickdowne]

This persists in 0.2.13 - I'm assuming #1686 did not make it into 0.2.13

beacon_1            | Sep 29 10:43:06.851 ERRO Failed to update eth1 cache             error: Failed to update eth1 cache: GetBlockNumberFailed("Request failed: reqwest::Error { kind: Request, url: \"http://geth:8545/\", source: hyper::Error(Connect, ConnectError(\"dns error\", Custom { kind: Interrupted, error: JoinError::Cancelled })) }"), retry_millis: 7000, service: eth1_rpc
beacon_1            | Failed to start beacon node: Failed to build beacon chain: Unable to init validator pubkey cache: ValidatorPubkeyCacheFileError("Persistence file already exists: \"/var/lib/lighthouse/beacon/pubkey_cache.ssz\"")

[michaelsproul]

No, as a temporary workaround you should delete the ~/.lighthouse/beacon directory and restart the node (and let it start without killing it)