Lighthouse: Beacon node is not able to produce a block: NoEth1ChainConnection

[q9f]

Description

I configured a custom testnet with custom deposit contract and try to run my own beacon chain. Testnet config: https://gist.github.com/q9f/d6eea3ea3356e41bde81864143284ce9

I made 16 validators and deposits on Görli testnet but my beacon node is not able to create blocks.

Version

Lighthouse testnet5 branch.

~/.opt/lighthouse testnet5*
❯ git log -n1
commit 90e456d845bc4b9964f906be59e7a523c998b5c8 (HEAD -> testnet5, origin/testnet5)
Merge: 45e85c8e 394de218
Author: Paul Hauner <[email protected]>
Date:   Thu Mar 19 00:30:37 2020 +1100

    Merge branch 'eth1-votes-tree-hash' into testnet5

~/.opt/lighthouse testnet5*
❯ rustc --version
rustc 1.41.1 (f3e1a954d 2020-02-24)

Present Behaviour

I run a local Görli Geth node:

geth --goerli \
  --rpc --rpcport 9779 --rpcapi admin,debug,eth,miner,net,personal,txpool,web3 \
  --allow-insecure-unlock --port 31537

I run a local beacon chain Lighthouse node:

lighthouse b --testnet-dir ~/.lighthouse/schlesi \
  --spec mainnet --eth1-endpoint http://127.0.0.1:9779 --http

I run my 16 validators locally:

lighthouse vc --allow-unsynced

The beacon node complains that there is no eth1 connection:

Mar 19 10:29:12.427 WARN Skipping more than an epoch             request_slot: 5792, head_slot: 0, service: beacon
Mar 19 10:29:12.481 WARN Skipping more than an epoch             request_slot: 5824, head_slot: 0, service: beacon
Mar 19 10:29:18.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 19 10:29:18.001 INFO Syncing                                 est_time: --, speed: 0.00 slots/sec, distance: 5846 slots (19 hrs 29 mins), peers: 0, service: slot_notifier
Mar 19 10:29:24.108 WARN Skipping more than an epoch             request_slot: 5792, head_slot: 0, service: beacon
Mar 19 10:29:24.158 WARN Skipping more than an epoch             request_slot: 5846, head_slot: 0, service: beacon
Mar 19 10:29:28.097 INFO Attestation from local validator        slot: 5847, index: 0, source: 0, target: 0, service: http
Mar 19 10:29:29.006 ERRO Error whilst producing block            error: NoEth1ChainConnection, service: http
Mar 19 10:29:29.008 WARN Skipping more than an epoch             request_slot: 5824, head_slot: 0, service: beacon
Mar 19 10:29:30.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 19 10:29:30.001 INFO Syncing                                 est_time: --, speed: 0.00 slots/sec, distance: 5847 slots (19 hrs 29 mins), peers: 0, service: slot_notifier
Mar 19 10:29:36.108 WARN Skipping more than an epoch             request_slot: 5847, head_slot: 0, service: beacon
Mar 19 10:29:36.112 WARN Skipping more than an epoch             request_slot: 5792, head_slot: 0, service: beacon
Mar 19 10:29:36.176 WARN Skipping more than an epoch             request_slot: 5824, head_slot: 0, service: beacon
Mar 19 10:29:41.972 ERRO Error whilst producing block            error: NoEth1ChainConnection, service: http
Mar 19 10:29:42.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 19 10:29:42.002 INFO Syncing                                 est_time: --, speed: 0.00 slots/sec, distance: 5848 slots (19 hrs 30 mins), peers: 0, service: slot_notifier

The validator node relays the same issue:

Mar 19 10:29:12.384 INFO Starting validator client               datadir: "/home/user/.lighthouse/validators", beacon_node: http://localhost:5052/
Mar 19 10:29:12.404 INFO Connected to beacon node                version: Lighthouse/v0.1.0-unstable/x86_64-linux
Mar 19 10:29:12.408 INFO Genesis has already occurred            seconds_ago: 70152
Mar 19 10:29:12.423 INFO Loaded validator keypair store          voting_validators: 16
Mar 19 10:29:28.098 INFO Successfully published attestation      slot: 5847, committee_index: 0, head_block: 0xd0ad…1b0d, signatures: 1, service: attestation
Mar 19 10:29:29.008 CRIT Error whilst producing block            message: Error from beacon node when producing block: DidNotSucceed { status: 500, body: "Beacon node is not able to produce a block: NoEth1ChainConnection" }, service: block
Mar 19 10:29:30.001 INFO All validators active                   slot: 5847, epoch: 182, total_validators: 16, active_validators: 16, proposers: 16, service: notifier
Mar 19 10:29:41.972 CRIT Error whilst producing block            message: Error from beacon node when producing block: DidNotSucceed { status: 500, body: "Beacon node is not able to produce a block: NoEth1ChainConnection" }, service: block
Mar 19 10:29:42.001 INFO All validators active                   slot: 5848, epoch: 182, total_validators: 16, active_validators: 16, proposers: 16, service: notifier
Mar 19 10:29:52.077 INFO Successfully published attestation      slot: 5849, committee_index: 0, head_block: 0xd0ad…1b0d, signatures: 1, service: attestation
Mar 19 10:29:54.001 INFO All validators active                   slot: 5849, epoch: 182, total_validators: 16, active_validators: 16, proposers: 16, service: notifier
Mar 19 10:29:54.312 CRIT Error whilst producing block            message: Error from beacon node when producing block: DidNotSucceed { status: 500, body: "Beacon node is not able to produce a block: NoEth1ChainConnection" }, service: block
Mar 19 10:30:05.283 CRIT Error whilst producing block            message: Error from beacon node when producing block: DidNotSucceed { status: 500, body: "Beacon node is not able to produce a block: NoEth1ChainConnection" }, service: block
Mar 19 10:30:06.001 INFO All validators active                   slot: 5850, epoch: 182, total_validators: 16, active_validators: 16, proposers: 16, service: notifier

Expected Behaviour

The beacon node should connect to eth1 and produce blocks. The beacon node was able to fetch genesis from the local Görli node, so there _must_ be an eth1 connection.

Steps to resolve

I'm unsure why this is happening.

[pawanjay176]

Hey @q9f Didn't replicate your testnet setup, but you might be missing the --eth1 flag.

lighthouse b --testnet-dir ~/.lighthouse/schlesi \
  --spec mainnet --eth1 --eth1-endpoint http://127.0.0.1:9779 --http

[pawanjay176]

@paulhauner seems redundant to have to specify the --eth1 flag if we are already specifying an endpoint with --eth1-endpoint. We can set sync_eth1_chain to true in both the cases I think.

[q9f]

Is there any case where you do not want to connect to eth1?

[pawanjay176]

If you are just syncing the beacon chain and don't have any validators connected, you don't need to connect to eth1.

[q9f]

I see. I added the --eth1 flag but I can no longer restart my validators: #938

[paulhauner]

@paulhauner seems redundant to have to specify the --eth1 flag if we are already specifying an endpoint with --eth1-endpoint. We can set sync_eth1_chain to true in both the cases I think.

Good point @q9f. This is addressed here: https://github.com/sigp/lighthouse/pull/941

[paulhauner]

I believe this is closed.. But I'm not sure, so feel free to reopen :)