Prysm: "eth1 node using incorrect chain id, 0 != 1" when geth is not fully synced

[victorelec14]

🐞 Bug Report

Description

beacon-chain does not start with a private node ETH1 (Docker)

Has this worked before in a previous version?

i dont know

🔬 Minimal Reproduction

🔥 Error

time="2020-11-23 23:31:55" level=error msg="Could not connect to powchain endpoint" error="could not dial eth1 nodes: eth1 node using incorrect chain id, 0 != 1" prefix=powchain

time="2020-11-23 23:31:54" level=warning msg="Running on ETH2 Mainnet" prefix=flags
time="2020-11-23 23:31:54" level=info msg="Using "max_cover" strategy on attestation aggregation" prefix=flags
time="2020-11-23 23:31:54" level=info msg="Checking DB" database-path="/data/beaconchaindata" prefix=node
time="2020-11-23 23:31:55" level=warning msg="No bootstrap addresses supplied" prefix=p2p
time="2020-11-23 23:31:55" level=info msg="Deposit contract: 0x00000000219ab540356cbb839cbe05303d7705fa" prefix=node
time="2020-11-23 23:31:55" level=info msg="Waiting for state to be initialized" prefix=initial-sync
time="2020-11-23 23:31:55" level=info msg="Starting beacon node" prefix=node version="Prysm/v1.0.0-beta.2/f361450e8da18033f5458b018e3e6784295f50ca. Built at: 2020-11-14 02:35:28+00:00"
time="2020-11-23 23:31:55" level=info msg="Waiting to reach the validator deposit threshold to start the beacon chain..." prefix=blockchain
time="2020-11-23 23:31:55" level=info msg="gRPC server listening on port" address="127.0.0.1:4000" prefix=rpc
time="2020-11-23 23:31:55" level=warning msg="You are using an insecure gRPC server. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc" prefix=rpc
time="2020-11-23 23:31:55" level=info msg="Starting JSON-HTTP API" address="127.0.0.1:3500" prefix=gateway
time="2020-11-23 23:31:55" level=error msg="Could not connect to powchain endpoint" error="could not dial eth1 nodes: eth1 node using incorrect chain id, 0 != 1" prefix=powchain

## 🌍 Your Environment **Operating System:**

  
   docker -v
Docker version 19.03.8, build afacb8b7f0

  

Eth1 Node: (Docker):

  
docker run -p 127.0.0.1:8545:8545 -p 30303:30303 -v $(HOME)/eth2/node:/root ethereum/client-go --http -http.addr "0.0.0.0"

--or-
docker run -p 0.0.0.0:8545:8545 -p 30303:30303 -v $(HOME)/eth2/node:/root ethereum/client-go --http -http.addr "0.0.0.0"

  

Beacon: (Docker):

  

docker run -it -v$(HOME)/eth2/beacon/.eth2:/data -p 4000:4000 -p 13000:13000 -p 12000:12000/udp --name beacon-node \
  gcr.io/prysmaticlabs/prysm/beacon-chain:stable \
  --datadir=/data \
  --rpc-host=127.0.0.1 \
  --http-web3provider="http://192.168.178.170:8545"

  

What version of Prysm are you running? (Which release)

  
gcr.io/prysmaticlabs/prysm/beacon-chain:stable
Prysm/v1.0.0-beta.2/f361450e8da18033f5458b018e3e6784295f50ca
  

Anything else relevant (validator index / public key)?

[BiggT76]

I am also experiencing this problem.

[victorelec14]

I have tried with "--mainnet" which was added in v1.0.0-beta.2 but it does not work either, it is as if I always want to start in the testnet and eth1 is in mainnet.

With prymont and medal I have had no problems being testnets.

Cleardb too nothing

--clear-db

[BiggT76]

I've heard others having this problem, if I had to guess its because Prysm isn't ready for mainnet yet. Given all the big bold warnings on their website _not to run_ prysm on mainnet yet...

Hopefully someone "in the know" can offer more insight.

[prestonvanloon]

Can you try with RPC enabled in geth? --rpc

[victorelec14]

Can you try with RPC enabled in geth? --rpc

--http -http.addr are the new parameter, rpc be deprecated in future, but i get the same response with --rpc

The flag --rpc is deprecated and will be removed in the future, please use --http
WARN [11-23|23:07:53.386] The flag --rpcaddr is deprecated and will be removed in the future, please use --http.addr

[victorelec14]

--verbosity=debug

time="2020-11-24 22:14:53" level=warning msg="Running on ETH2 Mainnet" prefix=flags
time="2020-11-24 22:14:53" level=info msg="Using "max_cover" strategy on attestation aggregation" prefix=flags
time="2020-11-24 22:14:53" level=info msg="Checking DB" database-path="/data/beaconchaindata" prefix=node
time="2020-11-24 22:14:54" level=warning msg="No bootstrap addresses supplied" prefix=p2p
time="2020-11-24 22:14:54" level=debug msg="ECDSA private key generated" prefix=p2p
time="2020-11-24 22:14:54" level=info msg="Deposit contract: 0x00000000219ab540356cbb839cbe05303d7705fa" prefix=node
time="2020-11-24 22:14:54" level=info msg="Waiting for state to be initialized" prefix=initial-sync
time="2020-11-24 22:14:54" level=info msg="Starting beacon node" prefix=node version="Prysm/v1.0.0-beta.2/f361450e8da18033f5458b018e3e6784295f50ca. Built at: 2020-11-14 02:35:28+00:00"
time="2020-11-24 22:14:54" level=debug msg="Starting 9 services: [*p2p.Service *powchain.Service *attestations.Service *blockchain.Service *initialsync.Service *sync.Service *rpc.Service *gateway.Gateway *prometheus.Service]" prefix=registry
time="2020-11-24 22:14:54" level=debug msg="Starting service type *p2p.Service" prefix=registry
time="2020-11-24 22:14:54" level=debug msg="Starting service type *powchain.Service" prefix=registry
time="2020-11-24 22:14:54" level=debug msg="Starting service type *attestations.Service" prefix=registry
time="2020-11-24 22:14:54" level=debug msg="Starting service type *blockchain.Service" prefix=registry
time="2020-11-24 22:14:54" level=debug msg="Starting service type *initialsync.Service" prefix=registry
time="2020-11-24 22:14:54" level=debug msg="Starting service type *sync.Service" prefix=registry
time="2020-11-24 22:14:54" level=debug msg="Starting service type *rpc.Service" prefix=registry
time="2020-11-24 22:14:54" level=debug msg="Starting service type *gateway.Gateway" prefix=registry
time="2020-11-24 22:14:54" level=info msg="Waiting to reach the validator deposit threshold to start the beacon chain..." prefix=blockchain
time="2020-11-24 22:14:54" level=debug msg="Starting service type *prometheus.Service" prefix=registry
time="2020-11-24 22:14:54" level=info msg="Starting JSON-HTTP API" address="127.0.0.1:3500" prefix=gateway
time="2020-11-24 22:14:54" level=info msg="gRPC server listening on port" address="127.0.0.1:4000" prefix=rpc
time="2020-11-24 22:14:54" level=warning msg="You are using an insecure gRPC server. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc" prefix=rpc
time="2020-11-24 22:14:54" level=debug msg="Starting prometheus service" address="127.0.0.1:8080" prefix=prometheus
time="2020-11-24 22:14:54" level=error msg="Could not connect to powchain endpoint" error="could not dial eth1 nodes: eth1 node using incorrect chain id, 0 != 1" prefix=powchain

[MAI1O]

I am experiencing this same issue. Only difference is I'm not using docker. I've tried a number of things but nothing works. Really hoping somebody comes through on this. Sorry I can't contribute anything else.

[BiggT76]

I might have a clue:
Just confirmed that the issue appears to be related to how the wallet is created/imported. I know how two people (myself included) that created the keys on one machine but imported them into another. Did anyone else here do the same?

Reference: https://www.reddit.com/r/ethstaker/comments/k0cblo/can_anyone_help_could_not_connect_to_powchain/gdhx69i/?context=3

[prestonvanloon]

I might have a clue:
Just confirmed that the issue appears to be related to how the wallet is created/imported. I know how two people (myself included) that created the keys on one machine but imported them into another. Did anyone else here do the same?

Reference: https://www.reddit.com/r/ethstaker/comments/k0cblo/can_anyone_help_could_not_connect_to_powchain/gdhx69i/?context=3

This specific error doesn't have anything to do with importing validator wallets.

For some reason, your eth1 node is returning a chain ID of 0, which doesn't make much sense. ETH1 mainnet chain ID is 1.

Could you kindly run this and paste the results?

curl -X POST --data '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":67}' -H "Content-Type:application/json" http://192.168.178.170:8545

You should see

{"jsonrpc":"2.0","id":67,"result":"0x1"}

The result is the chain ID.

[BiggT76]

invalid content type, only application/json is supported

[prestonvanloon]

@BiggT76, sorry, I edited it. Please try again

[BiggT76]

{"jsonrpc":"2.0","id":67,"result":"0x0"} although when I did a geth dump it said my ChainID was 1

[BiggT76]

Beginning of geth config:
INFO [11-24|19:11:53.154] Maximum peer count ETH=50 LES=0 total=50 INFO [11-24|19:11:53.177] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory" INFO [11-24|19:11:53.200] Set global gas cap cap=25000000 [Eth] NetworkId = 1 SyncMode = "fast"

[prestonvanloon]

OK, we have identified the issue. Your geth node is not fully synced and for some reason is returning 0x0 for the chain ID.

[BiggT76]

That's strange...I'll double check that. I swore it was updated.
Thanks for your help @prestonvanloon , if the problem persists after full sync I'll provide an update.

[prestonvanloon]

@BiggT76 No problem at all! I'm going to see if we can file an issue in go-ethereum.

[prestonvanloon]

See: https://github.com/ethereum/go-ethereum/issues/20894

[prestonvanloon]

It should be fixed by https://github.com/ethereum/go-ethereum/pull/21686

[egk10]

my geth is synced whith a 1 block delay comparing on etherscan. is this consider UNSYNC ???

[BiggT76]

@egk10 It's my understanding that its still processing all of the Imported new state entries records that keep popping up. This was what I found on the geth FAQ page:

You are just done with the block download phase and still running the state downloads. You can see this yourself via the seemingly endless Imported state entries [...] stream of logs. You’ll need to wait that out too before your node comes truly online.

[karalabe]

@prestonvanloon This is not a bug. As long as the node only has the genesis block available, the chain ID is for all intents and purposes 0, because the notion of a chain id did not exist. Whether we should change the behavior is debatable (and debated a lot really), but it's not a bug, it's intended to work like this.

[victorelec14]

curl -X POST --data '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":67}' -H "Content-Type:application/json" http://192.168.178.170:8545
{"jsonrpc":"2.0","id":67,"result":"0x0"}

Get Status:

INFO [11-25|08:36:41.027] Imported new state entries               count=224  elapsed="3.3µs"      processed=90087688 pending=150604 trieretry=0   coderetry=0  duplicate=7083 unexpected=28991
INFO [11-25|08:36:41.037] Imported new state entries               count=362  elapsed="3.08µs"     processed=90088050 pending=156396 trieretry=0   coderetry=0  duplicate=7083 unexpected=28991
INFO [11-25|08:36:41.099] Imported new state entries               count=384  elapsed="2.922µs"    processed=90088434 pending=162536 trieretry=0   coderetry=0  duplicate=7083 unexpected=28991
INFO [11-25|08:36:41.113] Imported new state entries               count=498  elapsed="3.238µs"    processed=90088932 pending=170504 trieretry=0   coderetry=0  duplicate=7083 unexpected=28991
INFO [11-25|08:36:41.122] Imported new state entries               count=378  elapsed="2.883µs"    processed=90089310 pending=176552 trieretry=0   coderetry=0  duplicate=7083 unexpected=28991
INFO [11-25|08:36:42.343] Imported new state entries               count=384  elapsed="3.928µs"    processed=90089694 pending=182606 trieretry=0   coderetry=0  duplicate=7083 unexpected=28991
INFO [11-25|08:36:42.722] Imported new block headers               count=1    elapsed=4.931ms      number=11326506 hash="a729cf…4f692f"
INFO [11-25|08:36:48.875] Imported new block headers               count=2    elapsed=5.695ms      number=11326508 hash="3200b5…bf9854"
INFO [11-25|08:37:38.103] Imported new block headers               count=1    elapsed=5.365ms      number=11326509 hash="bf4d3b…2715ae"
INFO [11-25|08:37:38.903] Downloader queue stats                   receiptTasks=0  blockTasks=0  itemSize=189.02KiB throttle=347
INFO [11-25|08:37:41.201] Imported new block headers               count=1    elapsed=5.115ms      number=11326510 hash="00d7a5…367f4c"

it seems to me that there is something wrong with geht? , it is the second time I synchronize the chain

[karalabe]

@victorelec14 There are 650 million state entries. You'll need to wait it out

[victorelec14]

@prestonvanloon @karalabe

What node do you recommend me to use?

• go-ethereum (Geth)
• parity
• openethereum
• nethermind

should the chain always sync in full? , could it be the fast?

Thank you

[victorelec14]

confirmed, I'm on the same BlockNumber as Etherscan, but the "state entries" haven't finished importing.

Maybe you could put an info message in Prism or check the status of the chain and show it?

thanks

[karalabe]

@victorelec14 You are not on the same block. As long as the states didn't download all the states, you are essentially on block 0. It's a painful process and takes a lot of time, sorry about that. There's a new sync protocol that will solve this, but it's not something served yet by nodes, so it won't help you yet.

[nisdas]

@victorelec14 Keep a lookout for this PR
https://github.com/prysmaticlabs/prysm/pull/7895

It will be merged in the next release. Unfortunately the current error log is confusing as you have noticed. With this PR it will inform you first if your node is 'synced' or not. We performed the sync check later on in the initial startup routine, which as we have realised misses this edge case with the chain id.

[prestonvanloon]

Thanks for the insight @karalabe! PR #7895 will handle this more gracefully.

[prestonvanloon]

Expect this to be released in the next patch version v1.0.1. ETA Friday November 27, by end of day US/Pacific.