Prysm: Unable to process past logs query returned more than 10000 results

[prestonvanloon]

🐞 Bug Report

Description

When running a beacon node on pyrmont with an infura endpoint, the powchain receives this error.

ERROR powchain: Unable to process past logs query returned more than 10000 results

### Has this worked before in a previous version? Yes, in other testnets. Although other testnets have not enjoyed such rapid deposits up to 100k. ## 🔬 Minimal Reproduction

beacon-chain --pyrmont --clear-db --http-web3provider=https://goerli.infura.io/...

## 🔥 Error

[2020-11-16 11:42:03]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/REDACTED
[2020-11-16 11:42:03] ERROR powchain: Unable to process past logs query returned more than 10000 results

## 🌍 Your Environment **Operating System:**

  
linux amd64
  

What version of Prysm are you running? (Which release)

  
v1.0.0-beta.3
  

Anything else relevant (validator index / public key)?

[prestonvanloon]

Note: this does not happen when running your own eth1 node. Infura is returning this error.

[fabdarice]

I can look into this. Is this a limitation from infura? or we are fetching too many blocks at once?
I faced similar issues in the past when I had to dynamically fetch N numbers of blocks back because we cannot predict how many events will be included in each block.
I think geth will crash around ~2.5K events.

[fabdarice]

@prestonvanloon browsing through my logs, seeing this as well with my local eth1 node.

time="2020-11-16 19:08:14" level=error msg="Unable to process past logs Post : context canceled" prefix=powchain
--

[xiaojian-zhou]

I switch to alchemy, and it looks fine for me

[jerryji]

I got the same error -- is it related to the 100,000 pre-deposit in Pyrmont? I wonder how the Prysm team runs its own Pyrmont beacon-chain

It's definitely an Infura issue as I've received the same error "RRO Failed to update eth1 cache error: Failed to update eth1 cache: GetDepositLogsFailed("Eth1 node returned error: {"code":-32005,"message":"query returned more than 10000 results"}"), retry_millis: 7000, service: eth1_rpc" from Lighthouse

But @fabdarice reported same issue with local geth. As many (most) ETH2 validators are running geth, it's going to be a tough time for them as well

[nisdas]

You can either run your own eth1 node or use another 3rd party eth1 provider. As for us, we have our goerli endpoint, which doesn't have any restrictions compared to infura.

https://goerli.prylabs.net/

[prestonvanloon]

@infura @timmyers @egalano
Any suggested workarounds or thoughts?
This seems to be an infura only issue, but may be pointing out an inappropriate use of the eth1 json-rpc API.

[prestonvanloon]

One user suggested to lower the request limit in #7857. Potentially, this could be configurable.

[jerryji]

Lighthouse did a similar commit -- https://github.com/sigp/lighthouse/commit/bcc7f6b143ccd46aa8e493e150af359b2d3639b8 , which works fine with Infura up to 150

[egalano]

Hey @prestonvanloon . This limit was put in place because certain queries can return huge amounts of data which caused instability both in our backend systems and also would force the client to deal with a huge chunk of data returned in the response. The JSON-RPC API doesn't prevent someone from asking for every event from the genesis block to the latest block which would timeout or error. Since there is no pagination in the JSON-RPC spec, our recommendation has been for devs to iterate over block ranges when requesting log data that exceeds the 10k limit. Some client libraries handle this for the user already.

Technically the ethereum clients don't limit response size like this, but we believe that they should. Some malformed getLogs queries can impact the stability of the node.

[vvavepacket]

I can confirm this is happening on Pyrmont, where if its a newly created beacon node, Infura or prysm is going to return results of more than 10000. But if you already have a beacon node running for some time, it doesn't happen. it seems its happening only for new folks wanting to run fresh validators

Verified /root/prysm/dist/beacon-chain-v1.0.0-beta.3-linux-amd64 has been signed by Prysmatic Labs.
Starting Prysm beacon-chain --http-web3provider=https://goerli.infura.io/v3/XXXXXXXXXXX --pyrmont
[2020-11-22 19:16:34]  WARN flags: Running on Pyrmont Testnet
[2020-11-22 19:16:34]  INFO flags: Using "max_cover" strategy on attestation aggregation
[2020-11-22 19:16:34]  INFO node: Checking DB database-path=/root/.eth2/beaconchaindata
[2020-11-22 19:16:34]  INFO node: Deposit contract: 0x8c5fecdc472e27bc447696f431e425d02dd46a8c
[2020-11-22 19:16:34]  INFO node: Starting beacon node version=Prysm/v1.0.0-beta.3/f361450e8da18033f5458b018e3e6784295f50ca. Built at: 2020-11-16 01:40:19+00:00
[2020-11-22 19:16:34]  INFO initial-sync: Waiting for state to be initialized
[2020-11-22 19:16:34]  INFO blockchain: Waiting to reach the validator deposit threshold to start the beacon chain...
[2020-11-22 19:16:34]  INFO gateway: Starting JSON-HTTP API address=127.0.0.1:3500
[2020-11-22 19:16:34]  INFO rpc: gRPC server listening on port address=127.0.0.1:4000
[2020-11-22 19:16:34]  WARN rpc: You are using an insecure gRPC server. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc
[2020-11-22 19:16:35]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/XXXXXX
[2020-11-22 19:16:35] ERROR powchain: Unable to process past logs query returned more than 10000 results
[2020-11-22 19:16:41]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/XXXXXX
[2020-11-22 19:16:41] ERROR powchain: Unable to process past logs query returned more than 10000 results
[2020-11-22 19:16:47]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/XXXXXX
[2020-11-22 19:16:48] ERROR powchain: Unable to process past logs query returned more than 10000 results
[2020-11-22 19:16:54]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/XXXXXX
[2020-11-22 19:16:54] ERROR powchain: Unable to process past logs query returned more than 10000 results
[2020-11-22 19:17:00]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/XXXXXX
[2020-11-22 19:17:00] ERROR powchain: Unable to process past logs query returned more than 10000 results
[2020-11-22 19:17:07]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/XXXXXX
[2020-11-22 19:17:07] ERROR powchain: Unable to process past logs query returned more than 10000 results
[2020-11-22 19:17:13]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/XXXXXX
[2020-11-22 19:17:13] ERROR powchain: Unable to process past logs query returned more than 10000 results
[2020-11-22 19:17:20]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/XXXXXX
[2020-11-22 19:17:20] ERROR powchain: Unable to process past logs query returned more than 10000 results
[2020-11-22 19:17:26]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/XXXXXX
[2020-11-22 19:17:26] ERROR powchain: Unable to process past logs query returned more than 10000 results
[2020-11-22 19:17:32]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/XXXXXX
[2020-11-22 19:17:33] ERROR powchain: Unable to process past logs query returned more than 10000 results
[2020-11-22 19:17:39]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/XXXXXX
[2020-11-22 19:17:39] ERROR powchain: Unable to process past logs query returned more than 10000 results
[2020-11-22 19:17:46]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/XXXXXX
[2020-11-22 19:17:46] ERROR powchain: Unable to process past logs query returned more than 10000 results
[2020-11-22 19:17:52]  INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/XXXXXX
[2020-11-22 19:17:52] ERROR powchain: Unable to process past logs query returned more than 10000 results

[edsonayllon]

+1

[Molkavi]

+1 on Windows, same repro...

[ChainlinkResources]

As @nisdas recommended:
You can either run your own eth1 node or use another 3rd party eth1 provider. As for us, we have our goerli endpoint, which doesn't have any restrictions compared to infura.

https://goerli.prylabs.net/

Using this endpoint worked great, no hiccups!

[vvavepacket]

Do you have an endpoint pointing to mainnet?

[image: Mailtrack]
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&
Sender
notified by
Mailtrack
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&
11/23/20,
08:55:40 PM

On Mon, Nov 23, 2020 at 8:55 PM Kirsten Imler [e-mail]
wrote:

As @nisdas https://github.com/nisdas recommended:
You can either run your own eth1 node or use another 3rd party eth1
provider. As for us, we have our goerli endpoint, which doesn't have any
restrictions compared to infura.

https://goerli.prylabs.net/

Using this endpoint worked great, no hiccups!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/prysmaticlabs/prysm/issues/7827#issuecomment-732535743,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB7JJWQLRPU2BIZF25P7UKLSRMHAPANCNFSM4TXTDAEA
.

[PJWilkinson]

Does it mean it will still sync (i.e. ignore the error) or would we need to switch away from Infura?

[jerryji]

FYI, I used my https://eth-goerli.alchemyapi.io/v2/ to get over the initial sync, then switch back to Infura (as alchemyapi.io free tier does not offer enough credits).

[PJWilkinson]

FYI, I used my https://eth-goerli.alchemyapi.io/v2/ to get over the initial sync, then switch back to Infura (as alchemyapi.io free tier does not offer enough credits).

Awesome - thanks @jerryji - didn't realise you could swap between them either.

[1hugoboss1]

I switch to alchemy, and it looks fine for me

Any tutorials on how to set up the beacon chain using Alchemy?

[vvavepacket]

Create an account at alchemy, get the URL and use that in your beacon node

On Tue, Nov 24, 2020, 11:06 AM 1hugoboss1 [e-mail] wrote:

I switch to alchemy, and it looks fine for me

Any tutorials on how to set up the beacon chain using Alchemy?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/prysmaticlabs/prysm/issues/7827#issuecomment-733075075,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB7JJWWZNXOQSNLYRGMMQP3SRPKZVANCNFSM4TXTDAEA
.

[prestonvanloon]

There are several 3rd party eth1 providers out there.

Take a look at https://ethereumnodes.com/. I am not sure if all of these support goerli or not and do your own due diligence as always.

Edit: we are working on a fix, but it is lower priority and will take some time.

[egalano]

@prestonvanloon do you think you'll be able to merge #7896 soon? Or do you need help from our team in getting this resolved. This close to Phase 0 launch, we want to be as helpful as possible to people in their preparations.

[prestonvanloon]

@egalano I've added a comment to that PR https://github.com/prysmaticlabs/prysm/pull/7896#issuecomment-733947363
Nishant had some feedback, I'll wait for them to respond before we continue forward on it. I think we can include this into the patch release later this week and revisit later, unless a graceful retry approach is easy to implement.

[emile2738]

A workaround that got me going after having the same issues was by first syncing with a free account at alchemy with --clear-db then reconnecting to local eth1 node.

[prestonvanloon]

When #7896 merged, please add --eth1-header-req-limit=150 to your beacon chain client.

This will be available in the patch release v1.0.1.

[sako013]

There are several 3rd party eth1 providers out there.

Take a look at https://ethereumnodes.com/. I am not sure if all of these support goerli or not and do your own due diligence as always.

Edit: we are working on a fix, but it is lower priority and will take some time.

i used Alchemy instead of Infura and it seems working but I could choose Eth2 in Infura,I am not sure about Alchemy is it the same!