Lighthouse: Handle RPC rate limits gracefully during sync

[AgeManning]

Description

In scenario's with smaller testnets, it can be the case that Lighthouse processes blocks and syncs too quickly that we hit other client's RPC rate limits. In the limiting case of processing blocks and receiving blocks instantly, we can hit every client's RPC rate limit.

This should only happen during syncing. As the current sync implementation is a giant state-machine, we currently don't have a mechanism in which to stall or wait a period of time before requesting further batches from a specific peer. Furthermore, RPC rate-limits are not spec'd and each client may have their own changing response to being a rate limit.

It might be a nice feature to acknowledge Lighthouse's and other clients rate limit messages and develop a mechanism that tracks rate limits for peers during sync and waits for the limit to be surpassed before requesting batches from a peer (if all peers in chain pool have requested us to be rate-limited).

Having this feature, will prevent us from banning all other peers in a small testnet due to them rate limiting us.

[divagant-martian]

this sounds like something I would like to work on (as soon as dispatch my current work)

[divagant-martian]

Starting with this one next week

[AgeManning]

When you get back to this. I've been having a chat with a few people and there might a new direction to handle this across clients. Lets have chat when you're back on this :)

[paulhauner]

@AgeManning feel free change this to A0 if you see fit.

[divagant-martian]

Revisiting this. I would prefer to use our own calculated rate limits. My reasoning is that rate limits from other peers are not to be trusted. If we realize a request would hit our own rate limits we can pick another peer.

Related to this, we could change our rate limits to allow bursts in all protocols. For instance, we have a 1 every 5secs rate limit for Ping. A peer that got slightly slowed down might hit this limits but recover quickly. By allowing 2pings every 10 seconds we still maintain the same average rate limit without punishing those peers.