Solana: Large amounts of traffic are being sent to offline nodes

[leoluk]

Problem

After shutting down a node, the remainder of the cluster keeps sending traffic. This has been the source of frequent complaints by users who ran Solana on a home connection, shut it down, and then continued to get DDoSed by the cluster.

Proposed Solution

Stop sending traffic to nodes that aren't up-to-date in gossip.

This will only help with nodes that are fully offline. Congestion control is needed as well: #12410

A malicious attacker could still create fake entries since the gossip IP isn't authenticated via a three-way handshake that proves that there's a valid return path, see https://github.com/solana-labs/solana/issues/9491.

[leoluk]

Did a little experiment and shut down our TdS node (which has average stake, like everyone else). Traffic spiked to >300 Mbps of sustained UDP traffic to 8000/udp, which is our gossip port:

300 Mbps is enough to kill many residential broadband connections, and there were no signs of it stopping until I turned the node back on after 20 minutes. It's easy to mistake such a spike of UDP traffic, especially to a common port like 8000/udp, for a DDoS attack. What's worse is that egress traffic from the node stops at the same time, so it looks like a _successful_ attack at that :-)

In fact, there was recent case of a validator's Hetzner server getting locked for a suspected DDoS attack on another validator.

[MBGBuzzer]

I confirm that our server received a block from the hardware provider Hetzner, said that they blocked outgoing traffic and will not unblock it until we fix this situation. There was a clear suspicion of a DDoS attack. No explanation that this is legitimate traffic, the reason they are not interested, the only fix.

From this we can say that this is the narrowest point of the network, attackers are not asleep, and if this happens again, it will cause a big problem for the network and validators.
4BLOCK.TEAM

[urb4n-thr34t]

Hello!
Couple days ago I have recieved the complaint from Hetzner about the outgoing attack from my server.
Host bans server due to outgoing traffic from port 8001/udp.
As you can see it was just a regular solana traffic.

We know, when solana node is down, incoming udp traffic doesn't stop, but become even more.
So if IP will be used by someone else, it will still recieve huge amount of traffic, which could be recognized as DoS attack,
and the complaints potentialy could be sent to all cluster validators.
As I know, I am not the one who have received the complaint about this server.

I think it's a critical to implement some mechanism to set node state to offline to stop unnecessary traffic after node's shut down, overwise it could lead to disaster.
At the current state someone could potentialy attack entire cluster by starting hundreds of VPS with solana at different locations, then shut them down.
Validators will send to these dead nodes UDP spam, and complaints from providers won't keep you waiting long.

You can even try to snipe a top-stake validators a with manual complaints (IP of all nodes are openly visible).

[pkrasam]

A call to/for the network domain security specialists ...how might one go about solving for such an issue that could potentially become a network halter ?!

Are there any temporary workarounds we can put in place ...while the long-term solution is being worked out ?!

[mvines]

Workaround: When your staked node is shut down, run this command for ~1 minute to ensure the cluster receives an update to your IP address marking it as invalid.

$ solana-gossip spy --identity ~/validator-keypair.json --gossip-host 0.0.0.0 --entrypoint testnet.solana.com:8001

[leoluk]

We're still seeing plenty of excess traffic even after waiting 20 minutes:

Much less but I think the only acceptable amount of traffic for an offline node is close to zero.

(traffic from validators < 1.3.15 has been filtered from the graph)

[behzadnouri]

Is it possible to check if this is still an issue even after https://github.com/solana-labs/solana/pull/12674?
Note that staked offline node should still receive some amount of traffic (but much less than before) in order to mitigate eclipse attack.