Substrate: Deprecation of Sentry Nodes

[mxinden]

Summary

With the upcoming release of Substrate and Pokadot support for sentry nodes will be deprecated. We are currently planning on removing support for sentry nodes by October 2020. Operators protecting their validators via sentry nodes today should decommission their sentry nodes and ensure their validator nodes are publicly routable before support has been removed.

Status Quo

Today one way of operating validator nodes in a secure fashion involves running one or many sentry nodes in front of ones validators. As an example setup see the current Polkadot Secure Validator project. Sentry nodes, operating as full nodes, act as proxies for the validator nodes, thus offer the following security improvements:

• Sentry nodes serve as a DOS protection mechanism as multiple sentry nodes can handle a large amount of traffic for a single validator node. Sentry nodes operate on application layer. Thereby they can filter out bogus messages e.g. invalid block announcements.
• Sentry nodes don't have access to relevant private key material. This indirection protects the key material against certain classes of software exploits, though not the most severe exploits that lead to remote code execution.

While sentry nodes can improve validator security, there are multiple trade-offs involved.

• The notion of sentry nodes adds complexity to both the node implementation itself as well as the overall network topology. One example concerns any component using the DHT. When publishing a validators addresses the authority discovery module on a validator node can not do so directly, but has to forward the signed addresses to one of its sentry nodes for it to publish them on the DHT (See https://github.com/paritytech/substrate/issues/6264).
• The extra hop for all traffic destined for a validator behind sentry nodes adds latency. This latency is not to be confused with the latency a commodity layer-4 proxy would introduce. Instead, as the sentry node operates as a full node, the additional latency does not only involve packet forwarding or transport-layer-security de- and encryption, but also things like block validation.

Reasoning for Deprecation

While the complexity required to support sentry nodes is manageable for simple blockchains that do most if not all communication through a gossip network, this complexity increases heavily for more sophisticated network topologies like the one required for Polkadot.

One example of the additional complexity can be found within Polkadot. In order to support parachains in a scalable manner, one can not do all collator node to validator node communication through gossiping as it would overwhelm the network. Instead collator nodes of parachains need to be able to talk to validator nodes of the relaychain directly.

With sentry nodes in mind, validators would not be directly reachable, but instead only reachable through their sentry nodes. A validator would need to tell its sentry node to allow traffic from a specific collator before that collator could forward messages through the sentry node to the validator. Collators would need to discover not validators but the sentry nodes of those validators. ... For details on the Polkadot topology you can e.g. consult the Polkadot overview paper.

Taking all the above into account we have decided to deprecate support for sentry nodes. This decision might be revisited in the long term future, e.g. once the parachain protocols have stabilized.

Actions Required / Recommended

Required

All operators of validator nodes are required to make the TCP port of the P2P protocol of their validator nodes routable via the public internet. The TCP port of the RPC endpoint should stay unchanged and protected.

Recommended

While the P2P protocol port of a validator node needs to be publicly routable, one can still protect the endpoint on layer 4 (TCP) and downwards. Depending on your required security level you might want to put a mature TCP proxy in-front of your validator (e.g. Nginx). You can operate a stateful firewall yourself or use a hosted firewall / DOS protection service by your favorite cloud provider. You can consider reaching out to a large CDN. ...

Once supported, we recommend using remote signing, doing all relevant cryptographic operations not on the validator node itself, but on a separate node. There might be an intermediate feature version allowing cryptographic operations to happen in a different OS process on the same machine.

Follow operational best practices. Only expose a minimal amount of ports. Make sure to record logs. Setup monitoring for each machine and application involved. Configure alerting software. ...

Timeline

Deprecation of support for sentry nodes will happen with the next release of Substrate and Polkadot. Updates to the Polkadot secure validator project will happen thereafter. We don't expect the actual removal before October 2020.

---

Deprecation warning will be introduced through https://github.com/paritytech/substrate/pull/6779.

[bkchr]

Duplicate of https://github.com/paritytech/substrate/issues/6762 ?

[lamafab]

@mxinden Regarding the firewall, are there some options like limiting the amount of libp2p messages or "weighing" those (such as preventing nodes from sending a lot of "expensive" traffic)? The nginx proxy you refer to is for RPC calls, I assume?

[LukeWheeldon]

Looking forward to some guidance on nginx so I can start testings, before sentry support is completely dropped. Thank you.

[mxinden]

Regarding the firewall, are there some options like limiting the amount of libp2p messages or "weighing" those (such as preventing nodes from sending a lot of "expensive" traffic)?

I am not aware of a Substrate specific (layer 7) firewall. Thus one needs to use firewalls operating on layer 4 (TCP) and below. Given that these firewalls do not understand the application specific traffic there is no way for them to "weight" the impact of a request.

With the ongoing _backpressure_ efforts the node itself would be able to "weight" per peer and thus ensure fairness.

The nginx proxy you refer to is for RPC calls, I assume?

This issue only addresses the P2P port. It does not address the RPC port. The RPC port should stay as it is: secured and private.

[amnay-mo]

Does this affect the rules for applying for "Kusama’s Thousand Validators Programme"?

In the rules, it is required that Validators should run "at least one sentry node"

[haikoschol]

@amnay-mo I believe it does and that those rules need updating. @wpank?