Tendermint: mempool: locally-defined maxMsgSize and stopping peers

[melekes]

Problem definition

While working on https://github.com/tendermint/tendermint/pull/5321, I saw that we're stopping the peer if it sends us a message bigger than maxMsgSize.

maxMsgSize = MaxBatchBytes + proto overhead
maxMsgSize = 10MB + 4 bytes

Because MaxBatchBytes is a config parameter (defined locally and not agreed by all nodes), peers can break the connection with our node if we increase this parameter and send them bigger messages.

Proposals

1. Adaptive flow control (similar to TCP): try bigger size, then smaller, then smaller until the node accepts the msg.
2. Learn about other nodes limits (during handshake) and send them smaller messages (according to their limits)

[erikgrinaker]

I prefer specifying it in the handshake; why guess when we can know?