Tendermint: mempool: Getting stuck in infinite loop of submitting txs (non-deterministic)

[thanethomson]

Tendermint version: v0.32.1 (official release)

ABCI app: kvstore

Environment:

• OS (e.g. from /etc/os-release): CentOS
• Install tools: tmtestnet
• Others:

I set up an 8-validator network on AWS using the tmtestnet script, where the mempool size limit was set to 5,000.

What happened:
After submitting transactions at a sufficiently high rate to saturate the mempool, it appears as though one or more validators have transactions "stuck" in their mempool and they somehow manage to keep successfully submitting these transactions to the network, and the network keeps committing these transactions ad infinitum.

The trouble is, this problem seems to be non-deterministic. It does not always occur when saturating the mempool with transactions, but it has only been observed when the mempool becomes saturated (thus far, out of testing on 6 distinct 8-validator networks).

Zooming in on the tail end of the graph:

What you expected to happen:
I would have expected all validators' mempools to be empty after such a load test, and that no further transactions be committed to the blockchain when nobody is submitting any transactions.

Have you tried the latest version: yes

How to reproduce it (as minimally and precisely as possible):
Use tmtestnet to deploy the network and then execute a load test against it:

./tmtestnet.py -c 08-validators.yaml network deploy
./tmtestnet.py -c 08-validators.yaml loadtest start 3200_txs_per_sec

Logs (paste a small part showing an error (< 10 lines) or link a pastebin, gist, etc. containing more of the log file):
I was unable to obtain any meaningful logs, and unable to reproduce now when attempting at greater logging verbosity.

Config (you can paste only the changes you've made):
See this gist

[melekes]

I would like to remind you & those who are reading this that application should also implement replay protection.

[marbar3778]

While load testing tm I was able to reproduce this. The application used did not have replay protection

[ebuchman]

Is it possible to replicate on an app that does have replay protection?