Tendermint: Mempool size full is there any why to fix this issue

[skdjango001]

Hi I am trying to figure out mempool size,
E[2020-05-28|14:13:02.431] Error on broadcastTxCommit module=rpc err="Mempool is full: number of txs 10000 (max: 10000), total txs bytes 4774160 (max: 1073741824)"
how can i manage this mempool size and i just want to know can we turnoff mempool ? @melekes @ebuchman

[erikgrinaker]

Hi @skdjango001! I believe you can control the mempool size with these options in the configuration file:

[mempool]

# Maximum number of transactions in the mempool
size = 10000

# Limit the total size of all txs in the mempool.
# This only accounts for raw transactions (e.g. given 1MB transactions and
# max_txs_bytes=5MB, mempool will only accept 5 transactions).
max_txs_bytes = 1073741824

The mempool cannot be disabled, as far as I know. Let us know if you have any further questions!

[marbar3778]

The informal way that nodes have been turnring the mempool off is by setting the size to 0.

[skdjango001]

Hi @erikgrinaker thanx for your replay, I already increased the mempool size to size = 10000 but still getting above error

[erikgrinaker]

So increase it to something higher?

[skdjango001]

Is there any permanent solution for this @erikgrinaker

[skdjango001]

I am doing performance testing with 4 nodes and 1st I tried with mempool size= 5000 at that time i am getting error mempool full then i reset the node and started again with size = 10000 , Now i am again getting mempool size full; what i am trying to say is is there any alternate for this problem?
@erikgrinaker

[erikgrinaker]

The mempool limit is a resource usage limit. If your usage requires more resources than you have allocated then you will either have to reduce your usage or increase the resource limit.

There have been some talk about mempool optimizations and/or better rate control, but I'm not sure if there are any concrete plans at the moment. You may be able to find some information about this by searching for issues about the mempool.

[skdjango001]

Thanks for your suggestion @erikgrinaker now i increased to 20000 but still getting same error
E[2020-06-01|10:09:31.851] Error on broadcastTxCommit module=rpc err="Mempool is full: number of txs 20000 (max: 20000), total txs bytes 12457360 (max: 1073741824)"
E[2020-06-01|10:09:33.213] Error on broadcastTxCommit module=rpc err="Mempool is full: number of txs 20000 (max: 20000), total txs bytes 12457360 (max: 1073741824)"

[skdjango001]

I just want to know why this happened again and again, Is there any config should i change in config.toml file ? please help me out of this problem @melekes @erikgrinaker @ebuchman

[melekes]

How many transactions are you sending to Tendermint? What's the average block time? transactions per block?

[skdjango001]

avg txns per newblock = 3.48;

home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
/home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
2020/05/28 18:21:13 {0xc001412780 0xc0019cdd90}
I[2020-05-28|18:21:13.672] Executed block                               module=state height=15 validTxs=1 invalidTxs=0
I[2020-05-28|18:21:13.673] Committed state                              module=state height=15 txs=1 appHash=4E6F205374617465
/home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
/home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
2020/05/28 18:21:13 {0xc0019cb100 0xc001aa6fe0}
I[2020-05-28|18:21:13.971] Executed block                               module=state height=16 validTxs=1 invalidTxs=0
I[2020-05-28|18:21:13.973] Committed state                              module=state height=16 txs=1 appHash=4E6F205374617465
/home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
/home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
2020/05/28 18:21:14 {0xc001bcea00 0xc001bf32a0}
I[2020-05-28|18:21:14.190] Executed block                               module=state height=17 validTxs=1 invalidTxs=0
I[2020-05-28|18:21:14.191] Committed state                              module=state height=17 txs=1 appHash=4E6F205374617465
/home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
/home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
2020/05/28 18:21:14 {0xc0011df600 0xc001169aa0}
I[2020-05-28|18:21:14.572] Executed block                               module=state height=18 validTxs=1 invalidTxs=0
I[2020-05-28|18:21:14.573] Committed state                              module=state height=18 txs=1 appHash=4E6F205374617465
/home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
/home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
2020/05/28 18:21:14 {0xc00160d680 0xc00174acb0}
I[2020-05-28|18:21:14.874] Executed block                               module=state height=19 validTxs=1 invalidTxs=0
I[2020-05-28|18:21:14.875] Committed state                              module=state height=19 txs=1 appHash=4E6F205374617465
/home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
/home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
2020/05/28 18:21:15 {0xc0000d9980 0xc000d42e40}
I[2020-05-28|18:21:15.080] Executed block                               module=state height=20 validTxs=1 invalidTxs=0
I[2020-05-28|18:21:15.081] Committed state                              module=state height=20 txs=1 appHash=4E6F205374617465
/home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
/home/ten1//go/src/properties.txtFile contents: ZSC_Test_0,ZSC_Test_0_OWNER,cmn_usr_pwd,My!s2qlm1,Test_Node0SHA512 op is  ORxrSC5rWhx-PtRNbojMLwfreX64_rnMXKKFAJJNZdrkM6IWXlScV2PHzAKdsqlm-byI439DQYhbT4lssDCU-A==
2020/05/28 18:21:15 {0xc00045ce00 

[skdjango001]

[2020-06-01|12:23:56.250] Executed block                               module=state height=4224 validTxs=1 invalidTxs=0
I[2020-06-01|12:23:56.251] Committed state                              module=state height=4224 txs=1 appHash=4E6F205374617465
2020/06/01 12:23:56 {0xc001dd4680 0xc0026e5db0}
I[2020-06-01|12:23:56.449] Executed block                               module=state height=4225 validTxs=1 invalidTxs=0
I[2020-06-01|12:23:56.450] Committed state                              module=state height=4225 txs=1 appHash=4E6F205374617465
2020/06/01 12:23:56 {0xc0018aad00 0xc001fa1120}
I[2020-06-01|12:23:56.743] Executed block                               module=state height=4226 validTxs=1 invalidTxs=0
I[2020-06-01|12:23:56.744] Committed state                              module=state height=4226 txs=1 appHash=4E6F205374617465
2020/06/01 12:23:57 {0xc000fc2400 0xc0017463d0}
I[2020-06-01|12:23:57.052] Executed block                               module=state height=4227 validTxs=1 invalidTxs=0
I[2020-06-01|12:23:57.053] Committed state                              module=state height=4227 txs=1 appHash=4E6F205374617465
2020/06/01 12:23:57 {0xc000fc3080 0xc0020b5280}
I[2020-06-01|12:23:57.351] Executed block                               module=state height=4228 validTxs=1 invalidTxs=0
I[2020-06-01|12:23:57.352] Committed state                              module=state height=4228 txs=1 appHash=4E6F205374617465
2020/06/01 12:23:57 {0xc001127300 0xc002392dc0}
I[2020-06-01|12:23:57.745] Executed block                               module=state height=4229 validTxs=1 invalidTxs=0
I[2020-06-01|12:23:57.746] Committed state                              module=state height=4229 txs=1 appHash=4E6F205374617465
2020/06/01 12:23:58 {0xc0020cbb00 0xc002c06730}
I[2020-06-01|12:23:58.053] Executed block                               module=state height=4230 validTxs=1 invalidTxs=0
I[2020-06-01|12:23:58.054] Committed state                              module=state height=4230 txs=1 appHash=4E6F205374617465

These are the logs @melekes @erikgrinaker @ebuchman

[skdjango001]

And also i am noted some readings

please check this also

[skdjango001]

Every time after mempool is full i need to do tendermint unsafe_reset_all and then again i am starting the nodes,
I don't want to reset every time so i need to fix mempool issue its getting headache to me @melekes @erikgrinaker @ebuchman

[skdjango001]

every 3 blocks in 1sec;

[skdjango001]

HI @erikgrinaker again getting same error for me
E[2020-06-01|23:22:46.479] Error on broadcastTxCommit module=rpc err="Mempool is full: number of txs 30000 (max: 30000), total txs bytes 8130000 (max: 1073741824)"
after increases mempool size;
please help me out of this

[erikgrinaker]

@skdjango001 You said you are doing performance testing, can you provide some more details about what you are testing and how you are doing it? 3.48 txs per block seems very low, is your application not able to handle higher throughput than this? My guess would be that you are sending far more transactions at a far higher rate than the application is able to process, which leads to them piling up in the mempool until it's full.

[skdjango001]

I have already attached the xl sheet plz look into that @erikgrinaker

[erikgrinaker]

From the sheet, I can see that you are processing about 3.5 txs per block. How many are you submitting per block?

[skdjango001]

per block 4 valid txs

[erikgrinaker]

Ok, so when submitting 4 tx/b and processing 3.5 tx/b a mempool of 10000 will be full after 20000 blocks. Try to reduce the submission rate to 3 tx/b.

[skdjango001]

actually i am changing flush_throttle_timeout to 10ms from 100ms in config.toml is there any effect with this change?

[erikgrinaker]

actually i am changing flush_throttle_timeout to 10ms from 100ms in config.toml is there any effect with this change?

I don't see how that would be related. That is a P2P networking setting which throttles the rate of outbound messages.