Neo: Majority of consensus nodes are issuing empty blocks

[EdgeDLT]

Most consensus nodes are failing to include any transactions in blocks. This seems to be the case for 5 out of 7 nodes.

I originally suspected it was a policy issue, related to the network fee changes, but I have tested both GAS and NEP-5 transfers with varying fees (0.001 up to 0.1 GAS).

Each time, the transfer remains pending until it is included in a block by one node that seems to be consistently behaving as expected (key 024c7b7fb6c310fccf1ba33b082519d82964ea93868d676662d4a59ad548df0e7d).

If this node does not include the transfer, the next node usually does (key 025bdf3f181f53e9696227843950deb72dcd374ded17c057159513c3d0abe20b64).

Perhaps the root cause is related to #799.

[EdgeDLT]

Another observation; I'm not that familiar with how the mempool functions, but there seem to be a lot of transactions pending that are very large in size with insufficient (or zero) network fee paid.

If the updated fee policy does not regard these transactions as valid, they should not be pending at all. Ideally, the fee required to accompany such a large transaction should be calculated and reported to the user before the transaction is submitted.

It may be best to remove or alter the new fee mechanism until UTXO transfers are removed in NEO3. The average user will not understand why two seemingly identical transactions (such as transferring NEO) could require vastly different fees for approval.

[vncoelho]

@EdgeDLT, we are investigating.
In fact, this issue used to happen before dBFT 2.0.
I remember that we already had a discussion about why nodes where not having transactions in their mempool.

We are trying to double check Akka Mailbox priorities.
The CN is operating nice! We are quite happy with its behavior.
These P2P are something that has been bordering us for some time.

CN with index 0 and 1 are producing blocks normally.

[15:15:18.484] relay block: height=3855354 hash=0x1b704ae100615f4c1fe24f15054353f680936ec99fd07cf1bfbeea57a34cae22 tx=1
[15:15:18.574] persist block: height=3855354 hash=0x1b704ae100615f4c1fe24f15054353f680936ec99fd07cf1bfbeea57a34cae22 tx=1
[15:15:18.582] initialize: height=3855355 view=0 index=-1 role=WatchOnly
[15:15:35.090] OnPrepareRequestReceived: height=3855355 view=0 index=0 tx=27
[15:15:49.502] persist block: height=3855355 hash=0x87853170fe4a2182bdbc6482857e7b8b0e7305b7e1c7eec4a71c939b96510d91 tx=27
[15:15:49.519] initialize: height=3855356 view=0 index=-1 role=WatchOnly
[15:16:03.557] OnPrepareRequestReceived: height=3855356 view=0 index=1 tx=6

[vncoelho]

@EdgeDLT, I do not know if you are aware, but there is this easy way to set WatchOnly nodes now with dBFT 2.0.

We ensured that any node in the network can type start consensus and follow all messages!

[vncoelho]

[EdgeDLT]

That's great to see, thanks Vitor!

I checked out some more of the transactions stuck in the mempool from O3 Explorer. There are lots (probably over a hundred) 6780-byte invocation transactions stuck with very large verification scripts.

Invocation Script:
000430323135

Verification Script:
011ac56b6c766b00527ac46c766b51527ac4616168164e656f2e52756e74696d652e47657454726967676572009c6c766b52527ac46c766b52c3640e0261611463ec04c935a0b7f1adcd244379c29650a13b98826168184e656f2e52756e74696d652e436865636b5769746e6573736c766b56527ac46c766b56c3640e00516c766b57527ac462a80561682953797374656d2e457865637574696f6e456e67696e652e476574536372697074436f6e7461696e65726c766b53527ac46c766b53c36168174e656f2e5472616e73616374696f6e2e476574547970656c766b54527ac46c766b54c36102d1009c009c6c766b58527ac46c766b58c3640e00006c766b57527ac46229056c766b53c36c766b55527ac46c766b55c36168234e656f2e496e766f636174696f6e5472616e73616374696f6e2e476574536372697074c0013d9c009c6c766b59527ac46c766b59c3640f0061006c766b57527ac462d1046c766b55c36168234e656f2e496e766f636174696f6e5472616e73616374696f6e2e47657453637269707400517f011c9c009c6c766b5a527ac46c766b5ac3640e00006c766b57527ac46284046c766b55c36168234e656f2e496e766f636174696f6e5472616e73616374696f6e2e476574536372697074011d01207f0351c1080877697468647261777e01677e61682d53797374656d2e457865637574696f6e456e67696e652e476574457865637574696e67536372697074486173687e9c009c6c766b5b527ac46c766b5bc3640f0061006c766b57527ac462f403516c766b57527ac462e9036168164e656f2e52756e74696d652e47657454726967676572609c6c766b5c527ac46c766b5cc364b403616c766b00c3076465706f736974876c766b5d527ac46c766b5dc3641700616c766b51c36165a0036c766b57527ac4628d036c766b00c3087769746864726177876c766b5e527ac46c766b5ec3641700616c766b51c3616576086c766b57527ac4625b036c766b00c317717565727941697244726f70546f74616c537570706c79876c766b5f527ac46c766b5fc3644800616168164e656f2e53746f726167652e476574436f6e746578740d61697264726f70537570706c79617c680f4e656f2e53746f726167652e4765746c766b57527ac462e9026c766b00c313717565727941697244726f7042616c616e6365876c766b60527ac46c766b60c3641700616c766b51c36165e50e6c766b57527ac462ac026c766b00c3157175657279417661696c61626c6542616c616e6365876c766b0111527ac46c766b0111c3641700616c766b51c3616574116c766b57527ac4626b026c766b00c31e7175657279417661696c61626c6542616c616e6365576974685068617365876c766b0112527ac46c766b0112c3641700616c766b51c361653f0b6c766b57527ac46221026c766b00c3117365745769746864726177537769746368876c766b0113527ac46c766b0113c3641700616c766b51c361656f0b6c766b57527ac462e4016c766b00c31371756572795769746864726177537769746368876c766b0114527ac46c766b0114c3644900616168164e656f2e53746f726167652e476574436f6e746578740e7769746864726177537769746368617c680f4e656f2e53746f726167652e4765746c766b57527ac46273016c766b00c31571756572794c617374576974686472617754696d65876c766b0115527ac46c766b0115c3641700616c766b51c36165ea0b6c766b57527ac46232016c766b00c316717565727941697264726f704163636f756e744e756d876c766b0116527ac46c766b0116c3644c00616168164e656f2e53746f726167652e476574436f6e746578741161697264726f704163636f756e744e756d617c680f4e656f2e53746f726167652e4765746c766b57527ac462bb006c766b00c313717565727941697264726f704163636f756e74876c766b0117527ac46c766b0117c3641700616c766b51c36165ec0b6c766b57527ac4627c006c766b00c309717565727944617461876c766b0118527ac46c766b0118c3644f00616c766b51c300c36c766b0119527ac46168164e656f2e53746f726167652e476574436f6e746578746c766b0119c3617c680f4e656f2e53746f726167652e4765746c766b57527ac4620f0061006c766b57527ac46203006c766b57c3616c75660111c56b6c766b00527ac461611463ec04c935a0b7f1adcd244379c29650a13b98826168184e656f2e52756e74696d652e436865636b5769746e657373009c6c766b56527ac46c766b56c3640e00006c766b57527ac462a9046c766b00c3c0539c009c6c766b58527ac46c766b58c364050061f06c766b00c300c36c766b51527ac46c766b51c3c001149c009c6c766b59527ac46c766b59c364050061f06c766b00c351c36c766b52527ac46c766b52c300a16314006c766b52c3070031cae8a0e909a0620400516c766b5a527ac46c766b5ac364050061f06168164e656f2e53746f726167652e476574436f6e746578740d61697264726f70537570706c79617c680f4e656f2e53746f726167652e4765746c766b53527ac4006c766b54527ac46c766b51c3616589116c766b5b527ac46c766b5bc364dc00616168164e656f2e53746f726167652e476574436f6e746578741161697264726f704163636f756e744e756d617c680f4e656f2e53746f726167652e47657451936c766b5c527ac46168164e656f2e53746f726167652e476574436f6e746578741161697264726f704163636f756e744e756d6c766b5cc37e6c766b51c3615272680f4e656f2e53746f726167652e507574616168164e656f2e53746f726167652e476574436f6e746578741161697264726f704163636f756e744e756d6c766b5cc3615272680f4e656f2e53746f726167652e50757461616c766b00c352c36c766b55527ac40a666972737450686173656c766b55c3876c766b5d527ac46c766b5dc3649200616168164e656f2e53746f726167652e476574436f6e74657874610a666972737450686173656c766b51c37e617c680f4e656f2e53746f726167652e4765746c766b54527ac46168164e656f2e53746f726167652e476574436f6e74657874610a666972737450686173656c766b51c37e6c766b52c3615272680f4e656f2e53746f726167652e50757461616267010b7365636f6e6450686173656c766b55c3876c766b5e527ac46c766b5ec3649400616168164e656f2e53746f726167652e476574436f6e74657874610b7365636f6e6450686173656c766b51c37e617c680f4e656f2e53746f726167652e4765746c766b54527ac46168164e656f2e53746f726167652e476574436f6e74657874610b7365636f6e6450686173656c766b51c37e6c766b52c3615272680f4e656f2e53746f726167652e507574616162b5000a746869726450686173656c766b55c3876c766b5f527ac46c766b5fc3649200616168164e656f2e53746f726167652e476574436f6e74657874610a746869726450686173656c766b51c37e617c680f4e656f2e53746f726167652e4765746c766b54527ac46168164e656f2e53746f726167652e476574436f6e74657874610a746869726450686173656c766b51c37e6c766b52c3615272680f4e656f2e53746f726167652e50757461616206006161f06c766b53c36c766b54c3946c766b52c3930700935ebae2bc1da06319006c766b53c36c766b54c3946c766b52c393009f620400516c766b60527ac46c766b60c364050061f06168164e656f2e53746f726167652e476574436f6e746578740d61697264726f70537570706c796c766b53c36c766b54c3946c766b52c393615272680f4e656f2e53746f726167652e50757461616c766b51c36c766b52c3617c096465706f736974656453c168124e656f2e52756e74696d652e4e6f7469667961516c766b57527ac46203006c766b57c3616c75660114c56b6c766b00527ac4616c766b00c3c0519c009c6c766b5b527ac46c766b5bc364050061f06c766b00c300c36c766b51527ac46c766b51c3c0011c9c009c6c766b5c527ac46c766b5cc364050061f06c766b51c35801147f6c766b52527ac46168164e656f2e53746f726167652e476574436f6e746578740e7769746864726177537769746368617c680f4e656f2e53746f726167652e4765746c766b53527ac46c766b53c3009c6c766b5d527ac46c766b5dc364050061f06c766b52c3610a66697273745068617365617c65ed076c766b54527ac46c766b52c3610b7365636f6e645068617365617c65cf076c766b55527ac46c766b52c3610a74686972645068617365617c65b2076c766b56527ac46c766b54c36c766b55c3936c766b56c3936c766b57527ac46c766b57c3519f6c766b5e527ac46c766b5ec364050061f061682d53797374656d2e457865637574696f6e456e67696e652e476574457865637574696e67536372697074486173686c766b58527ac4087472616e7366657253c576006c766b58c3c476516c766b52c3c476526c766b57c3c4617c675f89105fdcf8b97739c68f3e969d60b6e98bfa066c766b59527ac46c766b59c3519c6c766b5a527ac46c766b5ac36c766b5f527ac46c766b5fc3640602616168184e656f2e426c6f636b636861696e2e4765744865696768746168184e656f2e426c6f636b636861696e2e4765744865616465726168174e656f2e4865616465722e47657454696d657374616d706c766b60527ac46168164e656f2e53746f726167652e476574436f6e746578746c766b52c3106c617374576974686472617754696d657e6c766b60c3615272680f4e656f2e53746f726167652e507574616168164e656f2e53746f726167652e476574436f6e746578746c766b52c30a77697468647261774e6f7e617c680f4e656f2e53746f726167652e4765746c766b0111527ac46c766b0111c351936c766b0111527ac46168164e656f2e53746f726167652e476574436f6e746578746c766b52c30a77697468647261774e6f7e6c766b0111c3615272680f4e656f2e53746f726167652e5075746161682953797374656d2e457865637574696f6e456e67696e652e476574536372697074436f6e7461696e65726c766b0112527ac46168164e656f2e53746f726167652e476574436f6e746578746c766b52c3067478496e666f7e6c766b0111c37e6c766b0112c36168174e656f2e5472616e73616374696f6e2e47657448617368615272680f4e656f2e53746f726167652e50757461616c766b52c36c766b57c3617c08776974686472657753c168124e656f2e52756e74696d652e4e6f74696679616162100061006c766b0113527ac4620f00516c766b0113527ac46203006c766b0113c3616c756655c56b6c766b00527ac4616c766b00c3c0529c009c6c766b53527ac46c766b53c3640e00006c766b54527ac46238006c766b00c300c36c766b51527ac46c766b00c351c36c766b52527ac46c766b51c36c766b52c3617c6561046c766b54527ac46203006c766b54c3616c756655c56b6c766b00527ac461611463ec04c935a0b7f1adcd244379c29650a13b98826168184e656f2e52756e74696d652e436865636b5769746e657373009c6c766b51527ac46c766b51c3640e00006c766b52527ac462cf006c766b00c3c0519c009c6c766b53527ac46c766b53c3640e00006c766b52527ac462ab006c766b00c300c3026f6e876c766b54527ac46c766b54c3644600616168164e656f2e53746f726167652e476574436f6e746578740e776974686472617753776974636851615272680f4e656f2e53746f726167652e5075746161624300616168164e656f2e53746f726167652e476574436f6e746578740e776974686472617753776974636800615272680f4e656f2e53746f726167652e5075746161516c766b52527ac46203006c766b52c3616c756655c56b6c766b00527ac4616c766b00c3c0519c009c6c766b52527ac46c766b52c3640e00006c766b53527ac46283006c766b00c300c36c766b51527ac46c766b51c3c001149c009c6c766b54527ac46c766b54c3640e00006c766b53527ac46250006168164e656f2e53746f726167652e476574436f6e746578746c766b51c3106c617374576974686472617754696d657e617c680f4e656f2e53746f726167652e4765746c766b53527ac46203006c766b53c3616c756655c56b6c766b00527ac4616c766b00c3c0519c009c6c766b52527ac46c766b52c3640f0001006c766b53527ac46293006c766b00c300c36c766b51527ac46c766b51c3009f6311006c766b51c30400e1f505a0620400516c766b54527ac46c766b54c3640f0001006c766b53527ac46251006168164e656f2e53746f726167652e476574436f6e746578741161697264726f704163636f756e744e756d6c766b51c37e617c680f4e656f2e53746f726167652e4765746c766b53527ac46203006c766b53c3616c756658c56b6c766b00527ac4616c766b00c3c0529c009c6c766b53527ac46c766b53c3640e00006c766b54527ac46268016c766b00c300c36c766b51527ac46c766b00c351c36c766b52527ac40a666972737450686173656c766b52c3876c766b55527ac46c766b55c3644c00616168164e656f2e53746f726167652e476574436f6e74657874610a666972737450686173656c766b51c37e617c680f4e656f2e53746f726167652e4765746c766b54527ac462e3000b7365636f6e6450686173656c766b52c3876c766b56527ac46c766b56c3644d00616168164e656f2e53746f726167652e476574436f6e74657874610b7365636f6e6450686173656c766b51c37e617c680f4e656f2e53746f726167652e4765746c766b54527ac46278000a746869726450686173656c766b52c3876c766b57527ac46c766b57c3644c00616168164e656f2e53746f726167652e476574436f6e74657874610a746869726450686173656c766b51c37e617c680f4e656f2e53746f726167652e4765746c766b54527ac4620f0061006c766b54527ac46203006c766b54c3616c75665cc56b6c766b00527ac46c766b51527ac4616c766b00c3c001149c009c6c766b58527ac46c766b58c3640e00006c766b59527ac462f6006168164e656f2e53746f726167652e476574436f6e746578746c766b51c36c766b00c37e617c680f4e656f2e53746f726167652e4765746c766b52527ac46c766b52c3519f6c766b5a527ac46c766b5ac3640e00006c766b59527ac4629700006c766b53527ac4006c766b54527ac46c766b00c36c766b51c3617c656b016c766b55527ac46c766b00c36c766b51c3617c65b7026c766b56527ac46c766b56c36c766b55c3a06c766b5b527ac46c766b5bc3641c00616c766b56c36c766b55c39403805101966c766b53527ac4616c766b52c36c766b53c39502da02966c766b57527ac46c766b57c36c766b59527ac46203006c766b59c3616c756659c56b6c766b00527ac4616c766b00c3c0519c009c6c766b56527ac46c766b56c3640e00006c766b57527ac462b5006c766b00c300c36c766b51527ac46c766b51c3c001149c009c6c766b58527ac46c766b58c3640e00006c766b57527ac46282006c766b51c3610a66697273745068617365617c6558fe6c766b52527ac46c766b51c3610b7365636f6e645068617365617c653afe6c766b53527ac46c766b51c3610a74686972645068617365617c651dfe6c766b54527ac46c766b52c36c766b53c3936c766b54c3936c766b55527ac46c766b55c36c766b57527ac46203006c766b57c3616c75665ac56b6c766b00527ac46c766b51527ac4616168164e656f2e53746f726167652e476574436f6e746578746c766b00c3106c617374576974686472617754696d657e617c680f4e656f2e53746f726167652e4765746c766b52527ac46c766b51c3610a666972737450686173659c6c766b53527ac46c766b53c3643000616c766b52c304809d3b5ba16c766b54527ac46c766b54c36411006104809d3b5b6c766b52527ac4616162a3006c766b51c3610b7365636f6e6450686173659c6c766b55527ac46c766b55c3643000616c766b52c304805a8d5ba16c766b56527ac46c766b56c36411006104805a8d5b6c766b52527ac461616254006c766b51c3610a746869726450686173659c6c766b57527ac46c766b57c3643000616c766b52c30400c6dd5ba16c766b58527ac46c766b58c3641100610400c6dd5b6c766b52527ac461616206006161f06c766b52c36c766b59527ac46203006c766b59c3616c75665ac56b6c766b00527ac46c766b51527ac4616168184e656f2e426c6f636b636861696e2e4765744865696768746168184e656f2e426c6f636b636861696e2e4765744865616465726168174e656f2e4865616465722e47657454696d657374616d706c766b52527ac46c766b51c3610a666972737450686173659c6c766b53527ac46c766b53c3643000616c766b52c3048004fe5ea06c766b54527ac46c766b54c36411006104ff55ff5e6c766b52527ac4616162a3006c766b51c3610b7365636f6e6450686173659c6c766b55527ac46c766b55c3643000616c766b52c30480c14f5fa06c766b56527ac46c766b56c36411006104ff12515f6c766b52527ac461616254006c766b51c3610a746869726450686173659c6c766b57527ac46c766b57c3643000616c766b52c304002da05fa06c766b58527ac46c766b58c364110061047f7ea15f6c766b52527ac461616206006161f06c766b52c36c766b59527ac46203006c766b59c3616c756656c56b6c766b00527ac4616168164e656f2e53746f726167652e476574436f6e74657874610a666972737450686173656c766b00c37e617c680f4e656f2e53746f726167652e4765746c766b51527ac46168164e656f2e53746f726167652e476574436f6e74657874610b7365636f6e6450686173656c766b00c37e617c680f4e656f2e53746f726167652e4765746c766b52527ac46168164e656f2e53746f726167652e476574436f6e74657874610a746869726450686173656c766b00c37e617c680f4e656f2e53746f726167652e4765746c766b53527ac46c766b51c3009c6417006c766b52c3009c640d006c766b53c3009c620400006c766b54527ac46c766b54c3640f0061516c766b55527ac4620e00006c766b55527ac46203006c766b55c3616c7566

Disassembly with EcoLab led me to the address AeMy1tD8NvNjhVgnrGBC3RdDtUEcQqoWu3. Seems to be a Loopring address that sends 15-20 or so transactions a day. Looks like it hasn't been able to get a transaction validated since the update (due to the network fee update I expect).

Could this be causing any issues?

[vncoelho]

I do not known, Edge.

Our node just says that mempool verified is 22 right now:

[vncoelho]

This transaction with more than 1064 bytes needs to be high priority and attaches the necessary extra GAS for its size (this is the network fee), otherwise it will not pass the filter for entering in the mempool and will not be shared to the CN that have this filter.

Maybe this node from O3 is not updated with the most recent filter.
However, @EdgeDLT, this should not be a problem for CNs because they will filter it when this TX is rebroadcasted to them.

[saltyskip]

@vncoelho Can a node NOT running the filter broadcast a transaction via P2P to another node that IS running the filter and have it successfully be accepted into the mempool?

If the policy is only for RPC, than nodes can distribute bad transactions (ones that dont follow the consensus nodes policies) via p2p.

If this is the case is there a way to purge the mempool of these bad transactions?

[vncoelho]

Hiii Andreit, my friend, @saltyskip.

In theory, it is working perfect filtering any incoming tx.
The mempool class and the filter are well organized.

However, errors might exist, if any behavior was detected let's find the minor issue.
It would be better to have more UT there. But in fact, it has some good tests.
It was a nice work in team @jsolman, @erikzhang, @shargon, @igormcoelho and others.

[saltyskip]

Thanks @vncoelho let me know if there are is any additional diagnostic data I can provide. Just want to note that it is not only O3 node that has this issues.

Several nodes which I check in http://monitor.cityofzion.io have mempools with transactions that definitely would not pass the policy filter due to being too large and not attaching transaction fee.

[superboyiii]

Hi, Andreit. The NGD nodes works well via SimplePolicy till now. There might be something wrong with the crowded nodes which most likely that SimplePolicy isn't been installed or installed in a wrong path.
There's a nice monitor used for watching the tx numbers in mempool. Recommend you that.
https://happynodes.f27.ventures/table

[saltyskip]

We've removed all nodes that don't have simple policy enabled as candidates for submitting transactions to

[EdgeDLT]

Perhaps we should consider how to raise awareness of SimplePolicy or find a way to quickly filter out invalid transactions that are relayed by nodes that do not have it installed correctly. It's not good if normal nodes can affect consensus through such mistakes.

Seems we have gotten to the bottom of this, so I'm going to close this issue now, thanks all.

[sector420]

This issue appears to be happening again. Is anyone investigating this?

[vncoelho]

Good to reopen @shargon.

[EdgeDLT]

Maybe it would be good to move fee policy to native contracts as soon as it is realistic to do so. If not, then SimplyPolicy likely needs to be included directly into 2.10.3 instead of being a plugin.

Either way, it should not be 'opt-in' if it can cause issues with the network.

[vncoelho]

It is already moved for Neo 3, but we gonna not change this for Neo 2.x anymore, @EdgeDLT.

Maybe direct include the plugin in the release would be nice, @erikzhang.

[vncoelho]

But we still need to find how this is affecting mempool's like hapenning now.

[shargon]

Is solved?

[EdgeDLT]

I believe it was solved.

[lock9]

Hi, if new bugs were discovered while fixing this, please create a new issue to report it.
Thanks