Parity-ethereum: Subscription "newPendingTransactions" doesn't work properly

[madnadyka]

I'm running:

• Which Parity version?: 1.11.1
• Which operating system?: Linux
• How installed?: binaries
• Are you fully synchronized?: no
• Which network are you connected to?: ethereum
• Did you try to restart the node?: yes

Subscription "newPendingTransactions" doesn't work properly - I am receiving only small part of all new pending transactions. In version 1.10.4 this subscription is ok.

[Tbaut]

Can you provide an example as well as how you subscribe to new pending transactions and verify that it's not working as expected?

[iFA88]

I also use this, it seems it works without issue with Parity/v1.11.1-beta-6654d0216-20180515/x86_64-linux-gnu/rustc1.25.0. I receive about 4-10 new pending TX in every 10 sec.

[madnadyka]

I display in my program quantity of received by subscription pending tx per minute. Displaying data is a dictionary, where key is timestamp and value is tx quantity (not sorted by timestamp, yellow color) :
parity v1.10.4:

parity v1.11.1:

and after restart node parity v1.11.1:

[Tbaut]

Can you share how you subscribe? This visual way of checking is not very straightforward for us to pinpoint problems?
Did you have a look at the resources of the computer? It could also miss events because of lagging behind (v1.11.1 is more demanding in terms of CPU usage: https://github.com/paritytech/parity/issues/8696). A safer way to make sure not to miss any transaction would be to use the trace API: https://wiki.parity.io/JSONRPC-trace-module.html)

[madnadyka]

By websocket in my program (python):

def subscribe_transactions(self):
self.ws.send_str('{"jsonrpc":"2.0","id": 2, "method": "eth_subscribe", "params": ["newPendingTransactions"]}')

CPU of server with my program - 2 GB, CPU of server with node - 8 GB

[Tbaut]

This could be due to the new transaction queue, @tomusdrw can you have a look?

[Tbaut]

we can't downgrade due to the recent security alert

You can downgrade to the latest stable release, v1.10.6 that also has the security patch ;)

[tomusdrw]

@madnadyka I can't reproduce the issue on 11.4, pending subscription returns all transactions that are entering the pool. Perhaps your pool is full and only small portion of transactions actually get there? Try increasing the pool size with --txqueue-size=32000 for instance.

You can compare the results by running with -ltxqueue=debug (look for Added to the pool) and the pending filter, for me the amounts of the two are equal.
The only difference is that 1.11 returns imports in batches (for performance reasons) - if we are attempting to import a larger set of transactions the notification will be triggered only after the entire batch is imported.

[iFA88]

Few months before I have issued also, that theparity_pendingTransactions request sometimes hanging forever, but then I have used filter to get the pending transactions.

[tomusdrw]

Yeah, parity_pendingTransations is quite heavy (especially with large pools), cause it has to get the entire pending set and serialize all transactions to JSON, the response payload on it's own is huge.

In 1.11.x series it has an optional limit parameter, that allows you to query only the tip of the pool (like 100 best transactions) - that should work much faster.

[iFA88]

Getting 10-15k pending transaction have taken 6-12 sec. This is a great speed, idk why was hanging.

[tzapu]

@5chdn i don't think i have seen the limit param in the docs for parity_pendingTransations, is there some place specific we should look for the latest docs and not in the wiki? Does it have, or could it have a "just hashes" flag as well? :P

[5chdn]

Yeah, the wiki. We are having automated API docs soon (tm), we just need to make the CI pipeline actually finish.

[tzapu]

@madnadyka if still of any interest, i have tested newPendingTransactions with almost all versions of parity since it was added, it always seemed fine, including 1.11.xx

[tomusdrw]

Please re-open if the issues persists. We now also have #9745 to efficiently query the pool content.