Parity-ethereum: JSON-RPC sometimes takes too long to have a result

[jesinspirment]

_Before filing a new issue, please provide the following information._

_If you think that your issue is an exploitable security vulnerability, please mail your bugreport to [email protected] instead; your submission might be eligible for our Bug Bounty._
_You can find mode info on the reporting process in SECURITY.md_

• OpenEthereum version: 3.0.1
• Operating system: Ubuntu 18
• Installation: downloaded official Linux zip file
• Fully synchronized: yes
• Network: mainnet
• Restarted: yes

I have observed that for certain RPC requests, particularly eth_sendRawTransaction, eth_getTransactionByHash, eth_getTransactionReceipt, sometimes it takes extremely long to have a result, could be even a minute. This is quite disruptive for our production projects.

Is there a way we can minimise this? Initially, we have 2-core CPU (Xeon) and 8 GB RAM with --cache-size=3072. Suspecting that we may be able to overcome this by having more RAM, we upgraded to a 16 GB RAM node with --cache-size=8192. It seems that we still occasionally get this problem. We are using SSD too btw.

If it helps, we are running OpenEthereum with the following options:
--jsonrpc-interface="all" --jsonrpc-apis="all" --jsonrpc-cors="" --ws-interface="all" --ws-apis="all" --ws-origins="" --ws-hosts="all" --db-compaction=ssd --cache-size=8192 --min-peers=50 --max-peers=80 --snapshot-peers=50 --mode=active --release-track=stable --tx-queue-size=200000 --tx-queue-per-sender=2000 --tx-queue-mem-limit=128

[adria0]

Hi @jesinspirment, could you try running it with --no-periodic-snapshot flag activated, please? Your feedback about this change will be very appreciated.

[jesinspirment]

Ok will do for sure. But can I check why? Is there an issue if I allow the periodic snapshot creation? Also, what is the consequence if I do disable it, other than the fact that it will not generate the snapshot?

[adria0]

@jesinspirment we have been doing some internal checks and snapshot generation takes a lot of CPU and memory. We think that at this moment this makes sense for bootnodes and nodes with 4CPU/32GB RAM at least. You should not have any additional consequences by disabling this. We are going to disable this option by default in the next openethereum release.

[jesinspirment]

Ok I have just restarted my 16 GB server like this:

--jsonrpc-interface="all" --jsonrpc-apis="all" --jsonrpc-cors="" --ws-interface="all" --ws-apis="all" --ws-origins="" --ws-hosts="all" --db-compaction=ssd --cache-size=8192 --min-peers=50 --max-peers=80 --snapshot-peers=50 --mode=active --release-track=stable --tx-queue-size=200000 --tx-queue-per-sender=2000 --tx-queue-mem-limit=128 --no-periodic-snapshot

I will monitor for a while, and report back to you my findings. Hope this will help you guys.

[jesinspirment]

@adria0 I am still monitoring, but so far, disabling periodic snapshot seems to improve things. And it's a major improvement, at least in terms of memory usage.

[lingyun92]

The same issue, seems very slow when broadcast transactions.

[adria0]

The same issue, seems very slow when broadcast transactions.

Please, @niulingyun, try with --no-periodic-snapshot flag activated. Your feedback about this change will be very appreciated.

[lingyun92]

The same issue, seems very slow when broadcast transactions.

Please, @niulingyun, try with --no-periodic-snapshot flag activated. Your feedback about this change will be very appreciated.

not working, I am exchange.

[adria0]

The same issue, seems very slow when broadcast transactions.

Please, @niulingyun, try with --no-periodic-snapshot flag activated. Your feedback about this change will be very appreciated.

not working, I am exchange.

In which hardware are you running the client? CPU/RAM/Storage(SSD/NVM/HD)

[lingyun92]

The same issue, seems very slow when broadcast transactions.

Please, @niulingyun, try with --no-periodic-snapshot flag activated. Your feedback about this change will be very appreciated.

not working, I am exchange.

In which hardware are you running the client? CPU/RAM/Storage(SSD/NVM/HD)

AWS tokyo，m4.2xlarge with SSD , in docker

[adria0]

The same issue, seems very slow when broadcast transactions.

Please, @niulingyun, try with --no-periodic-snapshot flag activated. Your feedback about this change will be very appreciated.

not working, I am exchange.

In which hardware are you running the client? CPU/RAM/Storage(SSD/NVM/HD)

AWS tokyo，m4.2xlarge with SSD , in docker

We need more details to understand where the problem can be, please, can you provide information about:
1) CPU/RAM
2) Which transactions are you running? How much RPCs per second?
3) Are you running a fatdb or archive node?

[lingyun92]

1. 16 CORE 64G RAM
2. transfer and many read rpc calls, like get block, get receipt, trace transaction, get balance
3. ENTRYPOINT ["/root/openethereum", "--jsonrpc-apis", "safe,parity_set", "--no-ws", "--no-ipc", "--max-peers", "300", "--min-peers","200", "--tracing", "on", "--unsafe-expose", "--no-download", "--tx-queue-mem-limit=64", "--tx-queue-size=200000", "--no-periodic-snapshot"]

[lingyun92]

I change my send function to geth and keep read functions on parity, and works well.

[adria0]

I change my send function to geth and keep read functions on parity, and works well.

@niulingyun, we do not know what "works well" in terms of operations per second means for you, openethereum and geth have different database models and for some RPC operations, the same query has different costs. If you dou think that this information is confidential you can contact me in discoord https://discord.io/openethereum

[rakita]

Initial problem seems resolved