Parity-ethereum: Crash with "Too many open files"

[miguelmtzinf]

• Parity Ethereum version: Parity-Ethereum/v2.4.8-stable-25435c6-20190625/x86_64-linux-gnu/rustc1.35.0
• Operating system: Linux
• Installation: Docker official image parity/parity:stable
• Fully synchronized: no
• Network: private network
• Restarted: yes (once, automatically by node)

I am running a parity private network using docker-compose locally. The network is compound by 3 nodes (node0, node1 and node2) and using aura consensus with 1 signer (from node0). So, node0 is the gateway (RPC port opened) and in charge of consensus, and node1 and node2 are replicated peers. While network is running, the number of open files is increasing little by little in all nodes, more in node0 (I think because is the gateway). When it was running for some days (depends of rpc call load), it get a "too many open files" error because it gets the machine limit, which is 4096 (check the limit using ulimit -aH). Node0 crashed and automatically got restarted, reseting number of open files and now parity network seems work normally...

reserved_peers:

enode://6355b8589ae36fbfdc5f2a3acd1ec6a1ae5841299a9a0132906e73ac3c587bdab5ad6f0353112a3e58f85b91839f14840456345b492dc374f3d6cd92a0361fa2@172.16.0.10:30303
enode://7a859c5d4294136b1a01b604477ac28a5643ee39a59ce44a9588001883bf16aa262f6fcf56f0c527c536484ebf6a6a28b98a2faa2027764ad9949152ac2ef9ec@172.16.0.11:30303
enode://e65a91891394696db2faf2bffaa83925f7034fe3a81223c12d78b4badf9591956bebd4f4b886f63ff3ce089ed29600aa5441a8bce15c70429c28a410420f8199@172.16.0.12:30303

authority.toml:

[parity]
chain = "/parity/config/chain.json"
mode = "active"
auto_update = "none"
base_path = "/parity/.local/share/io.parity.ethereum"
keys_path = "/parity/keys"
no_persistent_txqueue = true

[rpc]
port = 8545
interface = "0.0.0.0"
cors = ["all"]
hosts = ["all"]
apis = ["web3", "eth", "net", "parity", "traces", "rpc", "personal", "parity_accounts", "signer", "parity_set"]

[secretstore]
disable = true

[ipfs]
enable = false

[network]
reserved_peers = "/parity/config/reserved_peers"
reserved_only = true
discovery = false

[websockets]
disable = true

[account]
password = ["/parity/authority.pwd"]

[mining]
usd_per_tx = "0"
force_sealing = true
gas_floor_target = "0"
reseal_min_period = 0

[misc]
logging = "miner=info,own_tx=info,sync=info,net=info"

chain.json:

{
        "name": "parityDockerized",
        "engine": {
                "authorityRound": {
                        "params": {
                                "stepDuration": "5",
                                "validators": {
                                        "multi": {
                                                "0": {
                                                        "list": [
                                                                "0x70d95cee845883e9fc6e1cd0359cec4ccd9e8621"
                                                        ]
                                                }
                                        }
                                }
                        }
                }
        },
        "params": {
                "gasLimitBoundDivisor": "0x400",
                "maximumExtraDataSize": "0x20",
                "minGasLimit": "0x7A1200",
                "networkID": "0x2323",
                "eip155Transition": 0,
                "validateChainIdTransition": 0,
                "eip140Transition": 0,
                "eip211Transition": 0,
                "eip214Transition": 0,
                "eip658Transition": 0,
                "eip150Transition": "0x0",
                "eip160Transition": "0x0",
                "eip161abcTransition": "0x0",
                "eip161dTransition": "0x0",
                "eip98Transition": "0x7fffffffffffff",
                "maxCodeSize": 24576,
                "maxCodeSizeTransition": "0x0"
        },
        "genesis": {
                "seal": {
                        "authorityRound": {
                                "step": "0x0",
                                "signature": "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
                        }
                },
                "difficulty": "0x20000",
                "gasLimit": "0x7A1200"
        },
        "accounts": {
                "0x0000000000000000000000000000000000000001": { "balance": "1", "builtin": { "name": "ecrecover", "pricing": { "linear": { "base": 3000, "word": 0 } } } },
          "0x0000000000000000000000000000000000000002": { "balance": "1", "builtin": { "name": "sha256", "pricing": { "linear": { "base": 60, "word": 12 } } } },
          "0x0000000000000000000000000000000000000003": { "balance": "1", "builtin": { "name": "ripemd160", "pricing": { "linear": { "base": 600, "word": 120 } } } },
          "0x0000000000000000000000000000000000000004": { "balance": "1", "builtin": { "name": "identity", "pricing": { "linear": { "base": 15, "word": 3 } } } },
          "0x0000000000000000000000000000000000000005": { "builtin": { "name": "modexp", "activate_at": 5067000, "pricing": { "modexp": { "divisor": 20 } } } },
          "0x0000000000000000000000000000000000000006": { "builtin": { "name": "alt_bn128_add", "activate_at": 5067000, "pricing": { "linear": { "base": 500, "word": 0 } } } },
          "0x0000000000000000000000000000000000000007": { "builtin": { "name": "alt_bn128_mul", "activate_at": 5067000, "pricing": { "linear": { "base": 40000, "word": 0 } } } },
          "0x0000000000000000000000000000000000000008": { "builtin": { "name": "alt_bn128_pairing", "activate_at": 5067000, "pricing": { "alt_bn128_pairing": { "base": 100000, "pair": 80000 } } } },
                "0x58f46a91865a1a4b2c77a18e8d22ebf3cc5178f9": {
                        "balance": "10000000000000000000000000000"
                },
                "0xeba885d922ec0577b3a3fcc9d14e6f819251799e": {
                        "balance": "10000000000000000000000000000"
                },
                "0x81f502f15beb550e7d53e9c49bbb6800828785e5": {
                        "balance": "10000000000000000000000000000"
                },
                "0x64207dfb8f26d1f77b23b63f69e372cbc450231c": {
                        "balance": "10000000000000000000000000000"
                },
                "0xe7e55f456b5c05c354261aead6da5f93e1dcf1e9": {
                        "balance": "10000000000000000000000000000"
                },
                "0xde2cda99a0d85bf8912c38bce04d5993d8029ea2": {
                        "balance": "10000000000000000000000000000"
                },
                "0x7db4b30ad23f6b6430192a2ecf427531d3335259": {
                        "balance": "10000000000000000000000000000"
                },
                "0x3fe4e4ef36aa7496d480389b96685169477b884d": {
                        "balance": "10000000000000000000000000000"
                },
                "0x06adc31b338bbc89fe79ac9927e6491b5f86198e": {
                        "balance": "10000000000000000000000000000"
                },
                "0x40534d72addd3355bf0286260357b09cffc43083": {
                        "balance": "10000000000000000000000000000"
                },
                "0x3e80e7e61def2c47e3d1fd3b0e2babb539ca0bf8": {
                        "balance": "10000000000000000000000000000"
                }
        }
}

Node logs are here (before and after the restart): https://github.com/miguelmartinezinf/parityLogs

Thanks in advance,

[joshua-mir]

@miguelmartinezinf it would be quite helpful if you were able to check if this still occurs in 2.6.0 beta as there have been significant changes in that release that may have addressed this issue

[miguelmtzinf]

@joshua-mir I will definitely try it on that version and I will come with feedback

[miguelmtzinf]

Hi, It is still happening with Parity-Ethereum/v2.6.0-beta-e38293b-20190708/x86_64-linux-gnu/rustc1.36.0

Number of open files increase little by little til reaching 4096 (open files limit). In this case, parity does not restart automatically (Idk why), and rejects any connections with no other reaction...
Logs: https://github.com/miguelmartinezinf/parityLogs/tree/beta

Thanks in advance,

[usd-yamazaki]

• Parity Ethereum version: Parity-Ethereum/v2.1.10-stable-b47e064-20181213/x86_64-linux-gnu/rustc1.31.0
• Operating system: Linux (Ubuntu 18.04.1 LTS)
• Installation: one-line installer
• Fully synchronized: yes
• Network: private (Aura)
• Restarted: yes

I am using a slightly older version, but I am experiencing similar symptoms.
If you monitor the number of file descriptors with the following command,
It seems to be increasing each time there is access in JSON-RPC over HTTP.

ls /proc/`pidof parity`/fd/ | wc -l

[usd-yamazaki]

I will supplement a little.
This number seems to increase when using eth_sendRawTransaction rather than accessing JSON-RPC.

[joshua-mir]

That's concerning, but quite helpful thanks!