Go-ethereum: geth in private network, Out of Memory even after using up 8 GB RAM + 100 GB swapfile

[lbrep]

Geth version: 1.7.3-stable-4bb3c89d
OS & Version: Linux - Ubuntu 16.04
Network : Private network
No. of Peers none,
Geth Command used to start: /geth --datadir ./cdata2/ --port 30304 --networkid 19114 --rpc --rpcport 8546 --rpcaddr 0.0.0.0 --rpccorsdomain "*" --rpcapi "web3,eth,personal,net" console

ISSUE STATEMENT:
It was running fine until the block height reached around 2,81,000. But now suddenly the memory usage of my server instance has shooted up TOO High and the geth process gets killed immediately after the 8GB RAM is exhausted.

To resolve this I created a 100GB swapfile on the server. But even all 100GB got exhausted within minutes and then again the process was killed, saying "out of memory".

This also happens even when mining is stopped. The rate at which RAM is exhausted is as high as 100mb/sec.

[ekryski]

You should try upgrading to v1.8.3. There were a bunch of memory improvements in 1.8.0.

[lbrep]

@karalabe , Can you kindly suggest me a way out, what could be the possible reason behind this? This occurs even when no JSON-RPC calls are being made to the running instance of geth.

[ligi]

@lbrep you should listen to @ekryski - he made a valid point - why do you stick to such an old version and do not try with a recent version?

[lbrep]

@ligi , @ekryski , Thanks for a quick response. I have already tried with geth version 1.8.3 but the issue persists. Can you help me get some insight into what could have triggered this behaviour?

[ligi]

Thanks for trying 1.8.3 - unfortunate it persisted - perhaps @fjl can chip in as he is doing some work on analyzing/optimizing memory footprint.
But somehow this sounds like a serious bug/leak then - to get to the bottom of this it would be great to have access to the chain - is this possible by any chance?

[lbrep]

@ligi , Yes how would you like to connect to the chain?

[tcc-world]

@ligi, the problem occurs only when I open port 8546 on the network. Can it be a denial of service attack?

[ligi]

@tcc-king are you @lbrep ? bit confused about this now - and sorry just came back to this issue - would be interesting to know what kind of requests are coming in via 8546 for you - can you start geth with the --debug flag and share the output?

[lbrep]

@ligi, Yes both accounts are being used by me. Sorry for the confusion.
Port 8546 was opened at rpc address 0.0.0.0 which attracted external traffic to the chain. My best guess till now is that it was a DoS attack.
do you want server access to have more insight ?

[ajayatgit]

@lbrep I will recommend in general try to have --nodiscovery flag while running geth in private network.
This may have nothing to do with your problem, but at time by chance other people may also run geth with same network id.

[fjl]

@lbrep are you memory issues resolved after closing the port?

[lbrep]

@fjl , Though the issue was resolved by closing the port (--rpcaddress flag 127.0.01), what is the suggested strategy to avoid DDoS attacks if I need to open the port ? Shouldn't we treat this issue as a vulnerability ?

[jack16329]

same issue

[ovcharovvladimir]

running 1.8.16 geth with --testnet and get the same issue, out of memory after bunch of rpc request kind of getballance

[ovcharovvladimir]

any suggestions? Can anybody define the most stable eth client?

[holiman]

Most likely it's internet attackers trying to bruteforce your password via calling unlock. Don't leave RPC open to the internet by exposing it on 0.0.0.0 and if you do, don't enable personal.

A stack trace from the crash would show if this was indeed the case.

[ovcharovvladimir]

Thanks a lot @holiman , I have just found that! Which EIP describes this bug? As I understand anti-brute-force mechanism cannot get in time with the array in memory overloaded it?

[holiman]

There's no bug -- decrypting (or attempting to) a keystore file is very heavy on memory.