Go-ethereum: Geth crashes often while syncing with blockchain

[Rohithzr]

System information

> geth version
Geth
Version: 1.8.6-stable
Git Commit: 12683feca7483f0b0bf425c3c520e2724f69f2aa
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.10
Operating System: linux
GOPATH=
GOROOT=/usr/lib/go-1.10

Instance type: AWS c5.large
CPU: 2
RAM: 4



md5-e8c8458d6e0b53571e5a4d1938e7fd8e



config.toml
...
NetworkId = 1
SyncMode = "fast"
LightPeers = 100
DatabaseCache = 1536
...



md5-d1540ce083f0c0fd830f4d73fe396ac8




error log (journalctl)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: internal/poll.(*FD).WaitWrite(0xc4865e5580, 0xf95b98, 0x1091760)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /usr/lib/go-1.10/src/internal/poll/fd_unix.go:440 +0x37
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: net.(*netFD).connect(0xc4865e5580, 0x1091760, 0xc43874bc80, 0x0, 0x0, 0x1087660, 0xc46b850ce0, 0x0, 0x0, 0x0, ...)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /usr/lib/go-1.10/src/net/fd_unix.go:152 +0x299
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: net.(*netFD).dial(0xc4865e5580, 0x1091760, 0xc43874bc80, 0x1094b20, 0x0, 0x1094b20, 0xc42d36a660, 0xc484592938, 0x5eed0e)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /usr/lib/go-1.10/src/net/sock_posix.go:142 +0xe9
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: net.socket(0x1091760, 0xc43874bc80, 0xf04e50, 0x3, 0x2, 0x1, 0x0, 0x0, 0x1094b20, 0x0, ...)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /usr/lib/go-1.10/src/net/sock_posix.go:93 +0x1a6
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: net.internetSocket(0x1091760, 0xc43874bc80, 0xf04e50, 0x3, 0x1094b20, 0x0, 0x1094b20, 0xc42d36a660, 0x1, 0x0, ...)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /usr/lib/go-1.10/src/net/ipsock_posix.go:141 +0x129
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: net.doDialTCP(0x1091760, 0xc43874bc80, 0xf04e50, 0x3, 0x0, 0xc42d36a660, 0x1b086b0, 0x0, 0x0)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /usr/lib/go-1.10/src/net/tcpsock_posix.go:62 +0xb9
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: net.dialTCP(0x1091760, 0xc43874bc80, 0xf04e50, 0x3, 0x0, 0xc42d36a660, 0x22e82c2de8, 0x1ae0040, 0x37e11b3f1)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /usr/lib/go-1.10/src/net/tcpsock_posix.go:58 +0xe4
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: net.dialSingle(0x1091760, 0xc43874bc80, 0xc4865e5500, 0x108ac20, 0xc42d36a660, 0x0, 0x0, 0x0, 0x0)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /usr/lib/go-1.10/src/net/dial.go:547 +0x375
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: net.dialSerial(0x1091760, 0xc43874bc80, 0xc4865e5500, 0xc4d4c35790, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /usr/lib/go-1.10/src/net/dial.go:515 +0x22d
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: net.(*Dialer).DialContext(0xc420103d40, 0x1091720, 0xc420032018, 0xf04e50, 0x3, 0xc46b850ca0, 0x13, 0x0, 0x0, 0x0, ...)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /usr/lib/go-1.10/src/net/dial.go:397 +0x678
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: net.(*Dialer).Dial(0xc420103d40, 0xf04e50, 0x3, 0xc46b850ca0, 0x13, 0x1, 0x0, 0x0, 0x1000000000001)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /usr/lib/go-1.10/src/net/dial.go:320 +0x75
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: github.com/ethereum/go-ethereum/p2p.TCPDialer.Dial(0xc420103d40, 0xc4487b6000, 0xc4a758f710, 0x60d456, 0x5, 0x44)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /build/ethereum-LEwR9U/ethereum-1.8.6+build13246+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/dial.go:65 +0x
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: github.com/ethereum/go-ethereum/p2p.(*dialTask).dial(0xc42d36a630, 0xc42026e000, 0xc4487b6000, 0xc42590f500, 0x5)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /build/ethereum-LEwR9U/ethereum-1.8.6+build13246+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/dial.go:349 +0
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: github.com/ethereum/go-ethereum/p2p.(*dialTask).Do(0xc42d36a630, 0xc42026e000)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /build/ethereum-LEwR9U/ethereum-1.8.6+build13246+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/dial.go:297 +0
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: github.com/ethereum/go-ethereum/p2p.(*Server).run.func2.1(0x1086320, 0xc42d36a630, 0xc42026e000, 0xc421951b60)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /build/ethereum-LEwR9U/ethereum-1.8.6+build13246+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:570
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: created by github.com/ethereum/go-ethereum/p2p.(*Server).run.func2
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /build/ethereum-LEwR9U/ethereum-1.8.6+build13246+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:570
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: goroutine 12154 [runnable]:
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: github.com/ethereum/go-ethereum/p2p.(*Server).runPeer(0xc42026e000, 0xc453b046c0)
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /build/ethereum-LEwR9U/ethereum-1.8.6+build13246+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:882
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]: created by github.com/ethereum/go-ethereum/p2p.(*Server).run
Apr 25 11:45:58 ip-172-31-40-7 geth[7735]:         /build/ethereum-LEwR9U/ethereum-1.8.6+build13246+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:646
Apr 25 11:45:58 ip-172-31-40-7 systemd[1]: geth.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 25 11:45:58 ip-172-31-40-7 systemd[1]: geth.service: Unit entered failed state.
Apr 25 11:45:58 ip-172-31-40-7 systemd[1]: geth.service: Failed with result 'exit-code'.
Apr 25 11:46:02 ip-172-31-40-7 systemd[1]: geth.service: Service hold-off time over, scheduling restart.
Apr 25 11:46:02 ip-172-31-40-7 systemd[1]: Stopped Ethereum Geth.
Apr 25 11:46:02 ip-172-31-40-7 systemd[1]: Started Ethereum Geth. (I have set it to auto-restart)
Apr 25 11:46:02 ip-172-31-40-7 geth[7780]: INFO [04-25|11:46:02] Maximum peer count                       ETH=25 LES=0 total=25
Apr 25 11:46:02 ip-172-31-40-7 geth[7780]: INFO [04-25|11:46:02] Starting peer-to-peer node               instance=Geth/v1.8.6-stable-12683fec/linux-amd64/go1.10
Apr 25 11:46:02 ip-172-31-40-7 geth[7780]: INFO [04-25|11:46:02] Allocated cache and file handles         database=/ethereumData/data/.ethereum/geth/chaindata cache=1536 handles=

Expected behaviour

/usr/bin/geth --config=/ethereumData/data/config.toml should start syncing in fast mode and run without crashing.

[karalabe]

How much memory does your VM have?

[Rohithzr]

@karalabe

How much memory does your VM have?

4 GB

> top
PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
7975 root      20   0 4809708 3.330g      0 S   1.6 91.9   4:55.63 geth

Hogs all the RAM and crashes, I end up getting this error

> cat /ethereumData/data/config.toml
-bash: fork: Cannot allocate memory

[karalabe]

Generally Go likes to use twice as much memory as you tell it can. This is the way the garbage collector works, it only cleans junk after it used at least as much as it previously left. If you permit 1.5GB for the database, and maybe 500MB for internal caches, then you have 2GB of active memory, so the garbage collector can easily blow that up to 4GB and crash your node.

My suggestion for a 4GB memory machine is to run with --cache=1024.

[karalabe]

Closing this, but please reopen if it does not solve the issue.