Go-ethereum: with 1.9 and 4GB - Out of Memory

[pooleja]

Is 4 GB RAM no longer supported with 1.9? I was using a smaller server for testnet in the past with no issues.

System information

AWS T2.Medium (4 GB RAM)

# geth version
INFO [07-15|15:05:22.297] Bumping default cache on mainnet         provided=1024 updated=4096
WARN [07-15|15:05:22.299] Sanitizing cache to Go's GC limits       provided=4096 updated=1316
Geth
Version: 1.9.0-stable
Git Commit: 52f2461774bcb8cdd310f86b4bc501df5b783852
Architecture: amd64
Protocol Versions: [63]
Network Id: 1
Go Version: go1.11.5
Operating System: linux
GOPATH=
GOROOT=/usr/lib/go-1.11

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:    16.04
Codename:   xenial

Expected behaviour

Actual behaviour

After upgrading from 1.8 to 1.9, the syncing process failed with OOM error.

Backtrace

Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: fatal error: runtime: out of memory Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime stack: Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.throw(0x10c7a61, 0x16) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/panic.go:608 +0x72 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.sysMap(0xc0d8000000, 0x4000000, 0x2376d98) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/mem_linux.go:156 +0xc7 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.(*mheap).sysAlloc(0x234f000, 0x4000000, 0x1, 0x84e3ffed38) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/malloc.go:619 +0x1c7 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.(*mheap).grow(0x234f000, 0x41, 0x0) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/mheap.go:920 +0x42 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.(*mheap).allocSpanLocked(0x234f000, 0x41, 0x2376da8, 0x4497fb) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/mheap.go:848 +0x337 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.(*mheap).alloc_m(0x234f000, 0x41, 0x430101, 0xc00004ef40) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/mheap.go:692 +0x119 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.(*mheap).alloc.func1() Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/mheap.go:759 +0x4c Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.(*mheap).alloc(0x234f000, 0x41, 0x7f57e3010101, 0x453195) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/mheap.go:758 +0x8a Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.largeAlloc(0x81bf1, 0x101, 0xc000001601) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/malloc.go:1019 +0x97 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.mallocgc.func1() Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/malloc.go:914 +0x46 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.systemstack(0x7f57e98b8bc0) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/asm_amd64.s:351 +0x66 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.mstart() Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/proc.go:1229 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: goroutine 43 [running]: Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.systemstack_switch() Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/asm_amd64.s:311 fp=0xc03636d350 sp=0xc03636d348 pc=0x470500 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.mallocgc(0x81bf1, 0xefd8a0, 0x1, 0x6) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/malloc.go:913 +0x896 fp=0xc03636d3f0 sp=0xc03636d350 pc=0x421fb6 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.makeslice(0xefd8a0, 0x81bf1, 0x81bf1, 0x0, 0x6, 0x117e360) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/slice.go:70 +0x77 fp=0xc03636d420 sp=0xc03636d3f0 pc=0x459367 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy.Encode(0x0, 0x0, 0x0, 0xc0ca58a000, 0x6f346, 0x70000, 0x0, 0x16520, 0x23370) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy/encode.go:22 +0x228 fp=0xc03636d470 sp=0xc03636d420 pc=0x84d128 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/core/rawdb.(*freezerTable).Append(0xc00048dce0, 0xdf38b, 0xc0ca58a000, 0x6f346, 0x70000, 0x0, 0x0) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/core/rawdb/freezer_table.go:470 +0x3f2 fp=0xc03636d518 sp=0xc03636d470 pc=0x8affe2 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/core/rawdb.(*freezer).AppendAncient(0xc00028a380, 0xdf38b, 0xc07e884540, 0x20, 0x20, 0xc06429e480, 0x212, 0x240, 0xc072e0c000, 0x1e2d7, ...) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/core/rawdb/freezer.go:209 +0x81c fp=0xc03636d810 sp=0xc03636d518 pc=0x8a9fbc Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/core/rawdb.(*freezer).freeze(0xc00028a380, 0x1358840, 0xc000aec180) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/core/rawdb/freezer.go:327 +0xa07 fp=0xc03636dfc8 sp=0xc03636d810 pc=0x8ab277 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: runtime.goexit() Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc03636dfd0 sp=0xc03636dfc8 pc=0x4725e1 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/core/rawdb.NewDatabaseWithFreezer Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/core/rawdb/database.go:176 +0x1f1 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: goroutine 1 [chan receive, 18 minutes]: Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/node.(*Node).Wait(0xc000228f00) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/node/node.go:492 +0x7d Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: main.geth(0xc0000f4000, 0x0, 0x0) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:312 +0x9e Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli%2ev1.HandleAction(0xf18c20, 0x117d938, 0xc0000f4000, 0xc0000f4000, 0xc0004cff58) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli.v1/app.go:490 +0xc8 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli%2ev1.(*App).Run(0xc0001be1a0, 0xc0000a2000, 0x10, 0x10, 0x0, 0x0) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli.v1/app.go:264 +0x59d Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: main.main() Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:296 +0x55 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: goroutine 20 [chan receive, 18 minutes]: Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*nonrecursiveTree).dispatch(0xc00009e3c0, 0xc00009e300) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:36 +0xb6 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.newNonrecursiveTree Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:29 +0xdc Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: goroutine 21 [chan receive, 18 minutes]: Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*nonrecursiveTree).internal(0xc00009e3c0, 0xc00009e360) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:81 +0x7b Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.newNonrecursiveTree Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:30 +0x108 Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: goroutine 22 [syscall, 18 minutes]: Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: os/signal.signal_recv(0x0) Jul 15 14:59:44 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/sigqueue.go:139 +0x9c Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: os/signal.loop() Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/os/signal/signal_unix.go:23 +0x22 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by os/signal.init.0 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/os/signal/signal_unix.go:29 +0x41 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 23 [chan receive, 14 minutes]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc000180820) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:63 +0x9c Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/core.newTxSenderCacher Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:55 +0x9f Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 24 [chan receive, 14 minutes]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc000180820) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:63 +0x9c Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/core.newTxSenderCacher Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:55 +0x9f Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 29 [chan receive, 18 minutes]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).send(0xc000104000, 0xc000402000) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:254 +0xae Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).lazyinit Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:137 +0x17b Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 8 [chan receive]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/metrics.(*meterArbiter).tick(0x2342920) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/metrics/meter.go:289 +0x78 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/metrics.NewMeterForced Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/metrics/meter.go:70 +0xfd Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 9 [select]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).remote(0xc0002aa000, 0x0, 0x0, 0x0, 0x0) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/sealer.go:307 +0x526 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/consensus/ethash.New Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/ethash.go:496 +0x340 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 28 [syscall, 18 minutes]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: syscall.Syscall6(0xe8, 0x5, 0xc0003cbf6c, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/syscall/asm_linux_amd64.s:44 +0x5 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/golang.org/x/sys/unix.EpollWait(0x5, 0xc0003cbf6c, 0x1, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0x0) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:1689 +0x72 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).loop(0xc000104000, 0xc000402000) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:194 +0x8b Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).lazyinit Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:134 +0x128 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 13 [select, 18 minutes]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/accounts/keystore.(*watcher).loop(0xc0001a33c0) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/watch.go:94 +0x45a Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/accounts/keystore.(*watcher).start Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/watch.go:52 +0x61 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 30 [chan receive, 18 minutes]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).send(0xc000104000, 0xc000402000) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:254 +0xae Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).lazyinit Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:137 +0x17b Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 14 [select]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/accounts/keystore.(*KeyStore).updater(0xc0000e25a0) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/keystore.go:205 +0xc9 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/accounts/keystore.(*KeyStore).Subscribe Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/keystore.go:192 +0x123 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 15 [sleep]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: time.Sleep(0x3b9aca00) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/time.go:105 +0x14f Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).updater(0xc000226500) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:265 +0x3e Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).Subscribe Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:254 +0x123 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 16 [sleep]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: time.Sleep(0x3b9aca00) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/time.go:105 +0x14f Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).updater(0xc000226640) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:265 +0x3e Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).Subscribe Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:254 +0x123 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 34 [sleep]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: time.Sleep(0x3b9aca00) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/runtime/time.go:105 +0x14f Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).updater(0xc000226780) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:265 +0x3e Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).Subscribe Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:254 +0x123 Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 35 [select, 18 minutes]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/accounts.(*Manager).update(0xc0002aa700) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/manager.go:110 +0x19c Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/accounts.NewManager Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/manager.go:78 +0x50c Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: goroutine 36 [select]: Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc0002aa7e0) Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x12a Jul 15 14:59:46 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool Jul 15 14:59:48 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x177 Jul 15 14:59:48 ip-10-0-79-182 geth[23853]: goroutine 37 [runnable]: Jul 15 14:59:48 ip-10-0-79-182 geth[23853]: fmt.Sprintf(0x10ca323, 0x18, 0xc0000c75a0, 0x1, 0x1, 0x0, 0xc0146fd438) Jul 15 14:59:48 ip-10-0-79-182 geth[23853]: /usr/lib/go-1.11/src/fmt/print.go:201 +0xdd Jul 15 14:59:49 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).logf(0xc0000e2690, 0x10ca323, 0x18, 0xc0000c75a0, 0x1, 0x1) Jul 15 14:59:50 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session_util.go:34 +0x53 Jul 15 14:59:50 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*tOps).remove.func1() Jul 15 14:59:51 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.go:491 +0x1cf Jul 15 14:59:51 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache.(*mBucket).delete(0xc02efd7e90, 0xc00029a100, 0xc02f04f200, 0xc0ed10998d, 0x0, 0x712238, 0xc0481e1f10) Jul 15 14:59:52 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache/cache.go:188 +0x21e Jul 15 14:59:52 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache.(*Cache).delete(0xc00029a100, 0xc0481e1f10, 0xc0531d2000) Jul 15 14:59:52 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache/cache.go:329 +0x6d Jul 15 14:59:52 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache.(*Node).unref(0xc0481e1f10) Jul 15 14:59:53 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache/cache.go:623 +0x4d Jul 15 14:59:53 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache.(*Cache).Delete(0xc00029a100, 0x0, 0x712238, 0xc0576ce2c0, 0x234d000) Jul 15 14:59:54 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache/cache.go:440 +0x139 Jul 15 14:59:54 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*tOps).remove(0xc0004e0660, 0x4, 0x712238) Jul 15 14:59:55 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.go:487 +0x84 Jul 15 14:59:55 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).refLoop.func3(0xc0094f66c0) Jul 15 14:59:55 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session_util.go:108 +0x115 Jul 15 14:59:56 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).refLoop.func4() Jul 15 14:59:56 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session_util.go:176 +0x40f Jul 15 14:59:56 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc0000e2690) Jul 15 14:59:56 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session_util.go:187 +0x44c Jul 15 14:59:56 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.newSession Jul 15 14:59:56 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session.go:93 +0x2b4 Jul 15 14:59:56 ip-10-0-79-182 geth[23853]: goroutine 38 [select]: Jul 15 14:59:57 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0000aeb60) Jul 15 14:59:57 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0xd3 Jul 15 14:59:57 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB Jul 15 14:59:57 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:142 +0x40c Jul 15 14:59:57 ip-10-0-79-182 geth[23853]: goroutine 39 [select]: Jul 15 14:59:58 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0000aeb60) Jul 15 14:59:58 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101 +0xe7 Jul 15 14:59:59 ip-10-0-79-182 geth[23853]: created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB Jul 15 15:00:00 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:143 +0x42e Jul 15 15:00:00 ip-10-0-79-182 geth[23853]: goroutine 40 [runnable]: Jul 15 15:00:01 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.(*Reader).newBlockIter(0xc00b96dee0, 0xc0094f7180, 0x13460c0, 0xc0094f7180, 0x0, 0x1346000, 0xc0094f7180) Jul 15 15:00:01 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table/reader.go:736 +0x67 Jul 15 15:00:02 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.(*Reader).getDataIter(0xc00b96dee0, 0x1346a, 0x100e, 0x0, 0x1, 0x100e, 0x2) Jul 15 15:00:02 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table/reader.go:769 +0x119 Jul 15 15:00:04 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.(*Reader).getDataIterErr(0xc00b96dee0, 0x1346a, 0x100e, 0x0, 0x1, 0x0, 0x0) Jul 15 15:00:05 ip-10-0-79-182 geth[23853]: /build/ethereum-XN0HVv/ethereum-1.9.0+build19419+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table/reader.go:780 +0x128 Jul 15 15:00:05 ip-10-0-79-182 geth[23853]: github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.(*indexIter).Get(0xc05c63cf00, 0x8400a9, 0xc077cf4870) ^C^C

[ligi]

Thanks for the report! With 4GB you are at the very minimum of the requirement. And it seems 1.9 the memory usage increased a bit (e.g. for the freezer cc @karalabe ). We want to support 4GB again (rough consensus in the triage call) - but it could take a bit.
So short term fix: have more memory (sorry for that)
And we will close the issue once we feel geth should work fine with 4GB again.

[zigguratt]

Increasing swap space also helps. Since we have to use SSDs anyway, you can create a swap file/partition there. I recently finished syncing Geth 1.8.27-stable on an RK3399 device with only 2GB RAM. After sync finished it couldn't catch up to mainnet due to being murdered by the OOM killer. I increased swap from 1GB to 4GB and it's working fine now. It has now almost caught up to mainnet. I can imagine doing the same on 1.9 would be effective as well.

[pooleja]

Ok thanks @ligi for the info. And thanks @zigguratt for the suggestion... I will give it a try.

[karalabe]

We've increased the default memory allocation from 1GB to 4GB on mainnet in Geth v1.9.0. Go likes to use more memory though than allowed. You can try running with --cache=1024 to get the old behavior back, though generally the more memory you give it, the less junk it pushes to disk, the faster it imports blocks, etc.

[zigguratt]

My most recent sync was on a 4GB device where Geth was given --cache=256. The sync took about nine days.