Go-ethereum: geth repeatedly crashes with SIGSEGV

Created on 21 Oct 2019  Â·  22Comments  Â·  Source: ethereum/go-ethereum

System information

Geth version: 1.9.6-stable
OS & Version: Linux 4.19.66-v7l+
Go: go1.13 linux/arm

Expected behaviour

Geth service syncs the chain.

Actual behaviour

Geth crashes repeatedly when service is started.

Steps to reproduce the behaviour

I'm not sure why this happens, but it did not appear during fast sync. When the node was (or almost) caught up with the chain, it started to crash repeatedly. Tried rebuilding geth today with the latest release (1.9.6), but crashes persist.

Backtrace

Oct 21 11:33:04 evl-geth systemd[1]: Started Geth Node. Oct 21 11:33:04 evl-geth geth[5793]: INFO [10-21|11:33:04.557] Maximum peer count ETH=50 LES=0 total=50 Oct 21 11:33:04 evl-geth geth[5793]: INFO [10-21|11:33:04.558] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory" Oct 21 11:33:04 evl-geth geth[5793]: INFO [10-21|11:33:04.562] Starting peer-to-peer node instance=Geth/v1.9.6-stable-bd059680/linux-arm/go1.13 Oct 21 11:33:04 evl-geth geth[5793]: INFO [10-21|11:33:04.563] Allocated trie memory caches clean=64.00MiB dirty=64.00MiB Oct 21 11:33:04 evl-geth geth[5793]: INFO [10-21|11:33:04.563] Allocated cache and file handles database=/mnt/ssd/ethereum/geth/chaindata cache=128.00MiB handles=262144 Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.000] Opened ancient database database=/mnt/ssd/ethereum/geth/chaindata/ancient Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.008] Initialised chain configuration config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: <nil> Engine: ethash}" Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.008] Disk storage enabled for ethash caches dir=/mnt/ssd/ethereum/geth/ethash count=3 Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.008] Disk storage enabled for ethash DAGs dir=/home/pi/.ethash count=2 Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.010] Initialising Ethereum protocol versions=[63] network=1 dbversion=7 Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.062] Loaded most recent local header number=8714751 hash=398925…4639c7 td=12312815174212912772480 age=1w3d19h Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.062] Loaded most recent local full block number=8714666 hash=6a936b…e922eb td=12312604899411814897379 age=1w3d19h Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.062] Loaded most recent local fast block number=8714751 hash=398925…4639c7 td=12312815174212912772480 age=1w3d19h Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.064] Loaded local transaction journal transactions=0 dropped=0 Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.065] Regenerated local transaction journal transactions=0 accounts=0 Oct 21 11:33:06 evl-geth geth[5793]: WARN [10-21|11:33:06.065] Switch sync mode from fast sync to full sync Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.159] New local node record seq=890 id=b0ae5886b9284f06 ip=127.0.0.1 udp=30303 tcp=30303 Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.162] Started P2P networking self=enode://e64123ecdd615fa43ba43ff014f2802afd5b39c70d2b9b48d2a8358063bfecd171dadbf042fd8c08239257cfbc34c6b3fbe5f79021a395776e8ee3ab4f0bbc57@127.0.0.1:30303 Oct 21 11:33:06 evl-geth geth[5793]: INFO [10-21|11:33:06.168] IPC endpoint opened url=/mnt/ssd/ethereum/geth.ipc Oct 21 11:33:16 evl-geth geth[5793]: INFO [10-21|11:33:16.162] Block synchronisation started Oct 21 11:33:17 evl-geth geth[5793]: panic: runtime error: invalid memory address or nil pointer dereference Oct 21 11:33:17 evl-geth geth[5793]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x1b8 pc=0x63d828] Oct 21 11:33:17 evl-geth geth[5793]: goroutine 263 [running]: Oct 21 11:33:17 evl-geth geth[5793]: github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestor(0x24bac80, 0x2672480, 0x4a5d440, 0x0, 0x0, 0xa, 0xa) Oct 21 11:33:17 evl-geth geth[5793]: #011/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:877 +0x1500 Oct 21 11:33:17 evl-geth geth[5793]: github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0x24bac80, 0x2672480, 0x5ca3af9d, 0x3c227bfb, 0xc13762f3, 0x5704518b, 0x7de03df6, 0xf0182a20, 0xc22975cd, 0x400256e9, ...) Oct 21 11:33:17 evl-geth geth[5793]: #011/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:443 +0x2a0 Oct 21 11:33:17 evl-geth geth[5793]: github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0x24bac80, 0x86fe140, 0x10, 0x5ca3af9d, 0x3c227bfb, 0xc13762f3, 0x5704518b, 0x7de03df6, 0xf0182a20, 0xc22975cd, ...) Oct 21 11:33:17 evl-geth geth[5793]: #011/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:411 +0x338 Oct 21 11:33:17 evl-geth geth[5793]: github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0x24bac80, 0x86fe140, 0x10, 0x5ca3af9d, 0x3c227bfb, 0xc13762f3, 0x5704518b, 0x7de03df6, 0xf0182a20, 0xc22975cd, ...) Oct 21 11:33:17 evl-geth geth[5793]: #011/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:323 +0x5c Oct 21 11:33:17 evl-geth geth[5793]: github.com/ethereum/go-ethereum/eth.(*ProtocolManager).synchronise(0x49a81b0, 0x8631500) Oct 21 11:33:17 evl-geth geth[5793]: #011/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:190 +0x1b0 Oct 21 11:33:17 evl-geth geth[5793]: created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer Oct 21 11:33:17 evl-geth geth[5793]: #011/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:155 +0x17c Oct 21 11:33:17 evl-geth systemd[1]: geth.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Oct 21 11:33:17 evl-geth systemd[1]: geth.service: Failed with result 'exit-code'.

picture m0ar

Most helpful comment

@diglos Yeah my Pi gets borked after a night of OOM crashes too, until it stops responding to SSH and needs a hard reboot

picture m0ar on 29 Nov 2019
👍4

All 22 comments

Fast sync block is 8714751, full sync block is 8714666. Then it switches to full-sync mode, and tries to import blocks one by one...

Could you try it with some more detailed logging in place?

holiman picture holiman on 21 Oct 2019

@holiman Absolutely, here is the output from --verbosity 5:


Click to expand

```
pi@evl-geth:~/go-ethereum $ geth --verbosity 5 --syncmode fast --cache 256 --datadir /mnt/ssd/ethereum
DEBUG[10-21|14:53:24.978] Sanitizing Go's GC trigger percent=100
INFO [10-21|14:53:24.981] Maximum peer count ETH=50 LES=0 total=50
INFO [10-21|14:53:24.982] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
DEBUG[10-21|14:53:24.982] FS scan times list=91.405µs set=2.741µs diff=4µs
TRACE[10-21|14:53:24.982] Started watching keystore folder path=/mnt/ssd/ethereum/keystore
INFO [10-21|14:53:24.986] Starting peer-to-peer node instance=Geth/v1.9.6-stable-bd059680/linux-arm/go1.13
INFO [10-21|14:53:24.986] Allocated trie memory caches clean=64.00MiB dirty=64.00MiB
INFO [10-21|14:53:24.986] Allocated cache and file handles database=/mnt/ssd/ethereum/geth/chaindata cache=128.00MiB handles=524288
DEBUG[10-21|14:53:26.418] Chain freezer table opened database=/mnt/ssd/ethereum/geth/chaindata/ancient table=hashes items=8624751 size=263.21MiB
DEBUG[10-21|14:53:26.419] Chain freezer table opened database=/mnt/ssd/ethereum/geth/chaindata/ancient table=bodies items=8624751 size=1.22GiB
DEBUG[10-21|14:53:26.419] Chain freezer table opened database=/mnt/ssd/ethereum/geth/chaindata/ancient table=receipts items=8624751 size=1.73GiB
DEBUG[10-21|14:53:26.420] Chain freezer table opened database=/mnt/ssd/ethereum/geth/chaindata/ancient table=diffs items=8624751 size=83.48MiB
DEBUG[10-21|14:53:26.420] Chain freezer table opened database=/mnt/ssd/ethereum/geth/chaindata/ancient table=headers items=8624751 size=1.59GiB
INFO [10-21|14:53:26.420] Opened ancient database database=/mnt/ssd/ethereum/geth/chaindata/ancient
DEBUG[10-21|14:53:26.422] Ancient blocks frozen already number=8714666 hash=6a936b…e922eb frozen=8624751
INFO [10-21|14:53:26.424] Initialised chain configuration config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: Engine: ethash}"
INFO [10-21|14:53:26.424] Disk storage enabled for ethash caches dir=/mnt/ssd/ethereum/geth/ethash count=3
INFO [10-21|14:53:26.424] Disk storage enabled for ethash DAGs dir=/home/pi/.ethash count=2
INFO [10-21|14:53:26.425] Initialising Ethereum protocol versions=[63] network=1 dbversion=7
INFO [10-21|14:53:26.485] Loaded most recent local header number=8714751 hash=398925…4639c7 td=12312815174212912772480 age=1w3d22h
INFO [10-21|14:53:26.485] Loaded most recent local full block number=8714666 hash=6a936b…e922eb td=12312604899411814897379 age=1w3d23h
INFO [10-21|14:53:26.485] Loaded most recent local fast block number=8714751 hash=398925…4639c7 td=12312815174212912772480 age=1w3d22h
DEBUG[10-21|14:53:26.487] Reinjecting stale transactions count=0
INFO [10-21|14:53:26.487] Loaded local transaction journal transactions=0 dropped=0
INFO [10-21|14:53:26.487] Regenerated local transaction journal transactions=0 accounts=0
WARN [10-21|14:53:26.488] Switch sync mode from fast sync to full sync
DEBUG[10-21|14:53:26.489] Recalculated downloader QoS values rtt=20s confidence=1.000 ttl=1m0s
DEBUG[10-21|14:53:26.519] UDP listener up addr=[::]:30303
DEBUG[10-21|14:53:26.519] TCP listener up addr=[::]:30303
TRACE[10-21|14:53:26.569] Found seed node in database id=15fd05f575b045c4 addr=45.77.242.229:30305 age=1h21m17.569187994s
INFO [10-21|14:53:26.572] New local node record seq=1350 id=b0ae5886b9284f06 ip=127.0.0.1 udp=30303 tcp=30303
TRACE[10-21|14:53:26.572] Found seed node in database id=2135b4036f3c0ce6 addr=185.137.234.120:30303 age=1h29m42.572587546s
TRACE[10-21|14:53:26.572] Found seed node in database id=27c4fbeca04fffa1 addr=45.77.187.182:30388 age=19m29.572800893s
TRACE[10-21|14:53:26.572] Found seed node in database id=32d4407180141db5 addr=52.59.208.29:30303 age=4m18.572998185s
TRACE[10-21|14:53:26.573] Found seed node in database id=42f90998419ffee0 addr=167.99.153.43:30303 age=3h21m34.573224476s
TRACE[10-21|14:53:26.573] Found seed node in database id=47892be54ad19f4c addr=165.22.85.95:30303 age=1h46m22.57345523s
TRACE[10-21|14:53:26.573] Found seed node in database id=4ab7d88b76997f33 addr=159.69.140.75:21226 age=6h50m12.573656336s
TRACE[10-21|14:53:26.573] Found seed node in database id=4bf7b1616e7e3916 addr=167.71.216.34:30303 age=1h26m18.573848573s
TRACE[10-21|14:53:26.573] Found seed node in database id=5b8494e9f8e1456b addr=103.29.71.224:28568 age=6h50m57.574037939s
TRACE[10-21|14:53:26.574] Found seed node in database id=62c1f10a22bc411b addr=95.217.47.59:21236 age=1h52m24.574233527s
TRACE[10-21|14:53:26.574] Found seed node in database id=6ffd7817bb47942b addr=80.158.38.240:30303 age=38m45.574427985s
TRACE[10-21|14:53:26.574] Found seed node in database id=78bca176bd6061dc addr=34.237.26.75:30303 age=3h46m14.574655165s
TRACE[10-21|14:53:26.574] Found seed node in database id=7ccb1a6b44086628 addr=68.183.181.212:30303 age=47m40.574847865s
TRACE[10-21|14:53:26.574] Found seed node in database id=860f00e69799a87f addr=42.232.29.50:20182 age=64h27m57.57503986s
TRACE[10-21|14:53:26.575] Found seed node in database id=8f1c6fc898f38a2d addr=174.138.22.71:30303 age=4h34m35.575235059s
TRACE[10-21|14:53:26.575] Found seed node in database id=9359cec51ae10cdb addr=178.62.238.75:30303 age=2h8m52.575458221s
TRACE[10-21|14:53:26.575] Found seed node in database id=9b51527bc7771288 addr=47.74.249.14:30303 age=5h30m54.57564805s
TRACE[10-21|14:53:26.575] Found seed node in database id=a9f5120426e780ba addr=46.101.195.141:30303 age=1h22m20.57583899s
TRACE[10-21|14:53:26.576] Found seed node in database id=aa2e2209dce8b6c8 addr=213.227.145.70:30303 age=46m54.576077596s
TRACE[10-21|14:53:26.576] Found seed node in database id=b36064d9d880e159 addr=173.199.115.77:30303 age=6h39m54.576268054s
TRACE[10-21|14:53:26.576] Found seed node in database id=bcf7c32b8bd78102 addr=54.92.202.189:30303 age=1h25m32.576446736s
TRACE[10-21|14:53:26.576] Found seed node in database id=bfee2a3bd6b668e0 addr=104.251.214.135:30303 age=3h5m38.57665299s
TRACE[10-21|14:53:26.576] Found seed node in database id=cd2162aa63111f2e addr=149.28.8.149:30303 age=6h45m39.576844023s
TRACE[10-21|14:53:26.577] Found seed node in database id=d4aafc4523eb5646 addr=178.63.69.19:30306 age=2h22m35.577060629s
TRACE[10-21|14:53:26.577] Found seed node in database id=d7237db08e3fc068 addr=52.28.245.66:21000 age=6h44m2.577246254s
TRACE[10-21|14:53:26.577] Found seed node in database id=e131359c042cb4c4 addr=94.176.236.103:30303 age=1h38m34.577523044s
TRACE[10-21|14:53:26.577] Found seed node in database id=edcab126072c2f95 addr=176.99.9.182:30334 age=3h46m49.577706985s
TRACE[10-21|14:53:26.577] Found seed node in database id=eedd3ad43113da8f addr=35.240.35.207:30304 age=2h2m50.577893499s
TRACE[10-21|14:53:26.578] Found seed node in database id=f1b907674c716bdd addr=75.152.97.59:30303 age=16m32.57807381s
TRACE[10-21|14:53:26.578] Found seed node in database id=0c6253589a8d244d addr=104.42.217.25:30303 age=9m3.578265324s
TRACE[10-21|14:53:26.578] Found seed node in database id=c845e51a5e470e44 addr=18.138.108.67:30303 age=3h45m42.578587853s
TRACE[10-21|14:53:26.578] Found seed node in database id=f23ac6da7c02f84a addr=3.209.45.79:30303 age=3h45m46.578832663s
TRACE[10-21|14:53:26.578] Found seed node in database id=71563f2370599df6 addr=34.255.23.113:30303 age=13m7.57907899s
TRACE[10-21|14:53:26.579] Found seed node in database id=2d1f1ff277435247 addr=35.158.244.151:30303 age=3m29.579319503s
TRACE[10-21|14:53:26.579] Found seed node in database id=1e1ea8b3d566d276 addr=52.187.207.27:30303 age=35m3.579655496s
TRACE[10-21|14:53:26.579] Found seed node in database id=32f7e89d30e83b68 addr=191.234.162.198:30303 age=30m26.579969952s
TRACE[10-21|14:53:26.580] Found seed node in database id=8d1da4c5bc682f8f addr=52.231.165.108:30303 age=6h44m30.580220798s
TRACE[10-21|14:53:26.580] Found seed node in database id=0c6253589a8d244d addr=104.42.217.25:30303 age=9m3.580414108s
TRACE[10-21|14:53:26.580] Found seed node in database id=29cca67d5db0fbb2 addr=5.1.83.226:30303 age=436573h53m26.580669213s
INFO [10-21|14:53:26.581] Started P2P networking self=enode://e64123ecdd615fa43ba43ff014f2802afd5b39c70d2b9b48d2a8358063bfecd171dadbf042fd8c08239257cfbc34c6b3fbe5f79021a395776e8ee3ab4f0bbc57@127.0.0.1:30303
TRACE[10-21|14:53:26.581] New dial task task="discovery lookup"
DEBUG[10-21|14:53:26.582] InProc registered namespace=admin
DEBUG[10-21|14:53:26.582] InProc registered namespace=admin
DEBUG[10-21|14:53:26.582] InProc registered namespace=debug
DEBUG[10-21|14:53:26.583] InProc registered namespace=web3
DEBUG[10-21|14:53:26.583] InProc registered namespace=eth
DEBUG[10-21|14:53:26.583] InProc registered namespace=eth
DEBUG[10-21|14:53:26.584] InProc registered namespace=eth
DEBUG[10-21|14:53:26.584] InProc registered namespace=txpool
DEBUG[10-21|14:53:26.584] InProc registered namespace=debug
DEBUG[10-21|14:53:26.584] InProc registered namespace=debug
DEBUG[10-21|14:53:26.584] InProc registered namespace=eth
DEBUG[10-21|14:53:26.585] InProc registered namespace=personal
DEBUG[10-21|14:53:26.585] InProc registered namespace=eth
DEBUG[10-21|14:53:26.585] InProc registered namespace=ethash
DEBUG[10-21|14:53:26.585] InProc registered namespace=eth
DEBUG[10-21|14:53:26.585] InProc registered namespace=eth
DEBUG[10-21|14:53:26.586] InProc registered namespace=eth
DEBUG[10-21|14:53:26.586] InProc registered namespace=miner
DEBUG[10-21|14:53:26.586] InProc registered namespace=eth
DEBUG[10-21|14:53:26.586] InProc registered namespace=admin
DEBUG[10-21|14:53:26.586] InProc registered namespace=debug
DEBUG[10-21|14:53:26.587] InProc registered namespace=debug
DEBUG[10-21|14:53:26.587] InProc registered namespace=net
DEBUG[10-21|14:53:26.587] IPC registered namespace=admin
DEBUG[10-21|14:53:26.588] IPC registered namespace=admin
DEBUG[10-21|14:53:26.588] IPC registered namespace=debug
DEBUG[10-21|14:53:26.588] IPC registered namespace=web3
DEBUG[10-21|14:53:26.588] IPC registered namespace=eth
DEBUG[10-21|14:53:26.588] IPC registered namespace=eth
DEBUG[10-21|14:53:26.589] IPC registered namespace=eth
DEBUG[10-21|14:53:26.589] IPC registered namespace=txpool
DEBUG[10-21|14:53:26.589] IPC registered namespace=debug
DEBUG[10-21|14:53:26.589] IPC registered namespace=debug
DEBUG[10-21|14:53:26.589] IPC registered namespace=eth
DEBUG[10-21|14:53:26.589] IPC registered namespace=personal
DEBUG[10-21|14:53:26.590] IPC registered namespace=eth
DEBUG[10-21|14:53:26.590] IPC registered namespace=ethash
DEBUG[10-21|14:53:26.590] IPC registered namespace=eth
DEBUG[10-21|14:53:26.590] IPC registered namespace=eth
DEBUG[10-21|14:53:26.590] IPC registered namespace=eth
DEBUG[10-21|14:53:26.590] IPC registered namespace=miner
DEBUG[10-21|14:53:26.591] IPC registered namespace=eth
DEBUG[10-21|14:53:26.591] IPC registered namespace=admin
DEBUG[10-21|14:53:26.591] IPC registered namespace=debug
DEBUG[10-21|14:53:26.591] IPC registered namespace=debug
DEBUG[10-21|14:53:26.591] IPC registered namespace=net
INFO [10-21|14:53:26.591] IPC endpoint opened url=/mnt/ssd/ethereum/geth.ipc
TRACE[10-21|14:53:26.593] >> FINDNODE/v4 id=d4aafc4523eb5646 addr=178.63.69.19:30306 err=nil
TRACE[10-21|14:53:26.593] >> FINDNODE/v4 id=c845e51a5e470e44 addr=18.138.108.67:30303 err=nil
TRACE[10-21|14:53:26.594] >> FINDNODE/v4 id=d7237db08e3fc068 addr=52.28.245.66:21000 err=nil
TRACE[10-21|14:53:26.616] Found seed node in database id=15fd05f575b045c4 addr=45.77.242.229:30305 age=1h21m17.61672728s
TRACE[10-21|14:53:26.616] Found seed node in database id=1a1bf2b0678671a5 addr=5.39.18.8:30303 age=19m2.616883814s
TRACE[10-21|14:53:26.617] Found seed node in database id=2135b4036f3c0ce6 addr=185.137.234.120:30303 age=1h29m42.617067513s
TRACE[10-21|14:53:26.617] Found seed node in database id=27c4fbeca04fffa1 addr=45.77.187.182:30388 age=19m29.617200362s
TRACE[10-21|14:53:26.617] Found seed node in database id=34df49bf45b4789c addr=167.114.153.134:30305 age=46m2.617343137s
TRACE[10-21|14:53:26.617] Found seed node in database id=3859f96d7a948090 addr=47.75.251.64:30303 age=2h26m5.617477689s
TRACE[10-21|14:53:26.617] Found seed node in database id=47892be54ad19f4c addr=165.22.85.95:30303 age=1h46m22.617615186s
TRACE[10-21|14:53:26.617] Found seed node in database id=4ab7d88b76997f33 addr=159.69.140.75:21226 age=6h50m12.617745109s
TRACE[10-21|14:53:26.617] Found seed node in database id=5b8494e9f8e1456b addr=103.29.71.224:28568 age=6h50m57.617853347s
TRACE[10-21|14:53:26.617] Found seed node in database id=5f3e7d5be31c4bfa addr=5.9.28.9:30303 age=61h49m40.617962123s
TRACE[10-21|14:53:26.618] Found seed node in database id=62c1f10a22bc411b addr=95.217.47.59:21236 age=1h52m24.61807799s
TRACE[10-21|14:53:26.618] Found seed node in database id=6cac7a531cb10c55 addr=138.201.30.201:30303 age=6h51m8.618187414s
TRACE[10-21|14:53:26.618] Found seed node in database id=7a3b129c12591094 addr=51.38.115.52:30303 age=34m42.618302393s
TRACE[10-21|14:53:26.618] Found seed node in database id=83141c6224866e6a addr=93.189.47.189:30303 age=3h27m23.618459352s
TRACE[10-21|14:53:26.618] Found seed node in database id=887e8f67b92e0550 addr=51.68.136.169:30303 age=6h49m16.618593534s
TRACE[10-21|14:53:26.618] Found seed node in database id=945041c794fa52f6 addr=154.197.27.248:30303 age=1h15m42.618752864s
TRACE[10-21|14:53:26.618] Found seed node in database id=94e7bb35e5cc4487 addr=165.22.16.119:30303 age=6h22m27.618868898s
TRACE[10-21|14:53:26.618] Found seed node in database id=999d4a9364794c24 addr=172.93.189.174:30303 age=27m11.618989877s
TRACE[10-21|14:53:26.619] Found seed node in database id=a62864a7d32f9860 addr=95.217.4.143:30303 age=3h16m20.619127226s
TRACE[10-21|14:53:26.619] Found seed node in database id=ae9be3dbe71c5259 addr=31.13.217.52:30303 age=1h31m57.619261241s
TRACE[10-21|14:53:26.619] Found seed node in database id=b43ec88b49748e65 addr=51.68.140.98:30303 age=38m9.619364869s
TRACE[10-21|14:53:26.619] Found seed node in database id=ba118cf9b6072ae6 addr=18.224.2.102:30304 age=13m16.6194657s
TRACE[10-21|14:53:26.619] Found seed node in database id=c0548e0d952703c0 addr=51.75.29.226:30303 age=6h34m53.619630696s
TRACE[10-21|14:53:26.619] Found seed node in database id=c321dcf5ddaaa424 addr=88.99.164.186:21210 age=6h45m50.619767711s
TRACE[10-21|14:53:26.619] Found seed node in database id=ce1d3ba7e9934f36 addr=54.37.85.202:30303 age=6h38m1.619884597s
TRACE[10-21|14:53:26.619] Found seed node in database id=dc137e09cd3dcd0f addr=13.124.207.8:30303 age=6h44m55.620019742s
TRACE[10-21|14:53:26.620] Found seed node in database id=e272f31a9ab9425a addr=157.245.94.240:30300 age=20m13.620132407s
TRACE[10-21|14:53:26.620] Found seed node in database id=e70dedc66e915c77 addr=164.68.103.48:30303 age=59m27.620266848s
TRACE[10-21|14:53:26.620] Found seed node in database id=e793c676acafd0a4 addr=207.246.124.59:30303 age=33m8.620370957s
TRACE[10-21|14:53:26.620] Found seed node in database id=e939aa4210b366ec addr=144.217.68.211:30306 age=6h23m38.62049001s
TRACE[10-21|14:53:26.620] Found seed node in database id=c845e51a5e470e44 addr=18.138.108.67:30303 age=3h45m42.620603322s
TRACE[10-21|14:53:26.620] Found seed node in database id=f23ac6da7c02f84a addr=3.209.45.79:30303 age=3h45m46.62070132s
TRACE[10-21|14:53:26.619] << NEIGHBORS/v4 id=d7237db08e3fc068 addr=52.28.245.66:21000 err=nil
TRACE[10-21|14:53:26.620] Found seed node in database id=71563f2370599df6 addr=34.255.23.113:30303 age=13m7.620829206s
TRACE[10-21|14:53:26.620] Found seed node in database id=2d1f1ff277435247 addr=35.158.244.151:30303 age=3m29.621050275s
TRACE[10-21|14:53:26.621] Found seed node in database id=1e1ea8b3d566d276 addr=52.187.207.27:30303 age=35m3.621157568s
TRACE[10-21|14:53:26.621] Found seed node in database id=32f7e89d30e83b68 addr=191.234.162.198:30303 age=30m26.621283528s
TRACE[10-21|14:53:26.621] Found seed node in database id=8d1da4c5bc682f8f addr=52.231.165.108:30303 age=6h44m30.621400396s
TRACE[10-21|14:53:26.621] Found seed node in database id=0c6253589a8d244d addr=104.42.217.25:30303 age=9m3.621520227s
TRACE[10-21|14:53:26.621] Found seed node in database id=29cca67d5db0fbb2 addr=5.1.83.226:30303 age=436573h53m26.621633391s
TRACE[10-21|14:53:26.622] >> FINDNODE/v4 id=ba118cf9b6072ae6 addr=18.224.2.102:30304 err=nil
TRACE[10-21|14:53:26.622] >> FINDNODE/v4 id=b36064d9d880e159 addr=173.199.115.77:30303 err=nil
TRACE[10-21|14:53:26.623] << NEIGHBORS/v4 id=d7237db08e3fc068 addr=52.28.245.66:21000 err=nil
TRACE[10-21|14:53:26.624] >> FINDNODE/v4 id=b43ec88b49748e65 addr=51.68.140.98:30303 err=nil
TRACE[10-21|14:53:26.625] >> FINDNODE/v4 id=d2a34a6e40eab598 addr=51.38.60.79:30303 err=nil
TRACE[10-21|14:53:26.640] << NEIGHBORS/v4 id=d4aafc4523eb5646 addr=178.63.69.19:30306 err=nil
TRACE[10-21|14:53:26.642] << NEIGHBORS/v4 id=d4aafc4523eb5646 addr=178.63.69.19:30306 err=nil
TRACE[10-21|14:53:26.643] >> PING/v4 id=d29dc715c5f43d88 addr=67.205.142.63:50000 err=nil
TRACE[10-21|14:53:26.656] << NEIGHBORS/v4 id=d2a34a6e40eab598 addr=51.38.60.79:30303 err=nil
TRACE[10-21|14:53:26.659] << NEIGHBORS/v4 id=d2a34a6e40eab598 addr=51.38.60.79:30303 err=nil
TRACE[10-21|14:53:26.660] >> PING/v4 id=d29bdc4db2e3dcb3 addr=182.61.108.243:21212 err=nil
TRACE[10-21|14:53:26.682] << NEIGHBORS/v4 id=b43ec88b49748e65 addr=51.68.140.98:30303 err=nil
TRACE[10-21|14:53:26.685] << NEIGHBORS/v4 id=b43ec88b49748e65 addr=51.68.140.98:30303 err=nil
TRACE[10-21|14:53:26.686] >> FINDNODE/v4 id=b0ccc92109355e95 addr=140.112.28.132:30303 err=nil
TRACE[10-21|14:53:26.733] << NEIGHBORS/v4 id=b36064d9d880e159 addr=173.199.115.77:30303 err=nil
TRACE[10-21|14:53:26.735] << NEIGHBORS/v4 id=b36064d9d880e159 addr=173.199.115.77:30303 err=nil
TRACE[10-21|14:53:26.736] >> FINDNODE/v4 id=b007271daaf38367 addr=52.36.69.215:30303 err=nil
TRACE[10-21|14:53:26.754] << NEIGHBORS/v4 id=ba118cf9b6072ae6 addr=18.224.2.102:30304 err=nil
TRACE[10-21|14:53:26.756] << NEIGHBORS/v4 id=ba118cf9b6072ae6 addr=18.224.2.102:30304 err=nil
TRACE[10-21|14:53:26.757] << PONG/v4 id=1fa8969fb5d07f60 addr=67.205.142.63:50000 err="unsolicited reply"
TRACE[10-21|14:53:26.759] >> FINDNODE/v4 id=b09d9f90cead8dcc addr=149.28.118.15:30303 err=nil
TRACE[10-21|14:53:26.850] << NEIGHBORS/v4 id=c845e51a5e470e44 addr=18.138.108.67:30303 err=nil
TRACE[10-21|14:53:26.853] << NEIGHBORS/v4 id=c845e51a5e470e44 addr=18.138.108.67:30303 err=nil
TRACE[10-21|14:53:26.854] >> PING/v4 id=d29772b9b5f29c15 addr=128.199.41.168:50000 err=nil
TRACE[10-21|14:53:26.873] << PONG/v4 id=40f17881667e7bff addr=128.199.41.168:50000 err="unsolicited reply"
TRACE[10-21|14:53:26.887] << NEIGHBORS/v4 id=b09d9f90cead8dcc addr=149.28.118.15:30303 err=nil
TRACE[10-21|14:53:26.889] << NEIGHBORS/v4 id=b09d9f90cead8dcc addr=149.28.118.15:30303 err=nil
TRACE[10-21|14:53:26.890] >> FINDNODE/v4 id=b0b4b7b863af4a30 addr=35.171.255.98:30303 err=nil
TRACE[10-21|14:53:26.917] << PING/v4 id=b007271daaf38367 addr=52.36.69.215:30303 err=nil
TRACE[10-21|14:53:26.918] >> PONG/v4 id=b007271daaf38367 addr=52.36.69.215:30303 err=nil
TRACE[10-21|14:53:26.969] << NEIGHBORS/v4 id=b0ccc92109355e95 addr=140.112.28.132:30303 err=nil
TRACE[10-21|14:53:26.971] << NEIGHBORS/v4 id=b0ccc92109355e95 addr=140.112.28.132:30303 err=nil
TRACE[10-21|14:53:26.972] >> FINDNODE/v4 id=b0a8a7fa57b40b1b addr=13.209.61.22:30303 err=nil
TRACE[10-21|14:53:26.998] << NEIGHBORS/v4 id=b0b4b7b863af4a30 addr=35.171.255.98:30303 err=nil
TRACE[10-21|14:53:27.001] << NEIGHBORS/v4 id=b0b4b7b863af4a30 addr=35.171.255.98:30303 err=nil
TRACE[10-21|14:53:27.002] >> FINDNODE/v4 id=b0a6ffc3ac4688f3 addr=206.189.224.11:30305 err=nil
TRACE[10-21|14:53:27.104] << NEIGHBORS/v4 id=b007271daaf38367 addr=52.36.69.215:30303 err=nil
TRACE[10-21|14:53:27.106] << NEIGHBORS/v4 id=b007271daaf38367 addr=52.36.69.215:30303 err=nil
TRACE[10-21|14:53:27.112] << NEIGHBORS/v4 id=b0a6ffc3ac4688f3 addr=206.189.224.11:30305 err=nil
TRACE[10-21|14:53:27.112] >> PING/v4 id=b0a77c36e1e28893 addr=157.245.205.31:50000 err=nil
TRACE[10-21|14:53:27.114] << NEIGHBORS/v4 id=b0a6ffc3ac4688f3 addr=206.189.224.11:30305 err=nil
TRACE[10-21|14:53:27.115] >> FINDNODE/v4 id=b0a75525c39086c4 addr=104.214.217.23:30303 err=nil
TRACE[10-21|14:53:27.142] << NEIGHBORS/v4 id=b0a75525c39086c4 addr=104.214.217.23:30303 err=nil
TRACE[10-21|14:53:27.145] << NEIGHBORS/v4 id=b0a75525c39086c4 addr=104.214.217.23:30303 err=nil
TRACE[10-21|14:53:27.146] >> FINDNODE/v4 id=b0a5ce1b3c3b951c addr=198.11.182.18:30303 err=nil
TRACE[10-21|14:53:27.283] << NEIGHBORS/v4 id=b0a8a7fa57b40b1b addr=13.209.61.22:30303 err=nil
TRACE[10-21|14:53:27.285] << NEIGHBORS/v4 id=b0a8a7fa57b40b1b addr=13.209.61.22:30303 err=nil
TRACE[10-21|14:53:27.286] >> PING/v4 id=b0a3ba5318812bf7 addr=47.244.205.240:30303 err=nil
TRACE[10-21|14:53:27.397] << PONG/v4 id=0cffad8b3bc8ee94 addr=157.245.205.31:50000 err="unsolicited reply"
TRACE[10-21|14:53:27.645] Findnode failed id=b0a5ce1b3c3b951c failcount=3 err="RPC timeout"
TRACE[10-21|14:53:27.645] >> FINDNODE/v4 id=d29dc715c5f43d88 addr=67.205.142.63:50000 err=nil
TRACE[10-21|14:53:27.646] >> FINDNODE/v4 id=b0bc0bae83bba53b addr=24.93.179.155:30303 err=nil
TRACE[10-21|14:53:27.661] >> FINDNODE/v4 id=d29bdc4db2e3dcb3 addr=182.61.108.243:21212 err=nil
TRACE[10-21|14:53:27.811] << NEIGHBORS/v4 id=b0bc0bae83bba53b addr=24.93.179.155:30303 err=nil
TRACE[10-21|14:53:27.818] << NEIGHBORS/v4 id=b0bc0bae83bba53b addr=24.93.179.155:30303 err=nil
TRACE[10-21|14:53:27.820] >> FINDNODE/v4 id=b0b35544b703ebff addr=95.168.186.250:30303 err=nil
TRACE[10-21|14:53:27.861] << NEIGHBORS/v4 id=b0b35544b703ebff addr=95.168.186.250:30303 err=nil
TRACE[10-21|14:53:27.865] >> FINDNODE/v4 id=d29772b9b5f29c15 addr=128.199.41.168:50000 err=nil
TRACE[10-21|14:53:27.867] << NEIGHBORS/v4 id=b0b35544b703ebff addr=95.168.186.250:30303 err=nil
TRACE[10-21|14:53:27.869] >> PING/v4 id=b0b04543ed500b0f addr=159.65.87.68:50000 err=nil
TRACE[10-21|14:53:27.899] << PONG/v4 id=efe0b09dba278d6d addr=159.65.87.68:50000 err="unsolicited reply"
TRACE[10-21|14:53:28.113] >> FINDNODE/v4 id=b0a77c36e1e28893 addr=157.245.205.31:50000 err=nil
TRACE[10-21|14:53:28.145] Findnode failed id=d29dc715c5f43d88 failcount=3 err="RPC timeout"
TRACE[10-21|14:53:28.147] >> PING/v4 id=d296e4811d30d123 addr=164.15.10.134:1052 err=nil
TRACE[10-21|14:53:28.161] Findnode failed id=d29bdc4db2e3dcb3 failcount=4 err="RPC timeout"
TRACE[10-21|14:53:28.163] >> PING/v4 id=d29587632448cb21 addr=167.99.82.0:50000 err=nil
TRACE[10-21|14:53:28.194] << PONG/v4 id=828c8262be1d3c61 addr=167.99.82.0:50000 err="unsolicited reply"
TRACE[10-21|14:53:28.289] >> FINDNODE/v4 id=b0a3ba5318812bf7 addr=47.244.205.240:30303 err=nil
TRACE[10-21|14:53:28.361] Findnode failed id=d29772b9b5f29c15 failcount=1 err="RPC timeout"
TRACE[10-21|14:53:28.363] >> PING/v4 id=d294cb9bcfb89fce addr=168.61.161.73:30303 err=nil
TRACE[10-21|14:53:28.613] Findnode failed id=b0a77c36e1e28893 failcount=109 err="RPC timeout"
TRACE[10-21|14:53:28.613] Too many findnode failures, dropping id=b0a77c36e1e28893 failcount=109
TRACE[10-21|14:53:28.616] >> FINDNODE/v4 id=b08e5ca5007081f8 addr=95.216.34.99:20954 err=nil
DEBUG[10-21|14:53:28.619] Couldn't add port mapping proto=tcp extport=30303 intport=30303 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered"
DEBUG[10-21|14:53:28.620] Couldn't add port mapping proto=udp extport=30303 intport=30303 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered"
TRACE[10-21|14:53:28.647] << NEIGHBORS/v4 id=b08e5ca5007081f8 addr=95.216.34.99:20954 err=nil
TRACE[10-21|14:53:28.654] << NEIGHBORS/v4 id=b08e5ca5007081f8 addr=95.216.34.99:20954 err=nil
TRACE[10-21|14:53:28.656] >> FINDNODE/v4 id=b08dd798c20605f9 addr=159.100.245.37:30303 err=nil
TRACE[10-21|14:53:28.709] << NEIGHBORS/v4 id=b08dd798c20605f9 addr=159.100.245.37:30303 err=nil
TRACE[10-21|14:53:28.715] << NEIGHBORS/v4 id=b08dd798c20605f9 addr=159.100.245.37:30303 err=nil
TRACE[10-21|14:53:28.717] >> PING/v4 id=b0ae133fa9174778 addr=138.68.252.162:50000 err=nil
TRACE[10-21|14:53:28.787] Findnode failed id=b0a3ba5318812bf7 failcount=24 err="RPC timeout"
TRACE[10-21|14:53:28.787] Too many findnode failures, dropping id=b0a3ba5318812bf7 failcount=24
TRACE[10-21|14:53:28.789] >> PING/v4 id=b0aa8e13085a58f2 addr=157.245.206.138:50000 err=nil
TRACE[10-21|14:53:28.871] >> FINDNODE/v4 id=b0b04543ed500b0f addr=159.65.87.68:50000 err=nil
TRACE[10-21|14:53:28.894] << PONG/v4 id=ba1435cac4cef010 addr=138.68.252.162:50000 err="unsolicited reply"
TRACE[10-21|14:53:29.079] << PONG/v4 id=29e22c15c08a6782 addr=157.245.206.138:50000 err="unsolicited reply"
TRACE[10-21|14:53:29.149] >> FINDNODE/v4 id=d296e4811d30d123 addr=164.15.10.134:1052 err=nil
TRACE[10-21|14:53:29.164] >> FINDNODE/v4 id=d29587632448cb21 addr=167.99.82.0:50000 err=nil
TRACE[10-21|14:53:29.365] >> FINDNODE/v4 id=d294cb9bcfb89fce addr=168.61.161.73:30303 err=nil
TRACE[10-21|14:53:29.370] Findnode failed id=b0b04543ed500b0f failcount=27 err="RPC timeout"
TRACE[10-21|14:53:29.370] Too many findnode failures, dropping id=b0b04543ed500b0f failcount=27
TRACE[10-21|14:53:29.372] >> PING/v4 id=b0a854822e6e3ffe addr=164.15.10.134:1123 err=nil
TRACE[10-21|14:53:29.649] Findnode failed id=d296e4811d30d123 failcount=4 err="RPC timeout"
TRACE[10-21|14:53:29.651] >> PING/v4 id=d293e23a0074d53f addr=159.89.20.82:30303 err=nil
TRACE[10-21|14:53:29.664] Findnode failed id=d29587632448cb21 failcount=2 err="RPC timeout"
TRACE[10-21|14:53:29.667] >> PING/v4 id=d29377beb3841c83 addr=167.99.12.22:50000 err=nil
TRACE[10-21|14:53:29.675] << PONG/v4 id=d293e23a0074d53f addr=159.89.20.82:30303 err=nil
TRACE[10-21|14:53:29.676] Accepted connection addr=31.164.124.43:40026
TRACE[10-21|14:53:29.719] >> FINDNODE/v4 id=b0ae133fa9174778 addr=138.68.252.162:50000 err=nil
TRACE[10-21|14:53:29.774] << PONG/v4 id=eb8686c8abc21417 addr=167.99.12.22:50000 err="unsolicited reply"
DEBUG[10-21|14:53:29.774] Adding p2p peer id=3b7062ace89b8232 conn=inbound addr=31.164.124.43:40026 peers=1 name=Geth/v1.9.3-unstable...
TRACE[10-21|14:53:29.774] Connection set up id=3b7062ace89b8232 addr=31.164.124.43:40026 conn=inbound inbound=true
TRACE[10-21|14:53:29.775] Starting protocol eth/63 id=3b7062ace89b8232 conn=inbound
DEBUG[10-21|14:53:29.775] Ethereum peer connected id=3b7062ace89b8232 conn=inbound name=Geth/v1.9.3-unstable-a978adfd-20190826/linux-amd64/go1.9.3
DEBUG[10-21|14:53:29.776] Ethereum handshake failed id=3b7062ace89b8232 conn=inbound err="Genesis block mismatch - b726b36f6d811b33 (!= d4e56740f876aef8)"
TRACE[10-21|14:53:29.776] Protocol eth/63 failed id=3b7062ace89b8232 conn=inbound err="Genesis block mismatch - b726b36f6d811b33 (!= d4e56740f876aef8)"
DEBUG[10-21|14:53:29.778] Removing p2p peer id=3b7062ace89b8232 conn=inbound addr=31.164.124.43:40026 peers=0 duration=4.110ms req=false err="Genesis block mismatch - b726b36f6d811b33 (!= d4e56740f876aef8)"
TRACE[10-21|14:53:29.791] >> FINDNODE/v4 id=b0aa8e13085a58f2 addr=157.245.206.138:50000 err=nil
TRACE[10-21|14:53:29.864] Findnode failed id=d294cb9bcfb89fce failcount=2 err="RPC timeout"
TRACE[10-21|14:53:29.867] >> PING/v4 id=d292d0bd601d724b addr=128.199.35.72:50000 err=nil
TRACE[10-21|14:53:29.887] << PONG/v4 id=d83b157765b00b81 addr=128.199.35.72:50000 err="unsolicited reply"
TRACE[10-21|14:53:30.177] >> FINDNODE/v4 id=d293e23a0074d53f addr=159.89.20.82:30303 err=nil
TRACE[10-21|14:53:30.201] << PING/v4 id=d293e23a0074d53f addr=159.89.20.82:30303 err=nil
TRACE[10-21|14:53:30.203] >> PONG/v4 id=d293e23a0074d53f addr=159.89.20.82:30303 err=nil
TRACE[10-21|14:53:30.218] Findnode failed id=b0ae133fa9174778 failcount=601 err="RPC timeout"
TRACE[10-21|14:53:30.218] Too many findnode failures, dropping id=b0ae133fa9174778 failcount=601
TRACE[10-21|14:53:30.220] >> PING/v4 id=b0a8ba2f2fc227e7 addr=178.62.40.90:50000 err=nil
TRACE[10-21|14:53:30.241] << NEIGHBORS/v4 id=d293e23a0074d53f addr=159.89.20.82:30303 err=nil
TRACE[10-21|14:53:30.245] Invalid neighbor node received ip=157.230.121.200 addr=159.89.20.82:30303 err="low port"
TRACE[10-21|14:53:30.246] << NEIGHBORS/v4 id=d293e23a0074d53f addr=159.89.20.82:30303 err=nil
DEBUG[10-21|14:53:30.246] IP exceeds bucket limit ip=164.15.10.134
TRACE[10-21|14:53:30.249] >> FINDNODE/v4 id=d29f82f0669d35b3 addr=164.15.10.134:1450 err=nil
TRACE[10-21|14:53:30.250] << PONG/v4 id=624beed8fbe560c7 addr=178.62.40.90:50000 err="unsolicited reply"
TRACE[10-21|14:53:30.290] Accepted connection addr=175.126.163.8:39436
TRACE[10-21|14:53:30.291] Findnode failed id=b0aa8e13085a58f2 failcount=537 err="RPC timeout"
TRACE[10-21|14:53:30.291] Too many findnode failures, dropping id=b0aa8e13085a58f2 failcount=537
TRACE[10-21|14:53:30.296] >> PING/v4 id=b0a51f21dd698b51 addr=13.126.85.24:30303 err=nil
TRACE[10-21|14:53:30.374] >> FINDNODE/v4 id=b0a854822e6e3ffe addr=164.15.10.134:1123 err=nil
DEBUG[10-21|14:53:30.606] Adding p2p peer id=8f112bd35cc4e73c conn=inbound addr=175.126.163.8:39436 peers=1 name=Geth/v1.8.23-stable-...
TRACE[10-21|14:53:30.607] Connection set up id=8f112bd35cc4e73c addr=175.126.163.8:39436 conn=inbound inbound=true
TRACE[10-21|14:53:30.607] Starting protocol eth/63 id=8f112bd35cc4e73c conn=inbound
DEBUG[10-21|14:53:30.608] Ethereum peer connected id=8f112bd35cc4e73c conn=inbound name=Geth/v1.8.23-stable-c9427004/linux-amd64/go1.11.5
TRACE[10-21|14:53:30.610] Registering sync peer peer=8f112bd35cc4e73c
DEBUG[10-21|14:53:30.610] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8552447 skip=0 reverse=false
TRACE[10-21|14:53:30.668] >> FINDNODE/v4 id=d29377beb3841c83 addr=167.99.12.22:50000 err=nil
TRACE[10-21|14:53:30.748] Findnode failed id=d29f82f0669d35b3 failcount=1 err="RPC timeout"
TRACE[10-21|14:53:30.750] >> PING/v4 id=d290414037655377 addr=157.245.206.68:50000 err=nil
TRACE[10-21|14:53:30.869] >> FINDNODE/v4 id=d292d0bd601d724b addr=128.199.35.72:50000 err=nil
TRACE[10-21|14:53:30.873] Findnode failed id=b0a854822e6e3ffe failcount=509 err="RPC timeout"
TRACE[10-21|14:53:30.873] Too many findnode failures, dropping id=b0a854822e6e3ffe failcount=509
TRACE[10-21|14:53:30.876] >> PING/v4 id=b0a0fb98bf2e3b3e addr=67.205.142.63:50000 err=nil
TRACE[10-21|14:53:30.982] << PONG/v4 id=1fa8969fb5d07f60 addr=67.205.142.63:50000 err="unsolicited reply"
TRACE[10-21|14:53:31.038] << PONG/v4 id=ddabfe72c4aeef0f addr=157.245.206.68:50000 err="unsolicited reply"
TRACE[10-21|14:53:31.167] Findnode failed id=d29377beb3841c83 failcount=2 err="RPC timeout"
TRACE[10-21|14:53:31.169] >> PING/v4 id=d28f4d17a1daf965 addr=164.15.10.134:1370 err=nil
TRACE[10-21|14:53:31.222] >> FINDNODE/v4 id=b0a8ba2f2fc227e7 addr=178.62.40.90:50000 err=nil
TRACE[10-21|14:53:31.298] >> FINDNODE/v4 id=b0a51f21dd698b51 addr=13.126.85.24:30303 err=nil
TRACE[10-21|14:53:31.368] Findnode failed id=d292d0bd601d724b failcount=1 err="RPC timeout"
TRACE[10-21|14:53:31.370] >> PING/v4 id=d28d9aa3da17090e addr=78.41.194.15:43026 err=nil
TRACE[10-21|14:53:31.721] Findnode failed id=b0a8ba2f2fc227e7 failcount=596 err="RPC timeout"
TRACE[10-21|14:53:31.721] Too many findnode failures, dropping id=b0a8ba2f2fc227e7 failcount=596
TRACE[10-21|14:53:31.724] >> PING/v4 id=b0b47c10f130cd71 addr=13.114.151.77:30303 err=nil
TRACE[10-21|14:53:31.752] >> FINDNODE/v4 id=d290414037655377 addr=157.245.206.68:50000 err=nil
TRACE[10-21|14:53:31.798] Findnode failed id=b0a51f21dd698b51 failcount=743 err="RPC timeout"
TRACE[10-21|14:53:31.798] Too many findnode failures, dropping id=b0a51f21dd698b51 failcount=743
TRACE[10-21|14:53:31.878] >> FINDNODE/v4 id=b0a0fb98bf2e3b3e addr=67.205.142.63:50000 err=nil
TRACE[10-21|14:53:32.170] >> FINDNODE/v4 id=d28f4d17a1daf965 addr=164.15.10.134:1370 err=nil
TRACE[10-21|14:53:32.251] Findnode failed id=d290414037655377 failcount=4 err="RPC timeout"
TRACE[10-21|14:53:32.252] >> PING/v4 id=d28d1c78c8340e36 addr=39.108.164.135:30303 err=nil
TRACE[10-21|14:53:32.371] >> FINDNODE/v4 id=d28d9aa3da17090e addr=78.41.194.15:43026 err=nil
TRACE[10-21|14:53:32.377] Findnode failed id=b0a0fb98bf2e3b3e failcount=572 err="RPC timeout"
TRACE[10-21|14:53:32.377] Too many findnode failures, dropping id=b0a0fb98bf2e3b3e failcount=572
TRACE[10-21|14:53:32.670] Findnode failed id=d28f4d17a1daf965 failcount=3 err="RPC timeout"
TRACE[10-21|14:53:32.671] >> PING/v4 id=d28918e9b337c0b0 addr=85.250.222.230:30303 err=nil
TRACE[10-21|14:53:32.724] >> FINDNODE/v4 id=b0b47c10f130cd71 addr=13.114.151.77:30303 err=nil
TRACE[10-21|14:53:32.790] Accepted connection addr=13.125.4.161:59046
TRACE[10-21|14:53:32.871] Findnode failed id=d28d9aa3da17090e failcount=12 err="RPC timeout"
TRACE[10-21|14:53:32.871] Too many findnode failures, dropping id=d28d9aa3da17090e failcount=12
TRACE[10-21|14:53:32.872] >> PING/v4 id=d28677e1d08b5cbd addr=47.56.185.214:30303 err=nil
DEBUG[10-21|14:53:33.099] Adding p2p peer id=649eb641681bc33b conn=inbound addr=13.125.4.161:59046 peers=2 name=Geth/v1.9.4-unstable...
TRACE[10-21|14:53:33.100] Connection set up id=649eb641681bc33b addr=13.125.4.161:59046 conn=inbound inbound=true
TRACE[10-21|14:53:33.100] Starting protocol eth/63 id=649eb641681bc33b conn=inbound
DEBUG[10-21|14:53:33.100] Ethereum peer connected id=649eb641681bc33b conn=inbound name=Geth/v1.9.4-unstable-b4a4a4db-20190903/linux-amd64/go1.12.7
DEBUG[10-21|14:53:33.101] Ethereum handshake failed id=649eb641681bc33b conn=inbound err="Genesis block mismatch - 71e0e1fcf9967616 (!= d4e56740f876aef8)"
TRACE[10-21|14:53:33.101] Protocol eth/63 failed id=649eb641681bc33b conn=inbound err="Genesis block mismatch - 71e0e1fcf9967616 (!= d4e56740f876aef8)"
DEBUG[10-21|14:53:33.103] Removing p2p peer id=649eb641681bc33b conn=inbound addr=13.125.4.161:59046 peers=1 duration=3.470ms req=false err="Genesis block mismatch - 71e0e1fcf9967616 (!= d4e56740f876aef8)"
TRACE[10-21|14:53:33.224] Findnode failed id=b0b47c10f130cd71 failcount=440 err="RPC timeout"
TRACE[10-21|14:53:33.224] Too many findnode failures, dropping id=b0b47c10f130cd71 failcount=440
TRACE[10-21|14:53:33.227] >> FINDNODE/v4 id=7ccb1a6b44086628 addr=68.183.181.212:30303 err=nil
TRACE[10-21|14:53:33.227] >> FINDNODE/v4 id=71563f2370599df6 addr=34.255.23.113:30303 err=nil
TRACE[10-21|14:53:33.229] >> FINDNODE/v4 id=78bca176bd6061dc addr=34.237.26.75:30303 err=nil
TRACE[10-21|14:53:33.255] >> FINDNODE/v4 id=d28d1c78c8340e36 addr=39.108.164.135:30303 err=nil
TRACE[10-21|14:53:33.295] << NEIGHBORS/v4 id=71563f2370599df6 addr=34.255.23.113:30303 err=nil
TRACE[10-21|14:53:33.301] << NEIGHBORS/v4 id=71563f2370599df6 addr=34.255.23.113:30303 err=nil
TRACE[10-21|14:53:33.304] >> FINDNODE/v4 id=72d48f27cd1ee7f6 addr=54.68.121.166:20203 err=nil
TRACE[10-21|14:53:33.367] << NEIGHBORS/v4 id=78bca176bd6061dc addr=34.237.26.75:30303 err=nil
TRACE[10-21|14:53:33.373] << NEIGHBORS/v4 id=78bca176bd6061dc addr=34.237.26.75:30303 err=nil
TRACE[10-21|14:53:33.376] >> FINDNODE/v4 id=72a321369de5e8fb addr=183.82.123.102:30304 err=nil
TRACE[10-21|14:53:33.460] >> PING/v4 id=999d4a9364794c24 addr=172.93.189.174:30303 err=nil
TRACE[10-21|14:53:33.497] << NEIGHBORS/v4 id=72d48f27cd1ee7f6 addr=54.68.121.166:20203 err=nil
TRACE[10-21|14:53:33.504] << NEIGHBORS/v4 id=72d48f27cd1ee7f6 addr=54.68.121.166:20203 err=nil
TRACE[10-21|14:53:33.507] >> PING/v4 id=72fb90e2e8cc8697 addr=35.158.55.23:30303 err=nil
TRACE[10-21|14:53:33.531] << NEIGHBORS/v4 id=72a321369de5e8fb addr=183.82.123.102:30304 err=nil
TRACE[10-21|14:53:33.538] << NEIGHBORS/v4 id=72a321369de5e8fb addr=183.82.123.102:30304 err=nil
TRACE[10-21|14:53:33.540] >> PING/v4 id=72f8970d203a3324 addr=74.125.182.17:64744 err=nil
TRACE[10-21|14:53:33.552] << NEIGHBORS/v4 id=7ccb1a6b44086628 addr=68.183.181.212:30303 err=nil
TRACE[10-21|14:53:33.559] << NEIGHBORS/v4 id=7ccb1a6b44086628 addr=68.183.181.212:30303 err=nil
TRACE[10-21|14:53:33.561] << PONG/v4 id=72fb90e2e8cc8697 addr=35.158.55.23:30303 err=nil
TRACE[10-21|14:53:33.561] >> PING/v4 id=72fe469d4a04c525 addr=107.172.208.100:30303 err=nil
TRACE[10-21|14:53:33.576] Accepted connection addr=35.224.108.27:33680
TRACE[10-21|14:53:33.602] Accepted connection addr=147.135.210.87:37144
DEBUG[10-21|14:53:33.668] Adding p2p peer id=b5549983afa1810a conn=inbound addr=147.135.210.87:37144 peers=2 name=Geth/v1.8.13-unstabl...
TRACE[10-21|14:53:33.669] Connection set up id=b5549983afa1810a addr=147.135.210.87:37144 conn=inbound inbound=true
TRACE[10-21|14:53:33.669] Starting protocol eth/63 id=b5549983afa1810a conn=inbound
DEBUG[10-21|14:53:33.670] Ethereum peer connected id=b5549983afa1810a conn=inbound name=Geth/v1.8.13-unstable-3b074515/linux-amd64/go1.10.3
DEBUG[10-21|14:53:33.672] Ethereum handshake failed id=b5549983afa1810a conn=inbound err="Genesis block mismatch - 325d96e6a3950ee7 (!= d4e56740f876aef8)"
TRACE[10-21|14:53:33.672] Protocol eth/63 failed id=b5549983afa1810a conn=inbound err="Genesis block mismatch - 325d96e6a3950ee7 (!= d4e56740f876aef8)"
DEBUG[10-21|14:53:33.673] Removing p2p peer id=b5549983afa1810a conn=inbound addr=147.135.210.87:37144 peers=1 duration=4.425ms req=false err="Genesis block mismatch - 325d96e6a3950ee7 (!= d4e56740f876aef8)"
TRACE[10-21|14:53:33.673] >> FINDNODE/v4 id=d28918e9b337c0b0 addr=85.250.222.230:30303 err=nil
TRACE[10-21|14:53:33.687] << PONG/v4 id=999d4a9364794c24 addr=172.93.189.174:30303 err=nil
DEBUG[10-21|14:53:33.687] Revalidated node b=14 id=999d4a9364794c24 checks=1
DEBUG[10-21|14:53:33.707] Adding p2p peer id=918bd3c3bcc5586f conn=inbound addr=35.224.108.27:33680 peers=2 name=Parity-Ethereum/v2.5...
TRACE[10-21|14:53:33.707] Connection set up id=918bd3c3bcc5586f addr=35.224.108.27:33680 conn=inbound inbound=true
TRACE[10-21|14:53:33.707] Starting protocol eth/63 id=918bd3c3bcc5586f conn=inbound
DEBUG[10-21|14:53:33.707] Ethereum peer connected id=918bd3c3bcc5586f conn=inbound name=Parity-Ethereum/v2.5.8-stable-c52a6c8-20190916/x86_64-linux-gnu/rustc1.36.0
TRACE[10-21|14:53:33.753] Findnode failed id=d28d1c78c8340e36 failcount=14 err="RPC timeout"
TRACE[10-21|14:53:33.753] Too many findnode failures, dropping id=d28d1c78c8340e36 failcount=14
TRACE[10-21|14:53:33.824] Registering sync peer peer=918bd3c3bcc5586f
DEBUG[10-21|14:53:33.824] Relaxed downloader QoS values rtt=20s confidence=0.500 ttl=1m0s
DEBUG[10-21|14:53:33.824] Fetching batch of headers id=918bd3c3bcc5586f conn=inbound count=1 fromnum=8552447 skip=0 reverse=false
TRACE[10-21|14:53:33.875] >> FINDNODE/v4 id=d28677e1d08b5cbd addr=47.56.185.214:30303 err=nil
TRACE[10-21|14:53:33.895] Accepted connection addr=18.197.97.197:56940
DEBUG[10-21|14:53:33.938] Adding p2p peer id=d501e73fc2b6ba8e conn=inbound addr=18.197.97.197:56940 peers=3 name=Geth/v1.8.23-stable-...
TRACE[10-21|14:53:33.939] Connection set up id=d501e73fc2b6ba8e addr=18.197.97.197:56940 conn=inbound inbound=true
TRACE[10-21|14:53:33.939] Starting protocol eth/63 id=d501e73fc2b6ba8e conn=inbound
DEBUG[10-21|14:53:33.939] Ethereum peer connected id=d501e73fc2b6ba8e conn=inbound name=Geth/v1.8.23-stable-c9427004/linux-amd64/go1.10.4
TRACE[10-21|14:53:33.944] Registering sync peer peer=d501e73fc2b6ba8e
DEBUG[10-21|14:53:33.944] Relaxed downloader QoS values rtt=20s confidence=0.333 ttl=1m0s
DEBUG[10-21|14:53:33.945] Fetching batch of headers id=d501e73fc2b6ba8e conn=inbound count=1 fromnum=8552447 skip=0 reverse=false
DEBUG[10-21|14:53:33.980] Failed to deliver headers err="no sync active"
TRACE[10-21|14:53:34.063] >> FINDNODE/v4 id=72fb90e2e8cc8697 addr=35.158.55.23:30303 err=nil
TRACE[10-21|14:53:34.160] << PING/v4 id=72fb90e2e8cc8697 addr=35.158.55.23:30303 err=nil
TRACE[10-21|14:53:34.161] >> PONG/v4 id=72fb90e2e8cc8697 addr=35.158.55.23:30303 err=nil
TRACE[10-21|14:53:34.172] Findnode failed id=d28918e9b337c0b0 failcount=3 err="RPC timeout"
TRACE[10-21|14:53:34.341] << NEIGHBORS/v4 id=72fb90e2e8cc8697 addr=35.158.55.23:30303 err=nil
TRACE[10-21|14:53:34.347] << NEIGHBORS/v4 id=72fb90e2e8cc8697 addr=35.158.55.23:30303 err=nil
TRACE[10-21|14:53:34.350] >> PING/v4 id=72f8b65a77a01c7f addr=18.130.136.41:30303 err=nil
TRACE[10-21|14:53:34.374] Findnode failed id=d28677e1d08b5cbd failcount=3 err="RPC timeout"
TRACE[10-21|14:53:34.374] Dial task done task="discovery lookup (16 results)"
TRACE[10-21|14:53:34.375] New dial task task="dyndial d29f82f0669d35b3 164.15.10.134:1450"
TRACE[10-21|14:53:34.375] New dial task task="dyndial d29dc715c5f43d88 67.205.142.63:50000"
TRACE[10-21|14:53:34.375] New dial task task="dyndial d29bdc4db2e3dcb3 182.61.108.243:21212"
TRACE[10-21|14:53:34.375] New dial task task="dyndial d29772b9b5f29c15 128.199.41.168:50000"
TRACE[10-21|14:53:34.375] New dial task task="dyndial d296e4811d30d123 164.15.10.134:1052"
TRACE[10-21|14:53:34.375] New dial task task="dyndial d29587632448cb21 167.99.82.0:50000"
TRACE[10-21|14:53:34.376] New dial task task="dyndial d294cb9bcfb89fce 168.61.161.73:30303"
TRACE[10-21|14:53:34.376] New dial task task="dyndial d293e23a0074d53f 159.89.20.82:30303"
TRACE[10-21|14:53:34.376] New dial task task="dyndial d29377beb3841c83 167.99.12.22:50000"
TRACE[10-21|14:53:34.376] New dial task task="dyndial d292d0bd601d724b 128.199.35.72:50000"
TRACE[10-21|14:53:34.376] New dial task task="dyndial d290414037655377 157.245.206.68:50000"
TRACE[10-21|14:53:34.376] New dial task task="dyndial d28f4d17a1daf965 164.15.10.134:1370"
TRACE[10-21|14:53:34.376] New dial task task="dyndial d28d9aa3da17090e 78.41.194.15:43026"
TRACE[10-21|14:53:34.376] New dial task task="dyndial d28d1c78c8340e36 39.108.164.135:30303"
TRACE[10-21|14:53:34.377] New dial task task="dyndial d28918e9b337c0b0 85.250.222.230:30303"
TRACE[10-21|14:53:34.377] New dial task task="dyndial d28677e1d08b5cbd 47.56.185.214:30303"
TRACE[10-21|14:53:34.542] >> FINDNODE/v4 id=72f8970d203a3324 addr=74.125.182.17:64744 err=nil
TRACE[10-21|14:53:34.552] Dial error task="dyndial d29772b9b5f29c15 128.199.41.168:50000" err="dial tcp 128.199.41.168:50000: connect: connection refused"
TRACE[10-21|14:53:34.552] Dial task done task="dyndial d29772b9b5f29c15 128.199.41.168:50000"
TRACE[10-21|14:53:34.553] New dial task task="discovery lookup"
TRACE[10-21|14:53:34.555] >> FINDNODE/v4 id=6ffd7817bb47942b addr=80.158.38.240:30303 err=nil
TRACE[10-21|14:53:34.557] >> FINDNODE/v4 id=62c1f10a22bc411b addr=95.217.47.59:21236 err=nil
TRACE[10-21|14:53:34.558] Dial error task="dyndial d292d0bd601d724b 128.199.35.72:50000" err="dial tcp 128.199.35.72:50000: connect: connection refused"
TRACE[10-21|14:53:34.558] Dial task done task="dyndial d292d0bd601d724b 128.199.35.72:50000"
TRACE[10-21|14:53:34.558] >> FINDNODE/v4 id=7ccb1a6b44086628 addr=68.183.181.212:30303 err=nil
TRACE[10-21|14:53:34.563] >> FINDNODE/v4 id=72fe469d4a04c525 addr=107.172.208.100:30303 err=nil
TRACE[10-21|14:53:34.565] Dial error task="dyndial d29587632448cb21 167.99.82.0:50000" err="dial tcp 167.99.82.0:50000: connect: connection refused"
TRACE[10-21|14:53:34.565] Dial task done task="dyndial d29587632448cb21 167.99.82.0:50000"
TRACE[10-21|14:53:34.642] Dial error task="dyndial d29377beb3841c83 167.99.12.22:50000" err="dial tcp 167.99.12.22:50000: connect: connection refused"
TRACE[10-21|14:53:34.642] Dial task done task="dyndial d29377beb3841c83 167.99.12.22:50000"
TRACE[10-21|14:53:34.644] Dial error task="dyndial d29dc715c5f43d88 67.205.142.63:50000" err="dial tcp 67.205.142.63:50000: connect: connection refused"
TRACE[10-21|14:53:34.644] Dial task done task="dyndial d29dc715c5f43d88 67.205.142.63:50000"
TRACE[10-21|14:53:34.645] Dial error task="dyndial d28918e9b337c0b0 85.250.222.230:30303" err="dial tcp 85.250.222.230:30303: connect: connection refused"
TRACE[10-21|14:53:34.645] Dial task done task="dyndial d28918e9b337c0b0 85.250.222.230:30303"
TRACE[10-21|14:53:34.655] Accepted connection addr=86.107.243.88:57700
TRACE[10-21|14:53:34.662] Dial error task="dyndial d290414037655377 157.245.206.68:50000" err="dial tcp 157.245.206.68:50000: connect: connection refused"
TRACE[10-21|14:53:34.663] Dial task done task="dyndial d290414037655377 157.245.206.68:50000"
TRACE[10-21|14:53:34.666] << NEIGHBORS/v4 id=62c1f10a22bc411b addr=95.217.47.59:21236 err=nil
TRACE[10-21|14:53:34.673] << NEIGHBORS/v4 id=62c1f10a22bc411b addr=95.217.47.59:21236 err=nil
DEBUG[10-21|14:53:34.675] Adding p2p peer id=d293e23a0074d53f conn=dyndial addr=159.89.20.82:30303 peers=4 name=Parity-Ethereum/v2.3...
TRACE[10-21|14:53:34.676] Connection set up id=d293e23a0074d53f addr=159.89.20.82:30303 conn=dyndial inbound=false
TRACE[10-21|14:53:34.676] Starting protocol eth/63 id=d293e23a0074d53f conn=dyndial
DEBUG[10-21|14:53:34.676] Ethereum peer connected id=d293e23a0074d53f conn=dyndial name=Parity-Ethereum/v2.3.4-beta-0e95db1-20190220/x86_64-linux-gnu/rustc1.32.0
TRACE[10-21|14:53:34.677] >> FINDNODE/v4 id=6dbec632440b9806 addr=164.68.121.126:30303 err=nil
TRACE[10-21|14:53:34.676] Dial task done task="dyndial d293e23a0074d53f 159.89.20.82:30303"
DEBUG[10-21|14:53:34.682] Ethereum handshake failed id=d293e23a0074d53f conn=dyndial err=EOF
DEBUG[10-21|14:53:34.683] Removing p2p peer id=d293e23a0074d53f conn=dyndial addr=159.89.20.82:30303 peers=3 duration=7.619ms req=true err="too many peers"
TRACE[10-21|14:53:34.691] Invalid neighbor node received ip=39.100.242.121 addr=80.158.38.240:30303 err="low port"
TRACE[10-21|14:53:34.692] << NEIGHBORS/v4 id=6ffd7817bb47942b addr=80.158.38.240:30303 err=nil
TRACE[10-21|14:53:34.698] << NEIGHBORS/v4 id=6ffd7817bb47942b addr=80.158.38.240:30303 err=nil
DEBUG[10-21|14:53:34.698] IP exceeds bucket limit ip=128.199.36.156
TRACE[10-21|14:53:34.701] >> PING/v4 id=6de702fafe659188 addr=178.130.44.248:30303 err=nil
TRACE[10-21|14:53:34.788] Invalid neighbor node received ip=178.130.44.248 addr=164.68.121.126:30303 err="low port"
TRACE[10-21|14:53:34.793] Invalid neighbor node received ip=39.100.242.121 addr=164.68.121.126:30303 err="low port"
TRACE[10-21|14:53:34.799] << NEIGHBORS/v4 id=6dbec632440b9806 addr=164.68.121.126:30303 err=nil
TRACE[10-21|14:53:34.804] << NEIGHBORS/v4 id=6dbec632440b9806 addr=164.68.121.126:30303 err=nil
TRACE[10-21|14:53:34.807] >> PING/v4 id=6de7d6b8d8cf03dc addr=141.20.21.166:34677 err=nil
TRACE[10-21|14:53:34.868] << NEIGHBORS/v4 id=7ccb1a6b44086628 addr=68.183.181.212:30303 err=nil
TRACE[10-21|14:53:34.874] << NEIGHBORS/v4 id=7ccb1a6b44086628 addr=68.183.181.212:30303 err=nil
TRACE[10-21|14:53:34.877] >> PING/v4 id=6de26d5a9c625384 addr=128.199.36.156:50000 err=nil
TRACE[10-21|14:53:34.887] Rejected peer id=d29bdc4db2e3dcb3 addr=182.61.108.243:21212 conn=dyndial err="useless peer"
TRACE[10-21|14:53:34.888] Setting up connection failed addr=182.61.108.243:21212 err="useless peer"
TRACE[10-21|14:53:34.888] Dial error task="dyndial d29bdc4db2e3dcb3 182.61.108.243:21212" err="useless peer"
TRACE[10-21|14:53:34.888] Dial task done task="dyndial d29bdc4db2e3dcb3 182.61.108.243:21212"
TRACE[10-21|14:53:35.042] Findnode failed id=72f8970d203a3324 failcount=1 err="RPC timeout"
TRACE[10-21|14:53:35.044] >> PING/v4 id=72ff347f38bb4e88 addr=54.236.62.205:30303 err=nil
TRACE[10-21|14:53:35.062] Findnode failed id=72fe469d4a04c525 failcount=9 err="RPC timeout"
TRACE[10-21|14:53:35.063] Too many findnode failures, dropping id=72fe469d4a04c525 failcount=9
TRACE[10-21|14:53:35.066] >> PING/v4 id=72fd17f82748ab4b addr=138.68.250.147:50000 err=nil
TRACE[10-21|14:53:35.237] << PONG/v4 id=6cd2a37407a80222 addr=138.68.250.147:50000 err="unsolicited reply"
TRACE[10-21|14:53:35.352] >> FINDNODE/v4 id=72f8b65a77a01c7f addr=18.130.136.41:30303 err=nil
TRACE[10-21|14:53:35.703] >> FINDNODE/v4 id=6de702fafe659188 addr=178.130.44.248:30303 err=nil
TRACE[10-21|14:53:35.808] >> FINDNODE/v4 id=6de7d6b8d8cf03dc addr=141.20.21.166:34677 err=nil
TRACE[10-21|14:53:35.851] Findnode failed id=72f8b65a77a01c7f failcount=2 err="RPC timeout"
TRACE[10-21|14:53:35.854] >> PING/v4 id=72f2c59ed464e9f6 addr=167.99.12.22:50000 err=nil
TRACE[10-21|14:53:35.880] >> FINDNODE/v4 id=6de26d5a9c625384 addr=128.199.36.156:50000 err=nil
TRACE[10-21|14:53:35.959] << PONG/v4 id=eb8686c8abc21417 addr=167.99.12.22:50000 err="unsolicited reply"
TRACE[10-21|14:53:36.046] >> FINDNODE/v4 id=72ff347f38bb4e88 addr=54.236.62.205:30303 err=nil
TRACE[10-21|14:53:36.068] >> FINDNODE/v4 id=72fd17f82748ab4b addr=138.68.250.147:50000 err=nil
TRACE[10-21|14:53:36.202] Findnode failed id=6de702fafe659188 failcount=4 err="RPC timeout"
TRACE[10-21|14:53:36.204] >> PING/v4 id=6de32e35712c8605 addr=128.199.36.156:50000 err=nil
TRACE[10-21|14:53:36.308] Findnode failed id=6de7d6b8d8cf03dc failcount=1 err="RPC timeout"
TRACE[10-21|14:53:36.310] >> PING/v4 id=6de35c272c6b3bd4 addr=167.71.61.184:30303 err=nil
TRACE[10-21|14:53:36.378] Findnode failed id=6de26d5a9c625384 failcount=1 err="RPC timeout"
TRACE[10-21|14:53:36.380] >> PING/v4 id=6de04dbe582d08c9 addr=164.15.10.134:1467 err=nil
TRACE[10-21|14:53:36.545] Findnode failed id=72ff347f38bb4e88 failcount=3 err="RPC timeout"
TRACE[10-21|14:53:36.548] >> FINDNODE/v4 id=72f03b9acc589a72 addr=178.238.225.47:30303 err=nil
TRACE[10-21|14:53:36.567] Findnode failed id=72fd17f82748ab4b failcount=3 err="RPC timeout"
TRACE[10-21|14:53:36.569] >> PING/v4 id=72f7a4f06b1c4b6a addr=34.250.50.222:30303 err=nil
INFO [10-21|14:53:36.582] Block synchronisation started
DEBUG[10-21|14:53:36.582] Synchronising with the network peer=8f112bd35cc4e73c eth=63 head=a9aa9f…401aba td=12483987806983869307610 mode=full
DEBUG[10-21|14:53:36.582] Retrieving remote chain height peer=8f112bd35cc4e73c
DEBUG[10-21|14:53:36.582] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromhash=a9aa9f…401aba skip=0 reverse=false
TRACE[10-21|14:53:36.591] << NEIGHBORS/v4 id=72f03b9acc589a72 addr=178.238.225.47:30303 err=nil
TRACE[10-21|14:53:36.597] << NEIGHBORS/v4 id=72f03b9acc589a72 addr=178.238.225.47:30303 err=nil
TRACE[10-21|14:53:36.600] >> PING/v4 id=72f2b457c05aad17 addr=128.199.35.72:50000 err=nil
TRACE[10-21|14:53:36.636] >> PING/v4 id=b7cfb51b7122692d addr=144.76.162.156:30303 err=nil
TRACE[10-21|14:53:36.651] << PONG/v4 id=d83b157765b00b81 addr=128.199.35.72:50000 err="unsolicited reply"
TRACE[10-21|14:53:36.683] << PONG/v4 id=b7cfb51b7122692d addr=144.76.162.156:30303 err=nil
DEBUG[10-21|14:53:36.683] Revalidated node b=11 id=b7cfb51b7122692d checks=1
TRACE[10-21|14:53:36.855] >> FINDNODE/v4 id=72f2c59ed464e9f6 addr=167.99.12.22:50000 err=nil
TRACE[10-21|14:53:36.881] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:36.881] Remote head header identified peer=8f112bd35cc4e73c number=8784300 hash=a9aa9f…401aba
DEBUG[10-21|14:53:36.881] Looking for common ancestor peer=8f112bd35cc4e73c local=8714666 remote=8784300
TRACE[10-21|14:53:36.882] Span searching for common ancestor peer=8f112bd35cc4e73c count=12 from=8784123 skip=15
DEBUG[10-21|14:53:36.882] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=12 fromnum=8784123 skip=15 reverse=false
TRACE[10-21|14:53:37.200] Binary searching for common ancestor peer=8f112bd35cc4e73c start=8624666 end=8784300
DEBUG[10-21|14:53:37.201] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8704483 skip=0 reverse=false
TRACE[10-21|14:53:37.206] >> FINDNODE/v4 id=6de32e35712c8605 addr=128.199.36.156:50000 err=nil
TRACE[10-21|14:53:37.312] >> FINDNODE/v4 id=6de35c272c6b3bd4 addr=167.71.61.184:30303 err=nil
TRACE[10-21|14:53:37.355] Findnode failed id=72f2c59ed464e9f6 failcount=1 err="RPC timeout"
TRACE[10-21|14:53:37.357] >> PING/v4 id=72f76c90ea44991f addr=94.79.55.28:30000 err=nil
TRACE[10-21|14:53:37.382] >> FINDNODE/v4 id=6de04dbe582d08c9 addr=164.15.10.134:1467 err=nil
TRACE[10-21|14:53:37.500] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:37.507] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8744391 skip=0 reverse=false
TRACE[10-21|14:53:37.572] >> FINDNODE/v4 id=72f7a4f06b1c4b6a addr=34.250.50.222:30303 err=nil
TRACE[10-21|14:53:37.601] >> FINDNODE/v4 id=72f2b457c05aad17 addr=128.199.35.72:50000 err=nil
TRACE[10-21|14:53:37.708] Findnode failed id=6de32e35712c8605 failcount=2 err="RPC timeout"
TRACE[10-21|14:53:37.711] >> PING/v4 id=6dee1df50008fa13 addr=128.199.36.156:50000 err=nil
TRACE[10-21|14:53:37.806] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:37.807] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8724437 skip=0 reverse=false
TRACE[10-21|14:53:37.810] Findnode failed id=6de35c272c6b3bd4 failcount=12 err="RPC timeout"
TRACE[10-21|14:53:37.811] Too many findnode failures, dropping id=6de35c272c6b3bd4 failcount=12
TRACE[10-21|14:53:37.813] >> PING/v4 id=6defe48b406759cf addr=157.245.206.138:50000 err=nil
TRACE[10-21|14:53:37.845] Accepted connection addr=40.68.81.47:34440
TRACE[10-21|14:53:37.881] Findnode failed id=6de04dbe582d08c9 failcount=3 err="RPC timeout"
DEBUG[10-21|14:53:37.882] Adding p2p peer id=d69a9fec3befba00 conn=inbound addr=40.68.81.47:34440 peers=4 name=Geth/v1.8.27-stable-...
TRACE[10-21|14:53:37.882] Connection set up id=d69a9fec3befba00 addr=40.68.81.47:34440 conn=inbound inbound=true
TRACE[10-21|14:53:37.882] Starting protocol eth/63 id=d69a9fec3befba00 conn=inbound
DEBUG[10-21|14:53:37.883] Ethereum peer connected id=d69a9fec3befba00 conn=inbound name=Geth/v1.8.27-stable-4bcc0a37/linux-amd64/go1.11.9
TRACE[10-21|14:53:37.884] >> FINDNODE/v4 id=6deb29c84e2b3758 addr=188.166.90.213:30303 err=nil
DEBUG[10-21|14:53:37.885] Ethereum handshake failed id=d69a9fec3befba00 conn=inbound err="Genesis block mismatch - 33477ffd19d181cb (!= d4e56740f876aef8)"
TRACE[10-21|14:53:37.885] Protocol eth/63 failed id=d69a9fec3befba00 conn=inbound err="Genesis block mismatch - 33477ffd19d181cb (!= d4e56740f876aef8)"
DEBUG[10-21|14:53:37.886] Removing p2p peer id=d69a9fec3befba00 conn=inbound addr=40.68.81.47:34440 peers=3 duration=4.344ms req=false err="Genesis block mismatch - 33477ffd19d181cb (!= d4e56740f876aef8)"
TRACE[10-21|14:53:37.919] << NEIGHBORS/v4 id=6deb29c84e2b3758 addr=188.166.90.213:30303 err=nil
TRACE[10-21|14:53:37.925] << NEIGHBORS/v4 id=6deb29c84e2b3758 addr=188.166.90.213:30303 err=nil
TRACE[10-21|14:53:37.928] >> PING/v4 id=6deb2fa8db66d98c addr=176.9.155.139:30303 err=nil
TRACE[10-21|14:53:38.070] Findnode failed id=72f7a4f06b1c4b6a failcount=8 err="RPC timeout"
TRACE[10-21|14:53:38.070] Too many findnode failures, dropping id=72f7a4f06b1c4b6a failcount=8
TRACE[10-21|14:53:38.072] >> PING/v4 id=72f5a6e13d21d938 addr=93.57.86.57:37111 err=nil
TRACE[10-21|14:53:38.101] Findnode failed id=72f2b457c05aad17 failcount=1 err="RPC timeout"
TRACE[10-21|14:53:38.102] >> PING/v4 id=72f52953ec145c5d addr=178.62.235.175:50000 err=nil
TRACE[10-21|14:53:38.109] << PONG/v4 id=29e22c15c08a6782 addr=157.245.206.138:50000 err="unsolicited reply"
TRACE[10-21|14:53:38.216] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:38.217] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8714460 skip=0 reverse=false
TRACE[10-21|14:53:38.358] >> FINDNODE/v4 id=72f76c90ea44991f addr=94.79.55.28:30000 err=nil
TRACE[10-21|14:53:38.429] Failed RLPx handshake addr=86.107.243.88:57700 conn=inbound err=EOF
TRACE[10-21|14:53:38.429] Setting up connection failed addr=86.107.243.88:57700 err=EOF
TRACE[10-21|14:53:38.534] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:38.542] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8719448 skip=0 reverse=false
TRACE[10-21|14:53:38.712] >> FINDNODE/v4 id=6dee1df50008fa13 addr=128.199.36.156:50000 err=nil
TRACE[10-21|14:53:38.815] >> FINDNODE/v4 id=6defe48b406759cf addr=157.245.206.138:50000 err=nil
TRACE[10-21|14:53:38.841] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:38.842] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8716954 skip=0 reverse=false
TRACE[10-21|14:53:38.858] Findnode failed id=72f76c90ea44991f failcount=1 err="RPC timeout"
TRACE[10-21|14:53:38.860] >> PING/v4 id=72eb9a8da7036fb8 addr=24.125.193.190:30303 err=nil
TRACE[10-21|14:53:38.930] >> FINDNODE/v4 id=6deb2fa8db66d98c addr=176.9.155.139:30303 err=nil
TRACE[10-21|14:53:39.074] >> FINDNODE/v4 id=72f5a6e13d21d938 addr=93.57.86.57:37111 err=nil
TRACE[10-21|14:53:39.104] >> FINDNODE/v4 id=72f52953ec145c5d addr=178.62.235.175:50000 err=nil
TRACE[10-21|14:53:39.143] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:39.144] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8715707 skip=0 reverse=false
TRACE[10-21|14:53:39.212] Findnode failed id=6dee1df50008fa13 failcount=2 err="RPC timeout"
TRACE[10-21|14:53:39.214] >> PING/v4 id=6df725320987e763 addr=178.62.40.90:50000 err=nil
TRACE[10-21|14:53:39.314] Findnode failed id=6defe48b406759cf failcount=2 err="RPC timeout"
TRACE[10-21|14:53:39.316] >> PING/v4 id=6df50697c60bc573 addr=178.62.27.60:50000 err=nil
TRACE[10-21|14:53:39.352] << PONG/v4 id=c0fffc49cb1a4b89 addr=178.62.27.60:50000 err="unsolicited reply"
TRACE[10-21|14:53:39.429] Findnode failed id=6deb2fa8db66d98c failcount=12 err="RPC timeout"
TRACE[10-21|14:53:39.429] Too many findnode failures, dropping id=6deb2fa8db66d98c failcount=12
TRACE[10-21|14:53:39.431] >> PING/v4 id=6df2383c819b9098 addr=128.199.55.156:50000 err=nil
TRACE[10-21|14:53:39.454] << PONG/v4 id=bc9e903f92ae4958 addr=128.199.55.156:50000 err="unsolicited reply"
TRACE[10-21|14:53:39.481] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:39.488] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8716330 skip=0 reverse=false
TRACE[10-21|14:53:39.573] Findnode failed id=72f5a6e13d21d938 failcount=6 err="RPC timeout"
TRACE[10-21|14:53:39.573] Too many findnode failures, dropping id=72f5a6e13d21d938 failcount=6
TRACE[10-21|14:53:39.576] >> PING/v4 id=72ef0f14b4b58e13 addr=35.195.106.62:20758 err=nil
TRACE[10-21|14:53:39.603] Findnode failed id=72f52953ec145c5d failcount=2 err="RPC timeout"
TRACE[10-21|14:53:39.605] >> PING/v4 id=72ec039292142ee2 addr=206.189.157.63:30303 err=nil
TRACE[10-21|14:53:39.795] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:39.796] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8716018 skip=0 reverse=false
TRACE[10-21|14:53:39.862] >> FINDNODE/v4 id=72eb9a8da7036fb8 addr=24.125.193.190:30303 err=nil
TRACE[10-21|14:53:40.094] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:40.100] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8716174 skip=0 reverse=false
TRACE[10-21|14:53:40.216] >> FINDNODE/v4 id=6df725320987e763 addr=178.62.40.90:50000 err=nil
TRACE[10-21|14:53:40.318] >> FINDNODE/v4 id=6df50697c60bc573 addr=178.62.27.60:50000 err=nil
TRACE[10-21|14:53:40.361] Findnode failed id=72eb9a8da7036fb8 failcount=3 err="RPC timeout"
TRACE[10-21|14:53:40.399] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:40.400] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8716096 skip=0 reverse=false
TRACE[10-21|14:53:40.433] >> FINDNODE/v4 id=6df2383c819b9098 addr=128.199.55.156:50000 err=nil
TRACE[10-21|14:53:40.578] >> FINDNODE/v4 id=72ef0f14b4b58e13 addr=35.195.106.62:20758 err=nil
TRACE[10-21|14:53:40.607] >> FINDNODE/v4 id=72ec039292142ee2 addr=206.189.157.63:30303 err=nil
TRACE[10-21|14:53:40.715] Findnode failed id=6df725320987e763 failcount=5 err="RPC timeout"
TRACE[10-21|14:53:40.715] Too many findnode failures, dropping id=6df725320987e763 failcount=5
TRACE[10-21|14:53:40.717] >> PING/v4 id=6dfad945754fe00b addr=13.233.201.54:23000 err=nil
TRACE[10-21|14:53:40.731] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:40.732] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8716057 skip=0 reverse=false
TRACE[10-21|14:53:40.770] << PING/v4 id=b0a6ffc3ac4688f3 addr=206.189.224.11:30305 err=nil
TRACE[10-21|14:53:40.772] >> PONG/v4 id=b0a6ffc3ac4688f3 addr=206.189.224.11:30305 err=nil
TRACE[10-21|14:53:40.817] Findnode failed id=6df50697c60bc573 failcount=5 err="RPC timeout"
TRACE[10-21|14:53:40.817] Too many findnode failures, dropping id=6df50697c60bc573 failcount=5
TRACE[10-21|14:53:40.828] >> PING/v4 id=6dc60a052cc54396 addr=35.226.157.25:30303 err=nil
TRACE[10-21|14:53:40.830] << PING/v4 id=b0b35544b703ebff addr=95.168.186.250:30303 err=nil
TRACE[10-21|14:53:40.832] >> PONG/v4 id=b0b35544b703ebff addr=95.168.186.250:30303 err=nil
TRACE[10-21|14:53:40.858] << ENRREQUEST/v4 id=b0b35544b703ebff addr=95.168.186.250:30303 err=nil
TRACE[10-21|14:53:40.860] >> ENRRESPONSE/v4 id=b0b35544b703ebff addr=95.168.186.250:30303 err=nil
TRACE[10-21|14:53:40.932] Findnode failed id=6df2383c819b9098 failcount=5 err="RPC timeout"
TRACE[10-21|14:53:40.932] Too many findnode failures, dropping id=6df2383c819b9098 failcount=5
TRACE[10-21|14:53:40.934] >> FINDNODE/v4 id=6dc6cf8757bad9c8 addr=54.162.153.64:30303 err=nil
TRACE[10-21|14:53:41.031] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:41.032] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8716037 skip=0 reverse=false
TRACE[10-21|14:53:41.042] << NEIGHBORS/v4 id=252cecb6dfba92ae addr=54.162.153.64:30303 err="unsolicited reply"
TRACE[10-21|14:53:41.045] << NEIGHBORS/v4 id=252cecb6dfba92ae addr=54.162.153.64:30303 err="unsolicited reply"
TRACE[10-21|14:53:41.077] Findnode failed id=72ef0f14b4b58e13 failcount=9 err="RPC timeout"
TRACE[10-21|14:53:41.077] Too many findnode failures, dropping id=72ef0f14b4b58e13 failcount=9
TRACE[10-21|14:53:41.106] Findnode failed id=72ec039292142ee2 failcount=1 err="RPC timeout"
TRACE[10-21|14:53:41.109] >> PING/v4 id=860f00e69799a87f addr=42.232.29.50:20182 err=nil
TRACE[10-21|14:53:41.109] >> FINDNODE/v4 id=887e8f67b92e0550 addr=51.68.136.169:30303 err=nil
TRACE[10-21|14:53:41.110] >> FINDNODE/v4 id=83141c6224866e6a addr=93.189.47.189:30303 err=nil
TRACE[10-21|14:53:41.178] << NEIGHBORS/v4 id=887e8f67b92e0550 addr=51.68.136.169:30303 err=nil
TRACE[10-21|14:53:41.181] << NEIGHBORS/v4 id=887e8f67b92e0550 addr=51.68.136.169:30303 err=nil
TRACE[10-21|14:53:41.182] >> FINDNODE/v4 id=80807c1c7b15253e addr=45.76.62.76:30305 err=nil
TRACE[10-21|14:53:41.195] Accepted connection addr=206.189.224.11:49234
TRACE[10-21|14:53:41.247] << NEIGHBORS/v4 id=83141c6224866e6a addr=93.189.47.189:30303 err=nil
TRACE[10-21|14:53:41.250] << NEIGHBORS/v4 id=83141c6224866e6a addr=93.189.47.189:30303 err=nil
TRACE[10-21|14:53:41.251] >> FINDNODE/v4 id=80ebec7d03036f90 addr=138.201.248.31:30303 err=nil
TRACE[10-21|14:53:41.300] << NEIGHBORS/v4 id=80ebec7d03036f90 addr=138.201.248.31:30303 err=nil
TRACE[10-21|14:53:41.303] << NEIGHBORS/v4 id=80ebec7d03036f90 addr=138.201.248.31:30303 err=nil
TRACE[10-21|14:53:41.304] >> PING/v4 id=80fe7bee1d8dc690 addr=145.239.133.36:30303 err=nil
DEBUG[10-21|14:53:41.305] Adding p2p peer id=b0a6ffc3ac4688f3 conn=inbound addr=206.189.224.11:49234 peers=4 name=Geth/v0.0.9.1-Ether1...
TRACE[10-21|14:53:41.305] Connection set up id=b0a6ffc3ac4688f3 addr=206.189.224.11:49234 conn=inbound inbound=true
TRACE[10-21|14:53:41.305] Starting protocol eth/63 id=b0a6ffc3ac4688f3 conn=inbound
DEBUG[10-21|14:53:41.306] Ethereum peer connected id=b0a6ffc3ac4688f3 conn=inbound name=Geth/v0.0.9.1-Ether1_MN_SN-Security-Update/linux-amd64/go1.11.5
DEBUG[10-21|14:53:41.307] Ethereum handshake failed id=b0a6ffc3ac4688f3 conn=inbound err="Genesis block mismatch - 70cefc67ff52eb3e (!= d4e56740f876aef8)"
TRACE[10-21|14:53:41.307] Protocol eth/63 failed id=b0a6ffc3ac4688f3 conn=inbound err="Genesis block mismatch - 70cefc67ff52eb3e (!= d4e56740f876aef8)"
DEBUG[10-21|14:53:41.308] Removing p2p peer id=b0a6ffc3ac4688f3 conn=inbound addr=206.189.224.11:49234 peers=3 duration=2.347ms req=false err="Genesis block mismatch - 70cefc67ff52eb3e (!= d4e56740f876aef8)"
TRACE[10-21|14:53:41.309] << NEIGHBORS/v4 id=80807c1c7b15253e addr=45.76.62.76:30305 err=nil
TRACE[10-21|14:53:41.312] << NEIGHBORS/v4 id=80807c1c7b15253e addr=45.76.62.76:30305 err=nil
TRACE[10-21|14:53:41.313] >> PING/v4 id=80ff25e95839d737 addr=123.118.62.70:20182 err=nil
TRACE[10-21|14:53:41.331] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:41.332] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8716047 skip=0 reverse=false
TRACE[10-21|14:53:41.433] Findnode failed id=6dc6cf8757bad9c8 failcount=3 err="RPC timeout"
TRACE[10-21|14:53:41.631] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:41.631] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8716042 skip=0 reverse=false
TRACE[10-21|14:53:41.718] >> FINDNODE/v4 id=6dfad945754fe00b addr=13.233.201.54:23000 err=nil
TRACE[10-21|14:53:41.829] >> FINDNODE/v4 id=6dc60a052cc54396 addr=35.226.157.25:30303 err=nil
TRACE[10-21|14:53:41.977] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:41.978] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8716039 skip=0 reverse=false
TRACE[10-21|14:53:42.111] >> FINDNODE/v4 id=860f00e69799a87f addr=42.232.29.50:20182 err=nil
TRACE[10-21|14:53:42.218] Findnode failed id=6dfad945754fe00b failcount=9 err="RPC timeout"
TRACE[10-21|14:53:42.218] Too many findnode failures, dropping id=6dfad945754fe00b failcount=9
TRACE[10-21|14:53:42.276] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:42.279] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8716040 skip=0 reverse=false
TRACE[10-21|14:53:42.306] >> FINDNODE/v4 id=80fe7bee1d8dc690 addr=145.239.133.36:30303 err=nil
TRACE[10-21|14:53:42.315] >> FINDNODE/v4 id=80ff25e95839d737 addr=123.118.62.70:20182 err=nil
TRACE[10-21|14:53:42.329] Findnode failed id=6dc60a052cc54396 failcount=7 err="RPC timeout"
TRACE[10-21|14:53:42.329] Too many findnode failures, dropping id=6dc60a052cc54396 failcount=7
TRACE[10-21|14:53:42.330] Dial task done task="discovery lookup (16 results)"
TRACE[10-21|14:53:42.330] New dial task task="dyndial 6de702fafe659188 178.130.44.248:30303"
TRACE[10-21|14:53:42.331] New dial task task="dyndial 6de7d6b8d8cf03dc 141.20.21.166:34677"
TRACE[10-21|14:53:42.331] New dial task task="dyndial 6de26d5a9c625384 128.199.36.156:50000"
TRACE[10-21|14:53:42.332] New dial task task="dyndial 6de32e35712c8605 128.199.36.156:50000"
TRACE[10-21|14:53:42.332] New dial task task="dyndial 6de35c272c6b3bd4 167.71.61.184:30303"
TRACE[10-21|14:53:42.333] New dial task task="dyndial 6de04dbe582d08c9 164.15.10.134:1467"
TRACE[10-21|14:53:42.333] New dial task task="dyndial 6dee1df50008fa13 128.199.36.156:50000"
TRACE[10-21|14:53:42.334] New dial task task="dyndial 6defe48b406759cf 157.245.206.138:50000"
TRACE[10-21|14:53:42.334] New dial task task="dyndial 6deb29c84e2b3758 188.166.90.213:30303"
TRACE[10-21|14:53:42.348] Dial error task="dyndial 6de26d5a9c625384 128.199.36.156:50000" err="dial tcp 128.199.36.156:50000: connect: connection refused"
TRACE[10-21|14:53:42.349] Dial task done task="dyndial 6de26d5a9c625384 128.199.36.156:50000"
TRACE[10-21|14:53:42.349] New dial task task="dyndial 6deb2fa8db66d98c 176.9.155.139:30303"
TRACE[10-21|14:53:42.349] Dial error task="dyndial 6de32e35712c8605 128.199.36.156:50000" err="dial tcp 128.199.36.156:50000: connect: connection refused"
TRACE[10-21|14:53:42.349] Dial task done task="dyndial 6de32e35712c8605 128.199.36.156:50000"
TRACE[10-21|14:53:42.349] New dial task task="dyndial 6df725320987e763 178.62.40.90:50000"
TRACE[10-21|14:53:42.351] Dial error task="dyndial 6dee1df50008fa13 128.199.36.156:50000" err="dial tcp 128.199.36.156:50000: connect: connection refused"
TRACE[10-21|14:53:42.351] Dial task done task="dyndial 6dee1df50008fa13 128.199.36.156:50000"
TRACE[10-21|14:53:42.351] New dial task task="dyndial 6df50697c60bc573 178.62.27.60:50000"
TRACE[10-21|14:53:42.376] Dial error task="dyndial 6df725320987e763 178.62.40.90:50000" err="dial tcp 178.62.40.90:50000: connect: connection refused"
TRACE[10-21|14:53:42.377] Dial task done task="dyndial 6df725320987e763 178.62.40.90:50000"
TRACE[10-21|14:53:42.377] New dial task task="dyndial 6df2383c819b9098 128.199.55.156:50000"
TRACE[10-21|14:53:42.377] Dial error task="dyndial 6df50697c60bc573 178.62.27.60:50000" err="dial tcp 178.62.27.60:50000: connect: connection refused"
TRACE[10-21|14:53:42.378] Dial task done task="dyndial 6df50697c60bc573 178.62.27.60:50000"
TRACE[10-21|14:53:42.378] New dial task task="dyndial 6dfad945754fe00b 13.233.201.54:23000"
TRACE[10-21|14:53:42.387] Failed proto handshake id=6deb29c84e2b3758 addr=188.166.90.213:30303 conn=dyndial err="too many peers"
TRACE[10-21|14:53:42.388] Setting up connection failed addr=188.166.90.213:30303 err="too many peers"
TRACE[10-21|14:53:42.388] Dial error task="dyndial 6deb29c84e2b3758 188.166.90.213:30303" err="too many peers"
TRACE[10-21|14:53:42.388] Dial task done task="dyndial 6deb29c84e2b3758 188.166.90.213:30303"
TRACE[10-21|14:53:42.389] New dial task task="dyndial 6dc60a052cc54396 35.226.157.25:30303"
TRACE[10-21|14:53:42.395] Dial error task="dyndial 6df2383c819b9098 128.199.55.156:50000" err="dial tcp 128.199.55.156:50000: connect: connection refused"
TRACE[10-21|14:53:42.396] Dial task done task="dyndial 6df2383c819b9098 128.199.55.156:50000"
TRACE[10-21|14:53:42.396] New dial task task="dyndial 6dc6cf8757bad9c8 54.162.153.64:1028"
TRACE[10-21|14:53:42.511] Dial error task="dyndial 6dfad945754fe00b 13.233.201.54:23000" err="dial tcp 13.233.201.54:23000: connect: connection refused"
TRACE[10-21|14:53:42.512] Dial task done task="dyndial 6dfad945754fe00b 13.233.201.54:23000"
TRACE[10-21|14:53:42.512] New dial task task="discovery lookup"
TRACE[10-21|14:53:42.514] >> FINDNODE/v4 id=e131359c042cb4c4 addr=94.176.236.103:30303 err=nil
TRACE[10-21|14:53:42.514] >> FINDNODE/v4 id=e793c676acafd0a4 addr=207.246.124.59:30303 err=nil
TRACE[10-21|14:53:42.515] >> FINDNODE/v4 id=e272f31a9ab9425a addr=157.245.94.240:30300 err=nil
TRACE[10-21|14:53:42.577] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:42.579] Fetching batch of headers id=8f112bd35cc4e73c conn=inbound count=1 fromnum=8716041 skip=0 reverse=false
TRACE[10-21|14:53:42.610] Findnode failed id=860f00e69799a87f failcount=7 err="RPC timeout"
TRACE[10-21|14:53:42.610] Too many findnode failures, dropping id=860f00e69799a87f failcount=7
TRACE[10-21|14:53:42.613] >> PING/v4 id=80fd7359b4999319 addr=61.147.122.42:61913 err=nil
TRACE[10-21|14:53:42.628] Dial error task="dyndial 6defe48b406759cf 157.245.206.138:50000" err="dial tcp 157.245.206.138:50000: connect: connection refused"
TRACE[10-21|14:53:42.629] Dial task done task="dyndial 6defe48b406759cf 157.245.206.138:50000"
TRACE[10-21|14:53:42.633] << NEIGHBORS/v4 id=e272f31a9ab9425a addr=157.245.94.240:30300 err=nil
TRACE[10-21|14:53:42.639] << NEIGHBORS/v4 id=e272f31a9ab9425a addr=157.245.94.240:30300 err=nil
TRACE[10-21|14:53:42.653] << NEIGHBORS/v4 id=e793c676acafd0a4 addr=207.246.124.59:30303 err=nil
TRACE[10-21|14:53:42.653] >> PING/v4 id=e08d7bbb9abf4c3b addr=171.217.52.134:30303 err=nil
TRACE[10-21|14:53:42.660] << NEIGHBORS/v4 id=e793c676acafd0a4 addr=207.246.124.59:30303 err=nil
TRACE[10-21|14:53:42.662] >> FINDNODE/v4 id=e08a00c9ae922ce9 addr=51.79.27.131:30303 err=nil
TRACE[10-21|14:53:42.674] << NEIGHBORS/v4 id=e131359c042cb4c4 addr=94.176.236.103:30303 err=nil
TRACE[10-21|14:53:42.681] << NEIGHBORS/v4 id=e131359c042cb4c4 addr=94.176.236.103:30303 err=nil
TRACE[10-21|14:53:42.684] >> FINDNODE/v4 id=e08a031a7bebb56b addr=3.208.69.150:30303 err=nil
TRACE[10-21|14:53:42.801] << NEIGHBORS/v4 id=e08a00c9ae922ce9 addr=51.79.27.131:30303 err=nil
TRACE[10-21|14:53:42.808] << NEIGHBORS/v4 id=e08a00c9ae922ce9 addr=51.79.27.131:30303 err=nil
TRACE[10-21|14:53:42.811] Findnode failed id=80fe7bee1d8dc690 failcount=11 err="RPC timeout"
TRACE[10-21|14:53:42.812] Too many findnode failures, dropping id=80fe7bee1d8dc690 failcount=11
TRACE[10-21|14:53:42.823] << NEIGHBORS/v4 id=e08a031a7bebb56b addr=3.208.69.150:30303 err=nil
TRACE[10-21|14:53:42.823] Findnode failed id=80ff25e95839d737 failcount=2 err="RPC timeout"
TRACE[10-21|14:53:42.823] >> PING/v4 id=e08174b91ac5a9e7 addr=157.245.206.68:50000 err=nil
TRACE[10-21|14:53:42.824] >> PING/v4 id=80fd30273665f2fa addr=138.68.246.170:50000 err=nil
TRACE[10-21|14:53:42.825] >> PING/v4 id=80fa8a11d65b83ad addr=52.194.227.28:30303 err=nil
TRACE[10-21|14:53:42.832] << NEIGHBORS/v4 id=e08a031a7bebb56b addr=3.208.69.150:30303 err=nil
TRACE[10-21|14:53:42.834] >> PING/v4 id=e0815e37ee2e7f9b addr=91.157.100.100:30303 err=nil
TRACE[10-21|14:53:42.878] Filtering headers peer=8f112bd35cc4e73c headers=1
DEBUG[10-21|14:53:42.881] Synchronisation terminated elapsed=6.298s
DEBUG[10-21|14:53:42.881] Reset ancient limit to zero
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1b8 pc=0x63d828]

goroutine 266 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(Downloader).findAncestor(0x20dab40, 0x21fc300, 0x8684240, 0x0, 0x0, 0xa, 0xa)
/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:877 +0x1500
github.com/ethereum/go-ethereum/eth/downloader.(Downloader).syncWithPeer(0x20dab40, 0x21fc300, 0xc69faaa9, 0x4ef38840, 0xb522428c, 0x16d006cd, 0x26f28c, 0x1620fb5d, 0x5991498f, 0xba1a4001, ...)
/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:443 +0x2a0
github.com/ethereum/go-ethereum/eth/downloader.(Downloader).synchronise(0x20dab40, 0x800d6b0, 0x10, 0xc69faaa9, 0x4ef38840, 0xb522428c, 0x16d006cd, 0x26f28c, 0x1620fb5d, 0x5991498f, ...)
/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:411 +0x338
github.com/ethereum/go-ethereum/eth/downloader.(Downloader).Synchronise(0x20dab40, 0x800d6b0, 0x10, 0xc69faaa9, 0x4ef38840, 0xb522428c, 0x16d006cd, 0x26f28c, 0x1620fb5d, 0x5991498f, ...)
/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:323 +0x5c
github.com/ethereum/go-ethereum/eth.(ProtocolManager).synchronise(0x21e63f0, 0x8124480)
/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:190 +0x1b0
created by github.com/ethereum/go-ethereum/eth.(ProtocolManager).syncer
/home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:155 +0x17c
```

m0ar picture m0ar on 21 Oct 2019

Interesting. It looks like the block body for block 8716041 exists, but the header seems to not be present.

holiman picture holiman on 22 Oct 2019

@holiman LMK if I can provide any other data to help find the culprit here.

Is there any way I can get through this impasse?

m0ar picture m0ar on 22 Oct 2019

Is there any way I can get through this impasse?

One thing that might work is to use debug.setHead at 8714666.

holiman picture holiman on 23 Oct 2019

@m0ar the first crash you had -- do you know if that was the exact _same_ crash as this one? I'm thinking that maybe the first crash left the db a bit corrupted, and now you're hitting this because some data is missing in the db.

Also, what hardware are you running on ? What's the ram size?

holiman picture holiman on 23 Oct 2019

@holiman How can I set this before the client crashes? It fails within seconds.

m0ar picture m0ar on 23 Oct 2019

Got it done. It definitely ran for longer, but crashed again:

[...]
TRACE[10-23|09:43:56.593] >> PING/v4                               id=87f1a269da09b133 addr=164.15.10.134:1439    err=nil
TRACE[10-23|09:43:56.602] >> FINDNODE/v4                           id=87f2609389c3c75f addr=74.71.193.225:30303   err=nil
TRACE[10-23|09:43:56.683] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:56.684] Remote head header identified            peer=fbecffbe2c173d8a number=8795682 hash=8952d0…f3c6e0
DEBUG[10-23|09:43:56.684] Looking for common ancestor              peer=fbecffbe2c173d8a local=8714666 remote=8795682
TRACE[10-23|09:43:56.684] Span searching for common ancestor       peer=fbecffbe2c173d8a count=12 from=8795505 skip=15
DEBUG[10-23|09:43:56.684] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=12 fromnum=8795505 skip=15 reverse=false
DEBUG[10-23|09:43:56.693] Peer discarded announcement              peer=fbecffbe2c173d8a number=8795684 hash=710e5d…04f5f2 distance=81018
TRACE[10-23|09:43:56.748] >> FINDNODE/v4                           id=dc1e03b53752c6d7 addr=113.52.133.48:30303   err=nil
TRACE[10-23|09:43:56.771] Binary searching for common ancestor     peer=fbecffbe2c173d8a start=8624666 end=8795682
DEBUG[10-23|09:43:56.772] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8710174 skip=0  reverse=false
TRACE[10-23|09:43:56.936] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:56.944] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8752928 skip=0  reverse=false
TRACE[10-23|09:43:56.990] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:56.991] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8731551 skip=0  reverse=false
TRACE[10-23|09:43:57.083] >> FINDNODE/v4                           id=dc1d32b41bfa9f2c addr=165.22.28.47:50000    err=nil
TRACE[10-23|09:43:57.083] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.084] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8720862 skip=0  reverse=false
TRACE[10-23|09:43:57.101] Findnode failed                          id=87f2609389c3c75f failcount=1   err="RPC timeout"
TRACE[10-23|09:43:57.103] >> PING/v4                               id=87f4880a9638d756 addr=94.79.55.28:30000     err=nil
TRACE[10-23|09:43:57.154] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.155] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8715518 skip=0  reverse=false
TRACE[10-23|09:43:57.234] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.240] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8718190 skip=0  reverse=false
TRACE[10-23|09:43:57.247] Findnode failed                          id=dc1e03b53752c6d7 failcount=3   err="RPC timeout"
TRACE[10-23|09:43:57.249] >> PING/v4                               id=dc1d079a808feccf addr=91.55.92.55:30303     err=nil
TRACE[10-23|09:43:57.349] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.350] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716854 skip=0  reverse=false
TRACE[10-23|09:43:57.397] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.398] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716186 skip=0  reverse=false
TRACE[10-23|09:43:57.444] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.445] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8715852 skip=0  reverse=false
DEBUG[10-23|09:43:57.466] Peer discarded announcement              peer=fbecffbe2c173d8a number=8795684 hash=d9d7fb…bded03 distance=81018
TRACE[10-23|09:43:57.475] >> FINDNODE/v4                           id=87f1d99d692cc00e addr=116.253.203.71:5818   err=nil
TRACE[10-23|09:43:57.481] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.488] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716019 skip=0  reverse=false
TRACE[10-23|09:43:57.514] >> FINDNODE/v4                           id=dc1d3faf109a9f5b addr=157.245.93.35:50000   err=nil
TRACE[10-23|09:43:57.541] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.543] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716102 skip=0  reverse=false
TRACE[10-23|09:43:57.573] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.575] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716060 skip=0  reverse=false
TRACE[10-23|09:43:57.582] Findnode failed                          id=dc1d32b41bfa9f2c failcount=2   err="RPC timeout"
TRACE[10-23|09:43:57.584] >> PING/v4                               id=dc1cc69c6a05eb8f addr=13.230.252.33:30303   err=nil
TRACE[10-23|09:43:57.595] >> FINDNODE/v4                           id=87f1a269da09b133 addr=164.15.10.134:1439    err=nil
TRACE[10-23|09:43:57.661] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.661] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716039 skip=0  reverse=false
TRACE[10-23|09:43:57.697] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.700] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716049 skip=0  reverse=false
TRACE[10-23|09:43:57.753] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.754] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716044 skip=0  reverse=false
TRACE[10-23|09:43:57.787] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.787] Fetching batch of headers                id=fbecffbe2c173d8a conn=dyndial         count=1  fromnum=8716041 skip=0  reverse=false
TRACE[10-23|09:43:57.847] Filtering headers                        peer=fbecffbe2c173d8a headers=1
DEBUG[10-23|09:43:57.851] Synchronisation terminated               elapsed=11.386s
DEBUG[10-23|09:43:57.851] Reset ancient limit to zero 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1b8 pc=0x63d828]

goroutine 376 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestor(0x24bac80, 0x266c6c0, 0x398b440, 0x0, 0x0, 0xa, 0xa)
    /home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:877 +0x1500
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0x24bac80, 0x266c6c0, 0xc3d05289, 0x15726f7b, 0x93e43866, 0xc042d0bd, 0x75832ac, 0xc6690d42, 0xd013c950, 0xe0c6f3ee, ...)
    /home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:443 +0x2a0
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0x24bac80, 0x89d1ca0, 0x10, 0xc3d05289, 0x15726f7b, 0x93e43866, 0xc042d0bd, 0x75832ac, 0xc6690d42, 0xd013c950, ...)
    /home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:411 +0x338
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0x24bac80, 0x89d1ca0, 0x10, 0xc3d05289, 0x15726f7b, 0x93e43866, 0xc042d0bd, 0x75832ac, 0xc6690d42, 0xd013c950, ...)
    /home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:323 +0x5c
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).synchronise(0x24921b0, 0x8a24e80)
    /home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:190 +0x1b0
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer
    /home/pi/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:155 +0x17c
m0ar picture m0ar on 23 Oct 2019

@m0ar the first crash you had -- do you know if that was the exact _same_ crash as this one? I'm thinking that maybe the first crash left the db a bit corrupted, and now you're hitting this because some data is missing in the db.

Also, what hardware are you running on ? What's the ram size?

I'm not sure, could be. Would a resync be the only solution then?

It's this setup: https://kauri.io/article/9695fcca217f46feb355245275835fc0
(RaspberryPi 4B /w 4GB and a 500GB samsung SSD)

m0ar picture m0ar on 23 Oct 2019

Hi.

I'm currently doing a fast sync on the Pi 4 and ran into this error (full log attached):

Oct 21 22:47:58 ethnode-9efbf7ff geth[526]: runtime: pointer 0x8f8fe000 to unallocated span span.base()=0x8c1be000 span.limit=0x2761dfe0 span.state=3
Oct 21 22:47:58 ethnode-9efbf7ff geth[526]: runtime: found in object at *(0x3834c00+0x320)
Oct 21 22:47:58 ethnode-9efbf7ff geth[526]: object=0x3834c00 s.base()=0x3830000 s.limit=0x3835f00 s.spanclass=88 s.elemsize=4864 s.state=mSpanInUse

It happened just once and the sync process seems normal (seems a pointer bug but I'm not sure if it is related to this issue). Just in case it helps. I will report back when the fast sync completes.

Setup:

  • Raspberry Pi 4 4GB RAM
  • --cache 512
  • Swap: 4GB
  • Disk: SSD Kingston 480GB with SATA-USB case (ASMedia ASM1153E chipset)

Geth: Official armv7 binary 1.9.6-stable

@m0ar, if you do a resync, would you mind to try this image?

https://ethraspbian.com/downloads/image_2019-10-20-EthRaspbian2.0-lite.zip

Just flash it and the fast sync process will start in 2-3 minutes (keep in mind that it will wipe out your entire USB disk and reinstall everything). Login ethereum:ethereum. You will be prompted to change the password on first login.

geth-bad-pointer-error-rp4.txt

diglos picture diglos on 24 Oct 2019

Hi.

Fast sync process was completed in 4 days. But, I got lots of errors like this one:

Oct 26 09:19:28 ethnode-9efbf7ff geth[16298]: runtime: out of memory: cannot allocate 279183360-byte block (2246934528 in use)
Oct 26 09:19:28 ethnode-9efbf7ff geth[16298]: fatal error: out of memory
Oct 26 09:19:28 ethnode-9efbf7ff geth[16298]: runtime stack:
Oct 26 09:19:28 ethnode-9efbf7ff geth[16298]: runtime.throw(0xb64f9a, 0xd)

It turns out that setting the cache to 256 MB (it was 512MB) solved this and Geth runs fine. But, somehow this is weird as with other 2 ARM boards (ARM64) I was using the default cache value (a little more that 1GB) and never ran into this (and both devices have 4GB RAM as well). Is it possible that this errors are caused by some problem with Go's GC?

diglos picture diglos on 26 Oct 2019
👍2

Hi, got another error:

Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: INFO [10-29|21:40:06.773] Imported new block headers               count=1    elapsed=20.904ms     number=8836401 hash=9993f4…8f53dc age=1m28s
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: panic: runtime error: slice bounds out of range [:37:34]
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: goroutine 1320563 [running]:
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy.load32(...)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy/encode_other.go:10
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy.encodeBlock(0x50148141, 0x46, 0x46, 0x50aee200, 0x22, 0x200, 0x12a684c8)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy/encode_other.go:176 +0x900
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy.Encode(0x50148140, 0x47, 0x47, 0x0, 0x0, 0x0, 0x200, 0x0, 0x0)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy/encode.go:37 +0x260
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/p2p.(*rlpxFrameRW).WriteMsg(0x22038b80, 0x1f, 0x0, 0x22, 0xdf1f98, 0x506e4440, 0x0, 0x0, 0x0, 0x0, ...)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:601 +0xc8
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/p2p.(*rlpx).WriteMsg(0x20b7f280, 0x1f, 0x0, 0x22, 0xdf1f98, 0x506e4440, 0x0, 0x0, 0x0, 0x0, ...)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:102 +0xe8
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/p2p.(*protoRW).WriteMsg(0x21d8c1e0, 0x1f, 0x0, 0x22, 0xdf1f98, 0x506e4440, 0x0, 0x0, 0x0, 0x0, ...)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/p2p/peer.go:412 +0x15c
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/p2p.Send(0xa5820030, 0x21d8c1e0, 0xf, 0x0, 0xa09728, 0x5954c5a0, 0x0, 0x0)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/p2p/message.go:101 +0xa0
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: github.com/ethereum/go-ethereum/eth.(*peer).RequestReceipts(0x20cffd00, 0x1c8063a0, 0x1, 0x1, 0xa19b711e, 0xdfeb7436)
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/peer.go:351 +0x128
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: created by github.com/ethereum/go-ethereum/eth/downloader.(*peerConnection).FetchReceipts
Oct 29 21:40:06 ethnode-9efbf7ff geth[7985]: #011/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/peer.go:203 +0x20c
Oct 29 21:40:07 ethnode-9efbf7ff systemd[1]: geth.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

I don't know if they are related. Let me know if you want me to open a separate issue.

diglos picture diglos on 30 Oct 2019

Hi,

I did another Fast Sync with Geth 1.9.7. This time I only ran into an unallocated span issue (see attached). It seems that the out of memory issues were a Go problem, I will keep monitoring the process in the coming days though.

geth-pointer-unallocated-span.txt

diglos picture diglos on 15 Nov 2019

@holiman

Did a re-sync, this is the initial crash (that once again occured, now on v1.9.8):
https://gist.github.com/m0ar/417aa0a4af2ba0f7e16760d29468e880

Run with geth --syncmode fast --cache 256 --datadir /mnt/ssd/ethereum

m0ar picture m0ar on 28 Nov 2019

Worth noting is also that I can get past these crashes when I restart the service, until it inevitably happens again. The amount of free memory is ~2GB at the time of crash.

@diglos Did you get anywhere investigating GC issues?

m0ar picture m0ar on 28 Nov 2019

Hi
I keep getting lots of out of memory errors once the sync is completed. It keeps happening until the Pi is unusable. So:

  • During sync, as reported above, throws a pointer 0x82d61000 to unallocated span error
  • Once synced, lots of fatal error: out of memory crashes

I'm currently doing another fast sync with 1.9.8 and already got a unallocated span error. Let's see how it ends but I think I will get the same errors.

Yesterday I got a report from a Eth on ARM Raspi 4 user (same problem, I think). Please attached find his log.

This doesn't happen at all on the NanoPC-T4 (4 GB RAM as well but ARM64)

Regards!

geth-out-of-memory.txt

diglos picture diglos on 28 Nov 2019
👍2

@diglos Yeah my Pi gets borked after a night of OOM crashes too, until it stops responding to SSH and needs a hard reboot

m0ar picture m0ar on 29 Nov 2019
👍4

Hi all.

It seems that this is a 32bit kernel thing. There are lots of issues about out of memory and oom crashes on the Raspberry Pi forums [1].

Raspbian enabled recently a 64bit kernel for the Raspberry Pi 4 which seems to work great handling 4GB of RAM (this is just the kernel, the userspace is still on 32bit). Once enabled, I completed a Fast Sync and it’s been running for 1 day with no issues whatsoever (with 256MB cache).

I’m releasing early next week a new image of Ethereum on ARM for the Raspberry Pi 4 with this kernel enabled by default.

@moar, you can enable the 64bit kernel just by adding the proper option to config.txt (first, make sure you have the latest firmware):

sudo rpi-eeprom-update
sudo echo arm_64bit=1 >> /boot/config.txt

And reboot.

This should fix the memory issues.

[1] https://www.raspberrypi.org/forums/viewtopic.php?t=255433
https://www.raspberrypi.org/forums/viewtopic.php?t=249924
https://www.raspberrypi.org/forums/viewtopic.php?t=244367
https://www.raspberrypi.org/forums/viewtopic.php?t=244849
https://www.raspberrypi.org/forums/viewtopic.php?t=244531

diglos picture diglos on 20 Dec 2019
❤2

Hi all.

It seems that this is a 32bit kernel thing. There are lots of issues about out of memory and oom crashes on the Raspberry Pi forums [1].

Raspbian enabled recently a 64bit kernel for the Raspberry Pi 4 which seems to work great handling 4GB of RAM (this is just the kernel, the userspace is still on 32bit). Once enabled, I completed a Fast Sync and it’s been running for 1 day with no issues whatsoever (with 256MB cache).

I’m releasing early next week a new image of Ethereum on ARM for the Raspberry Pi 4 with this kernel enabled by default.

@moar, you can enable the 64bit kernel just by adding the proper option to config.txt (first, make sure you have the latest firmware):

sudo rpi-eeprom-update
sudo echo arm_64bit=1 >> /boot/config.txt

And reboot.

This should fix the memory issues.

[1] https://www.raspberrypi.org/forums/viewtopic.php?t=255433
https://www.raspberrypi.org/forums/viewtopic.php?t=249924
https://www.raspberrypi.org/forums/viewtopic.php?t=244367
https://www.raspberrypi.org/forums/viewtopic.php?t=244849
https://www.raspberrypi.org/forums/viewtopic.php?t=244531

still getting the error even with those fixes

uname -a
Linux raspberrypi 4.19.93-v8+ #1286 SMP PREEMPT Mon Jan 6 13:29:56 GMT 2020 aarch64 GNU/Linux

sirnicolas21 picture sirnicolas21 on 10 Jan 2020
👍1

Hi :-)

It's been a while... We successfully synced a Raspberry Pi 4 running an Ubuntu 20.04 64 bit and we didn't get any crash or memory allocation problem. As we expected this seems to be a 32bit OS problem.

@m0ar please, take a look at this:

https://www.reddit.com/r/ethereum/comments/gf3nhg/ethereum_on_arm_raspberry_pi_4_images_release/

Regards!

diglos picture diglos on 19 May 2020
👍1

Thank you @diglos

adamschmideg picture adamschmideg on 28 May 2020

Hi @adamschmideg, just noting that the users moving to ARM64, but ARMv7 is still broken. This issue still persists and shouldn't be closed.

willianpaixao picture willianpaixao on 17 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

RoodyChan picture RoodyChan  Â·  3Comments
aakilfernandes picture aakilfernandes  Â·  3Comments
phpsamsb picture phpsamsb  Â·  3Comments
freshonline picture freshonline  Â·  3Comments
tymat picture tymat  Â·  3Comments