Go-ethereum: "Database compacting" for 10+ hours

[brandoncurtis]

My Geth archive node has been syncing fine for the last week on geth 1.8.9.
Yesterday I upgraded to geth 1.8.11 and restarted the node.
This morning, the log shows Database compacting, degraded performance for the past ten hours. No completion metrics are given, so I have no idea how long to expect this to take.

System information

Geth
Version: 1.8.11-stable
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.9.3
Operating System: linux
Ubuntu 16.04 LTS

[brandoncurtis]

Database compacting continued to display for ~24 hours until I killed Geth. This was on a high-performance SSD, so it surprised me. Upon restart the problem disappeared.

[rjl493456442]

@brandoncurtis Hey, have you noticed the eth.db.chaindata.compact the metrics? I want to make sure whether the 24 hours write delay is due many continuous compactions or a single very long compaction(I think it should be the former).

[brandoncurtis]

> debug.metrics(true).eth.db.chaindata.compact
{
  writedelay: {
    counter: {
      AvgRate01Min: 0,
      AvgRate05Min: 0,
      AvgRate15Min: 0,
      MeanRate: 0,
      Overall: 0
    },
    duration: {
      AvgRate01Min: 0,
      AvgRate05Min: 0,
      AvgRate15Min: 0,
      MeanRate: 0,
      Overall: 0
    }
  }
}

This may just be because I have not enabled metrics with --metrics.

Unfortunately this has happened again, this time on a different system syncing a full node, and I am unable to shut the node down with ctrl-c:

WARN [07-05|19:14:14] Database compacting, degraded performance database=/home/brandon/.ethereum/geth/chaindata
^CINFO [07-05|19:14:56] Got interrupt, shutting down... 
INFO [07-05|19:14:56] IPC endpoint closed                      endpoint=/home/brandon/.ethereum/geth.ipc
WARN [07-05|19:15:14] Database compacting, degraded performance database=/home/brandon/.ethereum/geth/chaindata
WARN [07-05|19:16:14] Database compacting, degraded performance database=/home/brandon/.ethereum/geth/chaindata
WARN [07-05|19:17:15] Database compacting, degraded performance database=/home/brandon/.ethereum/geth/chaindata
^CWARN [07-05|19:17:34] Already shutting down, interrupt more to panic. times=9
WARN [07-05|19:18:15] Database compacting, degraded performance database=/home/brandon/.ethereum/geth/chaindata

[rjl493456442]

@brandoncurtis Thank your for the feedback. First for the metrics, write delay metric data will be collected even without --metrics, the zero value is because these metrics are updated only after the write operation be resumed.

And the second, since write operation is paused due to the compaction, so blockchain.InsertChain operation is also sucked. Geth doesn't exit immediately before all inflight insert operations be done.

But for these problem, the main reason is leveldb write operation should be resumed when enough level0 files be compacted instead of waiting all compactions be finished.

I have a patch in my local to fix these problem, will open a PR for upstream repo and a PR for better UX for go-ethereum asap.

[brandoncurtis]

I had to force-kill it to get it to shut down. Here's what the panic printed, which includes a couple mentions of a leveldb.(*DB).compactionError:

INFO [07-05|14:58:01] Imported new chain segment               blocks=6    txs=939   mgas=37.486  elapsed=8.209s    mgasps=4.566   number=4356096 hash=5ebdb6…b1b836 cache=0.00B
INFO [07-05|14:58:09] Imported new chain segment               blocks=17   txs=1456  mgas=84.704  elapsed=8.337s    mgasps=10.159  number=4356113 hash=0a809a…8c215c cache=0.00B
WARN [07-05|14:58:11] Database compacting, degraded performance database=/home/brandon/.ethereum/geth/chaindata
WARN [07-05|14:59:11] Database compacting, degraded performance database=/home/brandon/.ethereum/geth/chaindata
WARN [07-05|15:00:12] Database compacting, degraded performance database=/home/brandon/.ethereum/geth/chaindata
...
^CWARN [07-05|19:22:32] Already shutting down, interrupt more to panic. times=1
^Cpanic: boom

goroutine 163 [running]:
github.com/ethereum/go-ethereum/internal/debug.LoudPanic(0xe106a0, 0x1159cd0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/internal/debug/loudpanic.go:26 +0x4e
github.com/ethereum/go-ethereum/cmd/utils.StartNode.func1(0xc420327180)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/utils/cmd.go:84 +0x26b
created by github.com/ethereum/go-ethereum/cmd/utils.StartNode
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/utils/cmd.go:70 +0xb7

goroutine 1 [chan receive, 4055 minutes]:
github.com/ethereum/go-ethereum/node.(*Node).Wait(0xc420327180)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/node/node.go:464 +0x7f
main.geth(0xc4204e2000, 0xc4204bfd48, 0xc4204e4000)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:238 +0x51
github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli%2ev1.HandleAction(0xe29980, 0x1030500, 0xc4204e2000, 0xc4204e2000, 0xc4204bff40)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli.v1/app.go:490 +0xd2
github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli%2ev1.(*App).Run(0xc42007b6c0, 0xc4200100b0, 0xb, 0xb, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli.v1/app.go:264 +0x635
main.main()
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:226 +0x55

goroutine 6 [chan receive, 4055 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*nonrecursiveTree).dispatch(0xc42005e1e0, 0xc42005e120)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:36 +0xc0
created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.newNonrecursiveTree
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:29 +0x149

goroutine 7 [chan receive, 4055 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*nonrecursiveTree).internal(0xc42005e1e0, 0xc42005e180)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:81 +0x88
created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.newNonrecursiveTree
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:30 +0x178
goroutine 8 [syscall]:                                                                                                                                                                                    [731/45080]
os/signal.signal_recv(0x187ab60)
        /usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 9 [chan receive, 264 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc420041560)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:55 +0xa4

goroutine 10 [chan receive, 264 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc420041560)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:55 +0xa4

goroutine 11 [chan receive, 264 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc420041560)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:55 +0xa4

goroutine 12 [chan receive, 264 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc420041560)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:55 +0xa4
goroutine 50 [syscall]:                                                                                                                                                                                   [699/45080]
syscall.Syscall6(0xe8, 0xc, 0xc42002976c, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0xffffffffffffffff, 0x0, 0x4)
        /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
github.com/ethereum/go-ethereum/vendor/golang.org/x/sys/unix.EpollWait(0xc, 0xc42002976c, 0x1, 0x1, 0xffffffffffffffff, 0xffffffffffffffff, 0x1876920, 0xc53e7f9400)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:1529 +0x77
github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).loop(0xc4200ba000, 0xc420486120)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:194 +0x8b
created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).lazyinit
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:134 +0x13a

goroutine 20 [select, locked to thread]:
runtime.gopark(0x1030e58, 0x0, 0xfa2be6, 0x6, 0x18, 0x1)
        /usr/local/go/src/runtime/proc.go:287 +0x12c
runtime.selectgo(0xc420029f50, 0xc420077bc0)
        /usr/local/go/src/runtime/select.go:395 +0x1149
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal_unix.go:511 +0x220
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 24 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/accounts/keystore.(*watcher).loop(0xc4201defc0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/watch.go:94 +0x511
created by github.com/ethereum/go-ethereum/accounts/keystore.(*watcher).start
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/watch.go:52 +0x5d

goroutine 25 [select]:
github.com/ethereum/go-ethereum/accounts/keystore.(*KeyStore).updater(0xc4200fc000)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/keystore.go:203 +0x104
created by github.com/ethereum/go-ethereum/accounts/keystore.(*KeyStore).Subscribe
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/keystore.go:190 +0x12c

goroutine 26 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/accounts.(*Manager).update(0xc4204d6b60)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/manager.go:95 +0x1d6
created by github.com/ethereum/go-ethereum/accounts.NewManager
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/manager.go:68 +0x5a7

goroutine 51 [chan receive, 4055 minutes]:                                                                                                                                                                [661/45080]
github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).send(0xc4200ba000, 0xc420486120)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:254 +0xc0
created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).lazyinit
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:137 +0x18d

goroutine 52 [chan receive, 4055 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).send(0xc4200ba000, 0xc420486120)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:254 +0xc0
created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).lazyinit
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:137 +0x18d

goroutine 27 [runnable]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc4202c6700)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x172
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x195

goroutine 28 [select]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc4204d0340)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x119
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:142 +0x4bd

goroutine 29 [select]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc4204d0340)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101 +0x13b
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:143 +0x4df

goroutine 30 [runnable]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*tFilesSortByKey).Less(0xc481888ca0, 0x3736, 0x0, 0xee01)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.go:278 +0x75
sort.doPivot(0x187e320, 0xc481888ca0, 0x0, 0x1dca5, 0xc43773c000, 0x7f41e0ec6458)
        /usr/local/go/src/sort/sort.go:119 +0xe8
sort.quickSort(0x187e320, 0xc481888ca0, 0x0, 0x1dca5, 0x22)
        /usr/local/go/src/sort/sort.go:192 +0x8a
sort.Sort(0x187e320, 0xc481888ca0)
        /usr/local/go/src/sort/sort.go:220 +0x79
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.tFiles.sortByKey(0xc5850ca000, 0x1dca5, 0x1dca5, 0xc4201e0e50)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.go:121 +0x8d
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*versionStaging).finish(0xc4443b74e0, 0xc482865860)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:493 +0x859
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*version).spawn(0xc519745ae0, 0xc482865860, 0xc519745ae0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:279 +0x7a
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).commit(0xc4200113f0, 0xc482865860, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session.go:195 +0x88
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionCommit.func1(0xc45168f428, 0xc46cfc4f01, 0xc45168f428)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:263 +0x37
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*compactionTransactFunc).run(0xc42d8265a0, 0xc45168f428, 0x1878860, 0xc42d8265a0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:241 +0x33
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransact(0xc4204d0340, 0xc4bf64aca0, 0x11, 0x1878860, 0xc42d8265a0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:185 +0x162
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransactFunc(0xc4204d0340, 0xc4bf64aca0, 0x11, 0xc481888c00, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:252 +0x99
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionCommit(0xc4204d0340, 0xfa75b0, 0xa, 0xc482865860)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:262 +0x120
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tableCompaction(0xc4204d0340, 0xc431f04b00, 0xc519745a00)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:551 +0xd26
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tableAutoCompaction(0xc4204d0340)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:633 +0x56
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc4204d0340)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:824 +0x3ba
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:149 +0x65e

goroutine 31 [select, 267 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc4204d0340)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x192
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:150 +0x680

goroutine 32 [select]:
github.com/ethereum/go-ethereum/ethdb.(*LDBDatabase).meter(0xc4200114a0, 0xb2d05e00)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/ethdb/database.go:361 +0x1b39
created by github.com/ethereum/go-ethereum/ethdb.(*LDBDatabase).Meter
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/ethdb/database.go:177 +0x1cc

goroutine 69 [select]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).qosTuner(0xc4204d0680)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1584 +0x352
created by github.com/ethereum/go-ethereum/eth/downloader.New
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:231 +0xa78

goroutine 70 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).stateFetcher(0xc4204d0680)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/statesync.go:76 +0x15a
created by github.com/ethereum/go-ethereum/eth/downloader.New
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:232 +0xa9d

goroutine 72 [chan receive, 4055 minutes]:
github.com/ethereum/go-ethereum/miner.(*worker).wait(0xc4200b7760)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:300 +0x345
created by github.com/ethereum/go-ethereum/miner.newWorker
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:160 +0x554

goroutine 74 [select, 5 minutes]:
github.com/ethereum/go-ethereum/p2p/nat.Map(0x18804a0, 0xc420248e80, 0xc420487980, 0xf9fda2, 0x3, 0x765f, 0x765f, 0xfae73f, 0x12)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:114 +0x4cb
created by github.com/ethereum/go-ethereum/p2p.(*Server).Start
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:435 +0x1007

goroutine 55 [select]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc4202c6540)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x172
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x195

goroutine 57 [select]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc42007ab60)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101 +0x13b
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:143 +0x4df

goroutine 56 [select, 5 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc42007ab60)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x119
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:142 +0x4bd

goroutine 58 [select, 5 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc42007ab60)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:804 +0x2c8
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:149 +0x65e

goroutine 59 [select, 33 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc42007ab60)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x192
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:150 +0x680

goroutine 60 [select, 35 minutes]:
github.com/ethereum/go-ethereum/p2p/discover.(*nodeDB).expirer(0xc42005f860)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/database.go:232 +0x128
created by github.com/ethereum/go-ethereum/p2p/discover.(*nodeDB).ensureExpirer.func1
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/database.go:223 +0x3e

goroutine 61 [select]:                                                                                                                                                                                    [523/45080]
github.com/ethereum/go-ethereum/p2p/discover.(*Table).loop(0xc4204c2140)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/table.go:381 +0x402
created by github.com/ethereum/go-ethereum/p2p/discover.newTable
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/table.go:151 +0x552

goroutine 62 [select]:
github.com/ethereum/go-ethereum/p2p/discover.(*udp).loop(0xc420098230)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/udp.go:385 +0x5e0
created by github.com/ethereum/go-ethereum/p2p/discover.newUDP
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/udp.go:262 +0x308

goroutine 63 [IO wait]:
internal/poll.runtime_pollWait(0x7f41e0e89f70, 0x72, 0x0)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4201e9018, 0x72, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4201e9018, 0xc4202ec500, 0x500, 0x500)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).ReadFrom(0xc4201e9000, 0xc4202ec500, 0x500, 0x500, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:170 +0x19a
net.(*netFD).readFrom(0xc4201e9000, 0xc4202ec500, 0x500, 0x500, 0xc7a5c8a22ebb77c7, 0xfab337b4e169f547, 0xbdf32e4b3b6d1220, 0x4e4bb71ccd52851f, 0x9e704fa889744aa8)
        /usr/local/go/src/net/fd_unix.go:208 +0x5e
net.(*UDPConn).readFrom(0xc4201ceb70, 0xc4202ec500, 0x500, 0x500, 0x0, 0xc42d245ed8, 0x81e351, 0xe5bf00)
        /usr/local/go/src/net/udpsock_posix.go:47 +0x6a
net.(*UDPConn).ReadFromUDP(0xc4201ceb70, 0xc4202ec500, 0x500, 0x500, 0x500, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/udpsock.go:106 +0x72
github.com/ethereum/go-ethereum/p2p/discover.(*udp).readLoop(0xc420098230, 0xc42005f020)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/udp.go:520 +0xec
created by github.com/ethereum/go-ethereum/p2p/discover.newUDP
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/udp.go:263 +0x33a

goroutine 64 [select]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc4202c68c0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x172
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x195

goroutine 65 [select, 4055 minutes]:                                                                                                                                                                      [485/45080]
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc42007b1e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x119
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:142 +0x4bd

goroutine 82 [select]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc42007b1e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101 +0x13b
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:143 +0x4df

goroutine 83 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc42007b1e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:804 +0x2c8
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:149 +0x65e

goroutine 84 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc42007b1e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x192
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:150 +0x680

goroutine 85 [select]:
github.com/ethereum/go-ethereum/p2p/discv5.(*Network).loop(0xc4203c4100)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discv5/net.go:415 +0x8e5
created by github.com/ethereum/go-ethereum/p2p/discv5.newNetwork
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discv5/net.go:168 +0x82b

goroutine 86 [chan receive]:
github.com/ethereum/go-ethereum/p2p.(*sharedUDPConn).ReadFromUDP(0xc4200408d0, 0xc42d17a000, 0x500, 0x500, 0x500, 0x0, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:365 +0x61
github.com/ethereum/go-ethereum/p2p/discv5.(*udp).readLoop(0xc42e34a4b0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discv5/udp.go:378 +0xdb
created by github.com/ethereum/go-ethereum/p2p/discv5.ListenUDP
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discv5/udp.go:244 +0x1dc

goroutine 443974 [select, 264 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compTriggerWait(0xc4204d0340, 0xc4200766c0, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:704 +0x353
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).flush.func1(0xbec7c2fc74f84800)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:94 +0x1e8
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).flush(0xc4204d0340, 0x1c0c3, 0xc4b54f85c0, 0x3a70, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:118 +0x171
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).writeLocked(0xc4204d0340, 0xc4297cca80, 0x0, 0x1, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:157 +0x6c
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).Write(0xc4204d0340, 0xc4297cca80, 0x0, 0x214, 0x214)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:317 +0x308
github.com/ethereum/go-ethereum/ethdb.(*ldbBatch).Write(0xc4aff61200, 0x19003, 0x20)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/ethdb/database.go:392 +0x40
github.com/ethereum/go-ethereum/trie.(*Database).commit(0xc4200fd680, 0xdee1b822e12a0167, 0xb86fe32b49dd21f1, 0x2ad6f5e19d304595, 0xff94bd0bf967e470, 0x18800a0, 0xc4aff61200, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/database.go:473 +0x2bd
github.com/ethereum/go-ethereum/trie.(*Database).commit(0xc4200fd680, 0xf6cc469813d54093, 0x3e65d00da92d563f, 0x80305a49e8b494b5, 0x63a29791b3feb549, 0x18800a0, 0xc4aff61200, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/database.go:464 +0x1b1
github.com/ethereum/go-ethereum/trie.(*Database).commit(0xc4200fd680, 0x7c5ee951472322e1, 0x47f47cbfd5d2e15b, 0x1963223be4975d92, 0x453246f3fe7dec96, 0x18800a0, 0xc4aff61200, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/database.go:464 +0x1b1
github.com/ethereum/go-ethereum/trie.(*Database).Commit(0xc4200fd680, 0x7c5ee951472322e1, 0x47f47cbfd5d2e15b, 0x1963223be4975d92, 0x453246f3fe7dec96, 0x453246f3fe7dec00, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/database.go:416 +0x579
github.com/ethereum/go-ethereum/core.(*BlockChain).WriteBlockWithState(0xc42034e000, 0xc50b7345a0, 0xc4a5633000, 0x94, 0x100, 0xc451ecd500, 0x100, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:910 +0x13b5
github.com/ethereum/go-ethereum/core.(*BlockChain).insertChain(0xc42034e000, 0xc43b3b0000, 0x800, 0x800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:1161 +0x19f5
github.com/ethereum/go-ethereum/core.(*BlockChain).InsertChain(0xc42034e000, 0xc43b3b0000, 0x800, 0x800, 0xc4a0580300, 0x1, 0x4)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:999 +0x67
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).importBlockResults(0xc4204d0680, 0xc422954000, 0x800, 0x800, 0x800, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1361 +0x462
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processFullSyncContent(0xc4204d0680, 0xc4202ea798, 0xfab8165d5d29e4fd)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1335 +0x79
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).(github.com/ethereum/go-ethereum/eth/downloader.processFullSyncContent)-fm(0x56efd0c900000008, 0x1030fa8)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:472 +0x2a
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1(0xc4204d0680, 0xc4e84012c0, 0xc424ac3770)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0x59
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0xb6

goroutine 79 [select, 3 minutes]:                                                                                                                                                                         [409/45080]
github.com/ethereum/go-ethereum/eth/fetcher.(*Fetcher).loop(0xc4204d7930)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/fetcher/fetcher.go:316 +0x7f5
created by github.com/ethereum/go-ethereum/eth/fetcher.(*Fetcher).Start
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/fetcher/fetcher.go:178 +0x3f

goroutine 80 [select, 35 minutes]:
github.com/ethereum/go-ethereum/p2p/discv5.(*nodeDB).expirer(0xc4200ff500)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discv5/database.go:245 +0x128
created by github.com/ethereum/go-ethereum/p2p/discv5.(*nodeDB).ensureExpirer.func1
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discv5/database.go:236 +0x3e

goroutine 88 [IO wait, 65 minutes]:
internal/poll.runtime_pollWait(0x7f41e0e89df0, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4203c1e98, 0x72, 0xc42049bd00, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4203c1e98, 0xffffffffffffff00, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc4203c1e80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e2
net.(*netFD).accept(0xc4203c1e80, 0x1492180, 0xc4952519e0, 0x1bd1a20)
        /usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc4201e2978, 0xc42049bf10, 0x419d5b, 0xc42d208900)
        /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4201e2978, 0x0, 0xc420327680, 0x1884460, 0xc4cb314d48)
        /usr/local/go/src/net/tcpsock.go:247 +0x49
github.com/ethereum/go-ethereum/p2p.(*Server).listenLoop(0xc420327680)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:762 +0x24a
created by github.com/ethereum/go-ethereum/p2p.(*Server).startListening
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:519 +0x159

goroutine 89 [select, 5 minutes]:
github.com/ethereum/go-ethereum/p2p/nat.Map(0x18804a0, 0xc420248e80, 0xc420487980, 0xf9fd96, 0x3, 0x765f, 0x765f, 0xfa8ea5, 0xc)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:114 +0x4cb
github.com/ethereum/go-ethereum/p2p.(*Server).startListening.func1(0xc420327680, 0xc42d174990)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:524 +0x8b
created by github.com/ethereum/go-ethereum/p2p.(*Server).startListening
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:523 +0x1fb

goroutine 90 [select, 172 minutes]:                                                                                                                                                                       [369/45080]
github.com/ethereum/go-ethereum/p2p.(*Server).run(0xc420327680, 0x18803e0, 0xc42ccf0e60)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:589 +0x61b
created by github.com/ethereum/go-ethereum/p2p.(*Server).Start
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:504 +0x7f6

goroutine 91 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 92 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 93 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 94 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 95 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 96 [select, 4055 minutes]:                                                                                                                                                                      [333/45080]
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 97 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 130 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 131 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 132 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 133 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 134 [select, 4055 minutes]:                                                                                                                                                                     [297/45080]
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 135 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 136 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 137 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 138 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc42e8620e0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x3c2
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 139 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).txBroadcastLoop(0xc42e86c300)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:757 +0x148
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).Start
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:209 +0xcb

goroutine 140 [chan receive, 4055 minutes]:                                                                                                                                                               [261/45080]
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).minedBroadcastLoop(0xc42e86c300)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:746 +0x59
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).Start
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:213 +0x15f

goroutine 141 [select]:
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer(0xc42e86c300)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:145 +0x1e8
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).Start
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:216 +0x181

goroutine 142 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).txsyncLoop(0xc42e86c300)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:109 +0x2bc
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).Start
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:217 +0x1a3

goroutine 143 [select, 430 minutes]:
github.com/ethereum/go-ethereum/eth/downloader.(*PublicDownloaderAPI).eventLoop(0xc42cd18ea0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/api.go:63 +0x336
created by github.com/ethereum/go-ethereum/eth/downloader.NewPublicDownloaderAPI
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/api.go:49 +0xe7

goroutine 145 [chan receive]:
github.com/ethereum/go-ethereum/eth/filters.(*PublicFilterAPI).timeoutLoop(0xc42e34a5f0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/filters/api.go:83 +0x124
created by github.com/ethereum/go-ethereum/eth/filters.NewPublicFilterAPI
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/filters/api.go:73 +0x1a8

goroutine 164 [chan receive, 4055 minutes]:
main.startNode.func1(0xc420327180, 0xc42d2e4360)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:280 +0x32d
created by main.startNode
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:265 +0x2c9

goroutine 165 [semacquire, 4055 minutes]:                                                                                                                                                                 [225/45080]
sync.runtime_notifyListWait(0xc420094340, 0xc400000000)
        /usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420094330)
        /usr/local/go/src/sync/cond.go:56 +0x80
io.(*pipe).read(0xc420094300, 0xc42e346a00, 0x200, 0x200, 0x0, 0x0, 0x0)
        /usr/local/go/src/io/pipe.go:47 +0xc6
io.(*PipeReader).Read(0xc4201e3d20, 0xc42e346a00, 0x200, 0x200, 0x0, 0x0, 0x0)
        /usr/local/go/src/io/pipe.go:130 +0x4c
encoding/json.(*Decoder).refill(0xc42d2e8000, 0x1, 0x1b0728)
        /usr/local/go/src/encoding/json/stream.go:152 +0x114
encoding/json.(*Decoder).readValue(0xc42d2e8000, 0x0, 0x0, 0x7f41e0ec5d90)
        /usr/local/go/src/encoding/json/stream.go:128 +0x275
encoding/json.(*Decoder).Decode(0xc42d2e8000, 0xe6ba00, 0xc42d2d5480, 0xc42cd31b01, 0xc42d2d5480)
        /usr/local/go/src/encoding/json/stream.go:57 +0x78
encoding/json.(*Decoder).Decode-fm(0xe6ba00, 0xc42d2d5480, 0xc42e34ab08, 0x2d79370)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/json.go:119 +0x3e
github.com/ethereum/go-ethereum/rpc.(*jsonCodec).ReadRequestHeaders(0xc42e34aaf0, 0x0, 0x0, 0x0, 0xc42d2c6e00, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/json.go:144 +0xe1
github.com/ethereum/go-ethereum/rpc.(*Server).readRequest(0xc42cd18f80, 0x18849e0, 0xc42e34aaf0, 0x1, 0xebc420, 0xc42d2a3900, 0x187fc60, 0xc42d2c97d0, 0x1)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:378 +0x5f
github.com/ethereum/go-ethereum/rpc.(*Server).serveRequest(0xc42cd18f80, 0x187fbe0, 0xc4200160b0, 0x18849e0, 0xc42e34aaf0, 0x0, 0x3, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:163 +0x308
github.com/ethereum/go-ethereum/rpc.(*Server).ServeCodec(0xc42cd18f80, 0x18849e0, 0xc42e34aaf0, 0x3)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:219 +0x94
created by github.com/ethereum/go-ethereum/rpc.DialInProc.func1
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/inproc.go:29 +0x2c1

goroutine 166 [select, 4055 minutes]:
github.com/ethereum/go-ethereum/rpc.(*Client).dispatch(0xc4203132d0, 0x1884580, 0xc42d2c6e70)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:541 +0x43d
created by github.com/ethereum/go-ethereum/rpc.newClient
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:244 +0x358

goroutine 167 [semacquire, 4055 minutes]:                                                                                                                                                                 [191/45080]
sync.runtime_notifyListWait(0xc420094400, 0xc400000000)
        /usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4200943f0)
        /usr/local/go/src/sync/cond.go:56 +0x80
io.(*pipe).read(0xc4200943c0, 0xc42e346800, 0x200, 0x200, 0x0, 0x0, 0x0)
        /usr/local/go/src/io/pipe.go:47 +0xc6
io.(*PipeReader).Read(0xc4201e3d30, 0xc42e346800, 0x200, 0x200, 0x0, 0x0, 0x0)
        /usr/local/go/src/io/pipe.go:130 +0x4c
encoding/json.(*Decoder).refill(0xc42d2e81c0, 0xdf5560, 0xc4204a48b8)
        /usr/local/go/src/encoding/json/stream.go:152 +0x114
encoding/json.(*Decoder).readValue(0xc42d2e81c0, 0x0, 0x0, 0x426b28)
        /usr/local/go/src/encoding/json/stream.go:128 +0x275
encoding/json.(*Decoder).Decode(0xc42d2e81c0, 0xe6ba00, 0xc42d2d5420, 0x45218f, 0xc42048aef8)
        /usr/local/go/src/encoding/json/stream.go:57 +0x78
github.com/ethereum/go-ethereum/rpc.(*Client).read.func1(0xec3cc0, 0xc42d2d5440, 0xc42d2c6e70, 0x7f41de1e7ec8, 0xc42d2c6e70)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:682 +0x79
github.com/ethereum/go-ethereum/rpc.(*Client).read(0xc4203132d0, 0x1884580, 0xc42d2c6e70, 0xc42048a460, 0x452422)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:695 +0x152
created by github.com/ethereum/go-ethereum/rpc.(*Client).dispatch
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:517 +0x71

goroutine 504086 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc42034e344)
        /usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc42034e338)
        /usr/local/go/src/sync/waitgroup.go:131 +0x72
github.com/ethereum/go-ethereum/core.(*BlockChain).Stop(0xc42034e000)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:654 +0xaa
github.com/ethereum/go-ethereum/eth.(*Ethereum).Stop(0xc42e8620e0, 0xc4201dca50, 0xc42d30df58)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/backend.go:413 +0x45
github.com/ethereum/go-ethereum/node.(*Node).Stop(0xc420327180, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/node/node.go:419 +0x25b
created by github.com/ethereum/go-ethereum/cmd/utils.StartNode.func1
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/utils/cmd.go:76 +0x15e

goroutine 488019 [select]:                                                                                                                                                                                [155/45080]
github.com/ethereum/go-ethereum/p2p.(*Peer).pingLoop(0xc4e8400420)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:239 +0x182
created by github.com/ethereum/go-ethereum/p2p.(*Peer).run
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:193 +0x117

goroutine 487989 [select, 172 minutes]:
github.com/ethereum/go-ethereum/eth.(*peer).broadcast(0xc4620d6e70)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:116 +0x1ec
created by github.com/ethereum/go-ethereum/eth.(*peerSet).Register
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:428 +0x12b

goroutine 488018 [IO wait]:
internal/poll.runtime_pollWait(0x7f41e0e89c70, 0x72, 0x0)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc442dfa918, 0x72, 0xffffffffffffff00, 0x1876920, 0x186f8a8)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc442dfa918, 0xc42b76c900, 0x20, 0x20)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc442dfa900, 0xc42b76c960, 0x20, 0x20, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
net.(*netFD).Read(0xc442dfa900, 0xc42b76c960, 0x20, 0x20, 0x0, 0xc4203becc0, 0xe5f4d08a8ce7)
        /usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc435347c50, 0xc42b76c960, 0x20, 0x20, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:176 +0x6d
io.ReadAtLeast(0x7f41cefc31d8, 0xc435347c50, 0xc42b76c960, 0x20, 0x20, 0x20, 0xf77e80, 0xdd4f8466c200, 0x7f41cefc31d8)
        /usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x7f41cefc31d8, 0xc435347c50, 0xc42b76c960, 0x20, 0x20, 0x20, 0x0, 0x6fc23a36a)
        /usr/local/go/src/io/io.go:327 +0x58
github.com/ethereum/go-ethereum/p2p.(*rlpxFrameRW).ReadMsg(0xc433eb8a10, 0xbec7d28093fbdbcd, 0xdd4f8466c27d, 0x1bd1a20, 0x0, 0x0, 0x1bd1a20, 0x0, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:656 +0x100
github.com/ethereum/go-ethereum/p2p.(*rlpx).ReadMsg(0xc42fd24db0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:95 +0x148
github.com/ethereum/go-ethereum/p2p.(*Peer).readLoop(0xc4e8400420, 0xc513315320)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:255 +0xad
created by github.com/ethereum/go-ethereum/p2p.(*Peer).run
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:192 +0xf2

goroutine 443712 [semacquire, 224 minutes]:                                                                                                                                                               [117/45080]
sync.runtime_Semacquire(0xc4204d07cc)
        /usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc4204d07c0)
        /usr/local/go/src/sync/waitgroup.go:131 +0x72
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Cancel(0xc4204d0680)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:525 +0x41
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync(0xc4204d0680, 0xc50a40bdc0, 0x5, 0x8, 0x5, 0xc50a40bdc0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:500 +0x174
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc4204d0680, 0xc50bb92e10, 0xe21e6c32b8f15071, 0x764fda8b4c7a25c7, 0x31d61f1885feca09, 0x6a0d6fc6d09623b9, 0xc48f334dc0, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:474 +0x6a3
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0xc4204d0680, 0xc49b800fd0, 0x10, 0xe21e6c32b8f15071, 0x764fda8b4c7a25c7, 0x31d61f1885feca09, 0x6a0d6fc6d09623b9, 0xc48f334dc0, 0x0, 0x0, ..
.)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:399 +0x3e2
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0xc4204d0680, 0xc49b800fd0, 0x10, 0xe21e6c32b8f15071, 0x764fda8b4c7a25c7, 0x31d61f1885feca09, 0x6a0d6fc6d09623b9, 0xc48f334dc0, 0x0, 0xc4202
e6760, ...)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:317 +0x97
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).synchronise(0xc42e86c300, 0xc45d0e93f0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:200 +0x392
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:155 +0x23c

goroutine 488020 [select, 3 minutes]:
github.com/ethereum/go-ethereum/p2p.(*protoRW).ReadMsg(0xc433eb8b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xdd319a4ae45a)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:402 +0x148
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handleMsg(0xc42e86c300, 0xc4620d6e70, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:320 +0xbd
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handle(0xc42e86c300, 0xc4620d6e70, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:309 +0x8ae
github.com/ethereum/go-ethereum/eth.NewProtocolManager.func1(0xc4e8400420, 0x1878060, 0xc433eb8b60, 0x0, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:142 +0x1f2
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1(0xc433eb8b60, 0xc4e8400420, 0x1878060, 0xc433eb8b60)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:348 +0x66
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:347 +0x228

goroutine 488016 [select, 7 minutes]:
github.com/ethereum/go-ethereum/p2p.(*Peer).run(0xc4e8400420, 0xdd7dc0, 0xc4200b2480, 0x0)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:202 +0x2fe
github.com/ethereum/go-ethereum/p2p.(*Server).runPeer(0xc420327680, 0xc4e8400420)
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:894 +0x122
created by github.com/ethereum/go-ethereum/p2p.(*Server).run
        /home/brandon/repos/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:646 +0x1443

[rjl493456442]

@brandoncurtis Hey, i pushed a fix and now it is merged(The main modification of this PR is spliting a huge write pause to several small one). Maybe you can have a try when new version is released.

[ryanschneider]

Hey @rjl493456442, for what it's worth, even with 1.8.13 running on an archive node that is currently synced to around block 5.3million (so still far behind head), I still get into a state where the node stops syncing for anywhere from 30 to 60 minutes, and emits the same warning every minute, but it does eventually recover:

Jul 31 21:28:23 ip-10-0-20-47 geth[21517]: INFO [07-31|21:28:23.730] Imported new chain segment               blocks=24 txs=2544 mgas=156.682 elapsed=8.450s     mgasps=18.541 number=5389082 hash=5a17f6…69
Jul 31 21:28:28 ip-10-0-20-47 geth[21517]: WARN [07-31|21:28:28.950] Database compacting, degraded performance database=/mnt/local/ethereum/geth/chaindata
Jul 31 21:29:29 ip-10-0-20-47 geth[21517]: WARN [07-31|21:29:29.272] Database compacting, degraded performance database=/mnt/local/ethereum/geth/chaindata
...truncating lots of duplicate log lines...
Jul 31 22:20:51 ip-10-0-20-47 geth[21517]: WARN [07-31|22:20:51.668] Database compacting, degraded performance database=/mnt/local/ethereum/geth/chaindata
Jul 31 22:21:39 ip-10-0-20-47 geth[21517]: INFO [07-31|22:21:39.835] Imported new chain segment               blocks=14 txs=1533 mgas=103.011 elapsed=53m16.104s mgasps=0.032  number=5389096 hash=5773ff…b3
Jul 31 22:21:48 ip-10-0-20-47 geth[21517]: INFO [07-31|22:21:48.291] Imported new chain segment               blocks=8  txs=943  mgas=51.582  elapsed=8.456s     mgasps=6.100  number=5389104 hash=9caf85…ab

However, this is an improvement over versions before your fix in #17144, those older versions would stay in the compacting state for multiple hours/days at a time, in fact I never actually saw a node recover, I always ended up having to restart the node manually.

[rjl493456442]

@ryanschneider Let me explain briefly what i have done in #17144. For the leveldb, if there are too many level0 files(more than 10 as the default settting), it will pause all new writing operations and wait compaction thread compacts these level0 files to higher level. This mechanism is to ensure a balance between writing speed and reading speed since too many level0 files will slow down reading speed.

Before my fix, the old version will resume writing operation only when all needed compactions are done. The compactions includes level0, level1 ... compactions. So it can cost you more than 1 day if the database size is large.
In my fix, i split these compactions to many pieces. Once leveldb finds there are not too many level0 files left(less than 10), it will resume writing operation and leave remain compactions in the backend.

However the main cause is for the archive node, the writing speed is larger than compaction speed. So it will still have a lot of pause phenomenon but just become smaller.

[ryanschneider]

Thanks for the explanation @rjl493456442 , that was very useful.

So, it might only surface during the initial sync, as hopefully the writing speed is less than the compaction speed once the node is caught up and only new blocks are writing to the chain.

I've started a new 1.8.13 archive node syncing from scratch as a test; I expect it will take over a week to catch up but will keep an eye on it.

I'm also going to take an older node w/ and existing chaindata and upgrade it to 1.8.13 and see if it reports any compaction warnings as well, as I said I suspect it won't, or if it does the interruptions will hopefully be short-lived.

[jotto]

--cache=30000 --gcmode=archive --syncmode=full

FWIW 1.8.15-unstable (we upgraded to 1.8.15-unstable around block 3,000,000) has been "Database compacting, degraded performance" for over 3 days on AWS i3.2xlarge (8 cores, 60GB of RAM, and 1900 GiB NVMe SSD). It started happening around block 5,700,000.

I'm not sure whether to restart the node or let it keep working.

It does not appear to be reducing disk usage, in fact it looks like disk usage has increased over the past 3 days by ~40GB.

edit: it's syncing again, so it looks like it may have been compacting for ~3.5 days

[rjl493456442]

@jotto Hey, so finally you restarted the node? Or the node is resumed after 3.5 days?

[jotto]

I did not restart, so it resumed after 3.5 days.

In the first hour after it finished compacting it looked like it was doing 0.5 blocks per second, and I just checked, and about ~5 hours since compacting it's doing about 3.5 blocks per second (promising).

[rjl493456442]

@jotto Thanks for the information! The weird thing is during the long compaction, the disk usage increases about 40GB. I'll check the leveldb codebase and post anything I find here.

[jotto]

Update: it's been going in and out of Database compacting, degraded performance (~10 hours at a time). Before it went into its latest compacting phase, it was around 6,050,000

[happyCoder1980]

Good morning.

I am new to Ehtereum and am using geth to simulate Ethereum nodes for a POC. I see the same error as the one seen here:

WARN [09-20|15:21:44.630] Database compacting, degraded performance database=/root/.ethereum/geth/chaindata
WARN [09-20|15:22:44.720] Database compacting, degraded performance database=/root/.ethereum/geth/chaindata

What's funny is I just set this up by installing geth on a couple of docker instances in my network. This is not a live production setup. In fact, I am yet to define a contract or even setup a web3 client to talk to the geth instance.

The insgtances do come up again in about 15 minutes though. However, I suspect I may be something really basic.

Is there something basic I am missing or could check.? Any logs in particular I can check to debug this?

Thanks,
Anand

[rjl493456442]

@happyCoder1980 It is because the underlying leveldb is evolved a long compaction which can pause all write operations of the system. So we print the warning logs for users.

[banteg]

From my tests, the database compaction currently takes up to ¹⁄₈ of the total sync time (4h19m of 1d10h in my case). I've also made a chart which shows when it occurs (red stripes) .

[ryanschneider]

_Very_ useful chart @banteg, thanks for sharing!

[plakat3]

Don't see the problem is solved so far. Got 1.8.20-stable-24d727b6 and this "Database compacting," goes forever. It get stuck at
age=11mo1w1d

@happyCoder1980 It is because the underlying leveldb is evolved a long compaction which can pause all write operations of the system. So we print the warning logs for users.

Warning logs doesn't help much as this can go forever. After restarting geth, the circle begins at
age=11mo1w1d
and is back to "Database compacting". I did a research about the problem, but there is nothing that can help.
If someone from the people who manage read this: Hire some people that do programming for a living, not for fun.

[ymonye]

I am on geth 1.8.20-stable-24d727b6 as well. Every now & then I experience the "Database compacting" message for several hours at a time when attempting to sync a full / archive node.

currentBlock: 5597323
highestBlock: 7024943

[yellow444]

4 blocks per second mee too
INFO [01-25|21:52:58.467] Imported new chain segment blocks=22 txs=2724 mgas=159.169 elapsed=8.426s mgasps=18.890 number=5596186 hash=0b9d94…589b70
( Samsung SSD 960 PRO 1TB and 128gb ram)

[PengFei-eth]

This is going to be a bigger problem in the future as the size of the blockchain continues to increase, if this problem isn't resolved correctly.

[fjl]

Closing this because the fix to leveldb was implemented and takes care of the original issue. We are looking to improve compaction times further in the upcoming 1.9 release.

[rotanid]

my compacting ran just over a week, i had to do a reboot then, now its already running for 4 days. core i5-3570k always at 100% load on one cpu core. no I/O load.

[rjl493456442]

@rotanid If no I/O load at all during the long compaction, I think there is a dead lock in the leveldb. I'll try to find it.