Cockroach: One node eats all CPU and an increasing amount of memory
Describe the problem
Doing the same insert again as in https://github.com/cockroachdb/cockroach/issues/38778 on the same setup, but now with different result.
Queries don't time out (yet?) as in the other issue, there are no logs about slow latch acquisitions, but one node eats up all CPU it can, causing the cluster to perform poorly. So far it seems to be a different issue, hence opening a separate ticket for it.
It's different in the given node consumes more and more memory and it uses not just one CPU core, but all of them. I could've observed this once a day ago, where it went to 30+ GiB until it brought the machine to its knees.
To Reproduce
The same as in the above linked ticket: filling up some tables.
Expected behavior
I would expect a constant performance without any steep drops.
Additional data / screenshots
The ingest job logs its progress. Columns:
- the number of operations executed so far (insert/deletes, deletes can affect nonexistent rows)
- transactions/sec for that batch
- insert statistics: min/max/avg/median (in ms)
- delete statistics: min/max/avg/median (in ms)
15600943, 400.98, ins:9.33/583.18/34.09/17.27, del:1.56/591.41/12.21/4.12
15700943, 373.21, ins:8.46/987.52/33.58/17.15, del:1.62/734.11/13.60/4.00
15800943, 398.64, ins:8.02/1020.05/39.92/17.04, del:1.37/853.66/12.34/4.05
15900951, 349.68, ins:8.48/1052.80/38.23/16.88, del:1.43/839.80/14.45/3.89
16000960, 375.46, ins:9.49/648.37/31.32/16.74, del:1.48/621.32/12.28/3.89
16100963, 391.48, ins:9.86/560.80/32.09/16.88, del:1.54/553.71/11.95/4.00
16200972, 417.35, ins:9.23/1756.25/42.95/16.97, del:1.44/789.27/11.51/4.05
16300978, 416.45, ins:10.09/968.13/34.97/17.05, del:1.44/977.60/11.51/4.10
16400992, 379.54, ins:9.93/657.86/34.22/17.17, del:1.60/682.42/12.49/4.01
16501001, 395.42, ins:8.02/1089.04/38.36/17.08, del:1.46/779.51/11.59/3.96
16601007, 406.95, ins:9.98/908.65/34.16/16.96, del:1.66/510.20/11.36/4.02
16701007, 323.47, ins:8.84/1212.30/36.79/16.90, del:1.52/1142.74/14.63/3.78
16801016, 316.66, ins:9.07/716.20/33.17/16.92, del:1.66/712.06/14.51/3.77
16901021, 302.12, ins:8.60/793.45/33.03/16.37, del:1.64/826.62/12.79/3.56
17001030, 277.57, ins:8.06/972.81/32.84/16.42, del:1.53/746.81/14.81/3.60
17101031, 295.58, ins:9.14/806.91/33.13/16.53, del:1.54/692.55/11.69/3.55
17201037, 259.20, ins:7.68/757.86/35.23/17.43, del:1.46/652.40/13.10/3.82
17301041, 259.08, ins:8.40/1372.07/35.24/16.86, del:1.13/914.09/14.30/3.72
17401060, 262.62, ins:8.04/698.44/30.19/15.69, del:1.33/543.42/9.55/3.22
17501060, 137.14, ins:7.96/717.65/25.42/14.91, del:1.35/506.85/14.59/3.34
17601063, 84.46, ins:7.88/649.38/21.43/14.39, del:1.15/383.98/10.83/3.03
17701064, 82.26, ins:7.70/564.12/21.73/14.41, del:1.37/350.27/10.64/3.05
As you can see, when the phenomenon started, the throughput declined drastically.
Memory usage increasing:
The only anomaly I could find is this:
From the CPU graph it can be seen that it's something happening inside the process (and doing much less work in the kernel) eating up all available resources (CRDB times are UTC, system graphs are UTC+2):
Saved debug zip and have a full goroutine dump at:
https://gist.github.com/bra-fsn/27848c0b2af2c6c90b9ba07fb1166746
According to a trace to the process (and to the graphs as well), it does very few syscalls.
The toplist is (ran the trace for 10 seconds):
(number of occurrences, and the actuall calls)
12866 6083 cockroachoss CALL _umtx_op(
3327 6083 cockroachoss CALL nanosleep(
1410 6083 cockroachoss CALL kevent(0x6
183 6083 cockroachoss CALL write(0x16
166 6083 cockroachoss CALL read(0x1b
154 6083 cockroachoss CALL read(0x1a
150 6083 cockroachoss CALL read(0x26
146 6083 cockroachoss CALL sched_yield
104 6083 cockroachoss CALL read(0x16
78 6083 cockroachoss CALL write(0x12
76 6083 cockroachoss CALL read(0x24
70 6083 cockroachoss CALL getdirentries(0x33
53 6083 cockroachoss CALL read(0x1f
49 6083 cockroachoss CALL fsync(0x650)
47 6083 cockroachoss CALL write(0x650
31 6083 cockroachoss CALL write(0x1d
27 6083 cockroachoss CALL read(0x25
26 6083 cockroachoss CALL write(0x1c
22 6083 cockroachoss CALL read(0x53a
21 6083 cockroachoss CALL close(0x33)
20 6083 cockroachoss CALL write(0x1b
18 6083 cockroachoss CALL write(0x53a
18 6083 cockroachoss CALL read(0x15
17 6083 cockroachoss CALL write(0x1a
17 6083 cockroachoss CALL getdirentries(0x3b
15 6083 cockroachoss CALL write(0x24
14 6083 cockroachoss CALL read(0x1d
13 6083 cockroachoss CALL write(0x26
13 6083 cockroachoss CALL write(0x25
12 6083 cockroachoss CALL write(0x1f
12 6083 cockroachoss CALL read(0x1c
11 6083 cockroachoss CALL write(0x15
10 6083 cockroachoss CALL read(0x12
It basically does some chatting with its cluster-buddies and doing very few writes to the log files (both in logs/ and /data/crdb/store2/034254.log like files. So judging from these, it doesn't do any IO intensive tasks, like compaction, merging or anything similar.
Also, I've stopped the SQL ingest and the process still eats all the CPUs. Leaving it in this state for a while to see what happens.
Environment:
- CockroachDB version 19.1.2
- Server OS: FreeBSD 11/amd64
- Client app python/psycopg2
bra-fsn
All 19 comments
OK, I killed the ingest script (so no SQL inserts/deletes come in) and left the cluster running.
It seems the process got into an endless loop or something:
(that bump in the system (green) times is when I killed the script, so now it does even less syscalls and just spins in userspace)
The memory usage remained constant BTW:
bra-fsn
on 10 Jul 2019
Huh, this is pretty weird. There are a bunch of stuck goroutines in C++ MVCCScan:
goroutine 69250073 [syscall, 72 minutes]:
github.com/cockroachdb/cockroach/pkg/storage/engine._Cfunc_MVCCScan(0x83791b3c0, 0xc00cceb7f0, 0xe, 0xc00cceb800, 0xe, 0x15b00934397a8d66, 0x0, 0x2, 0xc00cceb810, 0x10, ...)
_cgo_gotypes.go:1306 +0x6a
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBIterator).MVCCScan.func1(0xc00ecd57e8, 0xc00cceb7f0, 0xe, 0x10, 0xc00cceb800, 0xe, 0x10, 0x15b00934397a8d66, 0x0, 0x2, ...)
/home/bra/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:2434 +0x23f
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBIterator).MVCCScan(0xc00ecd57e8, 0xc00cceb7f0, 0xe, 0x10, 0xc00cceb800, 0xe, 0x10, 0x2, 0x15b00934397a8d66, 0x0, ...)
/home/bra/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:2429 +0x13d
github.com/cockroachdb/cockroach/pkg/storage/engine.MVCCScanToBytes(0x3b05220, 0xc008a605d0, 0x3b2b940, 0xc00ecd5770, 0xc00cceb7f0, 0xe, 0x10, 0xc00cceb800, 0xe, 0x10, ...)
/home/bra/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/mvcc.go:2042 +0x21a
@petermattis or @ajkr any clues about what this could mean?
jordanlewis
on 11 Jul 2019
I'm leaving the process running if you need it. BTW, still spinning.
bra-fsn
on 11 Jul 2019
This looks well qualified - I'm moving over to core. If there are any objections or basics to cover, let me know.
tim-o
on 11 Jul 2019
Thanks for reporting. The "debug/nodes/*/enginestats" files from the debug.zip could be useful to look at.
Also since the infinite loop appears to be inside a CGo call the Goroutine dump isn't enough. If you have time to run quickstack (https://github.com/yoshinorim/quickstack) on your process and report that output, it could be helpful as well.
ajkr
on 11 Jul 2019
Also it's worth noting that from the Goroutine dump the stuck scans are all ReverseScans. There was an infinite loop bug fix for that recently (31a81930be2), but I verified that fix is included in 19.1.2.
ajkr
on 11 Jul 2019
Attached enginestats outputs
enginestats.zip
bra-fsn
on 12 Jul 2019
@ajkr, sorry I couldn't get quickstack working. Issuing a pstack several times gave this:
# /tmp/pstack -O 6083
6083: /data/crdb/cockroachoss
----------------- thread -1 (running) -----------------
0x8069bc29d memcmp (27f0b0d, 0, d1b7121, 8, f, 0) + 2d in /lib/libc.so.7
0x27f0b0d _ZNK7rocksdb8MemTable13KeyComparatorclEPKcRKNS_5SliceE (288c994, 0, d1b7121, 8, 17, 0) + 7d in /data/crdb/cockroachoss
0x288c994 _ZN7rocksdb14InlineSkipListIRKNS_11MemTableRep13KeyComparatorEE8Iterator4SeekEPKc (27f8add, 0, ffffdee0, 7fff, e71f0fc, 0) + 94 in /data/crdb/cockroachoss
0x27f8add _ZN7rocksdb16MemTableIterator4SeekERKNS_5SliceE (2951ba4, 0, ffffdf10, 7fff, ffffff80, ffffffff) + 1cd in /data/crdb/cockroachoss
0x2951ba4 _ZN7rocksdb15MergingIterator4SeekERKNS_5SliceE (2793379, 0, 0, 0, 0, 0) + 194 in /data/crdb/cockroachoss
0x2793379 _ZN7rocksdb6DBIter4SeekERKNS_5SliceE (269cf47, 0, 3c30228, 0, ffffe168, 7fff) + 2e9 in /data/crdb/cockroachoss
0x269cf47 _ZN9cockroach11mvccScannerILb1EE21backwardLatestVersionERKN7rocksdb5SliceEi (269dbf3, 0, ffffe0c8, 7fff, ffffe2f0, 7fff) + 1d7 in /data/crdb/cockroachoss
0x269dbf3 _ZN9cockroach11mvccScannerILb1EE7prevKeyERKN7rocksdb5SliceE (269b608, 0, d444280, 8, ffffe0d9, 7fff) + 1f3 in /data/crdb/cockroachoss
0x269b608 _ZN9cockroach11mvccScannerILb1EE4scanEv (2696433, 0, 506b72ae, 15b008e7, 0, 0) + 78 in /data/crdb/cockroachoss
0x2696433 MVCCScan (254af3d, 0, 506b72ae, 15b008e7, 0, 0) + b3 in /data/crdb/cockroachoss
0x254af3d _cgo_46b63153f027_Cfunc_MVCCScan (60de60, 0, ffffe520, 7fff, 5e71c1, 0) + ad in /data/crdb/cockroachoss
# /tmp/pstack -O 6083
6083: /data/crdb/cockroachoss
----------------- thread -1 (running) -----------------
0x264c2a8 _ZNK9cockroach12DBComparator7CompareERKN7rocksdb5SliceES4_ (27a0b8c, 0, 20a880ed, 8, 1a, 0) + 88 in /data/crdb/cockroachoss
0x27a0b8c _ZNK7rocksdb21InternalKeyComparator7CompareERKNS_5SliceES3_ (2af18ef, 0, 768ec38, 8, 0, 0) + 4c in /data/crdb/cockroachoss
0x2af18ef _ZN7rocksdb14IndexBlockIter4SeekERKNS_5SliceE (291deba, 0, e, 0, 2d5fa4f8, 8) + 2bf in /data/crdb/cockroachoss
0x291deba _ZN7rocksdb23BlockBasedTableIteratorINS_13DataBlockIterENS_5SliceEE4SeekERKS2_ (283b6ee, 0, 2d5fa4e2, 8, 20, 0) + 17a in /data/crdb/cockroachoss
0x283b6ee _ZN7rocksdb12_GLOBAL__N_113LevelIterator4SeekERKNS_5SliceE (2951ba4, 0, 1e4f8af0, 8, ffffffe0, ffffffff) + 16e in /data/crdb/cockroachoss
0x2951ba4 _ZN7rocksdb15MergingIterator4SeekERKNS_5SliceE (2793379, 0, 0, 0, 0, 0) + 194 in /data/crdb/cockroachoss
0x2793379 _ZN7rocksdb6DBIter4SeekERKNS_5SliceE (269cf47, 0, 3c30228, 0, d99cab78, 7fff) + 2e9 in /data/crdb/cockroachoss
0x269cf47 _ZN9cockroach11mvccScannerILb1EE21backwardLatestVersionERKN7rocksdb5SliceEi (269dbf3, 0, d99caad8, 7fff, d99cad00, 7fff) + 1d7 in /data/crdb/cockroachoss
0x269dbf3 _ZN9cockroach11mvccScannerILb1EE7prevKeyERKN7rocksdb5SliceE (269b608, 0, 1c8aa140, 8, d99caae9, 7fff) + 1f3 in /data/crdb/cockroachoss
0x269b608 _ZN9cockroach11mvccScannerILb1EE4scanEv (2696433, 0, a5084546, 15b00896, 0, 0) + 78 in /data/crdb/cockroachoss
0x2696433 MVCCScan (254af3d, 0, a5084546, 15b00896, 0, 0) + b3 in /data/crdb/cockroachoss
0x254af3d _cgo_46b63153f027_Cfunc_MVCCScan (60de60, 0, 5d2ab80, 0, 1, 0) + ad in /data/crdb/cockroachoss
# /tmp/pstack -O 6083
6083: /data/crdb/cockroachoss
----------------- thread -1 (running) -----------------
0x2705292 _ZN7rocksdb12ShardedCache6LookupERKNS_5SliceEPNS_10StatisticsE (2907cef, 0, ffffda40, 7fff, 0, 0) + 22 in /data/crdb/cockroachoss
0x2907cef _ZN7rocksdb12_GLOBAL__N_117GetEntryFromCacheEPNS_5CacheERKNS_5SliceENS_7TickersES6_PmS7_PNS_10StatisticsEPNS_10GetContextE (290783c, 0, bfbc118, 8, 0, 0) + 2f in /data/crdb/cockroachoss
0x290783c _ZN7rocksdb15BlockBasedTable21GetDataBlockFromCacheERKNS_5SliceES3_PNS_5CacheES5_RKNS_18ImmutableCFOptionsERKNS_11ReadOptionsEPNS0_13CachableEntryINS_5BlockEEEjS3_mbPNS_10GetContextE (29063e7, 0, ac1cf30, 8, ffffdd10, 7fff) + 9c in /data/crdb/cockroachoss
0x29063e7 _ZN7rocksdb15BlockBasedTable25MaybeLoadDataBlockToCacheEPNS_18FilePrefetchBufferEPNS0_3RepERKNS_11ReadOptionsERKNS_11BlockHandleENS_5SliceEPNS0_13CachableEntryINS_5BlockEEEbPNS_10GetContextE (29180e1, 0, 3c5849a, 0, 0, 0) + 217 in /data/crdb/cockroachoss
0x29180e1 _ZN7rocksdb15BlockBasedTable20NewDataBlockIteratorINS_13DataBlockIterEEEPT_PNS0_3RepERKNS_11ReadOptionsERKNS_11BlockHandleES4_bbbPNS_10GetContextENS_6StatusEPNS_18FilePrefetchBufferE (291e74b, 0, 0, 0, 0, 0) + 1d1 in /data/crdb/cockroachoss
0x291e74b _ZN7rocksdb23BlockBasedTableIteratorINS_13DataBlockIterENS_5SliceEE13InitDataBlockEv (291ded3, 0, 2284cf74, 0, 76357d60, 8) + 23b in /data/crdb/cockroachoss
0x291ded3 _ZN7rocksdb23BlockBasedTableIteratorINS_13DataBlockIterENS_5SliceEE4SeekERKS2_ (283b6ee, 0, c278887, 8, 1f, 0) + 193 in /data/crdb/cockroachoss
0x283b6ee _ZN7rocksdb12_GLOBAL__N_113LevelIterator4SeekERKNS_5SliceE (2951ba4, 0, d48d2f0, 8, ffffffc0, ffffffff) + 16e in /data/crdb/cockroachoss
0x2951ba4 _ZN7rocksdb15MergingIterator4SeekERKNS_5SliceE (2793379, 0, ff000000, 7fff, 0, 0) + 194 in /data/crdb/cockroachoss
0x2793379 _ZN7rocksdb6DBIter4SeekERKNS_5SliceE (269d155, 0, ffffe2f0, 7fff, ffffe168, 7fff) + 2e9 in /data/crdb/cockroachoss
0x269d155 _ZN9cockroach11mvccScannerILb1EE12iterPeekPrevEPN7rocksdb5SliceE (269da8c, 0, ffffe0c8, 7fff, ffffe2f0, 7fff) + 145 in /data/crdb/cockroachoss
0x269da8c _ZN9cockroach11mvccScannerILb1EE7prevKeyERKN7rocksdb5SliceE (269b608, 0, d444280, 8, ffffe0d9, 7fff) + 8c in /data/crdb/cockroachoss
0x269b608 _ZN9cockroach11mvccScannerILb1EE4scanEv (2696433, 0, 506b72ae, 15b008e7, 0, 0) + 78 in /data/crdb/cockroachoss
0x2696433 MVCCScan (254af3d, 0, 506b72ae, 15b008e7, 0, 0) + b3 in /data/crdb/cockroachoss
0x254af3d _cgo_46b63153f027_Cfunc_MVCCScan (60de60, 0, ffffe520, 7fff, 5e71c1, 0) + ad in /data/crdb/cockroachoss
# /tmp/pstack -O 6083
6083: /data/crdb/cockroachoss
----------------- thread -1 (running) -----------------
0x8069bc27b memcmp (27a0b8c, 0, 14bb1d64, 8, 1a, 0) + b in /lib/libc.so.7
0x27a0b8c _ZNK7rocksdb21InternalKeyComparator7CompareERKNS_5SliceES3_ (2af7b6d, 0, 768ec38, 8, ffffddd0, 7fff) + 4c in /data/crdb/cockroachoss
0x2af7b6d _ZN7rocksdb9BlockIterINS_5SliceEE10BinarySeekINS_9DecodeKeyEEEbRKS1_jjPjPKNS_10ComparatorE (2af1369, 0, ffffde20, 7fff, 291e52b, 0) + 16d in /data/crdb/cockroachoss
0x2af1369 _ZN7rocksdb13DataBlockIter4SeekERKNS_5SliceE (291dee3, 0, e, 0, c268ad8, 8) + a9 in /data/crdb/cockroachoss
0x291dee3 _ZN7rocksdb23BlockBasedTableIteratorINS_13DataBlockIterENS_5SliceEE4SeekERKS2_ (283b6ee, 0, c268ac2, 8, 20, 0) + 1a3 in /data/crdb/cockroachoss
0x283b6ee _ZN7rocksdb12_GLOBAL__N_113LevelIterator4SeekERKNS_5SliceE (2951ba4, 0, d48d2f0, 8, 0, 0) + 16e in /data/crdb/cockroachoss
0x2951ba4 _ZN7rocksdb15MergingIterator4SeekERKNS_5SliceE (2793379, 0, ff000000, 7fff, 0, 0) + 194 in /data/crdb/cockroachoss
0x2793379 _ZN7rocksdb6DBIter4SeekERKNS_5SliceE (269d155, 0, ffffe2f0, 7fff, ffffe168, 7fff) + 2e9 in /data/crdb/cockroachoss
0x269d155 _ZN9cockroach11mvccScannerILb1EE12iterPeekPrevEPN7rocksdb5SliceE (269da8c, 0, ffffe0c8, 7fff, ffffe2f0, 7fff) + 145 in /data/crdb/cockroachoss
0x269da8c _ZN9cockroach11mvccScannerILb1EE7prevKeyERKN7rocksdb5SliceE (269b608, 0, d444280, 8, ffffe0d9, 7fff) + 8c in /data/crdb/cockroachoss
0x269b608 _ZN9cockroach11mvccScannerILb1EE4scanEv (2696433, 0, 506b72ae, 15b008e7, 0, 0) + 78 in /data/crdb/cockroachoss
0x2696433 MVCCScan (254af3d, 0, 506b72ae, 15b008e7, 0, 0) + b3 in /data/crdb/cockroachoss
0x254af3d _cgo_46b63153f027_Cfunc_MVCCScan (60de60, 0, ffffe520, 7fff, 5e71c1, 0) + ad in /data/crdb/cockroachoss
Saved a core for future reference.
Attached to it with gdb:
(gdb) info threads
Id Target Id Frame
* 1 LWP 102311 0x00000008069bc298 in memcmp () from /lib/libc.so.7
2 LWP 100193 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
3 LWP 100373 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
4 LWP 100442 FindGreaterOrEqual () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/memtable/inlineskiplist.h:473
5 LWP 100447 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
6 LWP 100475 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
7 LWP 101004 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
8 LWP 101032 0x00000008069bc298 in memcmp () from /lib/libc.so.7
9 LWP 101608 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
10 LWP 101641 GetVarint32Ptr () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/util/coding.h:158
11 LWP 101696 0x000000000264c29b in SplitKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/encoding.h:81
12 LWP 101714 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
13 LWP 101771 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
14 LWP 101781 0x00000008069c11dd in binuptime (bt=<optimized out>, tk=<optimized out>, abs=1) at /usr/src/lib/libc/sys/__vdso_gettimeofday.c:90
15 LWP 101849 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
16 LWP 101854 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
17 LWP 101866 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
18 LWP 101885 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
19 LWP 101893 0x00000008069bc296 in memcmp () from /lib/libc.so.7
20 LWP 101900 0x000000000264c287 in SplitKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/encoding.h:78
21 LWP 101914 0x00000008069bc29a in memcmp () from /lib/libc.so.7
22 LWP 101916 0x00000008069bc27b in memcmp () from /lib/libc.so.7
23 LWP 101918 Compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/dbformat.h:643
24 LWP 101919 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
25 LWP 101940 Status () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/include/rocksdb/status.h:27
26 LWP 101941 0x00000008069bc298 in memcmp () from /lib/libc.so.7
27 LWP 101942 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
28 LWP 101946 0x0000000805d2cf5c in _umtx_op_err () from /lib/libthr.so.3
29 LWP 101954 0x0000000805d2cf5c in _umtx_op_err () from /lib/libthr.so.3
30 LWP 101956 0x0000000805d2cf5c in _umtx_op_err () from /lib/libthr.so.3
31 LWP 101959 0x0000000805d2cf5c in _umtx_op_err () from /lib/libthr.so.3
32 LWP 101974 0x0000000805d2cf5c in _umtx_op_err () from /lib/libthr.so.3
33 LWP 101975 0x0000000805d2cf5c in _umtx_op_err () from /lib/libthr.so.3
34 LWP 101976 0x0000000805d2cf5c in _umtx_op_err () from /lib/libthr.so.3
35 LWP 101978 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
36 LWP 101990 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
37 LWP 102045 0x00000008069bc27b in memcmp () from /lib/libc.so.7
38 LWP 102046 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
39 LWP 102053 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
40 LWP 102058 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
41 LWP 102067 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
42 LWP 102146 0x0000000002811d5e in IsValueType () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/dbformat.h:72
43 LWP 102153 0x000000000283b639 in rocksdb::(anonymous namespace)::LevelIterator::Seek(rocksdb::Slice const&) ()
44 LWP 102154 0x00000008069bc296 in memcmp () from /lib/libc.so.7
45 LWP 102225 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
46 LWP 102229 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
47 LWP 102230 0x00000008069bc296 in memcmp () from /lib/libc.so.7
48 LWP 102236 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
49 LWP 100277 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
50 LWP 100329 0x0000000002af7a4e in BinarySeek<rocksdb::DecodeKey> () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/block.cc:618
51 LWP 103065 GetInternalKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/dbformat.h:357
52 LWP 103343 0x00000008069bc28b in memcmp () from /lib/libc.so.7
53 LWP 103337 0x000000000264c22d in Slice () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/../rocksdb/include/rocksdb/slice.h:38
54 LWP 103130 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
55 LWP 100344 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
56 LWP 100347 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
57 LWP 100348 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
58 LWP 100355 0x00000008069bc298 in memcmp () from /lib/libc.so.7
59 LWP 100359 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
60 LWP 100457 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
61 LWP 101687 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
62 LWP 101908 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
63 LWP 103396 0x000000000264c388 in Compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/../rocksdb/include/rocksdb/slice.h:244
64 LWP 103150 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
65 LWP 103184 0x000000000264c297 in SplitKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/encoding.h:81
66 LWP 103187 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
67 LWP 100346 0x00000008069bc27b in memcmp () from /lib/libc.so.7
68 LWP 100905 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
69 LWP 101020 0x0000000002af71e6 in ParseNextDataKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/block.cc:450
70 LWP 100516 0x00000008069bc27b in memcmp () from /lib/libc.so.7
71 LWP 100484 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
72 LWP 100275 ParseNextDataKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/block.cc:450
73 LWP 100281 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
74 LWP 100788 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
75 LWP 100795 SplitKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/encoding.h:82
76 LWP 101098 0x00000008069bc27b in memcmp () from /lib/libc.so.7
77 LWP 101101 0x00000008069bc29d in memcmp () from /lib/libc.so.7
78 LWP 101104 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
79 LWP 101105 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
80 LWP 101109 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
81 LWP 100256 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
82 LWP 100938 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
83 LWP 102485 0x00000008069bc27b in memcmp () from /lib/libc.so.7
84 LWP 101053 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
85 LWP 101847 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
86 LWP 101119 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
87 LWP 100192 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
88 LWP 100257 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
89 LWP 100266 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
90 LWP 101947 0x00000008069bc27b in memcmp () from /lib/libc.so.7
91 LWP 100966 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
92 LWP 100976 0x000000000264c22a in Slice () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/../rocksdb/include/rocksdb/slice.h:38
93 LWP 100866 runtime.kevent () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:436
94 LWP 100497 0x00000008069bc29d in memcmp () from /lib/libc.so.7
95 LWP 100848 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
96 LWP 100267 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
97 LWP 100965 0x0000000002af18e4 in BinarySeek<rocksdb::DecodeKey> () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/block.cc:624
98 LWP 100973 0x0000000805d2cf5c in _umtx_op_err () from /lib/libthr.so.3
99 LWP 101006 0x00000008069bc29d in memcmp () from /lib/libc.so.7
100 LWP 100980 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
101 LWP 101023 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
102 LWP 101058 0x00000008069c2996 in memcpy () from /lib/libc.so.7
103 LWP 101076 0x00000008069bc286 in memcmp () from /lib/libc.so.7
104 LWP 103435 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
105 LWP 103438 FindGreaterOrEqual () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/memtable/inlineskiplist.h:473
106 LWP 103440 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
107 LWP 101110 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
108 LWP 103489 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
109 LWP 103494 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
110 LWP 103496 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
111 LWP 103497 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
112 LWP 103498 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
113 LWP 103312 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
114 LWP 103321 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
115 LWP 103315 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
116 LWP 103322 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
117 LWP 103323 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
118 LWP 103324 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
119 LWP 103326 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
120 LWP 103330 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
121 LWP 103331 runtime.sys_umtx_op () at /usr/local/go/src/runtime/sys_freebsd_amd64.s:21
BTW, according to top, there are many threads which spin like hell, not just one:
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
6083 root 85 0 4467M 4462M RUN 5 20.3H 65.42% cockroachoss{cockroachoss}
6083 root 88 0 4467M 4462M RUN 12 22.0H 61.90% cockroachoss{cockroachoss}
6083 root 87 0 4467M 4462M RUN 10 21.4H 59.96% cockroachoss{cockroachoss}
6083 root 87 0 4467M 4462M RUN 8 21.7H 57.22% cockroachoss{cockroachoss}
6083 root 87 0 4467M 4462M RUN 4 21.6H 56.14% cockroachoss{cockroachoss}
6083 root 87 0 4467M 4462M RUN 13 21.4H 55.87% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M CPU15 15 20.9H 55.16% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M CPU1 1 21.5H 54.77% cockroachoss{cockroachoss}
6083 root 89 0 4467M 4462M CPU8 8 21.2H 53.76% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M RUN 17 20.7H 52.82% cockroachoss{cockroachoss}
6083 root 88 0 4467M 4462M CPU18 18 21.2H 50.62% cockroachoss{cockroachoss}
6083 root 89 0 4467M 4462M RUN 4 22.1H 49.86% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M CPU11 11 21.2H 49.64% cockroachoss{cockroachoss}
6083 root 87 0 4467M 4462M CPU17 17 21.3H 49.44% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M RUN 9 21.6H 49.28% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M RUN 2 21.3H 48.76% cockroachoss{cockroachoss}
6083 root 87 0 4467M 4462M RUN 18 21.6H 48.48% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M RUN 19 21.6H 48.24% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M RUN 14 20.6H 48.19% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M RUN 3 20.3H 47.20% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M RUN 1 21.1H 46.98% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M RUN 3 21.1H 46.06% cockroachoss{cockroachoss}
6083 root 85 0 4467M 4462M RUN 7 20.1H 45.95% cockroachoss{cockroachoss}
6083 root 84 0 4467M 4462M CPU12 12 20.6H 45.29% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M RUN 19 21.5H 45.20% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M RUN 11 21.0H 44.85% cockroachoss{cockroachoss}
6083 root 89 0 4467M 4462M CPU5 5 21.0H 43.84% cockroachoss{cockroachoss}
6083 root 85 0 4467M 4462M RUN 16 20.5H 42.48% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M RUN 9 22.1H 42.22% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M RUN 13 22.2H 42.19% cockroachoss{cockroachoss}
6083 root 85 0 4467M 4462M CPU9 9 20.9H 41.51% cockroachoss{cockroachoss}
6083 root 86 0 4467M 4462M RUN 6 21.0H 41.11% cockroachoss{cockroachoss}
6083 root 87 0 4467M 4462M CPU13 13 21.1H 41.07% cockroachoss{cockroachoss}
6083 root 87 0 4467M 4462M CPU14 14 20.6H 40.67% cockroachoss{cockroachoss}
6083 root 87 0 4467M 4462M CPU2 2 21.2H 40.43% cockroachoss{cockroachoss}
6083 root 85 0 4467M 4462M CPU3 3 20.8H 40.27% cockroachoss{cockroachoss}
6083 root 85 0 4467M 4462M RUN 0 20.6H 40.18% cockroachoss{cockroachoss}
6083 root 85 0 4467M 4462M CPU19 19 21.1H 38.33% cockroachoss{cockroachoss}
6083 root 84 0 4467M 4462M CPU7 7 20.3H 35.54% cockroachoss{cockroachoss}
6083 root 85 0 4467M 4462M CPU16 16 20.6H 35.26% cockroachoss{cockroachoss}
6083 root 85 0 4467M 4462M CPU0 0 20.7H 33.66% cockroachoss{cockroachoss}
6083 root 87 0 4467M 4462M CPU6 6 21.6H 27.21% cockroachoss{cockroachoss}
Thread IDs for them:
ps auwwx -H -o lwp | grep '[6]083'
root 6083 56.4 6.8 4566268 4559440 3 R+ Tue23 1285:40.96 ./cockroachoss s 101020
root 6083 54.6 6.8 4566268 4559440 3 R+ Tue23 1326:19.35 ./cockroachoss s 101058
root 6083 53.0 6.8 4566268 4559440 3 R+ Tue23 1301:09.01 ./cockroachoss s 103396
root 6083 51.8 6.8 4566268 4559440 3 R+ Tue23 1323:12.37 ./cockroachoss s 101101
root 6083 51.6 6.8 4566268 4559440 3 R+ Tue23 1296:09.39 ./cockroachoss s 101916
root 6083 51.3 6.8 4566268 4559440 3 R+ Tue23 1306:13.29 ./cockroachoss s 101006
root 6083 51.3 6.8 4566268 4559440 3 R+ Tue23 1270:24.96 ./cockroachoss s 102154
root 6083 51.1 6.8 4566268 4559440 3 R+ Tue23 1275:02.41 ./cockroachoss s 100275
root 6083 50.6 6.8 4566268 4559440 3 R+ Tue23 1260:08.35 ./cockroachoss s 101076
root 6083 50.2 6.8 4566268 4559440 3 R+ Tue23 1296:30.05 ./cockroachoss s 101032
root 6083 50.1 6.8 4566268 4559440 3 R+ Tue23 1296:52.39 ./cockroachoss s 101098
root 6083 49.7 6.8 4566268 4559440 3 R+ Tue23 1301:22.59 ./cockroachoss s 100355
root 6083 49.7 6.8 4566268 4559440 3 R+ Tue23 1273:52.24 ./cockroachoss s 102146
root 6083 49.7 6.8 4566268 4559440 3 R+ Tue23 1327:21.68 ./cockroachoss s 102311
root 6083 49.1 6.8 4566268 4559440 3 R+ Tue23 1241:14.19 ./cockroachoss s 103337
root 6083 49.0 6.8 4566268 4559440 3 R+ Tue23 1281:55.26 ./cockroachoss s 101641
root 6083 49.0 6.8 4566268 4559440 3 R+ Tue23 1267:04.84 ./cockroachoss s 101781
root 6083 48.8 6.8 4566268 4559440 3 R+ Tue23 1293:08.00 ./cockroachoss s 101914
root 6083 48.4 6.8 4566268 4559440 3 R+ Tue23 1263:48.79 ./cockroachoss s 101893
root 6083 48.4 6.8 4566268 4559440 3 R+ Tue23 1264:35.40 ./cockroachoss s 103343
root 6083 48.2 6.8 4566268 4559440 3 R+ Tue23 1237:46.09 ./cockroachoss s 100516
root 6083 47.9 6.8 4566268 4559440 3 R+ Tue23 1219:28.96 ./cockroachoss s 100965
root 6083 47.9 6.8 4566268 4559440 3 R+ Tue23 1242:06.67 ./cockroachoss s 103438
root 6083 47.8 6.8 4566268 4559440 3 R+ Tue23 1274:41.28 ./cockroachoss s 100442
root 6083 47.8 6.8 4566268 4559440 3 R+ Tue23 1240:16.50 ./cockroachoss s 103184
root 6083 47.6 6.8 4566268 4559440 3 R+ Tue23 1289:03.55 ./cockroachoss s 102485
root 6083 47.5 6.8 4566268 4559440 3 R+ Tue23 1237:07.89 ./cockroachoss s 101940
root 6083 47.0 6.8 4566268 4559440 3 R+ Tue23 1256:18.43 ./cockroachoss s 101918
root 6083 46.9 6.8 4566268 4559440 3 R+ Tue23 1234:33.20 ./cockroachoss s 100346
root 6083 46.7 6.8 4566268 4559440 3 R+ Tue23 1269:44.61 ./cockroachoss s 101900
root 6083 46.4 6.8 4566268 4559440 3 R+ Tue23 1210:24.73 ./cockroachoss s 100973
root 6083 46.4 6.8 4566268 4559440 3 R+ Tue23 1219:39.55 ./cockroachoss s 100976
root 6083 45.7 6.8 4566268 4559440 3 R+ Tue23 1334:58.23 ./cockroachoss s 102045
root 6083 45.6 6.8 4566268 4559440 3 R+ Tue23 1242:57.13 ./cockroachoss s 100329
root 6083 45.6 6.8 4566268 4559440 3 R+ Tue23 1258:38.56 ./cockroachoss s 101947
root 6083 45.4 6.8 4566268 4559440 3 R+ Tue23 1274:55.81 ./cockroachoss s 102153
root 6083 45.2 6.8 4566268 4559440 3 R+ Tue23 1278:20.63 ./cockroachoss s 102230
root 6083 45.0 6.8 4566268 4559440 3 R+ Tue23 1295:14.50 ./cockroachoss s 103065
root 6083 44.4 6.8 4566268 4559440 3 R+ Tue23 1252:46.70 ./cockroachoss s 101941
root 6083 44.2 6.8 4566268 4559440 3 R+ Tue23 1222:46.47 ./cockroachoss s 100497
root 6083 44.2 6.8 4566268 4559440 3 R+ Tue23 1238:23.98 ./cockroachoss s 100795
root 6083 43.4 6.8 4566268 4559440 3 R+ Tue23 1269:11.89 ./cockroachoss s 101696
root 6083 0.9 6.8 4566268 4559440 3 S+ Tue23 60:10.45 ./cockroachoss s 100193
root 6083 0.9 6.8 4566268 4559440 3 S+ Tue23 10:39.31 ./cockroachoss s 101023
root 6083 0.3 6.8 4566268 4559440 3 S+ Tue23 34:19.91 ./cockroachoss s 101687
root 6083 0.3 6.8 4566268 4559440 3 S+ Tue23 29:52.20 ./cockroachoss s 103187
root 6083 0.1 6.8 4566268 4559440 3 S+ Tue23 4:27.20 ./cockroachoss s 101110
root 6083 0.1 6.8 4566268 4559440 3 S+ Tue23 32:45.08 ./cockroachoss s 103150
root 6083 0.1 6.8 4566268 4559440 3 S+ Tue23 4:19.15 ./cockroachoss s 103494
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 15:51.97 ./cockroachoss s 100192
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 27:59.39 ./cockroachoss s 100256
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 13:09.40 ./cockroachoss s 100257
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 16:33.47 ./cockroachoss s 100266
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 9:10.63 ./cockroachoss s 100267
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 26:19.82 ./cockroachoss s 100277
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 33:19.25 ./cockroachoss s 100281
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 29:55.92 ./cockroachoss s 100344
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 23:35.83 ./cockroachoss s 100347
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 31:03.11 ./cockroachoss s 100348
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 27:35.74 ./cockroachoss s 100359
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 28:26.17 ./cockroachoss s 100373
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 0:00.00 ./cockroachoss s 100447
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 35:39.46 ./cockroachoss s 100457
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 0:00.00 ./cockroachoss s 100475
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 31:17.26 ./cockroachoss s 100484
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 27:23.20 ./cockroachoss s 100788
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 6:55.61 ./cockroachoss s 100848
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 11:45.63 ./cockroachoss s 100866
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 31:46.03 ./cockroachoss s 100905
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 33:43.84 ./cockroachoss s 100938
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 13:53.34 ./cockroachoss s 100966
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 6:36.49 ./cockroachoss s 100980
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 27:32.97 ./cockroachoss s 101004
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 19:21.45 ./cockroachoss s 101053
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 17:13.80 ./cockroachoss s 101104
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 21:39.02 ./cockroachoss s 101105
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 31:31.31 ./cockroachoss s 101109
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 20:35.60 ./cockroachoss s 101119
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 0:00.00 ./cockroachoss s 101608
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 33:23.01 ./cockroachoss s 101714
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 30:45.23 ./cockroachoss s 101771
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 16:41.64 ./cockroachoss s 101847
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 33:20.27 ./cockroachoss s 101849
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 31:49.38 ./cockroachoss s 101854
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 29:28.74 ./cockroachoss s 101866
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 35:34.44 ./cockroachoss s 101885
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 34:36.50 ./cockroachoss s 101908
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 33:47.96 ./cockroachoss s 101919
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 33:20.33 ./cockroachoss s 101942
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 40:52.25 ./cockroachoss s 101946
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 53:11.35 ./cockroachoss s 101954
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 39:31.52 ./cockroachoss s 101956
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 32:54.54 ./cockroachoss s 101959
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 9:18.72 ./cockroachoss s 101974
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 0:00.00 ./cockroachoss s 101975
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 0:00.00 ./cockroachoss s 101976
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 29:48.60 ./cockroachoss s 101978
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 28:07.16 ./cockroachoss s 101990
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 31:12.47 ./cockroachoss s 102046
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 29:48.31 ./cockroachoss s 102053
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 28:34.32 ./cockroachoss s 102058
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 34:47.11 ./cockroachoss s 102067
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 23:11.63 ./cockroachoss s 102225
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 33:48.70 ./cockroachoss s 102229
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 32:05.03 ./cockroachoss s 102236
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 27:38.78 ./cockroachoss s 103130
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 0:43.00 ./cockroachoss s 103312
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 0:00.01 ./cockroachoss s 103315
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 4:12.06 ./cockroachoss s 103321
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 0:00.00 ./cockroachoss s 103322
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 0:04.46 ./cockroachoss s 103323
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 1:10.16 ./cockroachoss s 103324
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 1:19.79 ./cockroachoss s 103326
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 0:00.00 ./cockroachoss s 103330
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 0:00.00 ./cockroachoss s 103331
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 5:11.97 ./cockroachoss s 103435
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 5:46.36 ./cockroachoss s 103440
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 0:08.23 ./cockroachoss s 103489
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 4:12.99 ./cockroachoss s 103496
root 6083 0.0 6.8 4566268 4559440 3 I+ Tue23 0:18.68 ./cockroachoss s 103497
root 6083 0.0 6.8 4566268 4559440 3 S+ Tue23 4:18.95 ./cockroachoss s 103498
Picking one with the most burned CPU credits:
77 LWP 101101 0x00000008069bc29d in memcmp () from /lib/libc.so.7
Printing backtrace for that thread:
(gdb) thread 77
[Switching to thread 77 (LWP 101101)]
#0 0x00000008069bc29d in memcmp () from /lib/libc.so.7
(gdb) bt
#0 0x00000008069bc29d in memcmp () from /lib/libc.so.7
#1 0x000000000264c303 in compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/../rocksdb/include/rocksdb/slice.h:241
#2 Compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/comparator.cc:29
#3 0x00000000027f0b0d in CompareKeySeq () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/dbformat.h:662
#4 operator() () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/memtable.cc:237
#5 0x000000000288c994 in FindGreaterOrEqual () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/memtable/inlineskiplist.h:475
#6 Seek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/memtable/inlineskiplist.h:392
#7 0x00000000027f8add in Seek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/memtable.cc:324
#8 0x0000000002951ba4 in Seek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/iterator_wrapper.h:68
#9 Seek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/merging_iterator.cc:137
#10 0x0000000002793379 in Seek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/db_iter.cc:1291
#11 0x000000000269cf47 in iterSeek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:539
#12 backwardLatestVersion () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:392
#13 0x000000000269dbf3 in prevKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:407
#14 0x000000000269b608 in scan () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:136
#15 0x0000000002696433 in MVCCScan () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.cc:294
#16 0x000000000254af3d in _cgo_46b63153f027_Cfunc_MVCCScan (v=0xc010dc2fe8) at cgo-gcc-prolog:1348
#17 0x000000000060de60 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:635
#18 0x00000000005e71c1 in runtime.park_m (gp=0x60c75f <runtime.morestack+143>) at /usr/local/go/src/runtime/proc.go:2605
#19 0x000000c00aae7980 in ?? ()
#20 0x000000000060c75f in runtime.morestack () at /usr/local/go/src/runtime/asm_amd64.s:429
#21 0x0000000807331fc0 in ?? ()
#22 0x0000000000200000 in ?? ()
#23 0x0000000000000000 in ?? ()
Taking a look at another:
(gdb) thread 58
[Switching to thread 58 (LWP 100355)]
#0 0x00000008069bc298 in memcmp () from /lib/libc.so.7
(gdb) bt
#0 0x00000008069bc298 in memcmp () from /lib/libc.so.7
#1 0x000000000264c303 in compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/../rocksdb/include/rocksdb/slice.h:241
#2 Compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/comparator.cc:29
#3 0x00000000027a0b8c in Compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/dbformat.h:642
#4 0x0000000002af7b6d in BinarySeek<rocksdb::DecodeKey> () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/block.cc:624
#5 0x0000000002af1bcf in SeekForPrev () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/block.cc:385
#6 0x000000000291e181 in rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::SeekForPrev(rocksdb::Slice const&) () at /usr/include/c++/v1/new:234
#7 0x000000000283b8c3 in rocksdb::(anonymous namespace)::LevelIterator::SeekForPrev(rocksdb::Slice const&) ()
#8 0x0000000002952886 in SeekForPrev () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/iterator_wrapper.h:71
#9 Prev () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/merging_iterator.cc:231
#10 0x00000000027919d2 in FindUserKeyBeforeSavedKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/db_iter.cc:1225
#11 0x0000000002791479 in ReverseToBackward () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/db_iter.cc:823
#12 0x000000000279122d in Prev () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/db_iter.cc:741
#13 0x000000000269d123 in iterPeekPrev () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:612
#14 0x000000000269da8c in prevKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:403
#15 0x000000000269b608 in scan () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:136
#16 0x0000000002696433 in MVCCScan () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.cc:294
#17 0x000000000254af3d in _cgo_46b63153f027_Cfunc_MVCCScan (v=0xc003f6afe8) at cgo-gcc-prolog:1348
#18 0x000000000060de60 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:635
#19 0x00000000005e71c1 in runtime.park_m (gp=0x60c75f <runtime.morestack+143>) at /usr/local/go/src/runtime/proc.go:2605
#20 0x000000c006271800 in ?? ()
#21 0x000000000060c75f in runtime.morestack () at /usr/local/go/src/runtime/asm_amd64.s:429
#22 0x0000000807331fc0 in ?? ()
#23 0x0000000000200000 in ?? ()
#24 0x0000000000000000 in ?? ()
And another:
(gdb) thread 22
[Switching to thread 22 (LWP 101916)]
#0 0x00000008069bc27b in memcmp () from /lib/libc.so.7
(gdb) bt
#0 0x00000008069bc27b in memcmp () from /lib/libc.so.7
#1 0x000000000264c303 in compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/../rocksdb/include/rocksdb/slice.h:241
#2 Compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/comparator.cc:29
#3 0x00000000027a0b8c in Compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/dbformat.h:642
#4 0x0000000002af178e in Compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/block.h:540
#5 Seek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/block.cc:372
#6 0x000000000291deba in rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::Seek(rocksdb::Slice const&) ()
#7 0x000000000283b6ee in rocksdb::(anonymous namespace)::LevelIterator::Seek(rocksdb::Slice const&) ()
#8 0x0000000002951ba4 in Seek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/iterator_wrapper.h:68
#9 Seek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/merging_iterator.cc:137
#10 0x0000000002793379 in Seek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/db_iter.cc:1291
#11 0x000000000269d155 in iterPeekPrev () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:620
#12 0x000000000269da8c in prevKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:403
#13 0x000000000269b608 in scan () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:136
#14 0x0000000002696433 in MVCCScan () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.cc:294
#15 0x000000000254af3d in _cgo_46b63153f027_Cfunc_MVCCScan (v=0xc00e6b8fe8) at cgo-gcc-prolog:1348
#16 0x000000000060de60 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:635
#17 0x0000000005d2ab80 in ?? ()
#18 0x0000000000000001 in ?? ()
#19 0x0000000805a50022 in ?? ()
#20 0x000000080d853b10 in ?? ()
#21 0x000000c00e6b9c88 in ?? ()
#22 0x00000000000030d8 in ?? ()
#23 0x000000c008756000 in ?? ()
#24 0x00000000005e3c00 in ?? () at /usr/local/go/src/runtime/proc.go:1082
#25 0x00000008072b1e40 in ?? ()
#26 0x0000000000200000 in ?? ()
#27 0x0000000000000000 in ?? ()
bt full for this thread:
(gdb) bt full
#0 0x00000008069bc27b in memcmp () from /lib/libc.so.7
No symbol table info available.
#1 0x000000000264c303 in compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/../rocksdb/include/rocksdb/slice.h:241
No locals.
#2 Compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/comparator.cc:29
No locals.
#3 0x00000000027a0b8c in Compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/dbformat.h:642
No locals.
#4 0x0000000002af178e in Compare () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/block.h:540
No locals.
#5 Seek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/block.cc:372
No locals.
#6 0x000000000291deba in rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::Seek(rocksdb::Slice const&) ()
No symbol table info available.
#7 0x000000000283b6ee in rocksdb::(anonymous namespace)::LevelIterator::Seek(rocksdb::Slice const&) ()
No symbol table info available.
#8 0x0000000002951ba4 in Seek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/iterator_wrapper.h:68
No locals.
#9 Seek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/merging_iterator.cc:137
No locals.
#10 0x0000000002793379 in Seek () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/db_iter.cc:1291
No locals.
#11 0x000000000269d155 in iterPeekPrev () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:620
No locals.
#12 0x000000000269da8c in prevKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:403
No locals.
#13 0x000000000269b608 in scan () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:136
No locals.
#14 0x0000000002696433 in MVCCScan () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.cc:294
No locals.
#15 0x000000000254af3d in _cgo_46b63153f027_Cfunc_MVCCScan (v=0xc00e6b8fe8) at cgo-gcc-prolog:1348
_cgo_a = 0xc00e6b8fe8
_cgo_stktop = 0xc00e6bc000 "\022\004\b"
_cgo_r = <optimized out>
#16 0x000000000060de60 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:635
No locals.
#17 0x0000000005d2ab80 in ?? ()
No symbol table info available.
#18 0x0000000000000001 in ?? ()
No symbol table info available.
#19 0x0000000805a50022 in ?? ()
No symbol table info available.
#20 0x000000080d853b10 in ?? ()
No symbol table info available.
#21 0x000000c00e6b9c88 in ?? ()
No symbol table info available.
#22 0x00000000000030d8 in ?? ()
No symbol table info available.
#23 0x000000c008756000 in ?? ()
No symbol table info available.
#24 0x00000000005e3c00 in ?? () at /usr/local/go/src/runtime/proc.go:1082
runtime.algarray = {{hash = {void (void *, uintptr, uintptr *)} 0x590d020 <runtime.algarray>, equal = {void (void *, void *, bool *)} 0x590d020 <runtime.algarray>}, {hash = {void (void *, uintptr,
uintptr *)} 0x590d020 <runtime.algarray>, equal = {void (void *, void *, bool *)} 0x590d020 <runtime.algarray>}, {hash = {void (void *, uintptr, uintptr *)} 0x590d020 <runtime.algarray>,
equal = {void (void *, void *, bool *)} 0x590d020 <runtime.algarray>}, {hash = {void (void *, uintptr, uintptr *)} 0x590d020 <runtime.algarray>, equal = {void (void *, void *,
bool *)} 0x590d020 <runtime.algarray>}, {hash = {void (void *, uintptr, uintptr *)} 0x590d020 <runtime.algarray>, equal = {void (void *, void *, bool *)} 0x590d020 <runtime.algarray>}, {
hash = {void (void *, uintptr, uintptr *)} 0x590d020 <runtime.algarray>, equal = {void (void *, void *, bool *)} 0x590d020 <runtime.algarray>}, {hash = {void (void *, uintptr,
uintptr *)} 0x590d020 <runtime.algarray>, equal = {void (void *, void *, bool *)} 0x590d020 <runtime.algarray>}, {hash = {void (void *, uintptr, uintptr *)} 0x590d020 <runtime.algarray>,
equal = {void (void *, void *, bool *)} 0x590d020 <runtime.algarray>}, {hash = {void (void *, uintptr, uintptr *)} 0x590d020 <runtime.algarray>, equal = {void (void *, void *,
bool *)} 0x590d020 <runtime.algarray>}, {hash = {void (void *, uintptr, uintptr *)} 0x590d020 <runtime.algarray>, equal = {void (void *, void *, bool *)} 0x590d020 <runtime.algarray>}, {
hash = {void (void *, uintptr, uintptr *)} 0x590d020 <runtime.algarray>, equal = {void (void *, void *, bool *)} 0x590d020 <runtime.algarray>}, {hash = {void (void *, uintptr,
uintptr *)} 0x590d020 <runtime.algarray>, equal = {void (void *, void *, bool *)} 0x590d020 <runtime.algarray>}, {hash = {void (void *, uintptr, uintptr *)} 0x590d020 <runtime.algarray>,
equal = {void (void *, void *, bool *)} 0x590d020 <runtime.algarray>}, {hash = {void (void *, uintptr, uintptr *)} 0x590d020 <runtime.algarray>, equal = {void (void *, void *,
bool *)} 0x590d020 <runtime.algarray>}}
runtime.hashkey = {0, 0, 0, 0}
runtime.useAeshash = true
runtime.writeBarrier = {enabled = false, pad = "\000\000", needed = false, cgo = false, alignme = 0}
runtime.aeskeysched = "u\267\245\320\345+\020\236\336z\003J\215\006\354\305RF\213m#\272\np\035\062 uT\247\r\273\205\256\366\230\035{\235\260\343#(HH\030\263\373\304\001\230\222\273<\034\212\373\026\347\365Xe\230\345\216)\026\303y\303\341\223\321\206P\367A\262\321\360l\204\277\233\036\363fJ\303\071\315\364\366>\323\262\260\232V\274\233\200\262=\277\300\375\333G\252\266a.o9E7\204\377\031ZL\023\214\025+\202+"
runtime.inForkedChild = false
runtime.framepointer_enabled = true
runtime.main_init_done = 0xc0000e0000
runtime.extraMWaiters = 0
runtime.debug = {allocfreetrace = 0, cgocheck = 1, clobberfree = 0, efence = 0, gccheckmark = 0, gcpacertrace = 0, gcshrinkstackoff = 0, gcstoptheworld = 0, gctrace = 0, invalidptr = 1,
madvdontneed = 0, sbrk = 0, scavenge = 0, scheddetail = 0, schedtrace = 0, tracebackancestors = 0}
runtime.modulesSlice = 0xc00000e000
runtime.mheap_ = {lock = {key = 0}, free = {treap = 0x809af7830}, scav = {treap = 0x817d962e0}, sweepgen = 35994, sweepdone = 1, sweepers = 0, allspans = {array = 0x819ee1000, len = 33977,
cap = 41472}, sweepSpans = {{spineLock = {key = 0}, spine = 0x80596b040, spineLen = 64, spineCap = 256, index = 0}, {spineLock = {key = 0}, spine = 0x8059ce640, spineLen = 65, spineCap = 256,
index = 21113}}, _ = 0, pagesInUse = 44660, pagesSwept = 52287, pagesSweptBasis = 0, sweepHeapLiveBasis = 196609368, sweepPagesPerByte = 0.00028152107397131916, reclaimIndex = 0,
reclaimCredit = 14017, scavengeCredit = 40960, largealloc = 250523959296, nlargealloc = 3440282, largefree = 250355179520, nlargefree = 3439759, nsmallfree = {0, 771684951, 3067953172, 3826969425,
3503638801, 1700274285, 1780443325, 1376528289, 525485688, 581108235, 422262060, 409925249, 118503899, 222632427, 187578140, 133935411, 249920358, 389058279, 307096356, 253790667, 225457374,
50580614, 70727476, 378891776, 67882520, 300555703, 80510686, 200383583, 2752230, 39126152, 53205440, 25364656, 24534122, 30847434, 7956912, 268257910, 17998419, 6808415, 9791638, 2315513, 8718479,
232033, 3432983, 7399091, 1982392, 3403750, 2689618, 277747, 81374, 194023, 2535064, 3823607, 85230, 576870, 111763, 204327, 275250, 148343, 884571, 313846, 14126, 221249, 46543, 1445917, 43280,
4291, 602562}, arenas = {0x807800000}, heapArenaAlloc = {next = 0, mapped = 0, end = 0}, arenaHints = 0x80595ebf0, arena = {next = 0, mapped = 0, end = 0}, allArenas = {array = 0x805962008,
len = 10, cap = 512}, sweepArenas = {array = 0x805962008, len = 10, cap = 512}, central = {{mcentral = {lock = {key = 0}, spanclass = 0 '\000', nonempty = {first = 0x0, last = 0x0}, empty = {
first = 0x0, last = 0x0}, nmalloc = 0}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 1 '\001', nonempty = {first = 0x0, last = 0x0}, empty = {first = 0x0,
last = 0x0}, nmalloc = 0}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 2 '\002', nonempty = {first = 0x8059d5c18, last = 0x809a595f0}, empty = {
first = 0x810883548, last = 0x809ac1d60}, nmalloc = 771701098}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 3 '\003', nonempty = {first = 0x0,
last = 0x0}, empty = {first = 0x0, last = 0x0}, nmalloc = 0}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 4 '\004', nonempty = {first = 0x819ea9368,
last = 0x815c7cf90}, empty = {first = 0x8059c6778, last = 0x809b19340}, nmalloc = 1451488753}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 5 '\005',
nonempty = {first = 0x815ca4468, last = 0x809ad4350}, empty = {first = 0x80d9bac20, last = 0x805ab3440}, nmalloc = 1616621674}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 6 '\006', nonempty = {first = 0x805a3acf8, last = 0x805a33470}, empty = {first = 0x8059ce160, last = 0x815c6a6d0}, nmalloc = 3066655340},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 7 '\a', nonempty = {first = 0x8108237a8, last = 0x809a65c08}, empty = {first = 0x8059c5268, last = 0x817d7faf8},
nmalloc = 760408392}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 8 '\b', nonempty = {first = 0x809a56d70, last = 0x809ba7620}, empty = {
first = 0x805966f08, last = 0x809a597a0}, nmalloc = 3111885297}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 9 '\t', nonempty = {first = 0x809adc888,
last = 0x815c437b0}, empty = {first = 0x815cd7a30, last = 0x815d47dc0}, nmalloc = 391848908}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 10 '\n',
nonempty = {first = 0x809a52568, last = 0x80d921e20}, empty = {first = 0x805a13128, last = 0x819eb9eb0}, nmalloc = 1256321750}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 11 '\v', nonempty = {first = 0x8059ca9b0, last = 0x817da9820}, empty = {first = 0x8059cb2b0, last = 0x809b26638}, nmalloc = 443988194},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 12 '\f', nonempty = {first = 0x819ea2c28, last = 0x810827448}, empty = {first = 0x8059c69b8, last = 0x817cf4300},
nmalloc = 1349120807}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 13 '\r', nonempty = {first = 0x809ab2c38, last = 0x809a13848}, empty = {
first = 0x817cb9738, last = 0x815d14608}, nmalloc = 431349402}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 14 '\016', nonempty = {first = 0x80d8c9350,
last = 0x819e1a140}, empty = {first = 0x8059c7bb8, last = 0x817d48148}, nmalloc = 1031948540}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 15 '\017',
nonempty = {first = 0x80d9c18e8, last = 0x805a6d608}, empty = {first = 0x805a15768, last = 0x809ba7bc0}, nmalloc = 344606787}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 16 '\020', nonempty = {first = 0x805a16e78, last = 0x815d41e60}, empty = {first = 0x805a131b8, last = 0x805a95cc8}, nmalloc = 449115919},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 17 '\021', nonempty = {first = 0x819f76e18, last = 0x809a38728}, empty = {first = 0x810870ac0,
last = 0x817d7f798}, nmalloc = 76410810}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 18 '\022', nonempty = {first = 0x8059d4f28, last = 0x805a365f0},
empty = {first = 0x8059c5bf8, last = 0x80d9a5848}, nmalloc = 360200837}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 19 '\023', nonempty = {
first = 0x80d859f98, last = 0x817cacea8}, empty = {first = 0x815c94e78, last = 0x809adefb0}, nmalloc = 220918063}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0},
spanclass = 20 '\024', nonempty = {first = 0x805a7b690, last = 0x805aa9340}, empty = {first = 0x8059c2ef8, last = 0x809af2770}, nmalloc = 305252926}, pad = "\000\000\000\000\000\000\000"}, {
mcentral = {lock = {key = 0}, spanclass = 21 '\025', nonempty = {first = 0x81a063e88, last = 0x80d913bf0}, empty = {first = 0x817d7a528, last = 0x815d86278}, nmalloc = 117015756},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 22 '\026', nonempty = {first = 0x80d8d33d0, last = 0x80d8b58e0}, empty = {first = 0x8059d3f88,
last = 0x817d7a888}, nmalloc = 346453331}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 23 '\027', nonempty = {first = 0x819f81d08, last = 0x805a620e8},
empty = {first = 0x0, last = 0x0}, nmalloc = 63477400}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 24 '\030', nonempty = {first = 0x80d912180,
last = 0x815c46a50}, empty = {first = 0x8059c2cb8, last = 0x805a14b98}, nmalloc = 65367326}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 25 '\031',
nonempty = {first = 0x8059ed228, last = 0x805a98ae8}, empty = {first = 0x817c67000, last = 0x80d938130}, nmalloc = 53138540}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 26 '\032', nonempty = {first = 0x805a71878, last = 0x805aacc78}, empty = {first = 0x8059d82e8, last = 0x819f7e108}, nmalloc = 39844077},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 27 '\033', nonempty = {first = 0x819f43018, last = 0x809b75a60}, empty = {first = 0x817d819e8,
last = 0x80d8b5b20}, nmalloc = 182791020}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 28 '\034', nonempty = {first = 0x8059e78d8, last = 0x809b763f0},
empty = {first = 0x805964958, last = 0x809aea530}, nmalloc = 107428605}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 29 '\035', nonempty = {
first = 0x809b79450, last = 0x80d8df090}, empty = {first = 0x805a29e88, last = 0x819e68360}, nmalloc = 80156800}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0},
spanclass = 30 '\036', nonempty = {first = 0x815c43960, last = 0x8059cb7c0}, empty = {first = 0x809b158d0, last = 0x809b14250}, nmalloc = 43046379}, pad = "\000\000\000\000\000\000\000"}, {
mcentral = {lock = {key = 0}, spanclass = 31 '\037', nonempty = {first = 0x809ad4500, last = 0x809a9ba40}, empty = {first = 0x810822488, last = 0x80d876450}, nmalloc = 90903106},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 32 ' ', nonempty = {first = 0x815c6f880, last = 0x817cc6630}, empty = {first = 0x809b94bc0, last = 0x809b3cbc0},
nmalloc = 169152182}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 33 '!', nonempty = {first = 0x805a82928, last = 0x80d847b80}, empty = {first = 0x0,
last = 0x0}, nmalloc = 80778295}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 34 '"', nonempty = {first = 0x809b6d118, last = 0x80d917410}, empty = {
first = 0x805a5af20, last = 0x817d32600}, nmalloc = 332433328}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 35 '#', nonempty = {first = 0x810833e20,
last = 0x81084e0b0}, empty = {first = 0x8059cd230, last = 0x809af1060}, nmalloc = 56629306}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 36 '$',
nonempty = {first = 0x80d8ebd28, last = 0x80d8829c0}, empty = {first = 0x819f7c848, last = 0x817c35cf0}, nmalloc = 204591840}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 37 '%', nonempty = {first = 0x815d80918, last = 0x810993348}, empty = {first = 0x8059d0a18, last = 0x80d8c8a50}, nmalloc = 102519499},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 38 '&', nonempty = {first = 0x819e194e0, last = 0x81a066168}, empty = {first = 0x8059c6538, last = 0x809b3b930},
nmalloc = 165709868}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 39 '\'', nonempty = {first = 0x80d952328, last = 0x809ac0800}, empty = {
first = 0x809b13290, last = 0x815c3e420}, nmalloc = 88084998}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 40 '(', nonempty = {first = 0x8059f3248,
last = 0x809b14910}, empty = {first = 0x815c5aba8, last = 0x817c71320}, nmalloc = 207288252}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 41 ')',
nonempty = {first = 0x80d89d2d0, last = 0x815c75e10}, empty = {first = 0x81084e1d0, last = 0x805a80eb8}, nmalloc = 18172565}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 42 '*', nonempty = {first = 0x817c2ebd0, last = 0x810829828}, empty = {first = 0x817cf33d0, last = 0x80d960090}, nmalloc = 37865350},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 43 '+', nonempty = {first = 0x8108726c0, last = 0x80d8eff80}, empty = {first = 0x8059c9118, last = 0x809a539a8},
nmalloc = 12724022}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 44 ',', nonempty = {first = 0x80d91baf8, last = 0x805a67568}, empty = {
first = 0x805a5a080, last = 0x815d26880}, nmalloc = 40246405}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 45 '-', nonempty = {first = 0x817cc5dc0,
last = 0x80d96e6d8}, empty = {first = 0x817cecd70, last = 0x8059d4e08}, nmalloc = 30482109}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 46 '.',
nonempty = {first = 0x817ca97b0, last = 0x80d9667c0}, empty = {first = 0x8059649e8, last = 0x81a0a5c38}, nmalloc = 375719979}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 47 '/', nonempty = {first = 0x817ce4768, last = 0x809adf430}, empty = {first = 0x815c70f00, last = 0x819e66470}, nmalloc = 3180807},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 48 '0', nonempty = {first = 0x817cf1600, last = 0x81082dae8}, empty = {first = 0x80d9cca38, last = 0x809a6d448},
nmalloc = 52450889}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 49 '1', nonempty = {first = 0x815c45730, last = 0x80d9c53f0}, empty = {
first = 0x809a97eb8, last = 0x817c7d510}, nmalloc = 15432201}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 50 '2', nonempty = {first = 0x815d58728,
last = 0x809ad9798}, empty = {first = 0x81087fa80, last = 0x80d8ca040}, nmalloc = 47829683}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 51 '3',
nonempty = {first = 0x8108f5680, last = 0x819e67ee0}, empty = {first = 0x81a0a4fd8, last = 0x819ea8698}, nmalloc = 252733016}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 52 '4', nonempty = {first = 0x80d856420, last = 0x809a6d688}, empty = {first = 0x8059d1828, last = 0x81a02bcd8}, nmalloc = 72687532},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 53 '5', nonempty = {first = 0x817d79568, last = 0x815d11ab8}, empty = {first = 0x8059c4e78, last = 0x819f32008},
nmalloc = 7824352}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 54 '6', nonempty = {first = 0x81087e0a0, last = 0x817cc6c60}, empty = {first = 0x805a56500,
last = 0x805a6a758}, nmalloc = 196998987}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 55 '7', nonempty = {first = 0x8059ed738, last = 0x815cb4388},
empty = {first = 0x809aa6a88, last = 0x815c787b0}, nmalloc = 3392552}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 56 '8', nonempty = {first = 0x80d813a50,
last = 0x809a53d08}, empty = {first = 0x805965918, last = 0x805a5c510}, nmalloc = 567041}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 57 '9',
nonempty = {first = 0x80d913ad0, last = 0x809b92bb0}, empty = {first = 0x809a9bf50, last = 0x80d8ca0d0}, nmalloc = 2185751}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 58 ':', nonempty = {first = 0x809abf450, last = 0x809b6fa98}, empty = {first = 0x805a13518, last = 0x8109bb5c0}, nmalloc = 37937578},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 59 ';', nonempty = {first = 0x809a92730, last = 0x80d919e48}, empty = {first = 0x80d84b510, last = 0x80d8b39f0},
nmalloc = 1189382}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 60 '<', nonempty = {first = 0x8059ee038, last = 0x809aa5918}, empty = {first = 0x8059cc9c0,
last = 0x815cb1688}, nmalloc = 51258225}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 61 '=', nonempty = {first = 0x805a6b838, last = 0x81087af88},
empty = {first = 0x8059c77c8, last = 0x809a3fd08}, nmalloc = 1952225}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 62 '>', nonempty = {first = 0x80d8f0c70,
last = 0x817c317b0}, empty = {first = 0x80d813030, last = 0x8109bc8e0}, nmalloc = 23490773}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 63 '?',
nonempty = {first = 0x81097a130, last = 0x819fb7ae0}, empty = {first = 0x805965f48, last = 0x81097e780}, nmalloc = 1877794}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 64 '@', nonempty = {first = 0x809ae38c0, last = 0x819e624d8}, empty = {first = 0x80d986fa0, last = 0x809b6c4b8}, nmalloc = 23182414},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 65 'A', nonempty = {first = 0x809a84ac8, last = 0x80d88c108}, empty = {first = 0x8059cb8e0, last = 0x809a45148},
nmalloc = 1352576}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 66 'B', nonempty = {first = 0x80d814620, last = 0x809a5cb88}, empty = {first = 0x805965408,
last = 0x809b12ea0}, nmalloc = 29991445}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 67 'C', nonempty = {first = 0x805a6bdd8, last = 0x80d8badb8},
empty = {first = 0x805a278f8, last = 0x810822a28}, nmalloc = 856698}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 68 'D', nonempty = {first = 0x810868830,
last = 0x809b19610}, empty = {first = 0x809b17720, last = 0x8059cbf10}, nmalloc = 6781246}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 69 'E',
nonempty = {first = 0x805a7bb10, last = 0x815d277b0}, empty = {first = 0x8059c8c08, last = 0x81086cd00}, nmalloc = 1176512}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 70 'F', nonempty = {first = 0x815cc3e98, last = 0x810815ca8}, empty = {first = 0x809b52be0, last = 0x809b53ba0}, nmalloc = 19878681},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 71 'G', nonempty = {first = 0x809ab3388, last = 0x810816ea8}, empty = {first = 0x80d958998, last = 0x805aa8f50},
nmalloc = 248382900}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 72 'H', nonempty = {first = 0x80d8786a0, last = 0x819fcf220}, empty = {
first = 0x805a149e8, last = 0x8059647a8}, nmalloc = 16994157}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 73 'I', nonempty = {first = 0x817cbda48,
last = 0x809a95d30}, empty = {first = 0x8059c4ba8, last = 0x80d871d68}, nmalloc = 1005232}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 74 'J',
nonempty = {first = 0x817ccec60, last = 0x809ab9a90}, empty = {first = 0x805a132d8, last = 0x80d993060}, nmalloc = 3908949}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 75 'K', nonempty = {first = 0x80d93c840, last = 0x819fbcb28}, empty = {first = 0x8059c7618, last = 0x80d8e8758}, nmalloc = 2900419},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 76 'L', nonempty = {first = 0x819eac2a8, last = 0x819e67280}, empty = {first = 0x81097b690, last = 0x80d9868e0},
nmalloc = 8782511}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 77 'M', nonempty = {first = 0x809a4a0b8, last = 0x817d813b8}, empty = {first = 0x815d3f700,
last = 0x805a649f8}, nmalloc = 1009698}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 78 'N', nonempty = {first = 0x805a3c0a8, last = 0x819e642a8},
empty = {first = 0x8059c3768, last = 0x8059f2798}, nmalloc = 1312328}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 79 'O', nonempty = {first = 0x80d93b5b0,
last = 0x805a2ba18}, empty = {first = 0x8059c4d58, last = 0x817c36170}, nmalloc = 1003805}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 80 'P',
nonempty = {first = 0x815d83a98, last = 0x815d293d0}, empty = {first = 0x8059efa18, last = 0x819e64c38}, nmalloc = 7951759}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 81 'Q', nonempty = {first = 0x809ba7860, last = 0x809a4cc48}, empty = {first = 0x809ae7e60, last = 0x810998378}, nmalloc = 767413},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 82 'R', nonempty = {first = 0x80d8c09e0, last = 0x819e11638}, empty = {first = 0x805964ef8, last = 0x805964ef8},
nmalloc = 153586}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 83 'S', nonempty = {first = 0x819f33cb8, last = 0x819eba180}, empty = {first = 0x817d6e1b0,
last = 0x80d946470}, nmalloc = 78555}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 84 'T', nonempty = {first = 0x809b67588, last = 0x817c4bed0}, empty = {
first = 0x8059d2a28, last = 0x805a3e148}, nmalloc = 2829048}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 85 'U', nonempty = {first = 0x80d9cb838,
last = 0x815c76c20}, empty = {first = 0x805a7e810, last = 0x81086fef0}, nmalloc = 604392}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 86 'V',
nonempty = {first = 0x80d9cbc28, last = 0x80d9cbc28}, empty = {first = 0x8059662a8, last = 0x80d812970}, nmalloc = 807000}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0},
spanclass = 87 'W', nonempty = {first = 0x817ce2a28, last = 0x80d8f0640}, empty = {first = 0x80d8b0a70, last = 0x80d96c638}, nmalloc = 6593408}, pad = "\000\000\000\000\000\000\000"}, {
mcentral = {lock = {key = 0}, spanclass = 88 'X', nonempty = {first = 0x816406560, last = 0x815c7cd50}, empty = {first = 0x8059c3888, last = 0x810882618}, nmalloc = 520622},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 89 'Y', nonempty = {first = 0x805a73598, last = 0x817cbe6a8}, empty = {first = 0x805964b08, last = 0x80d8c84b0},
nmalloc = 1462028}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 90 'Z', nonempty = {first = 0x805a8c0d0, last = 0x805a8c0d0}, empty = {first = 0x805a55e40,
last = 0x805a81968}, nmalloc = 2237352}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 91 '[', nonempty = {first = 0x809a61120, last = 0x809a61120},
empty = {first = 0x805a57310, last = 0x815c5bc88}, nmalloc = 1166771}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 92 '\\', nonempty = {
first = 0x809ba5070, last = 0x805a81ba8}, empty = {first = 0x805966218, last = 0x80d8f5478}, nmalloc = 2377370}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0},
spanclass = 93 ']', nonempty = {first = 0x819e1e3f0, last = 0x809ad2220}, empty = {first = 0x80d9654a0, last = 0x819fb8590}, nmalloc = 312297}, pad = "\000\000\000\000\000\000\000"}, {
mcentral = {lock = {key = 0}, spanclass = 94 '^', nonempty = {first = 0x815ca7c58, last = 0x809b77710}, empty = {first = 0x805a5ae00, last = 0x805a5ae00}, nmalloc = 109580},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 95 '_', nonempty = {first = 0x81097a370, last = 0x805a46178}, empty = {first = 0x0, last = 0x0},
nmalloc = 168182}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 96 '`', nonempty = {first = 0x8059d2758, last = 0x8059cd8f0}, empty = {first = 0x0,
last = 0x0}, nmalloc = 25731}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 97 'a', nonempty = {first = 0x809b76000, last = 0x809b76000}, empty = {
first = 0x0, last = 0x0}, nmalloc = 55657}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 98 'b', nonempty = {first = 0x80d925d00, last = 0x809ba61e0},
empty = {first = 0x0, last = 0x0}, nmalloc = 101252}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 99 'c', nonempty = {first = 0x0, last = 0x0}, empty = {
first = 0x0, last = 0x0}, nmalloc = 92780}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 100 'd', nonempty = {first = 0x0, last = 0x0}, empty = {
first = 0x8059d37a8, last = 0x817d81568}, nmalloc = 99657}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 101 'e', nonempty = {first = 0x0, last = 0x0},
empty = {first = 0x80d87f600, last = 0x817d74cb8}, nmalloc = 2435655}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 102 'f', nonempty = {
first = 0x81097c2f0, last = 0x809a9ca90}, empty = {first = 0x805963ab8, last = 0x817c71950}, nmalloc = 2462550}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0},
spanclass = 103 'g', nonempty = {first = 0x817c64b70, last = 0x817cc0ce8}, empty = {first = 0x8108db120, last = 0x817cc6fc0}, nmalloc = 1361361}, pad = "\000\000\000\000\000\000\000"}, {
mcentral = {lock = {key = 0}, spanclass = 104 'h', nonempty = {first = 0x0, last = 0x0}, empty = {first = 0x0, last = 0x0}, nmalloc = 32368}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {
lock = {key = 0}, spanclass = 105 'i', nonempty = {first = 0x815cc4ca8, last = 0x815cc4ca8}, empty = {first = 0x0, last = 0x0}, nmalloc = 52863}, pad = "\000\000\000\000\000\000\000"}, {
mcentral = {lock = {key = 0}, spanclass = 106 'j', nonempty = {first = 0x0, last = 0x0}, empty = {first = 0x0, last = 0x0}, nmalloc = 147262}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {
lock = {key = 0}, spanclass = 107 'k', nonempty = {first = 0x815c1c5d0, last = 0x815c1c5d0}, empty = {first = 0x80d987300, last = 0x817ca5068}, nmalloc = 429674},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 108 'l', nonempty = {first = 0x80d996010, last = 0x80d996010}, empty = {first = 0x805a5a740, last = 0x809aae3c8},
nmalloc = 67480}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 109 'm', nonempty = {first = 0x805ab47f0, last = 0x805ab47f0}, empty = {first = 0x0,
last = 0x0}, nmalloc = 44295}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 110 'n', nonempty = {first = 0x0, last = 0x0}, empty = {first = 0x8059caec0,
last = 0x815cd49d0}, nmalloc = 31113}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 111 'o', nonempty = {first = 0x809a52b08, last = 0x809a52b08},
empty = {first = 0x0, last = 0x0}, nmalloc = 173225}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 112 'p', nonempty = {first = 0x817ca3298,
last = 0x809ab2f98}, empty = {first = 0x8059d7328, last = 0x80d89d900}, nmalloc = 41881}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 113 'q',
nonempty = {first = 0x809b786d0, last = 0x805a7bde0}, empty = {first = 0x0, last = 0x0}, nmalloc = 233386}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0},
spanclass = 114 'r', nonempty = {first = 0x809a698c8, last = 0x809a698c8}, empty = {first = 0x8059cdd70, last = 0x80d87fd50}, nmalloc = 81153}, pad = "\000\000\000\000\000\000\000"}, {
mcentral = {lock = {key = 0}, spanclass = 115 's', nonempty = {first = 0x809b39ad0, last = 0x809b39ad0}, empty = {first = 0x0, last = 0x0}, nmalloc = 67206},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 116 't', nonempty = {first = 0x0, last = 0x0}, empty = {first = 0x8059cde00, last = 0x80d8e6fb8},
nmalloc = 33560}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 117 'u', nonempty = {first = 0x0, last = 0x0}, empty = {first = 0x810997a78,
last = 0x8108e8f00}, nmalloc = 851111}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 118 'v', nonempty = {first = 0x810975e50, last = 0x80d8262f0},
empty = {first = 0x815d1a178, last = 0x815d1a178}, nmalloc = 267327}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 119 'w', nonempty = {first = 0x809ad1f50,
last = 0x809ab03d8}, empty = {first = 0x0, last = 0x0}, nmalloc = 46615}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 120 'x', nonempty = {
first = 0x805a14958, last = 0x805a14958}, empty = {first = 0x805a14b08, last = 0x805a14b08}, nmalloc = 2824}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0},
spanclass = 121 'y', nonempty = {first = 0x809afdd78, last = 0x805aadde8}, empty = {first = 0x0, last = 0x0}, nmalloc = 11308}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {
key = 0}, spanclass = 122 'z', nonempty = {first = 0x819e67820, last = 0x819e67820}, empty = {first = 0x0, last = 0x0}, nmalloc = 13823}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {
lock = {key = 0}, spanclass = 123 '{', nonempty = {first = 0x817ca25a8, last = 0x817ca25a8}, empty = {first = 0x81086a900, last = 0x817d88cf8}, nmalloc = 207468},
pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 124 '|', nonempty = {first = 0x0, last = 0x0}, empty = {first = 0x8059c4338, last = 0x815d42130},
nmalloc = 43849}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 125 '}', nonempty = {first = 0x805ab2870, last = 0x805ab2870}, empty = {first = 0x805966068,
last = 0x805966068}, nmalloc = 2707}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 126 '~', nonempty = {first = 0x0, last = 0x0}, empty = {
first = 0x805a36050, last = 0x8059e8b68}, nmalloc = 33357}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 127 '\177', nonempty = {first = 0x0, last = 0x0},
empty = {first = 0x819ea4b18, last = 0x80d82c970}, nmalloc = 1412814}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0}, spanclass = 128 '\200', nonempty = {
first = 0x815c3f7b0, last = 0x80d9a88a8}, empty = {first = 0x8059d1318, last = 0x809ba1900}, nmalloc = 38784}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0},
spanclass = 129 '\201', nonempty = {first = 0x0, last = 0x0}, empty = {first = 0x0, last = 0x0}, nmalloc = 4505}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0},
spanclass = 130 '\202', nonempty = {first = 0x0, last = 0x0}, empty = {first = 0x0, last = 0x0}, nmalloc = 2877}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0},
spanclass = 131 '\203', nonempty = {first = 0x0, last = 0x0}, empty = {first = 0x0, last = 0x0}, nmalloc = 1414}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0},
spanclass = 132 '\204', nonempty = {first = 0x0, last = 0x0}, empty = {first = 0x0, last = 0x0}, nmalloc = 5631}, pad = "\000\000\000\000\000\000\000"}, {mcentral = {lock = {key = 0},
spanclass = 133 '\205', nonempty = {first = 0x0, last = 0x0}, empty = {first = 0x805a27628, last = 0x80d88b6e8}, nmalloc = 597033}, pad = "\000\000\000\000\000\000\000"}}, spanalloc = {
size = 144, first = {void (void *, void *)} 0x5d2ab80 <runtime.mheap_>, arg = 0x5d2ab80 <runtime.mheap_>, list = 0x815d40750, chunk = 34794401304, nchunk = 5296, inuse = 3765312,
stat = 0x5d623f8 <runtime.memstats+120>, zero = false}, cachealloc = {size = 1736, first = {void (void *, void *)} 0x5d2ab80 <runtime.mheap_>, arg = 0x0, list = 0x0, chunk = 34453534320,
nchunk = 12912, inuse = 34720, stat = 0x5d62408 <runtime.memstats+136>, zero = true}, treapalloc = {size = 48, first = {void (void *, void *)} 0x5d2ab80 <runtime.mheap_>, arg = 0x0,
list = 0x815d36d30, chunk = 34638486056, nchunk = 13504, inuse = 226320, stat = 0x5d62420 <runtime.memstats+160>, zero = true}, specialfinalizeralloc = {size = 48, first = {void (void *,
void *)} 0x5d2ab80 <runtime.mheap_>, arg = 0x0, list = 0x8059ba578, chunk = 34453827384, nchunk = 12496, inuse = 1536, stat = 0x5d62420 <runtime.memstats+160>, zero = true}, specialprofilealloc = {
size = 24, first = {void (void *, void *)} 0x5d2ab80 <runtime.mheap_>, arg = 0x0, list = 0x8059bf578, chunk = 34453855136, nchunk = 1336, inuse = 6768, stat = 0x5d62420 <runtime.memstats+160>,
zero = true}, speciallock = {key = 0}, arenaHintAlloc = {size = 24, first = {void (void *, void *)} 0x5d2ab80 <runtime.mheap_>, arg = 0x0, list = 0x0, chunk = 34453449736, nchunk = 13312,
inuse = 3072, stat = 0x5d62420 <runtime.memstats+160>, zero = true}, unused = 0x0}
runtime.blockprofilerate = 21974469
runtime.waitReasonStrings = {0x0 "", 0x335542d "GC assist marking", 0x333ef27 "IO wait", 0x3365327 "chan receive (nil chan)", 0x335d86e "chan send (nil chan)", 0x334a9c1 "dumping heap",
0x3358be2 "garbage collection", 0x3365a12 "garbage collection scan", 0x3344af0 "panicwait", 0x333e11e "select", 0x3356fcd "select (no cases)", 0x334e198 "GC assist wait", 0x334c0fa "GC sweep wait",
0x334a715 "chan receive", 0x334413c "chan send", 0x334f0a2 "finalizer wait", 0x33515af "force gc (idle)", 0x3346fcb "semacquire", 0x333c169 "sleep", 0x334fa7a "sync.Cond.Wait",
0x3363f20 "timer goroutine (idle)", 0x3363f62 "trace reader (blocked)", 0x3357451 "wait for GC cycle", 0x3352ae2 "GC worker (idle)"}
runtime.inf = inf
runtime.processorVersionInfo = 263921
runtime.isIntel = true
runtime.useAVXmemmove = true
runtime.cpuprof = {lock = {key = 0}, on = false, log = 0x0, extra = {0 <repeats 1000 times>}, numExtra = 0, lostExtra = 0}
runtime.prof = {signalLock = 0, hz = 0}
runtime.sched = {goidgen = 73794883, lastpoll = 0, lock = {key = 0}, midle = 825141010432, nmidle = 47, nmidlelocked = 1, mnext = 117, maxmcount = 10000, nmsys = 2, nmfreed = 0, ngsys = 47,
pidle = 824634238208, npidle = 20, nmspinning = 0, runq = {head = 0, tail = 0}, runqsize = 0, disable = {user = false, runnable = {head = 0, tail = 0}, n = 0}, gFree = {lock = {key = 0}, stack = {
head = 0}, noStack = {head = 824700649856}, n = 311}, sudoglock = {key = 0}, sudogcache = 0x0, deferlock = {key = 0}, deferpool = {0xc00384c1b0, 0xc0134c8640, 0xc01d086af0, 0xc00001f2c0,
0xc0126c25b0}, freem = 0x0, gcwaiting = 0, stopwait = 0, stopnote = {key = 0}, sysmonwait = 1, sysmonnote = {key = 0}, safePointFn = {void (runtime.p *)} 0x5d1b9e0 <runtime.sched>,
safePointWait = 0, safePointNote = {key = 0}, profilehz = 0, procresizetime = 243604571828901, totaltime = 4242864897248960}
runtime.gomaxprocs = 20
runtime.newprocs = 0
runtime.ncpu = 20
runtime.allm = 0xc00c39bc00
runtime.envs = {array = 0xc000050000, len = 27, cap = 27}
runtime.fastlog2Table = {0, 0.044394119358453499, 0.087462841250339429, 0.12928301694496647, 0.16992500144231248, 0.20945336562894989, 0.24792751344358555, 0.28540221886224837, 0.32192809488736229,
0.35755200461808367, 0.39231742277876036, 0.4262647547020979, 0.45943161863729731, 0.49185309632967478, 0.52356195605701283, 0.55458885167763738, 0.5849625007211563, 0.61470984411520824,
0.6438561897747247, 0.67242534197149562, 0.70043971814109218, 0.7279204545631992, 0.75488750216346856, 0.7813597135246596, 0.80735492205760417, 0.83289001416474173, 0.85798099512757209,
0.88264304936184124, 0.90689059560851848, 0.93073733756288624, 0.95419631038687525, 0.97727992349991644, 1}
runtime.allglen = 1523
runtime.allgs = {array = 0xc000a54000, len = 1523, cap = 1696}
runtime.firstmoduledata = {pclntable = {array = 0x3c79b60 <runtime.pclntab> "\373\377\377\377", len = 26255002, cap = 26255002}, ftab = {array = 0x3c79b70 <runtime.pclntab+16>, len = 74408,
cap = 74408}, filetab = {array = 0x5544220 <runtime.pclntab+25994944>, len = 3063, cap = 3063}, findfunctab = 62738720, minpc = 5970656, maxpc = 39096635, text = 5970656, etext = 39096635,
noptrdata = 93524224, enoptrdata = 97548384, data = 93218240, edata = 93522320, bss = 97554336, ebss = 97784272, noptrbss = 97903680, enoptrbss = 97925848, end = 97925848, gcdata = 61089024,
gcbss = 61086432, types = 46756736, etypes = 62900471, textsectmap = {array = 0x3a92c30 <runtime.textsectionmap>, len = 1, cap = 1}, typelinks = {array = 0x3c599a0 <runtime.typelink>, len = 22626,
cap = 22626}, itablinks = {array = 0x3c6fb28, len = 5125, cap = 5125}, ptab = {array = 0x0, len = 0, cap = 0}, pluginpath = 0x0 "", pkghashes = {array = 0x0, len = 0, cap = 0}, modulename = 0x0 "",
modulehashes = {array = 0x0, len = 0, cap = 0}, hasmain = 1 '\001', gcdatamask = {n = 38010,
bytedata = 0x80596c840 "\376", '\377' <repeats 91 times>, "\257VUUU\225UUVV\225\252\252\252\252\252\252UVUUUUUUU\245j\245\252V", 'U' <repeats 77 times>...}, gcbssmask = {n = 28742,
bytedata = 0x80596dad1 '\377' <repeats 164 times>, "\257", '\252' <repeats 22 times>, "\232V\252\252\252jYU\245\252\252\252\252"...}, typemap = 0x0, bad = false, next = 0x0}
runtime.memstats = {alloc = 285699288, total_alloc = 3305922919168, sys = 710695160, nlookup = 0, nmalloc = 23149589405, nfree = 23148956750, heap_alloc = 285699288, heap_sys = 659488768,
heap_idle = 293634048, heap_inuse = 365854720, heap_released = 222208000, heap_objects = 632655, stacks_inuse = 11599872, stacks_sys = 0, mspan_inuse = 3765600, mspan_sys = 4931584,
mcache_inuse = 34720, mcache_sys = 49152, buckhash_sys = 6881163, gc_sys = 23961600, other_sys = 3783021, next_gc = 393218736, last_gc_unix = 1562919451895485634, pause_total_ns = 7203903271,
pause_ns = {216877, 284590, 255523, 210529, 295025, 1541124, 118439, 209661, 136146, 176197, 153347, 149344, 53349039, 236922, 244070, 226729, 203822, 131041, 511860, 8576394, 189152, 223274, 236690,
291733, 129671, 166624, 240539, 130840, 267283, 213453, 201222, 236832, 439630, 260349, 222872, 292483, 318421, 207754, 949560, 128939, 256797, 788863, 219095, 238797, 176173, 291595, 194295,
213186, 231214, 363145, 261225, 134239, 301973, 161622, 278395, 258452, 167546, 335211, 229548, 208457, 272518, 183413, 240759, 168718, 267292, 159587, 375975, 170067, 210484, 365746, 135034,
188659, 264619, 170088, 190571, 188636, 269922, 186192, 273589, 2230561, 153457, 44097773, 208975, 171893, 250791, 316566, 226122, 289897, 162200, 215547, 419255, 460024, 168405, 201454, 141233,
223446, 686368, 211808, 243936, 237425, 278200, 219047, 324163, 212234, 261468, 252077, 656773, 123614, 263330, 309612, 131317, 243546, 241132, 309510, 195920, 138333, 170063, 148862, 160462,
281809, 396536, 188425, 283787, 191897, 280089, 1801891, 203466, 142167, 218116, 219620, 161066, 407379, 280760, 232678, 320121, 929291, 304344, 2231038, 290955, 181600, 405338, 148530, 303780,
144359, 864996, 267929, 161902, 196588, 613259, 154304, 214399, 266655, 557596, 215499, 295662, 186756, 132080, 10073202, 214565, 333244, 173118, 145660, 198726, 245820, 420951, 308187, 170953,
286338, 1032230, 199440, 219840, 298385, 168537, 652501, 301056, 526849, 320419, 294308, 248252, 225139, 381336, 535671, 253509, 319956, 202577, 311945, 482358, 158918, 331737, 311275, 192348,
277808, 199795, 204948, 188393, 484972, 154072, 700422, 557917, 467502...}, pause_end = {1562913412074353160, 1562913492069770427, 1562913572075038733, 1562913652236570851, 1562913732643859322,
1562913811881505665, 1562913892058112809, 1562913967053029030, 1562914043975576502, 1562914129411892992, 1562914203980570674, 1562914283977254060, 1562914369496475612, 1562914444020073395,
1562914531194944911, 1562914611932199140, 1562914691931329442, 1562914769171470869, 1562914843981248367, 1562914931894045541, 1562915012069753858, 1562915093974648843, 1562915178811915035,
1562915253976725656, 1562915334051127644, 1562915421889435877, 1562915491946042127, 1562915571935550457, 1562915651935948481, 1562915731918239068, 1562915811911862152, 1562915890903210964,
1562915963969670885, 1562916043954719057, 1562916122101607088, 1562916202026174224, 1562916282066356507, 1562916362287724238, 1562916442043381294, 1562916522010270201, 1562916601978654047,
1562916682230446805, 1562916763958887267, 1562916842072191207, 1562916922240253115, 1562917002352354062, 1562917071903216580, 1562917151880545308, 1562917231978882649, 1562917311973210987,
1562917392218610645, 1562917473955125777, 1562917552072659211, 1562917633945627447, 1562917711932191214, 1562917791975965390, 1562917872074676940, 1562917952231008351, 1562918033973854837,
1562918112224725645, 1562918192225390831, 1562918273967477365, 1562918352223648197, 1562918432514009496, 1562918511973180911, 1562918592045522690, 1562918672020815875, 1562918752153526847,
1562918831974388936, 1562918911933485337, 1562918985827496069, 1562919063982764387, 1562919142878799013, 1562919214236331916, 1562919293968426438, 1562919372485454404, 1562919451895485634,
1562899112084924539, 1562899192220480917, 1562899273988499452, 1562899353978822657, 1562899434025395450, 1562899513958734025, 1562899593928870146, 1562899672068399021, 1562899752225460578,
1562899833960206568, 1562899912583196792, 1562899993973250605, 1562900074230177627, 1562900153980164412, 1562900241884839738, 1562900322151582695, 1562900402071995426, 1562900483983867482,
1562900563970745955, 1562900643972966951, 1562900723959927699, 1562900802242550591, 1562900882141123083, 1562900962073051797, 1562901043943544991, 1562901122064564663, 1562901203974971317,
1562901282547365233, 1562901362219423264, 1562901443946942277, 1562901522132617997, 1562901602079458083, 1562901681979268308, 1562901762224440190, 1562901843960271084, 1562901922115883380,
1562902001944302624, 1562902082073157352, 1562902163941594597, 1562902242221096030, 1562902323960045676, 1562902403927095239, 1562902481989698372, 1562902562228468840, 1562902631994709133,
1562902713566650319, 1562902792048631666, 1562902871973045161, 1562902953962248036, 1562903033382668349, 1562903112071356293, 1562903192047387930, 1562903272076562232, 1562903353959934392,
1562903433937745530, 1562903512074451711, 1562903593947024794, 1562903672065455629, 1562903753974165140, 1562903836491245772, 1562903884125595703, 1562903963963205208, 1562904043987915207,
1562904131976106572, 1562904212288298115, 1562904292069466848, 1562904373978515868, 1562904453975816553, 1562904533979422070, 1562904615821797938, 1562904702044395504, 1562904783973531072,
1562904863928038365, 1562904942248672136, 1562905023958834549, 1562905103932777196, 1562905182071833751, 1562905263938835124, 1562905342130146459, 1562905422014760178, 1562905502121054240,
1562905582241287272, 1562905663971779247, 1562905743955928851, 1562905823945158006, 1562905902065669023, 1562905983958806958, 1562906062243111475, 1562906143965972741, 1562906223955804941,
1562906303927983303, 1562906382080316407, 1562906462138935347, 1562906543948088222, 1562906622217192837, 1562906703987324590, 1562906783960026718, 1562906863961505057, 1562906942227225691,
1562907023972780094, 1562907105825397444, 1562907191979748511, 1562907273949287497, 1562907352240649020, 1562907433978618564, 1562907521985410609, 1562907602241135156, 1562907683937293850,
1562907762228431736, 1562907843976453570, 1562907932002913031, 1562908012244113926, 1562908093961856118, 1562908173940638154, 1562908252073949008, 1562908333964134111, 1562908413959650510,
1562908493934891526, 1562908572106182797, 1562908652062918619, 1562908732068030586, 1562908814010994362, 1562908901978894027...}, numgc = 17997, numforcedgc = 0,
gc_cpu_fraction = 0.00054076827652344895, enablegc = true, debuggc = false, by_size = {{size = 0, nmalloc = 0, nfree = 0}, {size = 8, nmalloc = 771699326, nfree = 771684951}, {size = 16,
nmalloc = 3068109279, nfree = 3067953172}, {size = 32, nmalloc = 3827062491, nfree = 3826969425}, {size = 48, nmalloc = 3503733677, nfree = 3503638801}, {size = 64, nmalloc = 1700309405,
nfree = 1700274285}, {size = 80, nmalloc = 1780469809, nfree = 1780443325}, {size = 96, nmalloc = 1376554944, nfree = 1376528289}, {size = 112, nmalloc = 525526506, nfree = 525485688}, {
size = 128, nmalloc = 581118834, nfree = 581108235}, {size = 144, nmalloc = 422268569, nfree = 422262060}, {size = 160, nmalloc = 409930668, nfree = 409925249}, {size = 176, nmalloc = 118505794,
nfree = 118503899}, {size = 192, nmalloc = 222635024, nfree = 222632427}, {size = 208, nmalloc = 187585335, nfree = 187578140}, {size = 224, nmalloc = 133949476, nfree = 133935411}, {size = 240,
nmalloc = 249930464, nfree = 249920358}, {size = 256, nmalloc = 389062549, nfree = 389058279}, {size = 288, nmalloc = 307111275, nfree = 307096356}, {size = 320, nmalloc = 253794815,
nfree = 253790667}, {size = 352, nmalloc = 225460787, nfree = 225457374}, {size = 384, nmalloc = 50589372, nfree = 50580614}, {size = 416, nmalloc = 70728514, nfree = 70727476}, {size = 448,
nmalloc = 378900786, nfree = 378891776}, {size = 480, nmalloc = 67883080, nfree = 67882520}, {size = 512, nmalloc = 300562670, nfree = 300555703}, {size = 576, nmalloc = 80511857,
nfree = 80510686}, {size = 640, nmalloc = 200391528, nfree = 200383583}, {size = 704, nmalloc = 2752792, nfree = 2752230}, {size = 768, nmalloc = 39126960, nfree = 39126152}, {size = 896,
nmalloc = 53210450, nfree = 53205440}, {size = 1024, nmalloc = 25368567, nfree = 25364656}, {size = 1152, nmalloc = 24534982, nfree = 24534122}, {size = 1280, nmalloc = 30848143,
nfree = 30847434}, {size = 1408, nmalloc = 7957758, nfree = 7956912}, {size = 1536, nmalloc = 268261564, nfree = 268257910}, {size = 1792, nmalloc = 17999389, nfree = 17998419}, {size = 2048,
nmalloc = 6809368, nfree = 6808415}, {size = 2304, nmalloc = 9792209, nfree = 9791638}, {size = 2688, nmalloc = 2316127, nfree = 2315513}, {size = 3072, nmalloc = 8719172, nfree = 8718479}, {
size = 3200, nmalloc = 232141, nfree = 232033}, {size = 3456, nmalloc = 3433440, nfree = 3432983}, {size = 4096, nmalloc = 7400408, nfree = 7399091}, {size = 4864, nmalloc = 1982650,
nfree = 1982392}, {size = 5376, nmalloc = 3404117, nfree = 3403750}, {size = 6144, nmalloc = 2689667, nfree = 2689618}, {size = 6528, nmalloc = 277762, nfree = 277747}, {size = 6784,
nmalloc = 81388, nfree = 81374}, {size = 6912, nmalloc = 194032, nfree = 194023}, {size = 8192, nmalloc = 2535312, nfree = 2535064}, {size = 9472, nmalloc = 3823911, nfree = 3823607}, {
size = 9728, nmalloc = 85231, nfree = 85230}, {size = 10240, nmalloc = 576931, nfree = 576870}, {size = 10880, nmalloc = 111775, nfree = 111763}, {size = 12288, nmalloc = 204338, nfree = 204327},
{size = 13568, nmalloc = 275267, nfree = 275250}, {size = 14336, nmalloc = 148359, nfree = 148343}, {size = 16384, nmalloc = 884671, nfree = 884571}, {size = 18432, nmalloc = 313942,
nfree = 313846}, {size = 19072, nmalloc = 14132, nfree = 14126}, {size = 20480, nmalloc = 221291, nfree = 221249}, {size = 21760, nmalloc = 46556, nfree = 46543}, {size = 24576,
nmalloc = 1446171, nfree = 1445917}, {size = 27264, nmalloc = 43289, nfree = 43280}, {size = 28672, nmalloc = 4291, nfree = 4291}, {size = 32768, nmalloc = 602664, nfree = 602562}},
last_gc_nanotime = 243570161758237, tinyallocs = 1413031072, triggerRatio = 0.94999999999999996, gc_trigger = 383388267, heap_live = 288199160, heap_scan = 53603240, heap_marked = 196609368}
runtime.itabTable = 0xc00b5ce000
runtime.itabLock = {key = 0}
runtime.zeroVal = '\000' <repeats 1023 times>
runtime.uint16Type = 0x2e40480
runtime.uint32Type = 0x2e404c0
runtime.uint64Type = 0x2e40500
runtime.stringType = 0x2e40380
runtime.sliceType = 0x2eb68a0
runtime.cgo_yield = 0x3bfca28 <_cgo_yield>
runtime.class_to_size = {0, 8, 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 896, 1024, 1152, 1280, 1408, 1536,
1792, 2048, 2304, 2688, 3072, 3200, 3456, 4096, 4864, 5376, 6144, 6528, 6784, 6912, 8192, 9472, 9728, 10240, 10880, 12288, 13568, 14336, 16384, 18432, 19072, 20480, 21760, 24576, 27264, 28672, 32768}
runtime.physPageSize = 4096
runtime.gcphase = 0
runtime.gcBlackenEnabled = 0
runtime.MemProfileRate = 524288
runtime.panicking = 0
runtime.deferType = 0x31e6220
runtime.size_to_class8 = "\000\001\002\003\003\004\004\005\005\006\006\a\a\b\b\t\t\n\n\v\v\f\f\r\r\016\016\017\017\020\020\021\021\022\022\022\022\023\023\023\023\024\024\024\024\025\025\025\025\026\026\026\026\027\027\027\027\030\030\030\030\031\031\031\031\032\032\032\032\032\032\032\032\033\033\033\033\033\033\033\033\034\034\034\034\034\034\034\034\035\035\035\035\035\035\035\035", '\036' <repeats 16 times>, '\037' <repeats 16 times>
runtime.size_to_class128 = "\037 !\"#$$%%&&'''((()**+++++,,,,,,----......///0012222222222333333333344555566666", '7' <repeats 11 times>, "8888888888999999", ':' <repeats 16 times>, ';' <repeats 16 times>, "<<<<<", '=' <repeats 11 times>, ">>>>>>>>>>", '?' <repeats 22 times>, '@' <repeats 21 times>...
runtime.zerobase = 0
runtime.persistentChunks = 0x812368000
runtime.globalAlloc = {mutex = {key = 0}, persistentAlloc = {base = 0x80595a000, off = 116960}}
runtime.oneBitCount = "\000\001\001\002\001\002\002\003\001\002\002\003\002\003\003\004\001\002\002\003\002\003\003\004\002\003\003\004\003\004\004\005\001\002\002\003\002\003\003\004\002\003\003\004\003\004\004\005\002\003\003\004\003\004\004\005\003\004\004\005\004\005\005\006\001\002\002\003\002\003\003\004\002\003\003\004\003\004\004\005\002\003\003\004\003\004\004\005\003\004\004\005\004\005\005\006\002\003\003\004\003\004\004\005\003\004\004\005\004\005\005\006\003\004\004\005\004\005\005\006\004\005\005\006\005\006\006\a\001\002\002\003\002\003\003\004\002\003\003\004\003\004\004\005\002\003\003\004\003\004\004\005\003\004\004\005\004\005\005\006\002\003\003\004\003\004\004\005\003\004\004\005\004\005\005\006\003\004\004\005\004\005\005\006\004\005\005\006\005\006\006\a\002\003\003\004\003\004\004\005"...
runtime.emptymspan = {next = 0x0, prev = 0x0, list = 0x0, startAddr = 0, npages = 0, manualFreeList = 0, freeindex = 0, nelems = 0, allocCache = 0, allocBits = 0x0, gcmarkBits = 0x0, sweepgen = 0,
divMul = 0, baseMask = 0, allocCount = 0, spanclass = 0 '\000', state = 0 '\000', needzero = 0 '\000', divShift = 0 '\000', divShift2 = 0 '\000', scavenged = false, elemsize = 0, unusedsince = 0,
limit = 0, speciallock = {key = 0}, specials = 0x0}
runtime.class_to_allocnpages = "\000", '\001' <repeats 33 times>, "\002\001\002\001\002\001\003\002\003\001\003\002\003\004\005\006\001\a\006\005\004\003\005\a\002\t\a\005\b\003\n\a\004"
runtime.trace = <error reading variable runtime.trace (value of type `struct { runtime.lock runtime.mutex; runtime.lockOwner *runtime.g; runtime.enabled bool; runtime.shutdown bool; runtime.headerWritten bool; runtime.footerWritten bool; runtime.shutdownSema uint32; runtime.seqStart uint64; runtime.ticksStart int64; runtime.ticksEnd int64; runtime.timeStart int64; runtime.timeEnd int64; runtime.seqGC uint64; runtime.reading runtime.traceBufPtr; runtime.empty runtime.traceBufPtr; runtime.fullHead runtime.traceBufPtr; runtime.fullTail runtime.traceBufPtr; runtime.reader runtime.guintptr; runtime.stackTab runtime.traceStackTable; runtime.stringsLock runtime.mutex; runtime.strings map[string]uint64; runtime.stringSeq uint64; runtime.markWorkerLabels [3]uint64; runtime.bufLock runtime.mutex; runtime.buf runtime.traceBufPtr }' requires 65744 bytes, which is more than max-value-size)>
runtime.gcController = {scanWork = 30539768, bgScanCredit = 18922097, assistTime = 35150944, dedicatedMarkTime = 44136657, fractionalMarkTime = 0, idleMarkTime = 70921338,
markStartTime = 243570152160203, dedicatedMarkWorkersNeeded = 5, assistWorkPerByte = 3.7006982096378689, assistBytesPerWork = 0.27021927845822769, fractionalUtilizationGoal = 0, _ = {
_ = '\000' <repeats 63 times>}}
runtime.finlock = {key = 0}
runtime.finq = 0x0
runtime.finc = 0x819ecc3e0
runtime.allfin = 0x819ecc3e0
runtime.finptrmask = "{\357\275\367\336{\357\275\367\336{\357\275\367\336{\357\275\367\336{\357\275\367\336{\357\275\367\336{\357\275\367\336{\357\275\367\336{\357\275\367\336{\357\275\367\336{\357\275\367\336{\357\275\367\336{\357\275\367"
runtime.finalizer1 = "{\357\275\367\336"
runtime.fingwake = false
runtime.fingwait = true
runtime.fing = 0xc0000d0780
runtime.fingCreate = 1
runtime.fingRunning = false
runtime.cgoAlwaysFalse = false
runtime.heapminimum = 4194304
runtime.work = {full = 0, empty = 0, pad0 = {_ = '\000' <repeats 63 times>}, wbufSpans = {lock = {key = 0}, free = {first = 0x0, last = 0x0}, busy = {first = 0x0, last = 0x0}}, _ = 0,
bytesMarked = 196609368, markrootNext = 1595, markrootJobs = 1578, nproc = 4294967295, tstart = 243570161748971, nwait = 4294967295, ndone = 0, nFlushCacheRoots = 0, nDataRoots = 2, nBSSRoots = 1,
nSpanRoots = 50, nStackRoots = 1523, startSema = 1, markDoneSema = 1, bgMarkReady = {key = 0}, bgMarkDone = 0, mode = 0, userForced = false, totaltime = 2294034580515, initialHeapLive = 376404440,
assistQueue = {lock = {key = 0}, q = {head = 0, tail = 0}}, sweepWaiters = {lock = {key = 0}, list = {head = 0}}, cycles = 17997, stwprocs = 20, maxprocs = 20, tSweepTerm = 243570152160203,
tMark = 243570152417329, tMarkTerm = 243570161745441, tEnd = 243570161758237, pauseNS = 269922, pauseStart = 243570161745441, heap0 = 376404440, heap1 = 379397776, heap2 = 196609368,
heapGoal = 385981472}
runtime.gcpercent = 100
runtime.allp = {array = 0xc00005e000, len = 20, cap = 20}
runtime.sweep = {lock = {key = 0}, g = 0xc0000d0300, parked = true, started = false, nbgsweep = 24476, npausesweep = 0}
runtime.forcegcperiod = 120000000000
runtime.worldsema = 1
runtime.gcMarkDoneFlushed = 0
runtime.runtimeInitTime = 31461327652886
runtime.throwOnGCWork = false
runtime.allglock = {key = 0}
runtime.poolcleanup = {void (void)} 0x5d0b810 <runtime.poolcleanup>
runtime.useCheckmark = false
runtime.oneptrmask = "\001"
runtime.mSpanStateNames = {array = 0x58fc900 <runtime.statictmp_49>, len = 4, cap = 4}
runtime.class_to_divmagic = {{shift = 0 '\000', shift2 = 0 '\000', mul = 0, baseMask = 0}, {shift = 3 '\003', shift2 = 0 '\000', mul = 1, baseMask = 65528}, {shift = 4 '\004', shift2 = 0 '\000',
mul = 1, baseMask = 65520}, {shift = 5 '\005', shift2 = 0 '\000', mul = 1, baseMask = 65504}, {shift = 4 '\004', shift2 = 9 '\t', mul = 171, baseMask = 0}, {shift = 6 '\006', shift2 = 0 '\000',
mul = 1, baseMask = 65472}, {shift = 4 '\004', shift2 = 10 '\n', mul = 205, baseMask = 0}, {shift = 5 '\005', shift2 = 9 '\t', mul = 171, baseMask = 0}, {shift = 4 '\004', shift2 = 11 '\v',
mul = 293, baseMask = 0}, {shift = 7 '\a', shift2 = 0 '\000', mul = 1, baseMask = 65408}, {shift = 4 '\004', shift2 = 9 '\t', mul = 57, baseMask = 0}, {shift = 5 '\005', shift2 = 10 '\n',
mul = 205, baseMask = 0}, {shift = 4 '\004', shift2 = 12 '\f', mul = 373, baseMask = 0}, {shift = 6 '\006', shift2 = 7 '\a', mul = 43, baseMask = 0}, {shift = 4 '\004', shift2 = 13 '\r', mul = 631,
baseMask = 0}, {shift = 5 '\005', shift2 = 11 '\v', mul = 293, baseMask = 0}, {shift = 4 '\004', shift2 = 13 '\r', mul = 547, baseMask = 0}, {shift = 8 '\b', shift2 = 0 '\000', mul = 1,
baseMask = 65280}, {shift = 5 '\005', shift2 = 9 '\t', mul = 57, baseMask = 0}, {shift = 6 '\006', shift2 = 9 '\t', mul = 103, baseMask = 0}, {shift = 5 '\005', shift2 = 12 '\f', mul = 373,
baseMask = 0}, {shift = 7 '\a', shift2 = 7 '\a', mul = 43, baseMask = 0}, {shift = 5 '\005', shift2 = 10 '\n', mul = 79, baseMask = 0}, {shift = 6 '\006', shift2 = 10 '\n', mul = 147,
baseMask = 0}, {shift = 5 '\005', shift2 = 11 '\v', mul = 137, baseMask = 0}, {shift = 9 '\t', shift2 = 0 '\000', mul = 1, baseMask = 65024}, {shift = 6 '\006', shift2 = 9 '\t', mul = 57,
baseMask = 0}, {shift = 7 '\a', shift2 = 6 '\006', mul = 13, baseMask = 0}, {shift = 6 '\006', shift2 = 11 '\v', mul = 187, baseMask = 0}, {shift = 8 '\b', shift2 = 5 '\005', mul = 11,
baseMask = 0}, {shift = 7 '\a', shift2 = 8 '\b', mul = 37, baseMask = 0}, {shift = 10 '\n', shift2 = 0 '\000', mul = 1, baseMask = 64512}, {shift = 7 '\a', shift2 = 9 '\t', mul = 57, baseMask = 0},
{shift = 8 '\b', shift2 = 6 '\006', mul = 13, baseMask = 0}, {shift = 7 '\a', shift2 = 11 '\v', mul = 187, baseMask = 0}, {shift = 9 '\t', shift2 = 5 '\005', mul = 11, baseMask = 0}, {shift = 8 '\b',
shift2 = 8 '\b', mul = 37, baseMask = 0}, {shift = 11 '\v', shift2 = 0 '\000', mul = 1, baseMask = 63488}, {shift = 8 '\b', shift2 = 9 '\t', mul = 57, baseMask = 0}, {shift = 7 '\a',
shift2 = 10 '\n', mul = 49, baseMask = 0}, {shift = 10 '\n', shift2 = 5 '\005', mul = 11, baseMask = 0}, {shift = 7 '\a', shift2 = 10 '\n', mul = 41, baseMask = 0}, {shift = 7 '\a',
shift2 = 9 '\t', mul = 19, baseMask = 0}, {shift = 12 '\f', shift2 = 0 '\000', mul = 1, baseMask = 61440}, {shift = 8 '\b', shift2 = 9 '\t', mul = 27, baseMask = 0}, {shift = 8 '\b',
shift2 = 10 '\n', mul = 49, baseMask = 0}, {shift = 11 '\v', shift2 = 5 '\005', mul = 11, baseMask = 0}, {shift = 7 '\a', shift2 = 13 '\r', mul = 161, baseMask = 0}, {shift = 7 '\a',
shift2 = 13 '\r', mul = 155, baseMask = 0}, {shift = 8 '\b', shift2 = 9 '\t', mul = 19, baseMask = 0}, {shift = 13 '\r', shift2 = 0 '\000', mul = 1, baseMask = 57344}, {shift = 8 '\b',
shift2 = 12 '\f', mul = 111, baseMask = 0}, {shift = 9 '\t', shift2 = 9 '\t', mul = 27, baseMask = 0}, {shift = 11 '\v', shift2 = 6 '\006', mul = 13, baseMask = 0}, {shift = 7 '\a',
shift2 = 14 '\016', mul = 193, baseMask = 0}, {shift = 12 '\f', shift2 = 3 '\003', mul = 3, baseMask = 0}, {shift = 8 '\b', shift2 = 13 '\r', mul = 155, baseMask = 0}, {shift = 11 '\v',
shift2 = 8 '\b', mul = 37, baseMask = 0}, {shift = 14 '\016', shift2 = 0 '\000', mul = 1, baseMask = 49152}, {shift = 11 '\v', shift2 = 8 '\b', mul = 29, baseMask = 0}, {shift = 7 '\a',
shift2 = 13 '\r', mul = 55, baseMask = 0}, {shift = 12 '\f', shift2 = 5 '\005', mul = 7, baseMask = 0}, {shift = 8 '\b', shift2 = 14 '\016', mul = 193, baseMask = 0}, {shift = 13 '\r',
shift2 = 3 '\003', mul = 3, baseMask = 0}, {shift = 7 '\a', shift2 = 14 '\016', mul = 77, baseMask = 0}, {shift = 12 '\f', shift2 = 7 '\a', mul = 19, baseMask = 0}, {shift = 15 '\017',
shift2 = 0 '\000', mul = 1, baseMask = 32768}}
runtime.gcBitsArenas = {lock = {key = 0}, free = 0x81a002000, next = 0x819ff2000, current = 0x81a012000, previous = 0x815d91000}
runtime.bucketmem = 5432824
runtime.buckhash = 0x806e7f000
runtime.mbuckets = 0x81221cbe8
runtime.xbuckets = 0x819f6c6f0
runtime.bbuckets = 0x812368620
runtime.proflock = {key = 0}
runtime.mProf = {cycle = 17997, flushed = true}
runtime.mutexprofilerate = 1000
runtime.tracelock = {key = 0}
runtime.sizeof_C_MStats = 5776
runtime.netpollInited = 1
runtime.kq = 6
runtime.pollcache = {lock = {key = 0}, first = 0x810812240}
runtime.netpollWaiters = 259734249
runtime.sigset_all = {__bits = {4294967295, 4294967295, 4294967295, 4294967295}}
runtime.m0 = {g0 = 0x5d1bcc0 <runtime.g0>, morebuf = {sp = 0, pc = 0, g = 0, ctxt = 0x0, ret = 0, lr = 0, bp = 0}, divmod = 0, procid = 0, gsignal = 0xc000000000, goSigStack = {stack = {lo = 0,
hi = 0}, stackguard0 = 0, stackguard1 = 0, stktopsp = 0}, sigmask = {__bits = {0, 0, 0, 0}}, tls = {0, 0, 0, 0, 0, 0}, mstartfn = {void (void)} 0x5d1c780 <runtime.m0>, curg = 0xc009d0c300,
caughtsig = 0, p = 0, nextp = 0, oldp = 824634142464, id = 0, mallocing = 0, throwing = 0, preemptoff = 0x0 "", locks = 0, dying = 0, profilehz = 0, spinning = false, blocked = false, inwb = false,
newSigstack = true, printlock = 0 '\000', incgo = true, freeWait = 0, fastrand = {699966117, 1464327117}, needextram = false, traceback = 0 '\000', ncgocall = 13139754, ncgo = 1, cgoCallersUse = 0,
cgoCallers = 0xc00000a000, park = {key = 0}, alllink = 0x0, schedlink = 824739120896, mcache = 0x0, lockedg = 0, createstack = {0 <repeats 32 times>}, lockedExt = 0, lockedInt = 0, nextwaitm = 0,
waitunlockf = 0x0, waitlock = 0x0, waittraceev = 27 '\033', waittraceskip = 5, startingtrace = false, syscalltick = 92646321, thread = 0, freelink = 0x0, libcall = {fn = 0, n = 0, args = 0, r1 = 0,
r2 = 0, err = 0}, libcallpc = 0, libcallsp = 0, libcallg = 0, syscall = {fn = 0, n = 0, args = 0, r1 = 0, r2 = 0, err = 0}, vdsoSP = 0, vdsoPC = 0, mOS = {<No data fields>}}
runtime.urandom_dev = {array = 0x5933718 <runtime..gobytes> "/dev/urandom", len = 13, cap = 13}
runtime.isarchive = false
runtime.islibrary = false
runtime.timekeepSharedPage = 0x7ffffffff1b0
runtime.indexError = {tab = 0x3a9de80 <runtime.errorString,error>, data = 0x58eac00 <runtime.statictmp_59>}
runtime.sliceError = {tab = 0x3a9de80 <runtime.errorString,error>, data = 0x58eac10 <runtime.statictmp_60>}
runtime.divideError = {tab = 0x3a9de80 <runtime.errorString,error>, data = 0x58eabd0 <runtime.statictmp_51>}
runtime.overflowError = {tab = 0x3a9de80 <runtime.errorString,error>, data = 0x58eabe0 <runtime.statictmp_52>}
runtime.floatError = {tab = 0x3a9de80 <runtime.errorString,error>, data = 0x58eabf0 <runtime.statictmp_53>}
runtime.memoryError = {tab = 0x3a9de80 <runtime.errorString,error>, data = 0x58eabc0 <runtime.statictmp_50>}
runtime.runningPanicDefers = 0
runtime.paniclk = {key = 0}
runtime.traceback_cache = 4
runtime.didothers = false
runtime.deadlock = {key = 0}
runtime.sigtable = {{flags = 0, name = 0x3353142 "SIGNONE: no trap"}, {flags = 3, name = 0x3373ec6 "SIGHUP: terminal line hangup"}, {flags = 3, name = 0x33559b0 "SIGINT: interrupt"}, {flags = 5,
name = 0x334c60e "SIGQUIT: quit"}, {flags = 132, name = 0x33708c8 "SIGILL: illegal instruction"}, {flags = 132, name = 0x335aa0b "SIGTRAP: trace trap"}, {flags = 5,
name = 0x334e6f4 "SIGABRT: abort"}, {flags = 4, name = 0x338e67c "SIGEMT: emulate instruction executed"}, {flags = 136, name = 0x3380ed2 "SIGFPE: floating-point exception"}, {flags = 0,
name = 0x334c601 "SIGKILL: kill"}, {flags = 136, name = 0x335599f "SIGBUS: bus error"}, {flags = 136, name = 0x337ded4 "SIGSEGV: segmentation violation"}, {flags = 1,
name = 0x3364f05 "SIGSYS: bad system call"}, {flags = 1, name = 0x3377341 "SIGPIPE: write to broken pipe"}, {flags = 1, name = 0x335d396 "SIGALRM: alarm clock"}, {flags = 3,
name = 0x335d3aa "SIGTERM: termination"}, {flags = 257, name = 0x33879d1 "SIGURG: urgent condition on socket"}, {flags = 0, name = 0x334c61b "SIGSTOP: stop"}, {flags = 273,
name = 0x33625f2 "SIGTSTP: keyboard stop"}, {flags = 273, name = 0x3373eaa "SIGCONT: continue after stop"}, {flags = 385, name = 0x338429b "SIGCHLD: child status has changed"}, {flags = 273,
name = 0x33842bc "SIGTTIN: background read from tty"}, {flags = 273, name = 0x3380ef2 "SIGTTOU: background write to tty"}, {flags = 257, name = 0x3364eee "SIGIO: i/o now possible"}, {flags = 1,
name = 0x33708e3 "SIGXCPU: cpu limit exceeded"}, {flags = 1, name = 0x33842dd "SIGXFSZ: file size limit exceeded"}, {flags = 1, name = 0x337a66a "SIGVTALRM: virtual alarm clock"}, {flags = 129,
name = 0x337a610 "SIGPROF: profiling alarm clock"}, {flags = 257, name = 0x3373ee2 "SIGWINCH: window size change"}, {flags = 257, name = 0x33919d1 "SIGINFO: status request from keyboard"}, {
flags = 1, name = 0x337a62e "SIGUSR1: user-defined signal 1"}, {flags = 1, name = 0x337a64c "SIGUSR2: user-defined signal 2"}, {flags = 1, name = 0x3353152 "SIGTHR: reserved"}}
runtime.printBacklogIndex = 374
runtime.printBacklog = "/stop.(*Stopper).RunAsyncTask.func1(0xc0004f5d40, 0x3b051a0, 0xc0000fa010, 0xc014b3d780, 0x33, 0x0, 0x0, 0xc00c087290)\n\t/home/bra/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +"...
runtime.debuglock = {key = 0}
runtime.maxstacksize = 1000000000
runtime.mainStarted = true
runtime.newmHandoff = {lock = {key = 0}, newm = 0, waiting = true, wake = {key = 0}, haveTemplateThread = 1}
runtime.forcegc = {lock = {key = 0}, g = 0xc000000a80, idle = 1}
runtime.badmorestackg0Msg = 0x336592c "fatal: morestack on g0\n"
runtime.badmorestackgsignalMsg = 0x3374d1a "fatal: morestack on gsignal\n"
runtime.argc = 7
runtime.argv = 0x7fffffffe5e8
runtime.x86HasPOPCNT = true
runtime.x86HasSSE41 = true
runtime.arm64HasATOMICS = false
runtime.skipPC = 6341344
runtime.initSigmask = {__bits = {0, 0, 0, 0}}
runtime.buildVersion = 0x334228b "go1.12.2"
runtime.freezing = 0
runtime.cgoHasExtraM = true
runtime.extraMCount = 3
runtime.extram = 824647933952
runtime.earlycgocallback = {array = 0x5935620 <runtime..gobytes> "fatal error: cgo callback before cgo call\n", len = 42, cap = 42}
runtime.execLock = {rLock = {key = 0}, readers = 0, readerPass = 0, wLock = {key = 0}, writer = 0, readerCount = 0, readerWait = 0}
runtime.stealOrder = {count = 20, coprimes = {array = 0xc0000540a0, len = 8, cap = 8}}
runtime.sigprofCallers = {0 <repeats 32 times>}
runtime.sigprofCallersUse = 0
runtime.allpLock = {key = 0}
runtime.starttime = 0
runtime.overflowTag = {0x0}
runtime.ticks = {lock = {key = 0}, pad = 0, val = 2197446988}
runtime.argslice = {array = 0xc00004e000, len = 7, cap = 7}
runtime.test_z64 = 3298534883331
runtime.test_x64 = 42
runtime.dbgvars = {array = 0x590e9e0 <runtime.statictmp_61>, len = 16, cap = 16}
runtime.traceback_env = 4
runtime.reflectOffs = {lock = {key = 0}, next = -20, m = 0xc000a50f60, minv = 0xc000a50f90}
runtime.semtable = {{root = {lock = {key = 0}, treap = 0x0, nwait = 0}, pad = '\000' <repeats 39 times>} <repeats 99 times>, {root = {lock = {key = 0}, treap = 0xc000b60d80, nwait = 1},
pad = '\000' <repeats 39 times>}, {root = {lock = {key = 0}, treap = 0x0, nwait = 0}, pad = '\000' <repeats 39 times>} <repeats 151 times>}
runtime.testSigtrap = {void (runtime.siginfo *, runtime.sigctxt *, runtime.g *, bool *)} 0x5d0b828 <runtime.testSigtrap>
runtime.sig = {note = {key = 0}, mask = {0, 0}, wanted = {32782, 0}, ignored = {0, 0}, recv = {0, 0}, state = 1, delivering = 0, inuse = true}
runtime.crashing = 0
runtime.fwdSig = {0 <repeats 32 times>, 34457404144}
runtime.handlingSig = {0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}
runtime.signalsOK = true
runtime.enableSigChan = 0xc0000e0300
runtime.maskUpdatedChan = 0xc0000e0120
runtime.disableSigChan = 0xc0000e02a0
runtime.stackpool = {{first = 0x809a536d8, last = 0x805a22788}, {first = 0x809a56aa0, last = 0x809a97768}, {first = 0x80d9b9630, last = 0x805acddb0}, {first = 0x81098f270, last = 0x819eb13e0}}
runtime.stackLarge = {lock = {key = 0}, free = {{first = 0x0, last = 0x0} <repeats 35 times>}}
runtime.stackpoolmu = {key = 0}
runtime.staticbytes = "\000\001\002\003\004\005\006\a\b\t\n\v\f\r\016\017\020\021\022\023\024\025\026\027\030\031\032\033\034\035\036\037 !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\177\200\201\202\203\204\205\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237\240\241\242\243\244\245\246\247\250\251\252\253\254\255\256\257\260\261\262\263\264\265\266\267\270\271\272\273\274\275\276\277\300\301\302\303\304\305\306\307"...
runtime.badsystemstackMsg = 0x33c7e98 "fatal: systemstack called from unexpected goroutine"
runtime.cgoSymbolizer = 0x0
runtime.timers = {{timersBucket = {lock = {key = 0}, gp = 0xc0004dc780, created = true, sleeping = true, rescheduling = false, sleepUntil = 243609684820755, waitnote = {key = 0}, t = {
array = 0xc0035e6600, len = 10, cap = 64}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc0000d0a80, created = true, sleeping = true, rescheduling = false,
sleepUntil = 243605684469245, waitnote = {key = 0}, t = {array = 0xc00c0e3e00, len = 11, cap = 64}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc00049d800,
created = true, sleeping = true, rescheduling = false, sleepUntil = 243605679387831, waitnote = {key = 0}, t = {array = 0xc009229200, len = 6, cap = 64}}, pad = '\000' <repeats 63 times>}, {
timersBucket = {lock = {key = 0}, gp = 0xc0004dd500, created = true, sleeping = true, rescheduling = false, sleepUntil = 243605578503073, waitnote = {key = 0}, t = {array = 0xc00cbc0e00, len = 3,
cap = 64}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc003172000, created = true, sleeping = true, rescheduling = false, sleepUntil = 243622178530094,
waitnote = {key = 0}, t = {array = 0xc00cbc1000, len = 8, cap = 64}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc003174000, created = true, sleeping = true,
rescheduling = false, sleepUntil = 243612183857531, waitnote = {key = 0}, t = {array = 0xc00c61ac00, len = 5, cap = 64}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0},
gp = 0xc00328c000, created = true, sleeping = true, rescheduling = false, sleepUntil = 243612211064096, waitnote = {key = 0}, t = {array = 0xc00a8e4e00, len = 3, cap = 64}},
pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc000488900, created = true, sleeping = true, rescheduling = false, sleepUntil = 243612216565299, waitnote = {key = 0},
t = {array = 0xc000211400, len = 4, cap = 64}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc00328a780, created = true, sleeping = true, rescheduling = false,
sleepUntil = 243664082616858, waitnote = {key = 0}, t = {array = 0xc007a05c00, len = 2, cap = 32}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc000b42000,
created = true, sleeping = true, rescheduling = false, sleepUntil = 243664280436744, waitnote = {key = 0}, t = {array = 0xc007dfd800, len = 1, cap = 32}}, pad = '\000' <repeats 63 times>}, {
timersBucket = {lock = {key = 0}, gp = 0xc000ddc600, created = true, sleeping = true, rescheduling = false, sleepUntil = 243605678531906, waitnote = {key = 0}, t = {array = 0xc00a852c00, len = 4,
cap = 64}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc0004dc900, created = true, sleeping = true, rescheduling = false, sleepUntil = 243605578513935,
waitnote = {key = 0}, t = {array = 0xc01f06e400, len = 2, cap = 64}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc0004c0c00, created = true, sleeping = true,
rescheduling = false, sleepUntil = 243606111381527, waitnote = {key = 0}, t = {array = 0xc008522b00, len = 5, cap = 32}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0},
gp = 0xc0032c6000, created = true, sleeping = true, rescheduling = false, sleepUntil = 243606305588247, waitnote = {key = 0}, t = {array = 0xc00a636900, len = 4, cap = 32}},
pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc003424000, created = true, sleeping = true, rescheduling = false, sleepUntil = 243630678401183, waitnote = {key = 0},
t = {array = 0xc006679e00, len = 2, cap = 64}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc0032f4780, created = true, sleeping = true, rescheduling = false,
sleepUntil = 243660291660879, waitnote = {key = 0}, t = {array = 0xc008d74300, len = 0, cap = 32}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc00328aa80,
created = true, sleeping = true, rescheduling = false, sleepUntil = 243605638391354, waitnote = {key = 0}, t = {array = 0xc006bdb100, len = 4, cap = 32}}, pad = '\000' <repeats 63 times>}, {
timersBucket = {lock = {key = 0}, gp = 0xc00328af00, created = true, sleeping = true, rescheduling = false, sleepUntil = 243660294253780, waitnote = {key = 0}, t = {array = 0xc00ee0a800, len = 0,
cap = 32}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc003de2300, created = true, sleeping = true, rescheduling = false, sleepUntil = 243664084989821,
waitnote = {key = 0}, t = {array = 0xc008522300, len = 0, cap = 32}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0}, gp = 0xc001012900, created = true, sleeping = true,
rescheduling = false, sleepUntil = 243605690384207, waitnote = {key = 0}, t = {array = 0xc0074f2200, len = 9, cap = 64}}, pad = '\000' <repeats 63 times>}, {timersBucket = {lock = {key = 0},
gp = 0x0, created = false, sleeping = false, rescheduling = false, sleepUntil = 0, waitnote = {key = 0}, t = {array = 0x0, len = 0, cap = 0}}, pad = '\000' <repeats 63 times>} <repeats 44 times>}
runtime.faketime = 0
runtime.gcMarkWorkerModeStrings = {0x334e18a "GC (dedicated)", 0x3350691 "GC (fractional)", 0x33436e6 "GC (idle)"}
runtime.gStatusStrings = {0x333a21f "idle", 0x3342a93 "runnable", 0x33407f4 "running", 0x3340aa2 "syscall", 0x3340cee "waiting", 0x0 "", 0x333a057 "dead", 0x0 "", 0x3344238 "copystack"}
runtime.cgoTraceback = 0x0
runtime.pinnedTypemaps = {array = 0x0, len = 0, cap = 0}
runtime.binuptimeDummy = 0
runtime.zeroBintime = {sec = 0, frac = 0}
runtime.hpetDevMap = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}
runtime.uint16Eface = {_type = 0x2e40480, data = 0x5d5e6f6 <runtime.statictmp_54>}
runtime.uint32Eface = {_type = 0x2e404c0, data = 0x5d5e794 <runtime.statictmp_55>}
runtime.uint64Eface = {_type = 0x2e40500, data = 0x5d5ec50 <runtime.statictmp_56>}
runtime.stringEface = {_type = 0x2e40380, data = 0x5d0eec0 <runtime.statictmp_57>}
runtime.sliceEface = {_type = 0x2eb68a0, data = 0x5d10840 <runtime.statictmp_58>}
runtime.chansendpc = 5991248
runtime.chanrecvpc = 5994192
runtime.initdone路 = 2 '\002'
runtime..gobytes.4 = 1986356271
runtime..gobytes.5 = 1635017062
runtime.itabTableInit = {size = 512, count = 384, entries = {0x3a96180 <com/knz/go-libedit/common.CompletionGenerator>, 0x3a96200 <com/cockroachdb/cockroach/pkg/gossip.GossipClient>,
0x3ad97e0 <com/cockroachdb/cockroach/pkg/ccl/importccl.inputConverter>, 0x3ad9220 <PtrType,debug/dwarf.Type>, 0x3a95ea0 <*encoding/json.SyntaxError,error>,
0x3b28e20 <com/cockroachdb/cockroach/pkg/ccl/storageccl.ExportStorage>, 0x3b051e0 <timerCtx,context.Context>, 0x3ad9ca0 <com/spf13/pflag.Value>,
0x3b054a0 <com/cockroachdb/cockroach/pkg/ccl/changefeedccl.Sink>, 0x3b49620 <com/cockroachdb/cockroach/pkg/roachpb.Request>, 0x0, 0x3ada020 <com/gogo/protobuf/proto.Message>,
0x3b292a0 <com/cockroachdb/cockroach/pkg/util/protoutil.Message>, 0x3b29480 <com/cockroachdb/cockroach/pkg/kv.txnInterceptor>, 0x0, 0x0, 0x0, 0x0, 0x3b052a0 <gcm,crypto/cipher.AEAD>,
0x3adae20 <com/gogo/protobuf/proto.Message>, 0x3b3afa0 <Conn,net.Conn>, 0x0, 0x3a95ec0 <*encoding/json.UnmarshalTypeError,error>, 0x3b05660 <com/cockroachdb/cockroach/pkg/cli.logStream>,
0x3b3ee40 <com/cockroachdb/cockroach/pkg/roachpb.Response>, 0x3b28c40 <CurveParams,crypto/elliptic.Curve>, 0x3ac31a0 <serverHelloDoneMsg,crypto/tls.handshakeMessage>,
0x3b290c0 <com/cockroachdb/cockroach/pkg/util/protoutil.Message>, 0x3adaa20 <Payload_SchemaChange,interface { MarshalTo([]uint8) (int, error); Size() int; jobspb.isPayload_Details() }>,
0x3b49520 <com/cockroachdb/cockroach/pkg/roachpb.Request>, 0x0, 0x3ad92e0 <TypedefType,debug/dwarf.Type>, 0x3ad9060 <CharType,debug/dwarf.Type>, 0x3adae60 <com/gogo/protobuf/proto.Message>,
0x3a95fc0 <printer,io.Writer>, 0x0, 0x3a95d60 <*debug/elf.FormatError,error>, 0x3a96260 <com/cockroachdb/cockroach/pkg/internal/client.Sender>, 0x0,
0x3ac3120 <newSessionTicketMsgTLS13,crypto/tls.handshakeMessage>, 0x3ad9260 <QualType,debug/dwarf.Type>, 0x0, 0x3b053e0 <com/cockroachdb/cockroach/pkg/ccl/changefeedccl.Sink>, 0x0, 0x0, 0x0,
0x3a95ae0 <Reader,io.ByteReader>, 0x3b114e0 <com/cockroachdb/cockroach/pkg/sql/distsqlrun.RowSource>, 0x0, 0x0, 0x3a96040 <stringReader,io.Reader>,
0x3ac2fa0 <encryptedExtensionsMsg,crypto/tls.handshakeMessage>, 0x0, 0x3a95f40 <lineBreaker,io.Writer>, 0x3b29120 <com/cockroachdb/cockroach/pkg/util/protoutil.Message>,
0x3ad8d60 <byFreq,sort.Interface>, 0x3b11520 <spanFrontierHeap,container/heap.Interface>, 0x0, 0x3b11360 <cthWrapper,hash.Hash>, 0x3ac33e0 <dumper,io.WriteCloser>,
0x3a96360 <com/cockroachdb/cockroach/pkg/internal/client.Sender>, 0x0, 0x3ac2de0 <certificateMsgTLS13,crypto/tls.handshakeMessage>, 0x3ac30a0 <keyUpdateMsg,crypto/tls.handshakeMessage>,
0x3a960a0 <com/cockroachdb/cockroach/pkg/util/metric.Struct>, 0x3ad9320 <UcharType,debug/dwarf.Type>, 0x0, 0x0, 0x3adb1e0 <com/gogo/protobuf/proto.Message>, 0x0, 0x3a95de0 <Rpath,debug/macho.Load>,
0x3a963e0 <com/cockroachdb/cockroach/pkg/roachpb.combinable>, 0x3b28d60 <com/cockroachdb/cockroach/pkg/util/protoutil.Message>, 0x3ada420 <com/gogo/protobuf/proto.Message>,
0x3a95c40 <Writer,io.Writer>, 0x3a95be0 <reader,io.Reader>, 0x3ac3520 <intValue,flag.Value>, 0x3ac2fe0 <endOfEarlyDataMsg,crypto/tls.handshakeMessage>,
0x3b294e0 <com/cockroachdb/cockroach/pkg/kv.txnInterceptor>, 0x3a96320 <com/cockroachdb/cockroach/pkg/keys.errUglifyUnsupported,error>, 0x3a95cc0 <PSSOptions,crypto.SignerOpts>,
0x3a96300 <com/cockroachdb/cockroach/pkg/keys.InvalidRangeMetaKeyError,error>, 0x0, 0x3adb0a0 <com/gogo/protobuf/proto.Message>, 0x3a96120 <byteCounter,io.Reader>,
0x3ada760 <com/cockroachdb/cockroach/pkg/jobs/jobspb.isPayload_Details>, 0x0, 0x3a95b80 <Buffer,io.Writer>, 0x0, 0x0, 0x0, 0x3b05260 <gcmAsm,crypto/cipher.AEAD>,
0x3a95e00 <Segment,debug/macho.Load>, 0x3b056a0 <com/cockroachdb/cockroach/pkg/cli.logStream>, 0x3b29660 <com/cockroachdb/cockroach/pkg/kv.txnInterceptor>, 0x3ad9160 <FloatType,debug/dwarf.Type>,
0x3ada8a0 <Payload_CreateStats,interface { MarshalTo([]uint8) (int, error); Size() int; jobspb.isPayload_Details() }>, 0x3a95fa0 <*encoding/xml.UnsupportedTypeError,error>,
0x3ad9c60 <com/cockroachdb/cockroach/pkg/cli.rowStrIter>, 0x0, 0x3b11620 <mergedStream,container/heap.Interface>, 0x3ada060 <com/gogo/protobuf/proto.Message>,
0x3ac3760 <com/cockroachdb/cockroach/pkg/sql/distsqlrun.Processor>, 0x0, 0x0, 0x0, 0x3ad9920 <com/cockroachdb/cockroach/pkg/ccl/importccl.inputConverter>, 0x3b113e0 <xorNonceAEAD,crypto/tls.aead>,
0x3a96280 <com/cockroachdb/cockroach/pkg/storage/bulk.sender>, 0x0, 0x0, 0x3ad9020 <BoolType,debug/dwarf.Type>, 0x0, 0x3a95ba0 <Reader,io.Reader>, 0x3ac2ae0 <com/gogo/protobuf/proto.writer>,
0x3ac2b20 <com/golang/protobuf/proto.writer>, 0x3ad91a0 <FuncType,debug/dwarf.Type>, 0x3ac3860 <com/cockroachdb/cockroach/pkg/cli/systembench.worker>,
0x3b114a0 <com/cockroachdb/cockroach/pkg/sql/distsqlrun.RowSource>, 0x3ad9fa0 <com/gogo/protobuf/proto.Message>, 0x3b3b060 <com/cockroachdb/cockroach/pkg/gossip.Gossip_GossipServer>, 0x0,
0x3ad8d20 <com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.BufferedReader>, 0x3a95c20 <Reader,io.Reader>, 0x3ac2b60 <Reader,io.ByteScanner>,
0x3ada960 <com/cockroachdb/cockroach/pkg/jobs/jobspb.isPayload_Details>, 0x3adac20 <com/cockroachdb/cockroach/pkg/jobs/jobspb.isProgress_Progress>, 0x0, 0x3ac2c20 <Reader,io.ReadCloser>,
0x3ada320 <com/gogo/protobuf/proto.Message>, 0x3ada3a0 <com/gogo/protobuf/proto.Message>, 0x3adada0 <com/cockroachdb/cockroach/pkg/jobs/jobspb.isProgress_Details>, 0x0, 0x0,
0x3ac34e0 <int64Value,flag.Value>, 0x3b28f40 <com/cockroachdb/cockroach/pkg/ccl/storageccl.ExportStorage>, 0x0, 0x3ada6a0 <com/gogo/protobuf/proto.Message>,
0x3b28d00 <com/cockroachdb/cockroach/pkg/util/protoutil.Message>, 0x3ad8f60 <x25519Parameters,crypto/tls.ecdheParameters>, 0x3adaba0 <com/cockroachdb/cockroach/pkg/jobs/jobspb.isProgress_Details>,
0x3ac3360 <taggedEncoder,encoding/asn1.encoder>, 0x3ac3820 <com/cockroachdb/cockroach/pkg/cli/systembench.worker>, 0x0, 0x0, 0x0, 0x0, 0x3ada660 <com/gogo/protobuf/proto.Message>, 0x0, 0x0,
0x3ad8ea0 <tripleDESCipher,crypto/cipher.Block>, 0x3a961a0 <com/cockroachdb/cockroach/pkg/cli.initialSQLConnectionError,error>,
0x3adaae0 <Progress_Backup,interface { MarshalTo([]uint8) (int, error); Size() int; jobspb.isProgress_Details() }>, 0x0, 0x0, 0x0,
0x3a960e0 <com/cockroachdb/cockroach/pkg/ccl/changefeedccl.retryableError,error>, 0x3a96100 <com/cockroachdb/cockroach/pkg/sql/distsqlplan/replicaoracle.OracleFactory>,
0x3a95ca0 <devReader,io.Reader>, 0x3ad8ee0 <listener,net.Listener>, 0x3b28ee0 <com/cockroachdb/cockroach/pkg/ccl/storageccl.ExportStorage>, 0x3ac32e0 <driverConn,sync.Locker>,
0x3b29060 <com/cockroachdb/cockroach/pkg/util/protoutil.Message>, 0x0, 0x3ac3160 <nextProtoMsg,crypto/tls.handshakeMessage>, 0x3a95c80 <zr,io.Reader>,
0x3ada7a0 <Payload_Backup,interface { MarshalTo([]uint8) (int, error); Size() int; jobspb.isPayload_Details() }>, 0x3b11320 <digest,hash.Hash>,
0x3a95e60 <*encoding/json.InvalidUnmarshalError,error>, 0x3ada860 <com/cockroachdb/cockroach/pkg/jobs/jobspb.isPayload_Details>, 0x3ada4e0 <com/gogo/protobuf/proto.Message>,
0x3ad9ce0 <com/spf13/pflag.Value>, 0x0, 0x0, 0x0, 0x0, 0x3adab20 <com/cockroachdb/cockroach/pkg/jobs/jobspb.isProgress_Details>, 0x3adaf60 <com/gogo/protobuf/proto.Message>, 0x0, 0x0,
0x3ada1a0 <com/gogo/protobuf/proto.Message>, 0x3ada1e0 <com/golang/protobuf/proto.Message>, 0x3ad9a20 <com/gogo/protobuf/proto.Message>, 0x3b116e0 <com/cockroachdb/cockroach/pkg/cli.rowReporter>,
0x3a960c0 <com/cockroachdb/cockroach/pkg/base.ModuleTestingKnobs>, 0x3b057a0 <com/cockroachdb/cockroach/pkg/sqlmigrations.leaseManager>,
0x3a961c0 <com/cockroachdb/cockroach/pkg/cli/systembench/systembenchpb.PingerServer>, 0x3a95f20 <encodeState,io.Writer>, 0x0, 0x0,
0x3a96240 <com/cockroachdb/cockroach/pkg/internal/client.AutoCommitError,error>, 0x3ada9a0 <Payload_Restore,interface { MarshalTo([]uint8) (int, error); Size() int; jobspb.isPayload_Details() }>,
0x0, 0x3adaf20 <com/gogo/protobuf/proto.Message>, 0x3adaaa0 <com/cockroachdb/cockroach/pkg/jobs/jobspb.isProgress_Details>, 0x3b3eea0 <com/cockroachdb/cockroach/pkg/roachpb.Response>,
0x3ac2e20 <certificateRequestMsg,crypto/tls.handshakeMessage>, 0x3ac3260 <Conn,database/sql.stmtConnGrabber>, 0x0, 0x3ad91e0 <IntType,debug/dwarf.Type>...}}
runtime.statictmp_54 = 0
runtime.statictmp_55 = 0
runtime.statictmp_56 = 0
runtime.statictmp_57 = {str = 0x0, len = 0}
runtime.statictmp_58 = {array = 0x0, len = 0, cap = 0}
runtime.statictmp_49 = {0x33448d4 "mSpanDead", 0x3346c01 "mSpanInUse", 0x3348ce0 "mSpanManual", 0x33448dd "mSpanFree"}
runtime.statictmp_59 = {
str = 0x3358dfe "index out of rangeindicator_overflowinformation_schemainitially deferredinput/output errorinspecting enginesinvalid IP addressinvalid XML name: invalid bool valueinvalid character invalid criteria: in"..., len = 18}
runtime.statictmp_60 = {
str = 0x336c8e1 "slice bounds out of rangesnappy: unsupported inputsocket type not supportedsql.distsql.queries.totalsql.large_lshift_argumentsql.large_rshift_argumentsql.plan.hints.merge-joinsql.plan.ops.bin.%s %s %s"..., len = 25}
runtime.statictmp_51 = {
str = 0x336339c "integer divide by zerointents.abort-attemptsinterface conversion: internal inconsistencyinternal-planner.%s.%sinvalid UTF-8 sequenceinvalid address familyinvalid cast: %s -> %sinvalid csv chunk sizein"..., len = 22}
runtime.statictmp_52 = {
str = 0x3353c62 "integer overflowintent missing%sinternal error: internal_metricsinterrupted (%v)invalid %q valueinvalid %s valueinvalid CRC typeinvalid address invalid argumentinvalid bool: %dinvalid code: %qinvalid "..., len = 16}
runtime.statictmp_53 = {
str = 0x335dd46 "floating point errorforcegc: phase errorforeign-data wrapperform-data; name=\"%s\"found a delta of %+vfound certificate %sgRPC requires HTTP/2gc_trigger underflowget s3 object headerget-table-statistics"..., len = 20}
runtime.statictmp_50 = {
str = 0x33c0a7d "invalid memory address or nil pointer dereferenceinverted indexes don't support interleaved tableskafka server: The fetch session ID was not found.kafka server: The fetch session epoch is invalid.kafk"..., len = 49}
runtime.g0 = {stack = {lo = 140737486255416, hi = 140737488348512}, stackguard0 = 140737486256296, stackguard1 = 140737486256296, _panic = 0x0, _defer = 0x0, m = 0x5d1c780 <runtime.m0>, sched = {
sp = 140737488348472, pc = 6175855, g = 97631424, ctxt = 0x0, ret = 0, lr = 0, bp = 0}, syscallsp = 0, syscallpc = 0, stktopsp = 0, param = 0x0, atomicstatus = 0, stackLock = 0, goid = 0,
schedlink = 0, waitsince = 0, waitreason = 0 '\000', preempt = false, paniconfault = false, preemptscan = false, gcscandone = false, gcscanvalid = false, throwsplit = false, raceignore = 0 '\000',
sysblocktraced = false, sysexitticks = 0, traceseq = 0, tracelastp = 0, lockedm = 0, sig = 0, writebuf = {array = 0x0, len = 0, cap = 0}, sigcode0 = 0, sigcode1 = 0, sigpc = 0, gopc = 0,
ancestors = 0x0, startpc = 0, racectx = 0, waiting = 0x0, cgoCtxt = {array = 0x0, len = 0, cap = 0}, labels = 0x0, timer = 0x0, selectDone = 0, gcAssistBytes = -11520}
runtime.statictmp_61 = {{name = 0x334ea66 "allocfreetrace", value = 0x5d5f800 <runtime.debug>}, {name = 0x33485a8 "clobberfree", value = 0x5d5f808 <runtime.debug+8>}, {name = 0x3341ed3 "cgocheck",
value = 0x5d5f804 <runtime.debug+4>}, {name = 0x333d68c "efence", value = 0x5d5f80c <runtime.debug+12>}, {name = 0x334897b "gccheckmark", value = 0x5d5f810 <runtime.debug+16>}, {
name = 0x334aaf9 "gcpacertrace", value = 0x5d5f814 <runtime.debug+20>}, {name = 0x3353a52 "gcshrinkstackoff", value = 0x5d5f818 <runtime.debug+24>}, {name = 0x334f120 "gcstoptheworld",
value = 0x5d5f81c <runtime.debug+28>}, {name = 0x333fd2e "gctrace", value = 0x5d5f820 <runtime.debug+32>}, {name = 0x3346b11 "invalidptr", value = 0x5d5f824 <runtime.debug+36>}, {
--Type <RET> for more, q to quit, c to continue without paging--
name = 0x334ade1 "madvdontneed", value = 0x5d5f828 <runtime.debug+40>}, {name = 0x333a54b "sbrk", value = 0x5d5f82c <runtime.debug+44>}, {name = 0x3342aa3 "scavenge",
value = 0x5d5f830 <runtime.debug+48>}, {name = 0x334912c "scheddetail", value = 0x5d5f834 <runtime.debug+52>}, {name = 0x3346fad "schedtrace", value = 0x5d5f838 <runtime.debug+56>}, {
name = 0x3359974 "tracebackancestors", value = 0x5d5f83c <runtime.debug+60>}}
runtime.lfenceBeforeRdtsc = true
_cgo_callers = 0x0
#25 0x00000008072b1e40 in ?? ()
No symbol table info available.
#26 0x0000000000200000 in ?? ()
No symbol table info available.
#27 0x0000000000000000 in ?? ()
No symbol table info available.
Took a core some minutes later and inspected the last thread again:
(gdb) thread 22
[Switching to thread 22 (LWP 101916)]
#0 0x000000000283bae2 in rocksdb::(anonymous namespace)::LevelIterator::InitFileIterator(unsigned long) ()
(gdb) bt
#0 0x000000000283bae2 in rocksdb::(anonymous namespace)::LevelIterator::InitFileIterator(unsigned long) ()
#1 0x000000000283b8ae in rocksdb::(anonymous namespace)::LevelIterator::SeekForPrev(rocksdb::Slice const&) ()
#2 0x0000000002952886 in SeekForPrev () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/iterator_wrapper.h:71
#3 Prev () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/table/merging_iterator.cc:231
#4 0x00000000027919d2 in FindUserKeyBeforeSavedKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/db_iter.cc:1225
#5 0x0000000002791479 in ReverseToBackward () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/db_iter.cc:823
#6 0x000000000279122d in Prev () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/db_iter.cc:741
#7 0x000000000269d123 in iterPeekPrev () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:612
#8 0x000000000269da8c in prevKey () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:403
#9 0x000000000269b608 in scan () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.h:136
#10 0x0000000002696433 in MVCCScan () at /home/bra/go/src/github.com/cockroachdb/cockroach/c-deps/libroach/mvcc.cc:294
#11 0x000000000254af3d in _cgo_46b63153f027_Cfunc_MVCCScan (v=0xc00e6b8fe8) at cgo-gcc-prolog:1348
#12 0x000000000060de60 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:635
#13 0x0000000005d2ab80 in ?? ()
#14 0x0000000000000001 in ?? ()
#15 0x0000000805a50022 in ?? ()
#16 0x000000080d853b10 in ?? ()
#17 0x000000c00e6b9c88 in ?? ()
#18 0x00000000000030d8 in ?? ()
#19 0x000000c008756000 in ?? ()
#20 0x00000000005e3c00 in ?? () at /usr/local/go/src/runtime/proc.go:1082
#21 0x00000008072b1e40 in ?? ()
#22 0x0000000000200000 in ?? ()
#23 0x0000000000000000 in ?? ()
Please tell me if you need anything else.
Hope this helps.
bra-fsn
on 12 Jul 2019
Please tell me if you need anything else.
Hope this helps.
Thanks a lot for all the info you provided. Haven't found a cause yet but this certainly helps us know where to look.
ajkr
on 13 Jul 2019
👍1
@ajkr should I leave this cluster (and process) running, or the core file will be enough for further investigation?
I would like to try to reproduce https://github.com/cockroachdb/cockroach/issues/38778 to report back similar details.
bra-fsn
on 15 Jul 2019
@bra-fsn I can't think of anything more to ask for so IMO restarting is fine. Reporting similar details for #38778 would be great.
ajkr
on 15 Jul 2019
OK, restarting and trying to reproduce #38778.
bra-fsn
on 16 Jul 2019
So far I could only reproduce this one (now with v19.1.3). It's strange that it happened on the same machine again (they are equal in every aspect, both HW and SW wise) and AFAIR with the same node ID (1), the first one which I started the cluster with. Don't know whether it has any relevance, or not...
bra-fsn
on 17 Jul 2019
New infos:
in the last two runs, the problem started at around the same place during inserting (I'm inserting the same data for testing purposes):
First run:
7100477, 325.97, ins:8.46/425.60/24.60/16.01, del:1.78/307.55/8.57/3.76
7200484, 355.83, ins:8.30/683.17/26.52/15.83, del:1.64/453.58/8.06/3.73
7300484, 361.60, ins:8.09/687.86/25.74/15.37, del:1.59/707.56/8.07/3.51
7400517, 72.30, ins:7.15/179283.45/4861.63/16.29, del:1.37/122192.72/525.73/3.48
7500519, 66.40, ins:6.64/716673.78/5046.21/17.53, del:1.26/662574.21/743.98/3.47
7600548, 64.34, ins:6.69/658901.89/5259.56/18.50, del:1.28/839805.38/740.65/3.35
7700548, 64.12, ins:6.67/779129.70/5435.86/24.17, del:1.32/601857.04/767.23/3.39
7800598, 57.49, ins:6.58/721165.06/5292.19/29.08, del:1.26/1375491.88/984.29/3.91
Second run:
7200274, 135.35, ins:7.60/655.53/19.14/13.68, del:1.40/418.25/6.26/2.80
7300276, 131.77, ins:7.50/445.91/18.50/13.67, del:1.39/266.00/6.40/2.82
7400296, 126.96, ins:7.64/118804.71/247.50/13.78, del:1.53/115377.98/40.71/2.80
7500301, 57.52, ins:6.79/179163.88/5219.98/39.57, del:1.30/241990.84/632.32/4.41
7600320, 57.45, ins:6.97/418404.88/5247.66/33.54, del:1.25/657997.37/783.17/3.80
7700341, 61.70, ins:7.29/419806.13/5053.59/36.43, del:1.30/303213.16/769.29/3.79
7800344, 52.08, ins:6.36/1018092.44/5302.60/42.55, del:1.26/645580.14/833.97/4.90
So after around 7.2 million operations (these cover two kind of transactions, one for multiple inserts and another for deletes, details in #38778) the phenomenon kicks in reliably.
Also, maximum response times grow significantly, but that's no surprise in the light that one node eats all of the resources.
And so far it happened always on the first machine (now I started CRDB nodes in different order, so it's not node number 1). I don't know how CRDB works internally, but if it counts, it has the lowest IP number and lowest port in the cluster.
As mentioned above: I'm not aware of any differences between the machines, they are netbooted from exactly the same image and have the same HW.
Any news on this?
bra-fsn
on 18 Jul 2019
I did eventually find an infinite loop bug that produces the same stack traces as yours did. It is fixed in https://github.com/cockroachdb/cockroach/pull/39084. Thanks again for the bug report and all the info - without it this bug could've gone unfixed for a long time.
Are you building from source? If so it'd be very interesting to hear whether that patch fixes the issue for your workload. If not, well it should be available in an upcoming 19.1.x release.
ajkr
on 24 Jul 2019
Thanks @ajkr, restarted the data load and it still runs, never got so far before the fix.
bra-fsn
on 25 Jul 2019
🎉1
@ajkr : checked out v19.2.0-alpha.20190805 and I can't see this change in that. May I use that version for further testing (it contains a fix in a different form) or should I apply the fix by hand?
bra-fsn
on 6 Aug 2019
Hi @bra-fsn, thanks for pointing that out. I will try to make sure it's included in the next 19.2 alpha release. In the meantime you can apply the patch manually. Sorry for the inconvenience!
ajkr
on 6 Aug 2019
👍1
My issue is similar with 20.1.6: https://github.com/cockroachdb/cockroach/issues/56362. Usually, when this happens I saw KV store latency instability and growth of Goutines count. Experienced this at least 2 times.
sitano
on 28 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
otan
路
4Comments
xudongzheng
路
3Comments
melskyzy
路
3Comments
couchand
路
3Comments
ajwerner
路
4Comments