Etcd: Benchmarking is causing cluster to become unresponsive
Running into a very strange issue when benchmarking my ETCD 3.1.0 3 node cluster.
./benchmark --endpoints=$ENDPOINTS --conns=100 --clients=1000 put --key-size=100
--key-space-size=10 --sequential-keys --total=1000000 --val-size=20`
755885 / 1000000 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooom ! 75.59% 1m34s
There has been a few times where this completes without issue, but most of the time it ends up putting my cluster into an unresponsive state, and the benchmarking progress comes to a halt. There's no logs that indicate an issue, it simply just becomes unresponsive until I finally start restarting nodes.
The one thing that seems to be consistent is the logs before becoming unresponsive.
pkg/fileutil: purged file /data/etcd/member/snap/0000000000000003-00000000000b2807.snap successfully
pkg/fileutil: purged file /data/etcd/member/snap/0000000000000003-00000000000b00f0.snap successfully
pkg/fileutil: purged file /data/etcd/member/snap/0000000000000003-00000000000b2807.snap successfully
Please let me know if I can provide you any additional information.
arebug
davissp14
All 12 comments
putting my cluster into an unresponsive state
Can you double-check with other client requests?
If benchmark command hangs, you can check etcd server logs or do some etcdctl commands
ETCDCTL_API=3 etcdctl endpoint health
ETCDCTL_API=3 etcdctl put foo bar
ETCDCTL_API=3 etcdctl get foo
We use this tool quite a lot, but I've never had this issue.
gyuho
on 10 Mar 2017
@davissp14 Can you provide us detailed steps to reproduce this issue (including your platform information)
xiang90
on 10 Mar 2017
I am running 3 nodes in a containerized environment (LXC) fronted with 2 etcd proxies, each node is on a separate ec2 host running Ubuntu 14.04. The 3 member nodes are running with 256MB memory (limit set by cgroups) and has no CPU / Disk / IOPS limits in place.
The only node that appears to lock up is the leader. Commands run just fine on all non-leader nodes, while commands on the leader hang indefinitely.
I am able to reproduce the issue consistency by simply running the benchmark command against an empty Etcd on the setup described above.
./benchmark --endpoints=$ENDPOINTS --conns=100 --clients=1000 put --key-size=100
--key-space-size=10 --sequential-keys --total=1000000 --val-size=20
~ Boot Info ~
etcdmain: etcd Version: 3.1.0
etcdmain: Git SHA: 8ba2897
etcdmain: Go Version: go1.7.4
etcdmain: Go OS/Arch: linux/amd64
etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 16
etcdmain: the server is already initialized as member before, starting as etcd member...
embed: listening for peers on http://10.46.172.132:2380
embed: listening for client requests on 10.46.172.132:2379
embed: listening for client requests on 127.0.0.1:2379
etcdserver: name = xxxx
etcdserver: data dir = /data/etcd
etcdserver: member dir = /data/etcd/member
etcdserver: heartbeat = 500ms
etcdserver: election = 5000ms
etcdserver: snapshot count = 10000
~ Strace of hanging leader node~
[pid 55] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid 54] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid 55] <... write resumed> ) = 60
[pid 54] <... write resumed> ) = 60
[pid 28] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid 28] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 54] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid 54] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 29] write(16, "1\r\n\0\r\n", 6) = 6
[pid 54] write(17, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 42) = 42
[pid 29] write(13, "1\r\n\0\r\n", 6) = 6
[pid 54] write(14, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 42) = 42
[pid 29] write(20, "GET /version HTTP/1.1\r\nHost: 10."..., 106) = 106
[pid 28] read(20, "HTTP/1.1 200 OK\r\nContent-Type: a"..., 4096) = 152
[pid 28] read(20, 0xc421bfb000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28] write(21, "GET /version HTTP/1.1\r\nHost: 10."..., 106) = 106
[pid 55] read(21, "HTTP/1.1 200 OK\r\nContent-Type: a"..., 4096) = 152
[pid 55] read(21, 0xc421c03000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 29] read(19, "1\r\n\0\r\n", 4096) = 6
[pid 29] read(19, 0xc4215c9000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 29] read(15, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 4096) = 42
[pid 29] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 55] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid 28] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid 55] <... write resumed> ) = 60
[pid 28] <... write resumed> ) = 60
[pid 29] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid 28] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid 29] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 29] read(12, "1\r\n\0\r\n", 4096) = 6
[pid 29] read(12, 0xc420591000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 29] read(49, "GET /raft/probing HTTP/1.1\r\nHost"..., 4096) = 111
[pid 29] write(49, "HTTP/1.1 200 OK\r\nDate: Fri, 10 M"..., 168) = 168
[pid 29] read(49, 0xc420db3000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 29] read(10, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 4096) = 42
[pid 29] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 55] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid 54] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid 55] <... write resumed> ) = 60
[pid 54] <... write resumed> ) = 60
[pid 29] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid 54] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid 29] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 54] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 55] write(18, "GET /raft/probing HTTP/1.1\r\nHost"..., 111) = 111
[pid 54] read(18, "HTTP/1.1 200 OK\r\nDate: Fri, 10 M"..., 4096) = 168
[pid 54] read(18, 0xc420618000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 55] write(48, "GET /raft/probing HTTP/1.1\r\nHost"..., 111) = 111
[pid 29] read(48, "HTTP/1.1 200 OK\r\nDate: Fri, 10 M"..., 4096) = 168
[pid 29] read(48, 0xc421c0b000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 55] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60) = 60
[pid 28] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60) = 60
[pid 54] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid 28] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid 54] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 54] read(50, "GET /raft/probing HTTP/1.1\r\nHost"..., 4096) = 111
[pid 54] write(50, "HTTP/1.1 200 OK\r\nDate: Fri, 10 M"..., 168) = 168
[pid 54] read(50, 0xc421c13000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 29] write(17, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 42) = 42
[pid 55] write(16, "1\r\n\0\r\n", 6) = 6
[pid 29] write(13, "1\r\n\0\r\n", 6) = 6
[pid 55] write(14, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 42) = 42
[pid 54] read(19, "1\r\n\0\r\n", 4096) = 6
[pid 54] read(19, 0xc4215c9000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 54] read(15, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 4096) = 42
[pid 54] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 29] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid 28] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid 29] <... write resumed> ) = 60
[pid 28] <... write resumed> ) = 60
[pid 54] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid 54] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 54] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid 54] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28] read(12, "1\r\n\0\r\n", 4096) = 6
[pid 28] read(12, 0xc420591000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28] read(10, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 4096) = 42
[pid 28] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 29] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60) = 60
[pid 54] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60) = 60
[pid 28] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid 28] read(15, <unfinished ...>
[pid 54] read(10, <unfinished ...>
[pid 28] <... read resumed> 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 54] <... read resumed> "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid 54] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 29] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60) = 60
[pid 55] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60) = 60
[pid 28] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid 28] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 55] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid 55] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 55] write(16, "1\r\n\0\r\n", 6) = 6
[pid 54] write(17, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 42) = 42
[pid 55] write(13, "1\r\n\0\r\n", 6) = 6
[pid 54] write(14, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 42) = 42
[pid 55] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60) = 60
[pid 29] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60) = 60
[pid 28] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid 28] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 54] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid 54] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28] read(19, "1\r\n\0\r\n", 4096) = 6
[pid 28] read(19, 0xc4215c9000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28] read(15, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 4096) = 42
[pid 28] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 29] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid 55] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid 29] <... write resumed> ) = 60
[pid 55] <... write resumed> ) = 60
[pid 28] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid 55] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid 28] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 55] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 55] read(12, "1\r\n\0\r\n", 4096) = 6
[pid 55] read(12, 0xc420591000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 55] read(10, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 4096) = 42
[pid 55] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 54] write(20, "GET /version HTTP/1.1\r\nHost: 10."..., 106) = 106
[pid 55] read(20, "HTTP/1.1 200 OK\r\nContent-Type: a"..., 4096) = 152
[pid 55] read(20, 0xc421bfb000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 55] write(21, "GET /version HTTP/1.1\r\nHost: 10."..., 106) = 106
[pid 29] read(21, "HTTP/1.1 200 OK\r\nContent-Type: a"..., 4096) = 152
[pid 29] read(21, 0xc421c03000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 29] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid 54] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60) = 60
[pid 29] <... write resumed> ) = 60
[pid 55] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid 55] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 55] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid 55] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid 29] read(49, "GET /raft/probing HTTP/1.1\r\nHost"..., 4096) = 111
[pid 29] write(49, "HTTP/1.1 200 OK\r\nDate: Fri, 10 M"..., 168) = 168
[pid 29] read(49, 0xc420db3000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
I went ahead and updated to 3.1.2 to see if it helped. While it's not hanging like it was in 3.1.0, I am now seeing a panic.
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | panic: runtime error: invalid memory address or nil pointer dereference
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x8bdda8]
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 |
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | goroutine 2184 [running]:
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | panic(0xd15980, 0xc42000c0e0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | /usr/local/go/src/runtime/panic.go:500 +0x1a1
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).IsAdminPermitted(0xc4201f64d0, 0x0, 0x0, 0x0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:772 +0xc8
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*authMaintenanceServer).isAuthenticated(0xc420297c80, 0x7f88486169d0, 0xc4228e27e0, 0x12e0ec0, 0xc4215369c0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/maintenance.go:160 +0xae
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*authMaintenanceServer).Status(0xc420297c80, 0x7f88486169d0, 0xc4228e27e0, 0x132c548, 0x5, 0xe588f1, 0x18)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/maintenance.go:188 +0x43
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._Maintenance_Status_Handler.func1(0x7f88486169d0, 0xc4228e27e0, 0xd9f580, 0x132c548, 0xc420a90730, 0xc421536a38, 0x9bc1a6, 0xc4228e27e0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:3025 +0xa2
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor(0x7f88486169d0, 0xc4228e27e0, 0xd9f580, 0x132c548, 0xc421a4b4c0, 0xc421a4b4e0, 0x40f9e8, 0x20, 0xd479a0, 0x1)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29 +0xa4
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1(0x7f88486169d0, 0xc4228e27e0, 0xd9f580, 0x132c548, 0xc421a4b4c0, 0xc421a4b4e0, 0x0, 0xc420029000, 0x4f8934, 0xc421536b28)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:57 +0xc3
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._Maintenance_Status_Handler(0xd6a840, 0xc420297c80, 0x7f88486169d0, 0xc4228e27e0, 0xc420a90690, 0xc4202afe60, 0x0, 0x0, 0xc42008b6b0, 0x7f8ac87bdc08)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:3027 +0x166
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc42043e180, 0x12e3500, 0xc4201930e0, 0xc4215dc5a0, 0xc4202b5200, 0x1301398, 0x0, 0x0, 0x0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:638 +0xc14
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc42043e180, 0x12e3500, 0xc4201930e0, 0xc4215dc5a0, 0x0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:796 +0x6b0
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc421a2ec80, 0xc42043e180, 0x12e3500, 0xc4201930e0, 0xc4215dc5a0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:449 +0xab
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | created by github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19 | /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:450 +0xa3
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: start.13 | etcd.19 process exited.
Commands run just fine on all non-leader nodes, while commands on the leader hang indefinitely.
Which commands? Do you mean benchmark commands or etcdctl commands? Trying to figure out whether the etcd server is hanging or benchmark commands.
gyuho
on 10 Mar 2017
@davissp14 Is this cluster totally fresh? That path would only crash if auth is enabled...
Is the hang sensitive to the arguments? Does --conns=10 work? --clients=100? I ran it locally and the memory utilization is under 150MB.
Commands run just fine on all non-leader nodes, while commands on the leader hang indefinitely.
Commands sent to the leader node or commands running on the leader node? ETCDCTL_API=3 etcdctl get foo or ...? What is the output from ETCDCTL_API=3 ./bin/etcdctl --endpoints=$ENDPOINTS endpoint health?
Thanks!
heyitsanthony
on 10 Mar 2017
@gyuho Both the benchmark commands as well as the etcdctl commands hang on the leader node in 3.1.0. I am able to write to the non-leader nodes without issues, which seems strange. It appears that the server-to-server communication running on 2380 is working just fine, it's just client requests on 2379 are being blocked.
davissp14
on 10 Mar 2017
The last benchmark I ran actually managed to throw 2 of the 3 nodes into an unresponsive state (leader included). I am still able to read and write to the responsive node without problems.
root $ETCDCTL_API=3 etcdctl --endpoints=10.11.221.131:2379,10.11.221.130:2379,10.11.221.132:2379 --user=root:XXX endpoint health
10.11.221.132:2379 is healthy: successfully committed proposal: took = 3.081202ms
After reporting that the 1 responsive node is healthy, the endpoint health command just hangs. Running this on all 3 member nodes as well as the 2 proxies and each produce the same result.
@heyitsanthony This was a fresh cluster with auth enabled. I will play with the --conns value and see if it makes any difference.
UPDATE
@heyitsanthony Changing --conns 10 did not make a difference :(
davissp14
on 10 Mar 2017
Here is the output of curl localhost:2379/debug/pprof/goroutine?debug=1 from one of the unresponsive nodes.
goroutine profile: total 1163
996 @ 0x42cc8a 0x42cd7e 0x43ce8d 0x43cb20 0x74b39d 0x8bcace 0x8bfd99 0x5f6d9c 0x5f749a 0x5f2508 0x796788 0x79a3f2 0x70f6a2 0x61d224 0x79d783 0x6b2ae6 0x643254 0x644ef0 0x64a9fb 0x45de01
# 0x43cb1f sync.runtime_Semacquire+0x2f /usr/local/go/src/runtime/sema.go:47
# 0x74b39c sync.(*RWMutex).RLock+0x5c /usr/local/go/src/sync/rwmutex.go:43
# 0x8bcacd github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).AuthInfoFromToken+0x4d /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:649
# 0x8bfd98 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).AuthInfoFromCtx+0x108 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:989
# 0x5f6d9b github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).processInternalRaftRequestOnce+0x13b /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:655
# 0x5f7499 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).processInternalRaftRequest+0x89 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:703
# 0x5f2507 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).Put+0xc7 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:142
# 0x796787 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*kvServer).Put+0x87 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/key.go:67
# 0x79a3f1 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Put+0xa1 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/quota.go:62
# 0x70f6a1 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._KV_Put_Handler.func1+0xa1 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2305
# 0x61d223 github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor+0xa3 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29
# 0x79d782 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1+0xc2 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:57
# 0x6b2ae5 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._KV_Put_Handler+0x165 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2307
# 0x643253 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC+0xc13 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:638
# 0x644eef github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleStream+0x6af /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:796
# 0x64a9fa github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1+0xaa /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:449
14 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fa161 0x50b1e0 0x60e1eb 0x736d93 0x866aec 0x86714c 0x744074 0x7441d8 0x9283cb 0x928c7e 0x9b662f 0x9af450 0x641b10 0x64196a 0x641550 0x45de01
# 0x426ca8 net.runtime_pollWait+0x58 /usr/local/go/src/runtime/netpoll.go:160
# 0x4f88b7 net.(*pollDesc).wait+0x37 /usr/local/go/src/net/fd_poll_runtime.go:73
# 0x4f8923 net.(*pollDesc).waitRead+0x33 /usr/local/go/src/net/fd_poll_runtime.go:78
# 0x4fa160 net.(*netFD).Read+0x1a0 /usr/local/go/src/net/fd_unix.go:243
# 0x50b1df net.(*conn).Read+0x6f /usr/local/go/src/net/net.go:173
# 0x60e1ea github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*limitListenerConn).Read+0x6a <autogenerated>:16
# 0x736d92 github.com/coreos/etcd/cmd/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read+0x112 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/cockroachdb/cmux/cmux.go:204
# 0x866aeb bufio.(*Reader).fill+0x10b /usr/local/go/src/bufio/bufio.go:97
# 0x86714b bufio.(*Reader).Read+0x1bb /usr/local/go/src/bufio/bufio.go:209
# 0x744073 io.ReadAtLeast+0xa3 /usr/local/go/src/io/io.go:307
# 0x7441d7 io.ReadFull+0x57 /usr/local/go/src/io/io.go:325
# 0x9283ca github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2.readFrameHeader+0x7a /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2/frame.go:237
# 0x928c7d github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame+0x8d /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2/frame.go:469
# 0x9b662e github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*framer).readFrame+0x2e /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/http_util.go:508
# 0x9af44f github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams+0x1ff /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/http2_server.go:260
# 0x641b0f github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams+0x15f /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:451
# 0x641969 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveNewHTTP2Transport+0x3d9 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:438
# 0x64154f github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleRawConn+0x46f /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:415
14 @ 0x42cc8a 0x43c2b5 0x43b04c 0x9b365c 0x45de01
# 0x9b365b github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*http2Server).controller+0x69b /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/http2_server.go:674
12 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fa161 0x50b1e0 0x866aec 0x86714c 0x744074 0x7441d8 0x9283cb 0x928c7e 0x9b662f 0x9ac7c6 0x45de01
# 0x426ca8 net.runtime_pollWait+0x58 /usr/local/go/src/runtime/netpoll.go:160
# 0x4f88b7 net.(*pollDesc).wait+0x37 /usr/local/go/src/net/fd_poll_runtime.go:73
# 0x4f8923 net.(*pollDesc).waitRead+0x33 /usr/local/go/src/net/fd_poll_runtime.go:78
# 0x4fa160 net.(*netFD).Read+0x1a0 /usr/local/go/src/net/fd_unix.go:243
# 0x50b1df net.(*conn).Read+0x6f /usr/local/go/src/net/net.go:173
# 0x866aeb bufio.(*Reader).fill+0x10b /usr/local/go/src/bufio/bufio.go:97
# 0x86714b bufio.(*Reader).Read+0x1bb /usr/local/go/src/bufio/bufio.go:209
# 0x744073 io.ReadAtLeast+0xa3 /usr/local/go/src/io/io.go:307
# 0x7441d7 io.ReadFull+0x57 /usr/local/go/src/io/io.go:325
# 0x9283ca github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2.readFrameHeader+0x7a /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2/frame.go:237
# 0x928c7d github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame+0x8d /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2/frame.go:469
# 0x9b662e github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*framer).readFrame+0x2e /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/http_util.go:508
# 0x9ac7c5 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*http2Client).reader+0xb5 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/http2_client.go:937
12 @ 0x42cc8a 0x43c2b5 0x43b04c 0x63c6ae 0x64a4de 0x45de01
# 0x63c6ad github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*addrConn).transportMonitor+0x63d /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/clientconn.go:751
# 0x64a4dd github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn.func1+0x1dd /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/clientconn.go:506
12 @ 0x42cc8a 0x43c2b5 0x43b04c 0x9ad76d 0x45de01
# 0x9ad76c github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*http2Client).controller+0x59c /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/http2_client.go:1015
8 @ 0x42cc8a 0x43c2b5 0x43b04c 0x57c37d 0x45de01
# 0x57c37c net/http.(*persistConn).writeLoop+0x3bc /usr/local/go/src/net/http/transport.go:1646
8 @ 0x42cc8a 0x43c2b5 0x43b04c 0x7821b7 0x45de01
# 0x7821b6 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*pipeline).handle+0x686 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/pipeline.go:80
4 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fa161 0x50b1e0 0x60bb8d 0x60f43b 0x579bb4 0x866aec 0x866cd2 0x57a7d1 0x45de01
# 0x426ca8 net.runtime_pollWait+0x58 /usr/local/go/src/runtime/netpoll.go:160
# 0x4f88b7 net.(*pollDesc).wait+0x37 /usr/local/go/src/net/fd_poll_runtime.go:73
# 0x4f8923 net.(*pollDesc).waitRead+0x33 /usr/local/go/src/net/fd_poll_runtime.go:78
# 0x4fa160 net.(*netFD).Read+0x1a0 /usr/local/go/src/net/fd_unix.go:243
# 0x50b1df net.(*conn).Read+0x6f /usr/local/go/src/net/net.go:173
# 0x60bb8c github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.timeoutConn.Read+0x5c /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43
# 0x60f43a github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read+0x8a <autogenerated>:42
# 0x579bb3 net/http.(*persistConn).Read+0x153 /usr/local/go/src/net/http/transport.go:1261
# 0x866aeb bufio.(*Reader).fill+0x10b /usr/local/go/src/bufio/bufio.go:97
# 0x866cd1 bufio.(*Reader).Peek+0x61 /usr/local/go/src/bufio/bufio.go:129
# 0x57a7d0 net/http.(*persistConn).readLoop+0x1a0 /usr/local/go/src/net/http/transport.go:1418
4 @ 0x42cc8a 0x42cd7e 0x403838 0x4035fd 0x8bcb99 0x8bfd99 0x5f6d9c 0x5f749a 0x5f2508 0x796788 0x79a3f2 0x70f6a2 0x61d224 0x79d783 0x6b2ae6 0x643254 0x644ef0 0x64a9fb 0x45de01
# 0x8bcb98 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).AuthInfoFromToken+0x118 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:653
# 0x8bfd98 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).AuthInfoFromCtx+0x108 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:989
# 0x5f6d9b github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).processInternalRaftRequestOnce+0x13b /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:655
# 0x5f7499 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).processInternalRaftRequest+0x89 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:703
# 0x5f2507 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).Put+0xc7 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:142
# 0x796787 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*kvServer).Put+0x87 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/key.go:67
# 0x79a3f1 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Put+0xa1 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/quota.go:62
# 0x70f6a1 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._KV_Put_Handler.func1+0xa1 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2305
# 0x61d223 github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor+0xa3 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29
# 0x79d782 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1+0xc2 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:57
# 0x6b2ae5 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._KV_Put_Handler+0x165 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2307
# 0x643253 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC+0xc13 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:638
# 0x644eef github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleStream+0x6af /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:796
# 0x64a9fa github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1+0xaa /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:449
4 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x77ccb5 0x56ab3f 0x56ab3f 0x56b62d 0x567fd7 0x45de01
# 0x77ccb4 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*streamHandler).ServeHTTP+0xba4 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/http.go:323
# 0x56ab3e net/http.(*ServeMux).ServeHTTP+0x7e /usr/local/go/src/net/http/server.go:2022
# 0x56ab3e net/http.(*ServeMux).ServeHTTP+0x7e /usr/local/go/src/net/http/server.go:2022
# 0x56b62c net/http.serverHandler.ServeHTTP+0x7c /usr/local/go/src/net/http/server.go:2202
# 0x567fd6 net/http.(*conn).serve+0x4b6 /usr/local/go/src/net/http/server.go:1579
4 @ 0x42cc8a 0x43c2b5 0x43b04c 0x57aff9 0x45de01
# 0x57aff8 net/http.(*persistConn).readLoop+0x9c8 /usr/local/go/src/net/http/transport.go:1541
4 @ 0x42cc8a 0x43c2b5 0x43b04c 0x786519 0x45de01
# 0x786518 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*streamWriter).run+0x17e8 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/stream.go:149
2 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fa161 0x50b1e0 0x60bb8d 0x60f43b 0x562b44 0x866aec 0x867795 0x867917 0x86347e 0x86327f 0x55d5b5 0x563ebd 0x567ef3 0x45de01
# 0x426ca8 net.runtime_pollWait+0x58 /usr/local/go/src/runtime/netpoll.go:160
# 0x4f88b7 net.(*pollDesc).wait+0x37 /usr/local/go/src/net/fd_poll_runtime.go:73
# 0x4f8923 net.(*pollDesc).waitRead+0x33 /usr/local/go/src/net/fd_poll_runtime.go:78
# 0x4fa160 net.(*netFD).Read+0x1a0 /usr/local/go/src/net/fd_unix.go:243
# 0x50b1df net.(*conn).Read+0x6f /usr/local/go/src/net/net.go:173
# 0x60bb8c github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.timeoutConn.Read+0x5c /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43
# 0x60f43a github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read+0x8a <autogenerated>:42
# 0x562b43 net/http.(*connReader).Read+0x143 /usr/local/go/src/net/http/server.go:586
# 0x866aeb bufio.(*Reader).fill+0x10b /usr/local/go/src/bufio/bufio.go:97
# 0x867794 bufio.(*Reader).ReadSlice+0xb4 /usr/local/go/src/bufio/bufio.go:330
# 0x867916 bufio.(*Reader).ReadLine+0x36 /usr/local/go/src/bufio/bufio.go:359
# 0x86347d net/textproto.(*Reader).readLineSlice+0x5d /usr/local/go/src/net/textproto/reader.go:55
# 0x86327e net/textproto.(*Reader).ReadLine+0x2e /usr/local/go/src/net/textproto/reader.go:36
# 0x55d5b4 net/http.readRequest+0xa4 /usr/local/go/src/net/http/request.go:793
# 0x563ebc net/http.(*conn).readRequest+0x10c /usr/local/go/src/net/http/server.go:765
# 0x567ef2 net/http.(*conn).serve+0x3d2 /usr/local/go/src/net/http/server.go:1532
2 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fa161 0x50b1e0 0x60bb8d 0x60f43b 0x579bb4 0x866aec 0x867795 0x87db64 0x87d5a2 0x87d80b 0x571771 0x5716b6 0x57d816 0x744074 0x7441d8 0x77e60a 0x78764e 0x787154 0x45de01
# 0x426ca8 net.runtime_pollWait+0x58 /usr/local/go/src/runtime/netpoll.go:160
# 0x4f88b7 net.(*pollDesc).wait+0x37 /usr/local/go/src/net/fd_poll_runtime.go:73
# 0x4f8923 net.(*pollDesc).waitRead+0x33 /usr/local/go/src/net/fd_poll_runtime.go:78
# 0x4fa160 net.(*netFD).Read+0x1a0 /usr/local/go/src/net/fd_unix.go:243
# 0x50b1df net.(*conn).Read+0x6f /usr/local/go/src/net/net.go:173
# 0x60bb8c github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.timeoutConn.Read+0x5c /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43
# 0x60f43a github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read+0x8a <autogenerated>:42
# 0x579bb3 net/http.(*persistConn).Read+0x153 /usr/local/go/src/net/http/transport.go:1261
# 0x866aeb bufio.(*Reader).fill+0x10b /usr/local/go/src/bufio/bufio.go:97
# 0x867794 bufio.(*Reader).ReadSlice+0xb4 /usr/local/go/src/bufio/bufio.go:330
# 0x87db63 net/http/internal.readChunkLine+0x33 /usr/local/go/src/net/http/internal/chunked.go:110
# 0x87d5a1 net/http/internal.(*chunkedReader).beginChunk+0x31 /usr/local/go/src/net/http/internal/chunked.go:47
# 0x87d80a net/http/internal.(*chunkedReader).Read+0x7a /usr/local/go/src/net/http/internal/chunked.go:77
# 0x571770 net/http.(*body).readLocked+0x60 /usr/local/go/src/net/http/transfer.go:651
# 0x5716b5 net/http.(*body).Read+0xf5 /usr/local/go/src/net/http/transfer.go:643
# 0x57d815 net/http.(*bodyEOFSignal).Read+0x115 /usr/local/go/src/net/http/transport.go:1974
# 0x744073 io.ReadAtLeast+0xa3 /usr/local/go/src/io/io.go:307
# 0x7441d7 io.ReadFull+0x57 /usr/local/go/src/io/io.go:325
# 0x77e609 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*msgAppV2Decoder).decode+0xd9 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/msgappv2_codec.go:179
# 0x78764d github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*streamReader).decodeLoop+0x20d /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/stream.go:358
# 0x787153 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*streamReader).run+0x6f3 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/stream.go:311
2 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fa161 0x50b1e0 0x60bb8d 0x60f43b 0x579bb4 0x866aec 0x867795 0x87db64 0x87d5a2 0x87d80b 0x571771 0x5716b6 0x57d816 0x744074 0x7441d8 0x86e650 0x77d912 0x78764e 0x787154 0x45de01
# 0x426ca8 net.runtime_pollWait+0x58 /usr/local/go/src/runtime/netpoll.go:160
# 0x4f88b7 net.(*pollDesc).wait+0x37 /usr/local/go/src/net/fd_poll_runtime.go:73
# 0x4f8923 net.(*pollDesc).waitRead+0x33 /usr/local/go/src/net/fd_poll_runtime.go:78
# 0x4fa160 net.(*netFD).Read+0x1a0 /usr/local/go/src/net/fd_unix.go:243
# 0x50b1df net.(*conn).Read+0x6f /usr/local/go/src/net/net.go:173
# 0x60bb8c github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.timeoutConn.Read+0x5c /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43
# 0x60f43a github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read+0x8a <autogenerated>:42
# 0x579bb3 net/http.(*persistConn).Read+0x153 /usr/local/go/src/net/http/transport.go:1261
# 0x866aeb bufio.(*Reader).fill+0x10b /usr/local/go/src/bufio/bufio.go:97
# 0x867794 bufio.(*Reader).ReadSlice+0xb4 /usr/local/go/src/bufio/bufio.go:330
# 0x87db63 net/http/internal.readChunkLine+0x33 /usr/local/go/src/net/http/internal/chunked.go:110
# 0x87d5a1 net/http/internal.(*chunkedReader).beginChunk+0x31 /usr/local/go/src/net/http/internal/chunked.go:47
# 0x87d80a net/http/internal.(*chunkedReader).Read+0x7a /usr/local/go/src/net/http/internal/chunked.go:77
# 0x571770 net/http.(*body).readLocked+0x60 /usr/local/go/src/net/http/transfer.go:651
# 0x5716b5 net/http.(*body).Read+0xf5 /usr/local/go/src/net/http/transfer.go:643
# 0x57d815 net/http.(*bodyEOFSignal).Read+0x115 /usr/local/go/src/net/http/transport.go:1974
# 0x744073 io.ReadAtLeast+0xa3 /usr/local/go/src/io/io.go:307
# 0x7441d7 io.ReadFull+0x57 /usr/local/go/src/io/io.go:325
# 0x86e64f encoding/binary.Read+0x11f /usr/local/go/src/encoding/binary/binary.go:168
# 0x77d911 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*messageDecoder).decode+0x111 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/msg_codec.go:53
# 0x78764d github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*streamReader).decodeLoop+0x20d /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/stream.go:358
# 0x787153 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*streamReader).run+0x6f3 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/stream.go:311
2 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fba78 0x515ece 0x514619 0x60944f 0x6090b7 0x7363e4 0x49adb9 0x49cced 0x45de01
# 0x426ca8 net.runtime_pollWait+0x58 /usr/local/go/src/runtime/netpoll.go:160
# 0x4f88b7 net.(*pollDesc).wait+0x37 /usr/local/go/src/net/fd_poll_runtime.go:73
# 0x4f8923 net.(*pollDesc).waitRead+0x33 /usr/local/go/src/net/fd_poll_runtime.go:78
# 0x4fba77 net.(*netFD).accept+0x237 /usr/local/go/src/net/fd_unix.go:419
# 0x515ecd net.(*TCPListener).accept+0x2d /usr/local/go/src/net/tcpsock_posix.go:132
# 0x514618 net.(*TCPListener).Accept+0x48 /usr/local/go/src/net/tcpsock.go:222
# 0x60944e github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*limitListener).Accept+0x5e /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/limit_listen.go:46
# 0x6090b6 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*keepaliveListener).Accept+0x36 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/keepalive_listener.go:49
# 0x7363e3 github.com/coreos/etcd/cmd/vendor/github.com/cockroachdb/cmux.(*cMux).Serve+0xa3 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/cockroachdb/cmux/cmux.go:110
# 0x49adb8 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.(*serveCtx).serve+0x238 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/serve.go:119
# 0x49ccec github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.(*Etcd).serve.func2+0x6c /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/etcd.go:330
2 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x712068 0x45de01
# 0x712067 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterKVHandlerFromEndpoint.func1.1+0x57 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:586
2 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x7141b8 0x45de01
# 0x7141b7 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterWatchHandlerFromEndpoint.func1.1+0x57 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:783
2 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x714c68 0x45de01
# 0x714c67 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterLeaseHandlerFromEndpoint.func1.1+0x57 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:852
2 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x7168e8 0x45de01
# 0x7168e7 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterClusterHandlerFromEndpoint.func1.1+0x57 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1017
2 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x718448 0x45de01
# 0x718447 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterMaintenanceHandlerFromEndpoint.func1.1+0x57 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1182
2 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x71a6b8 0x45de01
# 0x71a6b7 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterAuthHandlerFromEndpoint.func1.1+0x57 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1379
2 @ 0x42cc8a 0x42cd7e 0x40452f 0x404235 0x737e1f 0x56baae 0x49ce0f 0x45de01
# 0x737e1e github.com/coreos/etcd/cmd/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept+0x7e <autogenerated>:9
# 0x56baad net/http.(*Server).Serve+0x1cd /usr/local/go/src/net/http/server.go:2273
# 0x49ce0e github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.(*serveCtx).serve.func2+0x3e /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/serve.go:87
2 @ 0x42cc8a 0x42cd7e 0x40452f 0x404235 0x737e1f 0x640c37 0x49cd7f 0x45de01
# 0x737e1e github.com/coreos/etcd/cmd/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept+0x7e <autogenerated>:9
# 0x640c36 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).Serve+0x186 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:354
# 0x49cd7e github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.(*serveCtx).serve.func1+0x3e /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/serve.go:70
2 @ 0x42cc8a 0x42cd7e 0x40452f 0x404235 0xb553b4 0x45de01
# 0xb553b3 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/logutil.(*MergeLogger).outputLoop+0x93 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/logutil/merge_logger.go:174
2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x606864 0x45de01
# 0x606863 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/fileutil.purgeFile.func1+0x853 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/fileutil/purge.go:70
2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x783080 0x45de01
# 0x78307f github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.monitorProbingStatus+0x54f /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/probing_status.go:51
2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x78dede 0x45de01
# 0x78dedd github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.startPeer.func1+0x28d /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/peer.go:156
2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x78e17e 0x45de01
# 0x78e17d github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.startPeer.func2+0x28d /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/peer.go:172
2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x79bbe5 0x6b39ab 0x61d411 0x79dab5 0x644116 0x645476 0x64a9fb 0x45de01
# 0x79bbe4 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*watchServer).Watch+0x4d4 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:139
# 0x6b39aa github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._Watch_Watch_Handler+0xba /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2459
# 0x61d410 github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.StreamServerInterceptor+0x100 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:40
# 0x79dab4 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1+0x114 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:93
# 0x644115 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).processStreamingRPC+0x2b5 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:723
# 0x645475 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleStream+0xc35 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:800
# 0x64a9fa github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1+0xaa /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:449
2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x79d265 0x79e32d 0x45de01
# 0x79d264 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*serverWatchStream).sendLoop+0xd94 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:255
# 0x79e32c github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*watchServer).Watch.func1+0x2c /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:125
2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x79e1c2 0x45de01
# 0x79e1c1 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.monitorLeader.func1+0x391 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:115
2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x9b706a 0x9b7a59 0x744074 0x7441d8 0x63d53f 0x63dffd 0x648c68 0x61d5fb 0x6b4a72 0x797d98 0x79fddd 0x6b494b 0x61d411 0x79dab5 0x644116 0x645476 0x64a9fb 0x45de01
# 0x9b7069 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read+0x699 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/transport.go:140
# 0x9b7a58 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*Stream).Read+0x58 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/transport.go:325
# 0x744073 io.ReadAtLeast+0xa3 /usr/local/go/src/io/io.go:307
# 0x7441d7 io.ReadFull+0x57 /usr/local/go/src/io/io.go:325
# 0x63d53e github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*parser).recvMsg+0x6e /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/rpc_util.go:231
# 0x63dffc github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.recv+0x4c /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/rpc_util.go:315
# 0x648c67 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*serverStream).RecvMsg+0xf7 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/stream.go:516
# 0x61d5fa github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*monitoredServerStream).RecvMsg+0x4a /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:69
# 0x6b4a71 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.(*leaseLeaseKeepAliveServer).Recv+0x61 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2657
# 0x797d97 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*LeaseServer).LeaseKeepAlive+0x37 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/lease.go:65
# 0x79fddc github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*quotaLeaseServer).LeaseKeepAlive+0x5c <autogenerated>:27
# 0x6b494a github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._Lease_LeaseKeepAlive_Handler+0xba /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2638
# 0x61d410 github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.StreamServerInterceptor+0x100 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:40
# 0x79dab4 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1+0x114 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:93
# 0x644115 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).processStreamingRPC+0x2b5 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:723
# 0x645475 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleStream+0xc35 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:800
# 0x64a9fa github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1+0xaa /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:449
2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x9b706a 0x9b7a59 0x744074 0x7441d8 0x63d53f 0x63dffd 0x648c68 0x79ed3d 0x61d5fb 0x6b3ad2 0x79bc8c 0x79e37d 0x45de01
# 0x9b7069 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read+0x699 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/transport.go:140
# 0x9b7a58 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*Stream).Read+0x58 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/transport.go:325
# 0x744073 io.ReadAtLeast+0xa3 /usr/local/go/src/io/io.go:307
# 0x7441d7 io.ReadFull+0x57 /usr/local/go/src/io/io.go:325
# 0x63d53e github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*parser).recvMsg+0x6e /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/rpc_util.go:231
# 0x63dffc github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.recv+0x4c /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/rpc_util.go:315
# 0x648c67 github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*serverStream).RecvMsg+0xf7 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/stream.go:516
# 0x79ed3c github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*serverStreamWithCtx).RecvMsg+0x5c <autogenerated>:1
# 0x61d5fa github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*monitoredServerStream).RecvMsg+0x4a /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:69
# 0x6b3ad1 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.(*watchWatchServer).Recv+0x61 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2478
# 0x79bc8b github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*serverWatchStream).recvLoop+0x4b /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:155
# 0x79e37c github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*watchServer).Watch.func2+0x2c /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:135
2 @ 0x42cc8a 0x43c2b5 0x43b04c 0xb57ec5 0x45de01
# 0xb57ec4 github.com/coreos/etcd/cmd/vendor/github.com/xiang90/probing.(*prober).AddHTTP.func1+0x4b4 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/xiang90/probing/prober.go:55
1 @ 0x40e152 0x440257 0x922772 0x45de01
# 0x440256 os/signal.signal_recv+0x156 /usr/local/go/src/runtime/sigqueue.go:116
# 0x922771 os/signal.loop+0x21 /usr/local/go/src/os/signal/signal_unix.go:22
1 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fba78 0x515ece 0x514619 0x60c037 0x56baae 0x49bbd2 0x49cc2d 0x45de01
# 0x426ca8 net.runtime_pollWait+0x58 /usr/local/go/src/runtime/netpoll.go:160
# 0x4f88b7 net.(*pollDesc).wait+0x37 /usr/local/go/src/net/fd_poll_runtime.go:73
# 0x4f8923 net.(*pollDesc).waitRead+0x33 /usr/local/go/src/net/fd_poll_runtime.go:78
# 0x4fba77 net.(*netFD).accept+0x237 /usr/local/go/src/net/fd_unix.go:419
# 0x515ecd net.(*TCPListener).accept+0x2d /usr/local/go/src/net/tcpsock_posix.go:132
# 0x514618 net.(*TCPListener).Accept+0x48 /usr/local/go/src/net/tcpsock.go:222
# 0x60c036 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*rwTimeoutListener).Accept+0x36 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/timeout_listener.go:49
# 0x56baad net/http.(*Server).Serve+0x1cd /usr/local/go/src/net/http/server.go:2273
# 0x49bbd1 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.servePeerHTTP+0x161 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/serve.go:142
# 0x49cc2c github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.(*Etcd).serve.func1+0x4c /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/etcd.go:317
1 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x6089c1 0x45de01
# 0x6089c0 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/osutil.HandleInterrupts.func1+0x70 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/osutil/interrupt_unix.go:51
1 @ 0x42cc8a 0x42cd7e 0x43ce8d 0x43cb20 0x74a4f0 0x74b4d1 0x8b7cf3 0x8b8f37 0x5d588a 0x6017a3 0x5fdbd3 0x5d10e1 0x601633 0x5d6d82 0x5ecd6e 0x5ecaa1 0x5e95e7 0x5e7884 0x5f9db0 0x91c20d 0x45de01
# 0x43cb1f sync.runtime_Semacquire+0x2f /usr/local/go/src/runtime/sema.go:47
# 0x74a4ef sync.(*Mutex).Lock+0xcf /usr/local/go/src/sync/mutex.go:85
# 0x74b4d0 sync.(*RWMutex).Lock+0x30 /usr/local/go/src/sync/rwmutex.go:86
# 0x8b7cf2 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).assignSimpleTokenToUser+0x52 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/simple_token.go:119
# 0x8b8f36 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).Authenticate+0x406 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:289
# 0x5d5889 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*applierV3backend).Authenticate+0x289 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply.go:622
# 0x6017a2 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*quotaApplierV3).Authenticate+0x52 <autogenerated>:174
# 0x5fdbd2 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*authApplierV3).Authenticate+0x52 <autogenerated>:66
# 0x5d10e0 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*applierV3backend).Apply+0x700 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply.go:118
# 0x601632 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*quotaApplierV3).Apply+0x52 <autogenerated>:171
# 0x5d6d81 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*authApplierV3).Apply+0xe1 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply_auth.go:55
# 0x5ecd6d github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyEntryNormal+0x28d /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1323
# 0x5ecaa0 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).apply+0x2c0 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1252
# 0x5e95e6 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyEntries+0xc6 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:891
# 0x5e7883 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyAll+0xc3 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:750
# 0x5f9daf github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).run.func6+0x3f /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:714
# 0x91c20c github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule.(*fifo).run+0xdc /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule/schedule.go:160
1 @ 0x42cc8a 0x42cd7e 0x43ce8d 0x43cb20 0x74a4f0 0x8dbc4d 0x8d9a1b 0x45de01
# 0x43cb1f sync.runtime_Semacquire+0x2f /usr/local/go/src/runtime/sema.go:47
# 0x74a4ef sync.(*Mutex).Lock+0xcf /usr/local/go/src/sync/mutex.go:85
# 0x8dbc4c github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend.(*batchTx).Commit+0x2c /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend/batch_tx.go:138
# 0x8d9a1a github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend.(*backend).run+0x13a /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend/backend.go:199
1 @ 0x42cc8a 0x42cd7e 0x43ce8d 0x43cb20 0x74b538 0x8bffac 0x8b7998 0x45de01
# 0x43cb1f sync.runtime_Semacquire+0x2f /usr/local/go/src/runtime/sema.go:47
# 0x74b537 sync.(*RWMutex).Lock+0x97 /usr/local/go/src/sync/rwmutex.go:91
# 0x8bffab github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.newDeleterFunc.func1+0x5b /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:182
# 0x8b7997 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*simpleTokenTTLKeeper).run+0x557 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/simple_token.go:91
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x455683 0x45de01
# 0x42cc89 runtime.gopark+0x139 /usr/local/go/src/runtime/proc.go:259
# 0x43c2b4 runtime.selectgoImpl+0x1234 /usr/local/go/src/runtime/select.go:423
# 0x43b04b runtime.selectgo+0x1b /usr/local/go/src/runtime/select.go:238
# 0x455682 runtime.ensureSigM.func1+0x2f2 /usr/local/go/src/runtime/signal1_unix.go:304
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x462ca0 0x468861 0x401014 0x42c824 0x45de01
# 0x462c9f github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2+0xc8f /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:171
# 0x468860 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.Main+0x60 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/main.go:39
# 0x401013 main.main+0x13 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/etcd/main.go:28
# 0x42c823 runtime.main+0x1f3 /usr/local/go/src/runtime/proc.go:183
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x5ddbfa 0x5f9861 0x5fb086 0x45de01
# 0x5ddbf9 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.monitorFileDescriptor+0x1b9 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/metrics.go:89
# 0x5f9860 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).Start.func2+0x30 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:515
# 0x5fb085 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach.func1+0x55 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1638
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x5e653a 0x5f981a 0x5fb086 0x45de01
# 0x5e6539 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).purgeFile+0x2c9 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:553
# 0x5f9819 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).(github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.purgeFile)-fm+0x29 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:514
# 0x5fb085 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach.func1+0x55 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1638
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x5e74b7 0x45de01
# 0x5e74b6 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).run+0x946 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:712
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x5ee19b 0x5f98aa 0x5fb086 0x45de01
# 0x5ee19a github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).monitorVersions+0x49a /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1488
# 0x5f98a9 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).(github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.monitorVersions)-fm+0x29 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:516
# 0x5fb085 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach.func1+0x55 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1638
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x5f7e41 0x5f98ea 0x5fb086 0x45de01
# 0x5f7e40 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).linearizableReadLoop+0x870 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:773
# 0x5f98e9 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).(github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.linearizableReadLoop)-fm+0x29 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:517
# 0x5fb085 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach.func1+0x55 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1638
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x5f941f 0x45de01
# 0x5f941e github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*raftNode).start.func1+0xdbe /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/raft.go:145
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x7af0e7 0x45de01
# 0x7af0e6 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/wal.(*filePipeline).run+0x196 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/wal/file_pipeline.go:89
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x8809f4 0x45de01
# 0x8809f3 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/lease.(*lessor).runLoop+0x193 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/lease/lessor.go:447
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x88d6e1 0x45de01
# 0x88d6e0 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc.(*watchableStore).syncWatchersLoop+0x200 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/watchable_store.go:281
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x88d8ff 0x45de01
# 0x88d8fe github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc.(*watchableStore).syncVictimsLoop+0x1ce /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/watchable_store.go:307
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x8e4e9b 0x45de01
# 0x8e4e9a github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*node).run+0x107a /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/node.go:309
1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x91c457 0x45de01
# 0x91c456 github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule.(*fifo).run+0x326 /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule/schedule.go:146
1 @ 0xb71f3f 0xb71d40 0xb6ea61 0x794116 0x794709 0x56ab3f 0x56b62d 0x567fd7 0x45de01
# 0xb71f3e runtime/pprof.writeRuntimeProfile+0x9e /usr/local/go/src/runtime/pprof/pprof.go:614
# 0xb71d3f runtime/pprof.writeGoroutine+0x9f /usr/local/go/src/runtime/pprof/pprof.go:576
# 0xb6ea60 runtime/pprof.(*Profile).WriteTo+0x340 /usr/local/go/src/runtime/pprof/pprof.go:298
# 0x794115 net/http/pprof.handler.ServeHTTP+0x1a5 /usr/local/go/src/net/http/pprof/pprof.go:209
# 0x794708 net/http/pprof.(*handler).ServeHTTP+0x78 <autogenerated>:1
# 0x56ab3e net/http.(*ServeMux).ServeHTTP+0x7e /usr/local/go/src/net/http/server.go:2022
# 0x56b62c net/http.serverHandler.ServeHTTP+0x7c /usr/local/go/src/net/http/server.go:2202
# 0x567fd6 net/http.(*conn).serve+0x4b6 /usr/local/go/src/net/http/server.go:1579
@davissp14 Seems like a but in Auth layer. Can you try to disable auth to see if it happens again?
xiang90
on 13 Mar 2017
There's a deadlock in simpleTokenTTLKeeper. assignSimpleTokenToUser acquires as.simpleTokensMu and posts to addSimpleTokenCh (suppose that the channel is full so it blocks). If the goroutine simpleTokenTTLKeeper.run happens to hit <-tokenTicker.C it will try to acquire simpleTokensMu while calling deleteTokenFunc. Since only run can drain addSimpleTokenCh, the lock is never released.
heyitsanthony
on 13 Mar 2017
👍2
@xiang90 Confirmed it works just fine when disabling Auth.
davissp14
on 13 Mar 2017
Was this page helpful?
0 / 5 - 0 ratings
Related issues
gek0
路
3Comments
itnikita
路
3Comments
kghost
路
4Comments
ghost
路
3Comments
yuchengwu
路
4Comments
Most helpful comment
There's a deadlock in
simpleTokenTTLKeeper.assignSimpleTokenToUseracquiresas.simpleTokensMuand posts toaddSimpleTokenCh(suppose that the channel is full so it blocks). If the goroutinesimpleTokenTTLKeeper.runhappens to hit<-tokenTicker.Cit will try to acquiresimpleTokensMuwhile callingdeleteTokenFunc. Since onlyruncan drainaddSimpleTokenCh, the lock is never released.