Go: runtime: SIGILL: illegal instruction from runtime.deductSweepCredit

Created on 22 Jan 2019 · 19Comments · Source: golang/go

What version of Go are you using (`go version`)?

$ go version
go version go1.11.4 linux/amd64

Does this issue reproduce with the latest release?

Yes, go 1.11.4. But happened only once.

What operating system and processor architecture are you using (`go env`)?

go env Output



$ go env

/usr/local/go1.11.4/go/bin# ./go env

GOARCH="amd64"

GOBIN=""

GOCACHE="/root/.cache/go-build"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GOOS="linux"

GOPATH="/var/jenkins_home/workspace"

GOPROXY=""

GORACE=""

GOROOT="/usr/local/go1.11.4/go"

GOTMPDIR=""

GOTOOLDIR="/usr/local/go1.11.4/go/pkg/tool/linux_amd64"

GCCGO="gccgo"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD=""

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build108139547=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Nothing specific was being done. This code has been running unmodified for few days without any issue.

What did you expect to see?

No panic

What did you see instead?

SIGILL: illegal instruction
PC=0x420c60 m=8 sigcode=2

goroutine 0 [idle]:
runtime.deductSweepCredit(0x2000, 0x0)
/usr/local/go1.11.4/go/src/runtime/mgcsweep.go:390
runtime.(mcentral).cacheSpan(0x1385618, 0xc0005cfca8)
/usr/local/go1.11.4/go/src/runtime/mcentral.go:43 +0x60
runtime.(mcache).refill(0x7f95da4e3000, 0xc0005cfc11)
/usr/local/go1.11.4/go/src/runtime/mcache.go:122 +0x95
runtime.(*mcache).nextFree.func1()
/usr/local/go1.11.4/go/src/runtime/malloc.go:749 +0x32
runtime.systemstack(0x0)
/usr/local/go1.11.4/go/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
/usr/local/go1.11.4/go/src/runtime/proc.go:1229

...
rax 0x2000
rbx 0x7f95d827c350
rcx 0x8
rdx 0x440
rdi 0x4553c0
rsi 0x137ddc0
rbp 0xc000399f70
rsp 0xc000399f30
r8 0x1
r9 0x11
r10 0x7f95d827c350
r11 0x7fffffffffffff
r12 0x0
r13 0x49
r14 0x49
r15 0x49
rip 0x420c60
rflags 0x10206
cs 0x33
fs 0x0
gs 0x0

Processor

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU @ 2.20GHz
stepping : 0
microcode : 0x1
cpu MHz : 2200.000
cache size : 56320 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb
rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid pni
pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes
xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch
invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle
avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat
arch_capabilities
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips : 4400.00
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

Runtime

Kubernetes 1.11.6 with Google COS os.

docker version
Client:
Version: 17.03.2-ce
API version: 1.27
Go version: go1.10.3
Git commit: f5ec1e2
Built: Thu Oct 25 10:42:32 2018
OS/Arch: linux/amd64
Server:
Version: 17.03.2-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.10.3
Git commit: f5ec1e2
Built: Thu Oct 25 10:42:32 2018
OS/Arch: linux/amd64
Experimental: false

Assembly a the PC

(gdb) disassemble 0x420c60
Dump of assembler code for function runtime.deductSweepCredit:
0x0000000000420c60 <+0>: mov %fs:0xfffffffffffffff8,%rcx
0x0000000000420c69 <+9>: cmp 0x10(%rcx),%rsp
0x0000000000420c6d <+13>: jbe 0x420dc8
0x0000000000420c73 <+19>: sub $0x20,%rsp
0x0000000000420c77 <+23>: mov %rbp,0x18(%rsp)
0x0000000000420c7c <+28>: lea 0x18(%rsp),%rbp
0x0000000000420c81 <+33>: movsd 0xf642cf(%rip),%xmm0 #
0x1384f58
0x0000000000420c89 <+41>: xorps %xmm1,%xmm1
0x0000000000420c8c <+44>: ucomisd %xmm1,%xmm0
0x0000000000420c90 <+48>: jne 0x420c98
0x0000000000420c92 <+50>: jnp 0x420dbe
0x0000000000420c98 <+56>: lea 0xf6a751(%rip),%rax #
0x138b3f0
0x0000000000420c9f <+63>: cmpb $0x0,(%rax)
0x0000000000420ca2 <+66>: jne 0x420daa
0x0000000000420ca8 <+72>: mov 0x28(%rsp),%rcx
0x0000000000420cad <+77>: mov 0x30(%rsp),%rdx
0x0000000000420cb2 <+82>: jmp 0x420d1c
0x0000000000420cb4 <+84>: lea 0xf6a735(%rip),%rax #
0x138b3f0
0x0000000000420cbb <+91>: mov 0x28(%rsp),%rcx
0x0000000000420cc0 <+96>: mov 0x30(%rsp),%rdx
0x0000000000420cc5 <+101>: mov 0x8(%rsp),%rbx
0x0000000000420cca <+106>: mov 0x10(%rsp),%rsi
0x0000000000420ccf <+111>: xorps %xmm1,%xmm1
0x0000000000420cd2 <+114>: mov 0xf64267(%rip),%rdi #
0x1384f40
0x0000000000420cd9 <+121>: sub %rbx,%rdi
0x0000000000420cdc <+124>: cmp %rdi,%rsi
0x0000000000420cdf <+127>: jle 0x420d72
0x0000000000420ce5 <+133>: callq 0x4201c0
0x0000000000420cea <+138>: cmpq $0xffffffffffffffff,(%rsp)
0x0000000000420cef <+143>: je 0x420d67
0x0000000000420cf1 <+145>: mov 0xf64250(%rip),%rax #
0x1384f48
0x0000000000420cf8 <+152>: mov 0x8(%rsp),%rcx
0x0000000000420cfd <+157>: cmp %rcx,%rax
0x0000000000420d00 <+160>: je 0x420cb4

FrozenDueToAge NeedsInvestigation

Source

vishwaspai

Most helpful comment

Yes, I am suggesting a hardware problem.

Of course if it happens on a different machine then my suggestion is almost certainly wrong. But a single occurrence of SIGILL on a single machine is likely a hardware problem. They do happen.

ianlancetaylor on 22 Jan 2019

👍4

All 19 comments

This code has been running unmodified for few days without any issue.

Which code ? How do we run it ? Could you show us a small self-contained code which shows the issue ? That will help us debug the issue further.

Have you tried with 1.12beta2 to see if the problem exists there too ?

Thanks.

agnivade on 22 Jan 2019

This code has been running unmodified for few days without any issue.

Which code ? How do we run it ? Could you show us a small self-contained code which shows the issue ? That will help us debug the issue further.

Have you tried with 1.12beta2 to see if the problem exists there too ?

Thanks.

Right now there nothing in this crash stack trace that gives me hints on creating a self contained snippet that makes this reproducible. Project is compiled as a static go binary running inside a pod in Kubernetes (environment details specified).

Problem happened once and if it starts repeating multiple times, we will experiment with 1.12 and see if this is specific to 1.11.4.

I would like to understand why the following go runtime is an illegal instruction (full disassembly provided).
runtime.deductSweepCredit:
0x0000000000420c60 <+0>: mov %fs:0xfffffffffffffff8,%rcx

vishwaspai on 22 Jan 2019

It maybe possible that there is a memory corruption somewhere. Please also try to run a race enabled binary to ensure there is no race anywhere.

Also, is that the entire stack trace ? There should also be some user-level code in it.

I will page some folks who can shed some light @ianlancetaylor @randall77 @aclements.

agnivade on 22 Jan 2019

Full stack trace.

https://github.com/golang/go/files/2782346/sigill.log

SIGILL: illegal instruction
PC=0x420c60 m=8 sigcode=2

goroutine 444 [running]:
runtime.systemstack_switch()
/usr/local/go1.11.4/go/src/runtime/asm_amd64.s:311 fp=0xc0005cf9a0 sp=0xc0005cf998 pc=0x4581d0
runtime.(mcache).nextFree(0x7f95da4e3000, 0x11, 0x1000100, 0xc0005cfa40, 0x40a03d)
/usr/local/go1.11.4/go/src/runtime/malloc.go:748 +0xb6 fp=0xc0005cf9f8 sp=0xc0005cf9a0 pc=0x40b756
runtime.mallocgc(0x70, 0xb96800, 0x1, 0x0)
/usr/local/go1.11.4/go/src/runtime/malloc.go:903 +0x793 fp=0xc0005cfa98 sp=0xc0005cf9f8 pc=0x40c0a3
runtime.makeslice(0xb96800, 0x65, 0x65, 0x0, 0x0, 0xa891ff)
/usr/local/go1.11.4/go/src/runtime/slice.go:70 +0x77 fp=0xc0005cfac8 sp=0xc0005cfa98 pc=0x442e07
github.com/1234567890123/ship/github.com/BruceLee/vishnu.encode(0xdb7160, 0xc0003cf140, 0xdc34a0, 0xc00056e800, 0x0, 0xb, 0x0, 0x0, 0xc0004c0c01)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/encoder_decoder.go:33 +0x1ba fp=0xc0005cfb40 sp=0xc0005cfac8 pc=0xa86c4a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).send(0xc0003ecb00, 0xdc0e60, 0xc0003526e0, 0xc000352701, 0x0, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:570 +0x24e fp=0xc0005cfc78 sp=0xc0005cfb40 pc=0xa6f53e
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).sendAndReceive(0xc0003ecb00, 0xdc0e60, 0xc0003526e0, 0xdb6f60, 0xc000352700, 0xc0003cebd0, 0x1371750)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:601 +0x6a fp=0xc0005cfd60 sp=0xc0005cfc78 pc=0xa6fa5a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).YeelK(0xc0003ecb00, 0xc0003526e0, 0xc000000006, 0xc00034d7d0, 0xc0004a2cb8)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:280 +0x70 fp=0xc0005cfdb0 sp=0xc0005cfd60 pc=0xa6df10
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).YeelKNewMessages(0xc000883130, 0x0, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:806 +0x3f7 fp=0xc0005cfea0 sp=0xc0005cfdb0 pc=0xa7e477
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionTTTTTTT(0xc000883130)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:692 +0x124 fp=0xc0005cffa0 sp=0xc0005cfea0 pc=0xa7ce74
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionTTTTTTT-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:633 +0x2a fp=0xc0005cffb8 sp=0xc0005cffa0 pc=0xaa98aa
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc000723430)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43 fp=0xc0005cffd8 sp=0xc0005cffb8 pc=0xaa5533
runtime.goexit()
/usr/local/go1.11.4/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0005cffe0 sp=0xc0005cffd8 pc=0x45a131
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(TTTTTTT).newsleeperTTTTTTT
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:633 +0x19c

goroutine 1 [IO wait, 16 minutes]:
internal/poll.runtime_pollWait(0x7f95d82773a0, 0x72, 0x0)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0002a3b18, 0x72, 0xc0004a1b00, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0002a3b18, 0xffffffffffffff00, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Accept(0xc0002a3b00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:384 +0x1a0
net.(netFD).accept(0xc0002a3b00, 0xc000646f60, 0x30, 0x50)
/usr/local/go1.11.4/go/src/net/fd_unix.go:238 +0x42
net.(TCPListener).accept(0xc0000948c8, 0x30, 0x30, 0xc6e400)
/usr/local/go1.11.4/go/src/net/tcpsock_posix.go:139 +0x2e
net.(TCPListener).AcceptTCP(0xc0000948c8, 0x0, 0xc00028e380, 0x50)
/usr/local/go1.11.4/go/src/net/tcpsock.go:247 +0x47
net/http.tcpKeepAliveListener.Accept(0xc0000948c8, 0xc000578000, 0xc000431990, 0xc000431988, 0x40c488)
/usr/local/go1.11.4/go/src/net/http/server.go:3232 +0x2f
crypto/tls.(listener).Accept(0xc000569800, 0xc6e400, 0xc000646f90, 0xbd1920, 0x1367060)
/usr/local/go1.11.4/go/src/crypto/tls/tls.go:52 +0x37
net/http.(Server).Serve(0xc000678270, 0xdbc120, 0xc000569800, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/http/server.go:2826 +0x22f
net/http.(Server).ServeTLS(0xc000678270, 0xdbf520, 0xc0000948c8, 0xce9444, 0x10, 0xce9454, 0x10, 0x0, 0x15)
/usr/local/go1.11.4/go/src/net/http/server.go:2891 +0x265
net/http.(Server).ListenAndServeTLS(0xc000678270, 0xce9444, 0x10, 0xce9454, 0x10, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/http/server.go:3048 +0x12f
github.com/1234567890123/lmn_xxx/httplib/svr.(*HttpSvc).Start(0xc0000a6000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/httplib/svr/httpsvr.go:567 +0x301
main.main()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/pqr_xx/poiuttr/main.go:202 +0x83d

goroutine 18 [syscall, 16 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go1.11.4/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
/usr/local/go1.11.4/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
/usr/local/go1.11.4/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 4 [chan receive, 16 minutes]:
github.com/1234567890123/lmn_xxx/st.dumpOnSig(0xc000096420)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/st/feelsleep.go:158 +0x5b
created by github.com/1234567890123/lmn_xxx/st.init.0
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/st/feelsleep.go:53 +0x16f

goroutine 5 [chan receive]:
github.com/1234567890123/lmn_xxx/st.dumpPeriodical()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/st/feelsleep.go:146 +0xba
created by github.com/1234567890123/lmn_xxx/st.init.0
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/st/feelsleep.go:54 +0x187

goroutine 6 [sleep]:
time.Sleep(0xdf8475800)
/usr/local/go1.11.4/go/src/runtime/time.go:105 +0x14f
github.com/1234567890123/lmn_xxx/st.FreeSleep()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/st/feelsleep.go:66 +0x30
created by github.com/1234567890123/lmn_xxx/st.init.0
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/st/feelsleep.go:55 +0x19f

goroutine 10 [select, 16 minutes]:
github.com/1234567890123/lmn_xxx/blog.(*blogger).drainingLogs(0xc00034dce0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/blog/blog.go:413 +0xb2
created by github.com/1234567890123/lmn_xxx/blog.LazyLog
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/blog/ntwrk_blog.go:103 +0x218

goroutine 130 [select, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(client).backgroundMetadataUpdater(0xc0003a4900)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/client.go:656 +0x12b
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(client).backgroundMetadataUpdater-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/client.go:165 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc00058fc00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.NewClient
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/client.go:165 +0x51d

goroutine 12 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(*healthBalancer).updateNotifyLoop(0xc000366200)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:339 +0x34e
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.newHealthBalancer
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:128 +0x2ed

goroutine 13 [select, 1 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(*healthBalancer).updateUnhealthy(0xc000366200)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:236 +0xb9
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.newHealthBalancer.func1(0xc000366200)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:132 +0x5a
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.newHealthBalancer
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:130 +0x32d

goroutine 14 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0003ce1e0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_conn_wrappers.go:117 +0xfa
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.newCCBalancerWrapper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_conn_wrappers.go:108 +0x120

goroutine 15 [chan receive, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(balancerWrapper).lbWatcher(0xc000077320)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_v1_wrapper.go:115 +0x186
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(balancerWrapperBuilder).Build
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_v1_wrapper.go:53 +0x1e4

goroutine 23 [IO wait, 16 minutes]:
internal/poll.runtime_pollWait(0x7f95d8277d60, 0x72, 0xc0003813e8)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0002a2d18, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0002a2d18, 0xc000365000, 0x400, 0x400)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0002a2d00, 0xc000365000, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0002a2d00, 0xc000365000, 0x400, 0x400, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc00000c300, 0xc000365000, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc0003cea80, 0xdb78e0, 0xc00000c300, 0x5, 0xc00000c300, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc00038a700, 0xd1d017, 0xc00038a820, 0x442e07)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc00038a700, 0xc0005a0800, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
github.com/1234567890123/lmn_xxx/blog.(ntwrkblog).conncheck(0xc00034dce0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/blog/ntwrk_blog.go:142 +0xad
created by github.com/1234567890123/lmn_xxx/blog.(ntwrkblog).reconnect
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/blog/ntwrk_blog.go:178 +0x3f2

goroutine 53 [IO wait]:
internal/poll.runtime_pollWait(0x7f95d8277f00, 0x72, 0xc0000aeaf0)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0002a2418, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0002a2418, 0xc0005bc000, 0x1000, 0x1000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0002a2400, 0xc0005bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0002a2400, 0xc0005bc000, 0x1000, 0x1000, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc0000943e8, 0xc0005bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc000716930, 0xdb78e0, 0xc0000943e8, 0x5, 0xc0000943e8, 0x44afba)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc0000bca80, 0xd1d017, 0xc0000bcba0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc0000bca80, 0xc0005dc9c0, 0x24, 0x24, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.fill(0xdc30e0, 0xc0000bca80, 0xc0005dc9c0, 0x24, 0x24, 0x0, 0x11)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:535 +0x53
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.(*mongoSocket).readLoop(0xc0005c2a80)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:551 +0x5f7
created by github.com/1234567890123/ship/gopkg.in/mgo%2ev2.newSocket
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:194 +0x1df

goroutine 35 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(addrConn).transportMonitor(0xc00033c600)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/clientconn.go:1012 +0x122
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(addrConn).connect.func1(0xc00033c600)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/clientconn.go:703 +0x1b2
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(*addrConn).connect
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/clientconn.go:694 +0x3a3

goroutine 43 [IO wait, 16 minutes]:
internal/poll.runtime_pollWait(0x7f95d8277c90, 0x72, 0xc0006c6998)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc000400218, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc000400218, 0xc000365800, 0x400, 0x400)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc000400200, 0xc000365800, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc000400200, 0xc000365800, 0x400, 0x400, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc00000c308, 0xc000365800, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc0003ceb40, 0xdb78e0, 0xc00000c308, 0x5, 0xc00000c308, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc00038aa80, 0xd1d017, 0xc00038aba0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc00038aa80, 0xc000744000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
bufio.(Reader).Read(0xc0003fe4e0, 0xc0005c2498, 0x9, 0x9, 0x30, 0x28, 0xc60160)
/usr/local/go1.11.4/go/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0xdb6960, 0xc0003fe4e0, 0xc0005c2498, 0x9, 0x9, 0x9, 0xc00071a280, 0x402a6e, 0xc0006c6eab)
/usr/local/go1.11.4/go/src/io/io.go:310 +0x88
io.ReadFull(0xdb6960, 0xc0003fe4e0, 0xc0005c2498, 0x9, 0x9, 0x81f22a, 0xc000716840, 0xc000710004)
/usr/local/go1.11.4/go/src/io/io.go:329 +0x58
github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2.readFrameHeader(0xc0005c2498, 0x9, 0x9, 0xdb6960, 0xc0003fe4e0, 0x0, 0x0, 0xc000716840, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2.(Framer).ReadFrame(0xc0005c2460, 0xc000716840, 0xc000716840, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(*http2Client).reader(0xc000099200)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:1118 +0xbf
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.newHTTP2Client
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:266 +0xb7d

goroutine 44 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.loopyWriter(0xdbf720, 0xc000072640, 0xc000716240, 0xc0006c7fb8)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:750 +0x281
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc000099200)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:298 +0x66
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.newHTTP2Client
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:297 +0xce1

goroutine 46 [select, 1 minutes]:
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.(*mongoCluster).syncServersLoop(0xc000366400)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/cluster.go:394 +0x2f0
created by github.com/1234567890123/ship/gopkg.in/mgo%2ev2.newCluster
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/cluster.go:78 +0x158

goroutine 180 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(*healthBalancer).updateNotifyLoop(0xc000366000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:339 +0x34e
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.newHealthBalancer
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:128 +0x2ed

goroutine 24 [sleep]:
time.Sleep(0x37e11d600)
/usr/local/go1.11.4/go/src/runtime/time.go:105 +0x14f
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.(*mongoServer).pinger(0xc000544460, 0xc0005e2101)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/server.go:301 +0x4a4
created by github.com/1234567890123/ship/gopkg.in/mgo%2ev2.newServer
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/server.go:89 +0x106

goroutine 63 [sleep]:
time.Sleep(0x37e11d600)
/usr/local/go1.11.4/go/src/runtime/time.go:105 +0x14f
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.(*mongoServer).pinger(0xc0005c2e00, 0xc0005b4001)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/server.go:301 +0x4a4
created by github.com/1234567890123/ship/gopkg.in/mgo%2ev2.newServer
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/server.go:89 +0x106

goroutine 128 [sleep, 1 minutes]:
time.Sleep(0x45d964b800)
/usr/local/go1.11.4/go/src/runtime/time.go:105 +0x14f
github.com/1234567890123/lmn_xxx/errq/TTTTTTT.sendTTTTTTTCounters()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:203 +0xae
created by github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:195 +0x7a3

goroutine 140 [select, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(client).backgroundMetadataUpdater(0xc0001645a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/client.go:656 +0x12b
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(client).backgroundMetadataUpdater-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/client.go:165 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc00058fdc0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.NewClient
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/client.go:165 +0x51d

goroutine 141 [chan receive, 16 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).dispatcher(0xc0002b8cb0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:246 +0xc7
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).dispatcher-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:99 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc00058fde0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.NewAsyncSleeperFromClient
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:99 +0x1d2

goroutine 127 [chan receive, 16 minutes]:
github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init.func3(0xc0006b60f0, 0xc0004d6000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:165 +0x103
created by github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:147 +0x597

goroutine 194 [chan receive, 16 minutes]:
main.handleSigUser(0xc0004778c0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/pqr_xx/poiuttr/main.go:111 +0x41
created by main.handleSignals
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/pqr_xx/poiuttr/main.go:106 +0xab

goroutine 191 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.loopyWriter(0xdbf720, 0xc000072a00, 0xc000418570, 0xc0000acfb8)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:750 +0x281
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc0004e0f00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:298 +0x66
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.newHTTP2Client
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:297 +0xce1

goroutine 70 [IO wait]:
internal/poll.runtime_pollWait(0x7f95d8277950, 0x72, 0xc0006c2af0)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc000400618, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc000400618, 0xc000484000, 0x1000, 0x1000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc000400600, 0xc000484000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc000400600, 0xc000484000, 0x1000, 0x1000, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc000094528, 0xc000484000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc000341dd0, 0xdb78e0, 0xc000094528, 0x5, 0xc000094528, 0x44afba)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc0000bd880, 0xd1d017, 0xc0000bd9a0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc0000bd880, 0xc0000415c0, 0x24, 0x24, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.fill(0xdc30e0, 0xc0000bd880, 0xc0000415c0, 0x24, 0x24, 0x0, 0x11)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:535 +0x53
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.(*mongoSocket).readLoop(0xc000545a40)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:551 +0x5f7
created by github.com/1234567890123/ship/gopkg.in/mgo%2ev2.newSocket
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:194 +0x1df

goroutine 121 [chan receive, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver(0xc0002b71e0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:683 +0xdc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005abd80)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*sleeper).Open.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x809

goroutine 134 [chan receive, 16 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(syncSleeper).handleErrors(0xc000585c60)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sync_sleeper.go:139 +0xb1
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(syncSleeper).handleErrors-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sync_sleeper.go:77 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc00058fc50)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.newSyncSleeperFromAsyncSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sync_sleeper.go:77 +0x106

goroutine 441 [chan receive, 15 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher(0xc00080e5a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:336 +0x5a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc000723400)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2ae

goroutine 193 [sleep]:
time.Sleep(0x1bf08eb000)
/usr/local/go1.11.4/go/src/runtime/time.go:105 +0x14f
github.com/1234567890123/lmn_xxx/rmsg.IterateOverMsgsinStore(0xcea171, 0x11, 0xd1b648)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/rmsg/reliablemsger.go:166 +0x207
created by github.com/1234567890123/lmn_xxx/rmsg.RegisterKeyPrefixHandler
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/rmsg/reliablemsger.go:104 +0x6f

goroutine 113 [sleep]:
time.Sleep(0x6fc23ac00)
/usr/local/go1.11.4/go/src/runtime/time.go:105 +0x14f
github.com/1234567890123/lmn_xxx/db/mongodb.RefreshSession(0xc0000b6000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/db/mongodb/connector.go:264 +0x5a
created by github.com/1234567890123/lmn_xxx/db/mongodb.getSession
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/db/mongodb/connector.go:198 +0x2d2

goroutine 69 [sleep]:
time.Sleep(0x37e11d600)
/usr/local/go1.11.4/go/src/runtime/time.go:105 +0x14f
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.(*mongoServer).pinger(0xc000544620, 0xc0005e2a01)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/server.go:301 +0x4a4
created by github.com/1234567890123/ship/gopkg.in/mgo%2ev2.newServer
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/server.go:89 +0x106

goroutine 133 [chan receive, 1 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(syncSleeper).handleSuccesses(0xc000585c60)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sync_sleeper.go:131 +0xab
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(syncSleeper).handleSuccesses-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sync_sleeper.go:76 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc00058fc40)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.newSyncSleeperFromAsyncSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sync_sleeper.go:76 +0xb8

goroutine 64 [IO wait]:
internal/poll.runtime_pollWait(0x7f95d8277bc0, 0x72, 0xc0006c3af0)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0002a2c98, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0002a2c98, 0xc0006dc000, 0x8000, 0x8000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0002a2c80, 0xc0006dc000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0002a2c80, 0xc0006dc000, 0x8000, 0x8000, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc000094520, 0xc0006dc000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc000717c80, 0xdb78e0, 0xc000094520, 0x5, 0xc000094520, 0x44afba)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc0000bd500, 0xd1d017, 0xc0000bd620, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc0000bd500, 0xc0005dd200, 0x24, 0x24, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.fill(0xdc30e0, 0xc0000bd500, 0xc0005dd200, 0x24, 0x24, 0x0, 0x11)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:535 +0x53
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.(*mongoSocket).readLoop(0xc0005c3260)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:551 +0x5f7
created by github.com/1234567890123/ship/gopkg.in/mgo%2ev2.newSocket
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:194 +0x1df

goroutine 182 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000716600)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_conn_wrappers.go:117 +0xfa
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.newCCBalancerWrapper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_conn_wrappers.go:108 +0x120

goroutine 132 [chan receive, 16 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).retryHandler(0xc0002b8a80)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:813 +0x216
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).retryHandler-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:100 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc00058fc30)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.NewAsyncSleeperFromClient
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:100 +0x220

goroutine 120 [chan receive, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver(0xc0003ec580)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:683 +0xdc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005abcb0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*sleeper).Open.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x809

goroutine 114 [chan receive, 16 minutes]:
github.com/1234567890123/lmn_xxx/db/mongodb.(*DbSessionPool).RefreshSessionLoop(0xc0004a05c0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/db/mongodb/connector.go:104 +0x110
created by github.com/1234567890123/lmn_xxx/db/mongodb.GetClusterSessionPool
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/db/mongodb/connector.go:78 +0x25f

goroutine 131 [chan receive, 1 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).dispatcher(0xc0002b8a80)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:246 +0xc7
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).dispatcher-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:99 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc00058fc20)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.NewAsyncSleeperFromClient
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:99 +0x1d2

goroutine 190 [IO wait, 16 minutes]:
internal/poll.runtime_pollWait(0x7f95d82777b0, 0x72, 0xc000600998)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0002a2798, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0002a2798, 0xc0004ebc00, 0x400, 0x400)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0002a2780, 0xc0004ebc00, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0002a2780, 0xc0004ebc00, 0x400, 0x400, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc00000c050, 0xc0004ebc00, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc000743e90, 0xdb78e0, 0xc00000c050, 0x5, 0xc00000c050, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc0000bc700, 0xd1d017, 0xc0000bc820, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc0000bc700, 0xc0006ac000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
bufio.(Reader).Read(0xc000477800, 0xc0000e23b8, 0x9, 0x9, 0x20, 0x0, 0x0)
/usr/local/go1.11.4/go/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0xdb6960, 0xc000477800, 0xc0000e23b8, 0x9, 0x9, 0x9, 0xc0005f8360, 0x402a6e, 0xc000600eab)
/usr/local/go1.11.4/go/src/io/io.go:310 +0x88
io.ReadFull(0xdb6960, 0xc000477800, 0xc0000e23b8, 0x9, 0x9, 0x44b948, 0x5ceca1357da, 0xc0245ea91b)
/usr/local/go1.11.4/go/src/io/io.go:329 +0x58
github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2.readFrameHeader(0xc0000e23b8, 0x9, 0x9, 0xdb6960, 0xc000477800, 0x0, 0x0, 0x137dba0, 0xbf0999d26419c4c7)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2.(Framer).ReadFrame(0xc0000e2380, 0xc00056dd20, 0xc00056dd20, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(*http2Client).reader(0xc0004e0f00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:1118 +0xbf
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.newHTTP2Client
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:266 +0xb7d

goroutine 179 [chan receive, 16 minutes]:
github.com/1234567890123/pqr_xx/poiuttr/msghandler.consumeMessages.func1(0xc0004d6000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/pqr_xx/poiuttr/msghandler/msgTTTTTTT.go:101 +0x9d
created by github.com/1234567890123/pqr_xx/poiuttr/msghandler.consumeMessages
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/pqr_xx/poiuttr/msghandler/msgTTTTTTT.go:100 +0x91

goroutine 185 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(addrConn).transportMonitor(0xc00033c200)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/clientconn.go:1012 +0x122
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(addrConn).connect.func1(0xc00033c200)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/clientconn.go:703 +0x1b2
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(*addrConn).connect
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/clientconn.go:694 +0x3a3

goroutine 195 [sleep, 1 minutes]:
time.Sleep(0x45d964b800)
/usr/local/go1.11.4/go/src/runtime/time.go:105 +0x14f
github.com/1234567890123/pqr_xx/poiuttr/metrics.startMetricsCollection()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/pqr_xx/poiuttr/metrics/metrics.go:55 +0x162
created by github.com/1234567890123/pqr_xx/poiuttr/metrics.Init
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/pqr_xx/poiuttr/metrics/metrics.go:48 +0x152

goroutine 184 [select]:
github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init.func3.1(0xc0006b60f0, 0xc0003fe780, 0xc0003fe720, 0xc000094410)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:154 +0xb3
created by github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init.func3
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:152 +0xcc

goroutine 181 [select]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(*healthBalancer).updateUnhealthy(0xc000366000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:236 +0xb9
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.newHealthBalancer.func1(0xc000366000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:132 +0x5a
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.newHealthBalancer
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:130 +0x32d

goroutine 183 [chan receive, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(balancerWrapper).lbWatcher(0xc0003fe5a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_v1_wrapper.go:115 +0x186
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(balancerWrapperBuilder).Build
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_v1_wrapper.go:53 +0x1e4

goroutine 119 [sleep]:
time.Sleep(0x6fc23ac00)
/usr/local/go1.11.4/go/src/runtime/time.go:105 +0x14f
github.com/1234567890123/lmn_xxx/db/mongodb.(DbSessionPool).refreshSessionPool(0xc0004a05c0, 0xc0004d6cc0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/db/mongodb/connector.go:284 +0x130
created by github.com/1234567890123/lmn_xxx/db/mongodb.(DbSessionPool).RefreshSessionLoop
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/db/mongodb/connector.go:102 +0xf9

goroutine 177 [chan receive, 1 minutes]:
github.com/1234567890123/pqr_xx/poiuttr/msghandler.SendMessageToMntrSvc(0xc0003de7e0, 0x1, 0x1)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/pqr_xx/poiuttr/msghandler/msgTTTTTTT.go:87 +0x5b
created by github.com/1234567890123/pqr_xx/poiuttr/msghandler.Initerrq
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/pqr_xx/poiuttr/msghandler/msgTTTTTTT.go:63 +0x513

goroutine 178 [chan receive, 16 minutes]:
github.com/1234567890123/pqr_xx/poiuttr/msghandler.consumeMessages.func1(0xc000081800)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/pqr_xx/poiuttr/msghandler/msgTTTTTTT.go:101 +0x9d
created by github.com/1234567890123/pqr_xx/poiuttr/msghandler.consumeMessages
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/pqr_xx/poiuttr/msghandler/msgTTTTTTT.go:100 +0x91

goroutine 305 [IO wait]:
internal/poll.runtime_pollWait(0x7f95d80e40b8, 0x72, 0xc0006d9b60)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0007d0918, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0007d0918, 0xc000811000, 0x1000, 0x1000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0007d0900, 0xc000811000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0007d0900, 0xc000811000, 0x1000, 0x1000, 0x4859cd, 0xc0007d0900, 0xc000620b00)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc000094a58, 0xc000811000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
bufio.(Reader).Read(0xc000791f80, 0xc000016dd8, 0x8, 0x8, 0xc0006d9d38, 0x52dc8b, 0xc0007d0900)
/usr/local/go1.11.4/go/src/bufio/bufio.go:216 +0x22f
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(bufConn).Read(0xc00054da80, 0xc000016dd8, 0x8, 0x8, 0x5408a8, 0xc0007d0900, 0x408a6b)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:109 +0x4d
io.ReadAtLeast(0x7f95d8287f28, 0xc00054da80, 0xc000016dd8, 0x8, 0x8, 0x8, 0xc65640, 0x0, 0x7f95d8287f28)
/usr/local/go1.11.4/go/src/io/io.go:310 +0x88
io.ReadFull(0x7f95d8287f28, 0xc00054da80, 0xc000016dd8, 0x8, 0x8, 0x0, 0x137dba0, 0x0)
/usr/local/go1.11.4/go/src/io/io.go:329 +0x58
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver(0xc0002b7b80)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:696 +0x29d
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc000546220)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).Open.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x809

goroutine 142 [chan receive, 16 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).retryHandler(0xc0002b8cb0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:813 +0x216
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).retryHandler-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:100 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc00058fdf0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.NewAsyncSleeperFromClient
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:100 +0x220

goroutine 143 [chan receive, 16 minutes]:
github.com/1234567890123/lmn_xxx/errq/sleeper.doAsyncMsgAccounting.func1(0xc00058fe00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/sleeper/errqsleeper.go:214 +0x32
created by github.com/1234567890123/lmn_xxx/errq/sleeper.doAsyncMsgAccounting
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/sleeper/errqsleeper.go:211 +0x3f

goroutine 144 [sleep, 1 minutes]:
time.Sleep(0x45d964b800)
/usr/local/go1.11.4/go/src/runtime/time.go:105 +0x14f
github.com/1234567890123/lmn_xxx/errq/sleeper.sendSleeperCounters()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/sleeper/errqsleeper.go:162 +0x2a2
created by github.com/1234567890123/lmn_xxx/errq/sleeper.Init
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/sleeper/errqsleeper.go:143 +0x799

goroutine 145 [chan receive, 1 minutes]:
github.com/1234567890123/lmn_xxx/errq/sleeper.writeMetricsOnKafka()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/sleeper/errqsleeper.go:153 +0x36
created by github.com/1234567890123/lmn_xxx/errq/sleeper.Init
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/sleeper/errqsleeper.go:144 +0x7b1

goroutine 125 [chan receive, 16 minutes]:
github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init.func1(0xc0006b60f0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:109 +0x7c
created by github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:108 +0x540

goroutine 122 [chan receive, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver(0xc0002b7760)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:683 +0xdc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005abe40)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*sleeper).Open.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x809

goroutine 126 [chan receive, 15 minutes]:
github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init.func2(0xc0006b60f0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:124 +0x77
created by github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:123 +0x565

goroutine 151 [select, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(client).backgroundMetadataUpdater(0xc000164870)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/client.go:656 +0x12b
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(client).backgroundMetadataUpdater-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/client.go:165 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc00058fff0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.NewClient
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/client.go:165 +0x51d

goroutine 152 [select, 15 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).nextTick(0xc0006b6000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:391 +0x2fe
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).mainLoop(0xc0006b6000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:328 +0x97
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.NewTTTTTTTFromClient
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:96 +0x36e

goroutine 153 [chan receive, 16 minutes]:
github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init.func1(0xc0006b6000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:109 +0x7c
created by github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:108 +0x540

goroutine 154 [chan receive, 15 minutes]:
github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init.func2(0xc0006b6000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:124 +0x77
created by github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:123 +0x565

goroutine 155 [chan receive, 16 minutes]:
github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init.func3(0xc0006b6000, 0xc000081800)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:165 +0x103
created by github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:147 +0x597

goroutine 124 [select, 15 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).nextTick(0xc0006b60f0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:391 +0x2fe
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).mainLoop(0xc0006b60f0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:328 +0x97
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.NewTTTTTTTFromClient
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:96 +0x36e

goroutine 123 [chan receive, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver(0xc0003ec160)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:683 +0xdc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc000722020)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*sleeper).Open.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x809

goroutine 162 [select, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(client).backgroundMetadataUpdater(0xc0003a4000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/client.go:656 +0x12b
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(client).backgroundMetadataUpdater-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/client.go:165 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005aa270)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.NewClient
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/client.go:165 +0x51d

goroutine 169 [runnable]:
internal/poll.runtime_pollWait(0x7f95d82776e0, 0x72, 0xc0006c1b60)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc000400898, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc000400898, 0xc0003be000, 0x1000, 0x1000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc000400880, 0xc0003be000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc000400880, 0xc0003be000, 0x1000, 0x1000, 0x4859cd, 0xc000400880, 0xc0002bed00)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc00000c128, 0xc0003be000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
bufio.(Reader).Read(0xc000354840, 0xc0005e29f0, 0x8, 0x8, 0xc0006c1d38, 0x52dc8b, 0xc000400880)
/usr/local/go1.11.4/go/src/bufio/bufio.go:216 +0x22f
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(bufConn).Read(0xc00059bde0, 0xc0005e29f0, 0x8, 0x8, 0x5408a8, 0xc000400880, 0x408a6b)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:109 +0x4d
io.ReadAtLeast(0x7f95d8287f28, 0xc00059bde0, 0xc0005e29f0, 0x8, 0x8, 0x8, 0xc65640, 0x0, 0x7f95d8287f28)
/usr/local/go1.11.4/go/src/io/io.go:310 +0x88
io.ReadFull(0x7f95d8287f28, 0xc00059bde0, 0xc0005e29f0, 0x8, 0x8, 0x0, 0x137dba0, 0x0)
/usr/local/go1.11.4/go/src/io/io.go:329 +0x58
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver(0xc0002b7a20)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:696 +0x29d
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005aa830)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).Open.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x809

goroutine 160 [select]:
github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init.func3.1(0xc0006b6000, 0xc0003de480, 0xc0003de420, 0xc00000c028)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:154 +0xb3
created by github.com/1234567890123/lmn_xxx/errq/TTTTTTT.Init.func3
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/errq/TTTTTTT/errqTTTTTTT.go:152 +0xcc

goroutine 453 [chan receive]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder(0xc00080e750)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:451 +0x27a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d9670)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x300

goroutine 457 [chan receive]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder(0xc00080e480)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:451 +0x27a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d9720)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x300

goroutine 196 [chan receive, 16 minutes]:
github.com/1234567890123/pqr_xx/poiuttr/metrics.StartStatsProcess()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/pqr_xx/poiuttr/metrics/metrics.go:113 +0x61
created by github.com/1234567890123/pqr_xx/poiuttr/metrics.Init
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/pqr_xx/poiuttr/metrics/metrics.go:49 +0x16a

goroutine 197 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(*healthBalancer).updateNotifyLoop(0xc000366300)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:339 +0x34e
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.newHealthBalancer
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:128 +0x2ed

goroutine 198 [select]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(*healthBalancer).updateUnhealthy(0xc000366300)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:236 +0xb9
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.newHealthBalancer.func1(0xc000366300)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:132 +0x5a
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.newHealthBalancer
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:130 +0x32d

goroutine 199 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000418930)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_conn_wrappers.go:117 +0xfa
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.newCCBalancerWrapper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_conn_wrappers.go:108 +0x120

goroutine 200 [chan receive, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(balancerWrapper).lbWatcher(0xc000477b00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_v1_wrapper.go:115 +0x186
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(balancerWrapperBuilder).Build
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_v1_wrapper.go:53 +0x1e4

goroutine 201 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(addrConn).transportMonitor(0xc00033ca00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/clientconn.go:1012 +0x122
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(addrConn).connect.func1(0xc00033ca00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/clientconn.go:703 +0x1b2
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(*addrConn).connect
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/clientconn.go:694 +0x3a3

goroutine 217 [IO wait]:
internal/poll.runtime_pollWait(0x7f95d8277540, 0x72, 0xc0006c0998)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc000400c98, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc000400c98, 0xc000365400, 0x400, 0x400)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc000400c80, 0xc000365400, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc000400c80, 0xc000365400, 0x400, 0x400, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc0000945c0, 0xc000365400, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc000419500, 0xdb78e0, 0xc0000945c0, 0x5, 0xc0000945c0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc0006a2a80, 0xd1d017, 0xc0006a2ba0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc0006a2a80, 0xc0006ea000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
bufio.(Reader).Read(0xc0006bae40, 0xc0000e2578, 0x9, 0x9, 0x20, 0x0, 0x0)
/usr/local/go1.11.4/go/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0xdb6960, 0xc0006bae40, 0xc0000e2578, 0x9, 0x9, 0x9, 0xc000646210, 0x402a6e, 0xc0006c0eab)
/usr/local/go1.11.4/go/src/io/io.go:310 +0x88
io.ReadFull(0xdb6960, 0xc0006bae40, 0xc0000e2578, 0x9, 0x9, 0x44b948, 0x6ae67da1935, 0xc00215207b)
/usr/local/go1.11.4/go/src/io/io.go:329 +0x58
github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2.readFrameHeader(0xc0000e2578, 0x9, 0x9, 0xdb6960, 0xc0006bae40, 0x0, 0xe000000000, 0x137dba0, 0xbf099ac28207fe65)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2.(Framer).ReadFrame(0xc0000e2540, 0xc00014d080, 0xc00014d080, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(*http2Client).reader(0xc000559980)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:1118 +0xbf
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.newHTTP2Client
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:266 +0xb7d

goroutine 2547 [select, 1 minutes]:
net/http.(persistConn).writeLoop(0xc0007a2000)
/usr/local/go1.11.4/go/src/net/http/transport.go:1885 +0x113
created by net/http.(Transport).dialConn
/usr/local/go1.11.4/go/src/net/http/transport.go:1339 +0x966

goroutine 171 [chan receive]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver(0xc0003ecb00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:683 +0xdc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005abed0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*sleeper).Open.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x809

goroutine 498 [select, 15 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(RoomsTTTTTTT).multiplex(0xc0003da3c0, 0xc0004d7920, 0xc000081e00, 0xc00069faa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/Roomss.go:129 +0x182
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).createTTTTTTT.func1(0xc0004d7920)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:829 +0x85
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc00001b140, 0xc00001b840)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 288 [IO wait]:
internal/poll.runtime_pollWait(0x7f95d80e4258, 0x72, 0xc0006d8b60)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0007d0818, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0007d0818, 0xc000810000, 0x1000, 0x1000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0007d0800, 0xc000810000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0007d0800, 0xc000810000, 0x1000, 0x1000, 0x4859cd, 0xc0007d0800, 0xc0002bed00)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc000094a50, 0xc000810000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
bufio.(Reader).Read(0xc000791ec0, 0xc000016db0, 0x8, 0x8, 0xc0006d8d38, 0x52dc8b, 0xc0007d0800)
/usr/local/go1.11.4/go/src/bufio/bufio.go:216 +0x22f
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(bufConn).Read(0xc00054d920, 0xc000016db0, 0x8, 0x8, 0x5408a8, 0xc0007d0800, 0x408a6b)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:109 +0x4d
io.ReadAtLeast(0x7f95d8287f28, 0xc00054d920, 0xc000016db0, 0x8, 0x8, 0x8, 0xc65640, 0x0, 0x7f95d8287f28)
/usr/local/go1.11.4/go/src/io/io.go:310 +0x88
io.ReadFull(0x7f95d8287f28, 0xc00054d920, 0xc000016db0, 0x8, 0x8, 0x0, 0x137dba0, 0x0)
/usr/local/go1.11.4/go/src/io/io.go:329 +0x58
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver(0xc0003ecc60)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:696 +0x29d
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005461a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).Open.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x809

goroutine 242 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(watchGrpcStream).serveSubstream(0xc00008dee0, 0xc0002ad970, 0xc0006b9560)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/watch.go:630 +0x2a5
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(watchGrpcStream).run
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/watch.go:470 +0xf6f

goroutine 211 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(recvBufferReader).read(0xc000578820, 0xc000585f30, 0x5, 0x5, 0x0, 0xa, 0x5e)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:133 +0x1ff
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(recvBufferReader).Read(0xc000578820, 0xc000585f30, 0x5, 0x5, 0xc000667c40, 0x8ce135, 0x3)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:122 +0x5a
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(transportReader).Read(0xc000341cb0, 0xc000585f30, 0x5, 0x5, 0x4639ae, 0xc000366ac8, 0xd1d108)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:403 +0x55
io.ReadAtLeast(0xdb7500, 0xc000341cb0, 0xc000585f30, 0x5, 0x5, 0x5, 0xc0004e0f00, 0xc000366a00, 0x5)
/usr/local/go1.11.4/go/src/io/io.go:310 +0x88
io.ReadFull(0xdb7500, 0xc000341cb0, 0xc000585f30, 0x5, 0x5, 0xc0000e6480, 0xdbd220, 0xc000667d18)
/usr/local/go1.11.4/go/src/io/io.go:329 +0x58
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(Stream).Read(0xc000366a00, 0xc000585f30, 0x5, 0x5, 0x0, 0xc00008cd08, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:387 +0xbf
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(parser).recvMsg(0xc000585f20, 0x7fffffff, 0x4, 0xc000667ee0, 0xc000559500, 0xc000667e18, 0x203000, 0x203000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/rpc_util.go:270 +0x65
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.recv(0xc000585f20, 0xdbf1e0, 0x139b850, 0xc000366a00, 0x0, 0x0, 0xc847a0, 0xc0005b6860, 0x7fffffff, 0x0, ...)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/rpc_util.go:356 +0x4d
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(clientStream).RecvMsg(0xc0005d62d0, 0xc847a0, 0xc0005b6860, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/stream.go:392 +0xbb
github.com/1234567890123/ship/github.com/coreos/etcd/etcdserver/etcdserverpb.(leaseLeaseKeepAliveClient).Recv(0xc000723bb0, 0xc000585f40, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:3630 +0x62
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(lessor).recvKeepAliveLoop(0xc00070e6e0, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/lease.go:441 +0x9e
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(*lessor).KeepAlive.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/lease.go:293 +0x43

goroutine 212 [select]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(lessor).deadlineLoop(0xc00070e6e0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/lease.go:527 +0x190
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(lessor).KeepAlive.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/lease.go:294 +0x65

goroutine 235 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.loopyWriter(0xdbf720, 0xc00072e540, 0xc0005f8c90, 0xc0005d1fb8)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:750 +0x281
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc0006d3b00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:298 +0x66
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.newHTTP2Client
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:297 +0xce1

goroutine 214 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.newClientStream.func4(0xdc3380, 0xc0004e0f00, 0xc00033e5a0, 0xc0005d62d0, 0xc000366a00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/stream.go:255 +0x174
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.newClientStream
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/stream.go:254 +0xb18

goroutine 215 [select]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(lessor).sendKeepAliveLoop(0xc00070e6e0, 0xdc31a0, 0xc000723bb0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/lease.go:567 +0x334
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(lessor).resetRecv
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/lease.go:484 +0x18a

goroutine 218 [select]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.loopyWriter(0xdbf720, 0xc000073d80, 0xc000341f20, 0xc0005fcfb8)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:750 +0x281
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc000559980)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:298 +0x66
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.newHTTP2Client
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:297 +0xce1

goroutine 225 [chan receive, 16 minutes]:
github.com/1234567890123/lmn_xxx/etcdclipool.EtcdV3MonitorWithoutPrefix(0xc000038cd0, 0x11, 0xc0005da820, 0x20, 0xd1b4f8, 0xcc6940, 0xc0000a6000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/etcdclipool/etcdcli.go:863 +0x52f
created by github.com/1234567890123/lmn_xxx/etcdclipool.RegisterCallBkWithoutPrefix
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/etcdclipool/etcdcli.go:331 +0x94

goroutine 220 [sleep, 6 minutes]:
time.Sleep(0x8bb2c97000)
/usr/local/go1.11.4/go/src/runtime/time.go:105 +0x14f
github.com/1234567890123/lmn_xxx/httplib/svr.(HttpSvc).PollAndWatchLogLevel(0xc0000a6000, 0xc000038cd0, 0x11)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/httplib/svr/loglevel.go:218 +0x110
created by github.com/1234567890123/lmn_xxx/httplib/svr.(HttpSvc).RegisterAccessPoint
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/httplib/svr/httpsvr.go:219 +0x1ad

goroutine 446 [select, 15 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(RoomsTTTTTTT).multiplex(0xc0002c3b80, 0xc0004d7920, 0xc000081e00, 0xc00069faa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/Roomss.go:129 +0x182
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).createTTTTTTT.func1(0xc0004d7920)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:829 +0x85
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc00001b140, 0xc00001b3e0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 295 [runnable]:
internal/poll.runtime_pollWait(0x7f95d80e4188, 0x72, 0xc00079fb60)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0007d0618, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0007d0618, 0xc0007fd000, 0x1000, 0x1000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0007d0600, 0xc0007fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0007d0600, 0xc0007fd000, 0x1000, 0x1000, 0x4859cd, 0xc0007d0600, 0xc0002bed00)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc00000c260, 0xc0007fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
bufio.(Reader).Read(0xc000800d20, 0xc0005119d8, 0x8, 0x8, 0xc00079fd38, 0x52dc8b, 0xc0007d0600)
/usr/local/go1.11.4/go/src/bufio/bufio.go:216 +0x22f
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(bufConn).Read(0xc0005c5b20, 0xc0005119d8, 0x8, 0x8, 0x5408a8, 0xc0007d0600, 0x408a6b)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:109 +0x4d
io.ReadAtLeast(0x7f95d8287f28, 0xc0005c5b20, 0xc0005119d8, 0x8, 0x8, 0x8, 0xc65640, 0x0, 0x7f95d8287f28)
/usr/local/go1.11.4/go/src/io/io.go:310 +0x88
io.ReadFull(0x7f95d8287f28, 0xc0005c5b20, 0xc0005119d8, 0x8, 0x8, 0x0, 0x137dba0, 0x0)
/usr/local/go1.11.4/go/src/io/io.go:329 +0x58
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver(0xc0003ec9a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:696 +0x29d
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005c1c90)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).Open.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x809

goroutine 299 [IO wait]:
internal/poll.runtime_pollWait(0x7f95d80e3fe8, 0x72, 0xc0008ccaf0)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0002a2098, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0002a2098, 0xc000691000, 0x1000, 0x1000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0002a2080, 0xc000691000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0002a2080, 0xc000691000, 0x1000, 0x1000, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc0000942e8, 0xc000691000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc0008980f0, 0xdb78e0, 0xc0000942e8, 0x5, 0xc0000942e8, 0x44afba)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc0006a2000, 0xd1d017, 0xc0006a2120, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc0006a2000, 0xc00037eae0, 0x24, 0x24, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.fill(0xdc30e0, 0xc0006a2000, 0xc00037eae0, 0x24, 0x24, 0x0, 0x11)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:535 +0x53
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.(*mongoSocket).readLoop(0xc0000e1880)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:551 +0x5f7
created by github.com/1234567890123/ship/gopkg.in/mgo%2ev2.newSocket
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:194 +0x1df

goroutine 226 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(*healthBalancer).updateNotifyLoop(0xc000366d00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:339 +0x34e
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.newHealthBalancer
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:128 +0x2ed

goroutine 227 [select]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(*healthBalancer).updateUnhealthy(0xc000366d00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:236 +0xb9
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.newHealthBalancer.func1(0xc000366d00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:132 +0x5a
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.newHealthBalancer
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/health_balancer.go:130 +0x32d

goroutine 228 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00068c8a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_conn_wrappers.go:117 +0xfa
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.newCCBalancerWrapper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_conn_wrappers.go:108 +0x120

goroutine 229 [chan receive, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(balancerWrapper).lbWatcher(0xc0006bb9e0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_v1_wrapper.go:115 +0x186
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(balancerWrapperBuilder).Build
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/balancer_v1_wrapper.go:53 +0x1e4

goroutine 230 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(addrConn).transportMonitor(0xc00033cc00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/clientconn.go:1012 +0x122
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(addrConn).connect.func1(0xc00033cc00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/clientconn.go:703 +0x1b2
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(*addrConn).connect
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/clientconn.go:694 +0x3a3

goroutine 234 [IO wait, 16 minutes]:
internal/poll.runtime_pollWait(0x7f95d82772d0, 0x72, 0xc0005fd998)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc000401098, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc000401098, 0xc0006d1000, 0x1000, 0x1000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc000401080, 0xc0006d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc000401080, 0xc0006d1000, 0x1000, 0x1000, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc00000c188, 0xc0006d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc0005f6300, 0xdb78e0, 0xc00000c188, 0x5, 0xc00000c188, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc000734700, 0xd1d017, 0xc000734820, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc000734700, 0xc00060a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
bufio.(Reader).Read(0xc0005f47e0, 0xc0000e28f8, 0x9, 0x9, 0x20, 0x0, 0x0)
/usr/local/go1.11.4/go/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0xdb6960, 0xc0005f47e0, 0xc0000e28f8, 0x9, 0x9, 0x9, 0xc0005f8c00, 0x402a6e, 0xc0005fdeab)
/usr/local/go1.11.4/go/src/io/io.go:310 +0x88
io.ReadFull(0xdb6960, 0xc0005f47e0, 0xc0000e28f8, 0x9, 0x9, 0x44b948, 0x5ceca4745c6, 0xc024929710)
/usr/local/go1.11.4/go/src/io/io.go:329 +0x58
github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2.readFrameHeader(0xc0000e28f8, 0x9, 0x9, 0xdb6960, 0xc0005f47e0, 0x0, 0x0, 0x137dba0, 0xbf0999d2647d3ba2)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2.(Framer).ReadFrame(0xc0000e28c0, 0xc0005dbd40, 0xc0005dbd40, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(*http2Client).reader(0xc0006d3b00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:1118 +0xbf
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.newHTTP2Client
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/http2_client.go:266 +0xb7d

goroutine 296 [runnable]:
internal/poll.runtime_pollWait(0x7f95d8277060, 0x72, 0xc0006d6b60)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0007d0718, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0007d0718, 0xc00081a000, 0x1000, 0x1000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0007d0700, 0xc00081a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0007d0700, 0xc00081a000, 0x1000, 0x1000, 0x4859cd, 0xc0007d0700, 0xc000620b00)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc00000c270, 0xc00081a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
bufio.(Reader).Read(0xc000800de0, 0xc000511a30, 0x8, 0x8, 0xc0006d6d38, 0x52dc8b, 0xc0007d0700)
/usr/local/go1.11.4/go/src/bufio/bufio.go:216 +0x22f
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(bufConn).Read(0xc0005c5c60, 0xc000511a30, 0x8, 0x8, 0x5408a8, 0xc0007d0700, 0x408a6b)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:109 +0x4d
io.ReadAtLeast(0x7f95d8287f28, 0xc0005c5c60, 0xc000511a30, 0x8, 0x8, 0x8, 0xc65640, 0x0, 0x7f95d8287f28)
/usr/local/go1.11.4/go/src/io/io.go:310 +0x88
io.ReadFull(0x7f95d8287f28, 0xc0005c5c60, 0xc000511a30, 0x8, 0x8, 0x0, 0x137dba0, 0x0)
/usr/local/go1.11.4/go/src/io/io.go:329 +0x58
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver(0xc0002b78c0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:696 +0x29d
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005c1cd0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).Open.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x809

goroutine 237 [select, 16 minutes]:
context.propagateCancel.func1(0xdbfa60, 0xc00058f6b0, 0xdb9460, 0xc00072f880)
/usr/local/go1.11.4/go/src/context/context.go:260 +0xd8
created by context.propagateCancel
/usr/local/go1.11.4/go/src/context/context.go:259 +0x18d

goroutine 238 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(watchGrpcStream).run(0xc00008dee0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/watch.go:455 +0x33e
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(watcher).newWatcherGrpcStream
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/watch.go:252 +0x2de

goroutine 241 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(recvBufferReader).read(0xc000579f40, 0xc00056e9b0, 0x5, 0x5, 0x1d, 0x1f, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:133 +0x1ff
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(recvBufferReader).Read(0xc000579f40, 0xc00056e9b0, 0x5, 0x5, 0x2, 0xbec8c0, 0x3)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:122 +0x5a
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(transportReader).Read(0xc0005f9050, 0xc00056e9b0, 0x5, 0x5, 0x4639ae, 0xc0003670c8, 0xd1d108)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:403 +0x55
io.ReadAtLeast(0xdb7500, 0xc0005f9050, 0xc00056e9b0, 0x5, 0x5, 0x5, 0xc0006d3b00, 0xc000367000, 0x5)
/usr/local/go1.11.4/go/src/io/io.go:310 +0x88
io.ReadFull(0xdb7500, 0xc0005f9050, 0xc00056e9b0, 0x5, 0x5, 0xc0000e6400, 0xdbde20, 0xc0005d2ce8)
/usr/local/go1.11.4/go/src/io/io.go:329 +0x58
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(Stream).Read(0xc000367000, 0xc00056e9b0, 0x5, 0x5, 0x0, 0xc0005d2ce8, 0x43d1dd)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:387 +0xbf
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(parser).recvMsg(0xc00056e9a0, 0x7fffffff, 0xc0006d3e00, 0xc0005d2eb8, 0x43dbfd, 0xc0006b9380, 0x203000, 0x203000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/rpc_util.go:270 +0x65
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.recv(0xc00056e9a0, 0xdbf1e0, 0x139b850, 0xc000367000, 0x0, 0x0, 0xc9f1a0, 0xc00049f900, 0x7fffffff, 0x0, ...)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/rpc_util.go:356 +0x4d
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(clientStream).RecvMsg(0xc0005d6690, 0xc9f1a0, 0xc00049f900, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/stream.go:392 +0xbb
github.com/1234567890123/ship/github.com/coreos/etcd/etcdserver/etcdserverpb.(watchWatchClient).Recv(0xc00058f700, 0xc0005d2ef0, 0x2, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:3499 +0x62
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(watchGrpcStream).serveWatchClient(0xc00008dee0, 0xdc3260, 0xc00058f700)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/watch.go:584 +0x4b
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(*watchGrpcStream).newWatchClient
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/watch.go:739 +0x52f

goroutine 240 [select, 16 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.newClientStream.func4(0xdc3380, 0xc0006d3b00, 0xc00033ef00, 0xc0005d6690, 0xc000367000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/stream.go:255 +0x174
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.newClientStream
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/stream.go:254 +0xb18

goroutine 243 [sleep, 1 minutes]:
time.Sleep(0x45d964b800)
/usr/local/go1.11.4/go/src/runtime/time.go:105 +0x14f
github.com/1234567890123/lmn_xxx/httplib/svr.(HttpSvc).invokeRegisterCmd(0xc0000a6000, 0xce0548, 0x4, 0xc0000360af, 0xe, 0xc0000360d1, 0xe, 0xc00037cd80, 0x3a, 0xc00037cdc0, ...)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/httplib/svr/httpsvr.go:487 +0xcd3
created by github.com/1234567890123/lmn_xxx/httplib/svr.(HttpSvc).RegisterAccessPoint
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/lmn_xxx/httplib/svr/httpsvr.go:236 +0x37b

goroutine 415 [IO wait]:
internal/poll.runtime_pollWait(0x7f95d80e3a38, 0x72, 0xc0008d1af0)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0007d1198, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0007d1198, 0xc000504000, 0x8000, 0x8000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0007d1180, 0xc000504000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0007d1180, 0xc000504000, 0x8000, 0x8000, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc00000c030, 0xc000504000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc0007f6240, 0xdb78e0, 0xc00000c030, 0x5, 0xc00000c030, 0x44afba)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc00028f180, 0xd1d017, 0xc00028f2a0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc00028f180, 0xc000041a40, 0x24, 0x24, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.fill(0xdc30e0, 0xc00028f180, 0xc000041a40, 0x24, 0x24, 0x0, 0x11)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:535 +0x53
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.(*mongoSocket).readLoop(0xc0000e3500)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:551 +0x5f7
created by github.com/1234567890123/ship/gopkg.in/mgo%2ev2.newSocket
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:194 +0x1df

goroutine 442 [chan receive]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder(0xc00080e5a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:451 +0x27a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc000723410)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x300

goroutine 461 [chan receive]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder(0xc00080e360)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:451 +0x27a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d9870)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x300

goroutine 459 [select]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).sendAndReceive(0xc0002b7a20, 0xdc0e60, 0xc000352560, 0xdb6f60, 0xc000352580, 0xc0004182d0, 0x1371750)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:611 +0x133
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).YeelK(0xc0002b7a20, 0xc000352560, 0xc000000002, 0xc00034d3d8, 0xc0004a2568)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:280 +0x70
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).YeelKNewMessages(0xc0007e57c0, 0x0, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:806 +0x3f7
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionTTTTTTT(0xc0007e57c0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:692 +0x124
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionTTTTTTT-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:633 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d9740)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(TTTTTTT).newsleeperTTTTTTT
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:633 +0x19c

goroutine 447 [chan receive, 15 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher(0xc00080e7e0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:336 +0x5a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc000723470)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2ae

goroutine 483 [select, 15 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(RoomsTTTTTTT).multiplex(0xc0002c3cc0, 0xc0006b9aa0, 0xc0004d6780, 0xc0004bdda0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/Roomss.go:129 +0x182
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).createTTTTTTT.func1(0xc0006b9aa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:829 +0x85
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc0003d6d00, 0xc00001b480)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 491 [select]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionManager(0xc0008834a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:659 +0x399
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionManager-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:632 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0007235b0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).newsleeperTTTTTTT
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:632 +0x14e

goroutine 1046 [chan receive, 11 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch(0xc0003df980)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:439 +0xcc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005aa420)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*asyncSleeper).newRoomsSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x1c2

goroutine 497 [chan receive]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder(0xc00080e3f0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:451 +0x27a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc000723600)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x300

goroutine 494 [chan receive]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder(0xc00080e510)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:451 +0x27a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0007235e0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x300

goroutine 493 [chan receive, 15 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher(0xc00080e510)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:336 +0x5a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0007235d0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2ae

goroutine 486 [select, 15 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(RoomsTTTTTTT).multiplex(0xc0003da000, 0xc0006b9aa0, 0xc0004d6780, 0xc0004bdda0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/Roomss.go:129 +0x182
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).createTTTTTTT.func1(0xc0006b9aa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:829 +0x85
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc0003d6d00, 0xc00001b4a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 485 [chan receive]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder(0xc00080e630)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:451 +0x27a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0007234f0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x300

goroutine 488 [select]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).cmLoop(0xc0006b60f0, 0xc0006b9aa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:456 +0x120
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).cmLoop-fm(0xc0006b9aa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:387 +0x34
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc0003d6d00, 0xc000723510)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 490 [chan receive]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder(0xc00080e2d0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:451 +0x27a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0007235a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x300

goroutine 492 [select]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).sendAndReceive(0xc0002b7b80, 0xdc0e60, 0xc0000d8be0, 0xdb6f60, 0xc0000d8c00, 0xc0003cf1a0, 0x1371750)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:611 +0x133
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).YeelK(0xc0002b7b80, 0xc0000d8be0, 0xc000000001, 0xc0000b38b0, 0xc000819b58)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:280 +0x70
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).YeelKNewMessages(0xc0008834a0, 0x0, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:806 +0x3f7
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionTTTTTTT(0xc0008834a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:692 +0x124
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionTTTTTTT-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:633 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0007235c0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(TTTTTTT).newsleeperTTTTTTT
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:633 +0x19c

goroutine 500 [select, 1 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).twLoop(0xc0006b6000, 0xc0004d7920)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:427 +0x143
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).twLoop-fm(0xc0004d7920)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:384 +0x34
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc00001b140, 0xc000723690)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 439 [select, 15 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(RoomsTTTTTTT).multiplex(0xc000349720, 0xc0006b9aa0, 0xc0004d6780, 0xc0004bdda0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/Roomss.go:129 +0x182
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).createTTTTTTT.func1(0xc0006b9aa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:829 +0x85
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc0003d6d00, 0xc00001b240)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 438 [select]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).hbLoop(0xc0006b6000, 0xc0004d7920)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:403 +0x120
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).hbLoop-fm(0xc0004d7920)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:369 +0x34
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc00001b140, 0xc0007231c0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 443 [runnable]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionManager(0xc000883130)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:659 +0x399
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionManager-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:632 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc000723420)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).newsleeperTTTTTTT
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:632 +0x14e

goroutine 460 [chan receive, 15 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher(0xc00080e360)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:336 +0x5a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d9860)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2ae

goroutine 489 [chan receive, 15 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher(0xc00080e2d0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:336 +0x5a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc000723590)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2ae

goroutine 452 [chan receive, 15 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher(0xc00080e750)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:336 +0x5a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d9660)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2ae

goroutine 484 [chan receive, 15 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher(0xc00080e630)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:336 +0x5a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0007234e0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2ae

goroutine 440 [select, 15 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(RoomsTTTTTTT).multiplex(0xc0002c2c80, 0xc0004d7920, 0xc000081e00, 0xc00069faa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/Roomss.go:129 +0x182
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).createTTTTTTT.func1(0xc0004d7920)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:829 +0x85
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc00001b140, 0xc00001b320)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 448 [chan receive]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder(0xc00080e7e0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:451 +0x27a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc000723480)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x300

goroutine 458 [select]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionManager(0xc0007e57c0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:659 +0x399
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionManager-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:632 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d9730)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).newsleeperTTTTTTT
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:632 +0x14e

goroutine 456 [chan receive, 15 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher(0xc00080e480)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:336 +0x5a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d9700)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2ae

goroutine 455 [select]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).sendAndReceive(0xc0003ec9a0, 0xdc0e60, 0xc0003525c0, 0xdb6f60, 0xc0003525e0, 0xc000418030, 0x1371750)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:611 +0x133
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).YeelK(0xc0003ec9a0, 0xc0003525c0, 0xc000000007, 0xc00034d4b8, 0xc0004a2708)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:280 +0x70
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).YeelKNewMessages(0xc0007e56d0, 0x0, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:806 +0x3f7
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionTTTTTTT(0xc0007e56d0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:692 +0x124
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionTTTTTTT-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:633 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d96a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(TTTTTTT).newsleeperTTTTTTT
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:633 +0x19c

goroutine 495 [select, 15 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(RoomsTTTTTTT).multiplex(0xc0003da140, 0xc0004d7920, 0xc000081e00, 0xc00069faa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/Roomss.go:129 +0x182
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).createTTTTTTT.func1(0xc0004d7920)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:829 +0x85
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc00001b140, 0xc00001b760)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 454 [select]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionManager(0xc0007e56d0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:659 +0x399
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionManager-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:632 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d9690)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).newsleeperTTTTTTT
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:632 +0x14e

goroutine 482 [select]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).sendAndReceive(0xc0003ecc60, 0xdc0e60, 0xc000352680, 0xdb6f60, 0xc0003526a0, 0xc0003cee10, 0x1371750)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:611 +0x133
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).YeelK(0xc0003ecc60, 0xc000352680, 0xc000000005, 0xc00034d680, 0xc0004a2a48)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:280 +0x70
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).YeelKNewMessages(0xc000883270, 0x0, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:806 +0x3f7
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionTTTTTTT(0xc000883270)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:692 +0x124
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionTTTTTTT-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:633 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0007234a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(TTTTTTT).newsleeperTTTTTTT
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:633 +0x19c

goroutine 481 [select]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionManager(0xc000883270)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:659 +0x399
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionManager-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:632 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc000723490)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).newsleeperTTTTTTT
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:632 +0x14e

goroutine 445 [select, 15 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(RoomsTTTTTTT).multiplex(0xc0002c2f00, 0xc0006b9aa0, 0xc0004d6780, 0xc0004bdda0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/Roomss.go:129 +0x182
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).createTTTTTTT.func1(0xc0006b9aa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:829 +0x85
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc0003d6d00, 0xc00001b3c0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 487 [select, 1 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).twLoop(0xc0006b60f0, 0xc0006b9aa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:427 +0x143
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).twLoop-fm(0xc0006b9aa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:384 +0x34
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc0003d6d00, 0xc000723500)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 499 [select, 15 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(RoomsTTTTTTT).multiplex(0xc0003dba40, 0xc0004d7920, 0xc000081e00, 0xc00069faa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/Roomss.go:129 +0x182
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).createTTTTTTT.func1(0xc0004d7920)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:829 +0x85
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc00001b140, 0xc00001b860)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 496 [chan receive, 15 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher(0xc00080e3f0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:336 +0x5a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0007235f0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2ae

goroutine 464 [chan receive, 15 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher(0xc00080e6c0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:336 +0x5a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).dispatcher-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d98d0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:163 +0x2ae

goroutine 605 [select, 14 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.newClientStream.func4(0xdc3380, 0xc000559980, 0xc00033e780, 0xc0005d6000, 0xc000366800)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/stream.go:255 +0x174
created by github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.newClientStream
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/stream.go:254 +0xb18

goroutine 501 [select]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).cmLoop(0xc0006b6000, 0xc0004d7920)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:456 +0x120
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).cmLoop-fm(0xc0004d7920)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:387 +0x34
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc00001b140, 0xc000723730)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 1062 [chan receive, 1 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver(0xc0002b6dc0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:683 +0xdc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005c08a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*sleeper).Open.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x809

goroutine 465 [chan receive]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder(0xc00080e6c0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:451 +0x27a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsTTTTTTT).responseFeeder-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d98e0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).TTTTTTTooms
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:164 +0x300

goroutine 416 [IO wait]:
internal/poll.runtime_pollWait(0x7f95d80e3bd8, 0x72, 0xc0008c6af0)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc000400298, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc000400298, 0xc000910000, 0x8000, 0x8000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc000400280, 0xc000910000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc000400280, 0xc000910000, 0x8000, 0x8000, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc0000947e8, 0xc000910000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc0008fe030, 0xdb78e0, 0xc0000947e8, 0x5, 0xc0000947e8, 0x44afba)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc0006a3c00, 0xd1d017, 0xc0006a3d20, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc0006a3c00, 0xc0005dc240, 0x24, 0x24, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.fill(0xdc30e0, 0xc0006a3c00, 0xc0005dc240, 0x24, 0x24, 0x0, 0x11)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:535 +0x53
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.(*mongoSocket).readLoop(0xc0000e3ce0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:551 +0x5f7
created by github.com/1234567890123/ship/gopkg.in/mgo%2ev2.newSocket
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:194 +0x1df

goroutine 462 [select]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionManager(0xc0007e5900)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:659 +0x399
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionManager-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:632 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d9880)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*TTTTTTT).newsleeperTTTTTTT
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:632 +0x14e

goroutine 301 [IO wait]:
internal/poll.runtime_pollWait(0x7f95d80e3b08, 0x72, 0xc00085caf0)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0007d1398, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0007d1398, 0xc0009d6000, 0x1000, 0x1000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0007d1380, 0xc0009d6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0007d1380, 0xc0009d6000, 0x1000, 0x1000, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc000094820, 0xc0009d6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc0009c9200, 0xdb78e0, 0xc000094820, 0x5, 0xc000094820, 0x44afba)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc0007a5500, 0xd1d017, 0xc0007a5620, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc0007a5500, 0xc000739f50, 0x24, 0x24, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.fill(0xdc30e0, 0xc0007a5500, 0xc000739f50, 0x24, 0x24, 0x0, 0x11)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:535 +0x53
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.(*mongoSocket).readLoop(0xc000938380)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:551 +0x5f7
created by github.com/1234567890123/ship/gopkg.in/mgo%2ev2.newSocket
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:194 +0x1df

goroutine 1814 [chan receive, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch(0xc0004435c0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:439 +0xcc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc00050c640)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*asyncSleeper).newRoomsSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x1c2

goroutine 425 [select]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).hbLoop(0xc0006b60f0, 0xc0006b9aa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:403 +0x120
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).hbLoop-fm(0xc0006b9aa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:369 +0x34
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc0003d6d00, 0xc0003d9130)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 466 [select, 15 minutes]:
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(RoomsTTTTTTT).multiplex(0xc0000d2a00, 0xc0006b9aa0, 0xc0004d6780, 0xc0004bdda0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/Roomss.go:129 +0x182
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(TTTTTTT).createTTTTTTT.func1(0xc0006b9aa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/TTTTTTT.go:829 +0x85
github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go.func1(0xc0003d6d00, 0xc0003d71e0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:73 +0x89
created by github.com/1234567890123/ship/github.com/bsm/vishnu-cluster.(loopTomb).Go
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/bsm/vishnu-cluster/util.go:69 +0x66

goroutine 629 [select]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(lessor).deadlineLoop(0xc00070eaa0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/lease.go:527 +0x190
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(lessor).KeepAlive.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/lease.go:294 +0x65

goroutine 463 [select]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).sendAndReceive(0xc0002b78c0, 0xdc0e60, 0xc0000d8b20, 0xdb6f60, 0xc0000d8b60, 0xc000418ba0, 0x1371750)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:611 +0x133
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).YeelK(0xc0002b78c0, 0xc0000d8b20, 0xc000000003, 0xc0000b3840, 0xc000819a88)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:280 +0x70
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).YeelKNewMessages(0xc0007e5900, 0x0, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:806 +0x3f7
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionTTTTTTT(0xc0007e5900)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:692 +0x124
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperTTTTTTT).rtyyeusptionTTTTTTT-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:633 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003d9890)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(TTTTTTT).newsleeperTTTTTTT
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/TTTTTTT.go:633 +0x19c

goroutine 393 [IO wait]:
internal/poll.runtime_pollWait(0x7f95d80e3f18, 0x72, 0xc00082baf0)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0002a2a18, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0002a2a18, 0xc0007bc000, 0x8000, 0x8000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0002a2a00, 0xc0007bc000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0002a2a00, 0xc0007bc000, 0x8000, 0x8000, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc0000945f0, 0xc0007bc000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc000730060, 0xdb78e0, 0xc0000945f0, 0x5, 0xc0000945f0, 0x44afba)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc0006a2e00, 0xd1d017, 0xc0006a2f20, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc0006a2e00, 0xc000041410, 0x24, 0x24, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.fill(0xdc30e0, 0xc0006a2e00, 0xc000041410, 0x24, 0x24, 0x0, 0x11)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:535 +0x53
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.(*mongoSocket).readLoop(0xc0000e2fc0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:551 +0x5f7
created by github.com/1234567890123/ship/gopkg.in/mgo%2ev2.newSocket
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:194 +0x1df

goroutine 300 [IO wait]:
internal/poll.runtime_pollWait(0x7f95d80e3ca8, 0x72, 0xc0008cfaf0)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0007d1298, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0007d1298, 0xc000944000, 0x8000, 0x8000)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0007d1280, 0xc000944000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0007d1280, 0xc000944000, 0x8000, 0x8000, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc00000c040, 0xc000944000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc0007f7650, 0xdb78e0, 0xc00000c040, 0x5, 0xc00000c040, 0x44afba)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc000734000, 0xd1d017, 0xc000734120, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc000734000, 0xc000739ef0, 0x24, 0x24, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.fill(0xdc30e0, 0xc000734000, 0xc000739ef0, 0x24, 0x24, 0x0, 0x11)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:535 +0x53
github.com/1234567890123/ship/gopkg.in/mgo%2ev2.(*mongoSocket).readLoop(0xc00080ae00)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:551 +0x5f7
created by github.com/1234567890123/ship/gopkg.in/mgo%2ev2.newSocket
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/gopkg.in/mgo.v2/socket.go:194 +0x1df

goroutine 2540 [chan receive, 1 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch(0xc000a6cd80)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:439 +0xcc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0000d4320)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*asyncSleeper).newRoomsSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x1c2

goroutine 2539 [chan receive, 1 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch(0xc000a6cc60)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:439 +0xcc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0000d4260)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*asyncSleeper).newRoomsSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x1c2

goroutine 2546 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f95d80e37c8, 0x72, 0xc0005fe858)
/usr/local/go1.11.4/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(pollDesc).wait(0xc0002a2698, 0x72, 0xffffffffffffff00, 0xdb8c00, 0x132e548)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(pollDesc).waitRead(0xc0002a2698, 0xc0005a1000, 0x400, 0x400)
/usr/local/go1.11.4/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(FD).Read(0xc0002a2680, 0xc0005a1000, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/internal/poll/fd_unix.go:169 +0x179
net.(netFD).Read(0xc0002a2680, 0xc0005a1000, 0x400, 0x400, 0x408a6b, 0xc00001c000, 0xbedb40)
/usr/local/go1.11.4/go/src/net/fd_unix.go:202 +0x4f
net.(conn).Read(0xc00000c118, 0xc0005a1000, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/net/net.go:177 +0x68
crypto/tls.(block).readFromUntil(0xc0009c8cf0, 0xdb78e0, 0xc00000c118, 0x5, 0xc00000c118, 0x30)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(Conn).readRecord(0xc0006a2700, 0xd1d017, 0xc0006a2820, 0x456ea0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(Conn).Read(0xc0006a2700, 0xc000595000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go1.11.4/go/src/crypto/tls/conn.go:1145 +0xf1
net/http.(persistConn).Read(0xc0007a2000, 0xc000595000, 0x1000, 0x1000, 0xc0005fec70, 0x404c95, 0xc0007f4960)
/usr/local/go1.11.4/go/src/net/http/transport.go:1497 +0x75
bufio.(Reader).fill(0xc000a6dc80)
/usr/local/go1.11.4/go/src/bufio/bufio.go:100 +0x10f
bufio.(Reader).Peek(0xc000a6dc80, 0x1, 0x0, 0x0, 0x1, 0xc0008b54a0, 0x0)
/usr/local/go1.11.4/go/src/bufio/bufio.go:132 +0x3f
net/http.(persistConn).readLoop(0xc0007a2000)
/usr/local/go1.11.4/go/src/net/http/transport.go:1645 +0x1a2
created by net/http.(*Transport).dialConn
/usr/local/go1.11.4/go/src/net/http/transport.go:1338 +0x941

goroutine 1053 [chan receive, 1 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver(0xc0002b69a0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:683 +0xdc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005aa790)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*sleeper).Open.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x809

goroutine 1049 [chan receive, 1 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).newsleeperSleeper.func1()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:569 +0x7c
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003f4740)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).newsleeperSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:568 +0x1f5

goroutine 1078 [chan receive, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch(0xc0000969c0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:439 +0xcc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005aaab0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*asyncSleeper).newRoomsSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x1c2

goroutine 1054 [chan receive, 11 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch(0xc0003dfec0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:439 +0xcc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005aa940)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*asyncSleeper).newRoomsSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x1c2

goroutine 1080 [select, 1 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperSleeper).run(0xc000096ae0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:615 +0x1e5
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperSleeper).run-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:565 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005aaad0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*asyncSleeper).newsleeperSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:565 +0x195

goroutine 1048 [select, 1 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperSleeper).run(0xc0003df9e0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:615 +0x1e5
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperSleeper).run-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:565 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005aa4f0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*asyncSleeper).newsleeperSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:565 +0x195

goroutine 628 [select, 14 minutes]:
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(recvBufferReader).read(0xc0008aca50, 0xc000724d90, 0x5, 0x5, 0xc000559980, 0xc000366800, 0x2a)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:133 +0x1ff
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(recvBufferReader).Read(0xc0008aca50, 0xc000724d90, 0x5, 0x5, 0xc0005edc40, 0x8ce135, 0x3)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:122 +0x5a
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(transportReader).Read(0xc0007ce000, 0xc000724d90, 0x5, 0x5, 0x4639ae, 0xc0003668c8, 0xd1d108)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:403 +0x55
io.ReadAtLeast(0xdb7500, 0xc0007ce000, 0xc000724d90, 0x5, 0x5, 0x5, 0xc000559980, 0xc000366800, 0x5)
/usr/local/go1.11.4/go/src/io/io.go:310 +0x88
io.ReadFull(0xdb7500, 0xc0007ce000, 0xc000724d90, 0x5, 0x5, 0xc000366700, 0xdbd220, 0xc0005edd18)
/usr/local/go1.11.4/go/src/io/io.go:329 +0x58
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport.(Stream).Read(0xc000366800, 0xc000724d90, 0x5, 0x5, 0x0, 0xc000157ee8, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/transport/transport.go:387 +0xbf
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(parser).recvMsg(0xc000724d80, 0x7fffffff, 0x4, 0xc0005edee0, 0xc000832f00, 0xc0005ede18, 0x203000, 0x203000)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/rpc_util.go:270 +0x65
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.recv(0xc000724d80, 0xdbf1e0, 0x139b850, 0xc000366800, 0x0, 0x0, 0xc847a0, 0xc000724ea0, 0x7fffffff, 0x0, ...)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/rpc_util.go:356 +0x4d
github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc.(clientStream).RecvMsg(0xc0005d6000, 0xc847a0, 0xc000724ea0, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/ship/google.golang.org/grpc/stream.go:392 +0xbb
github.com/1234567890123/ship/github.com/coreos/etcd/etcdserver/etcdserverpb.(leaseLeaseKeepAliveClient).Recv(0xc00050c430, 0xc000724da0, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:3630 +0x62
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(lessor).recvKeepAliveLoop(0xc00070eaa0, 0x0, 0x0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/lease.go:441 +0x9e
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(*lessor).KeepAlive.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/lease.go:293 +0x43

goroutine 1081 [chan receive, 1 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).newsleeperSleeper.func1()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:569 +0x7c
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003f4ce0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).newsleeperSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:568 +0x1f5

goroutine 1813 [chan receive, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch(0xc0003dd200)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:439 +0xcc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc00050c390)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*asyncSleeper).newRoomsSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x1c2

goroutine 606 [select]:
github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(lessor).sendKeepAliveLoop(0xc00070eaa0, 0xdc31a0, 0xc00050c430)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/lease.go:567 +0x334
created by github.com/1234567890123/ship/github.com/coreos/etcd/clientv3.(lessor).resetRecv
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/coreos/etcd/clientv3/lease.go:484 +0x18a

goroutine 1045 [chan receive, 1 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(topicSleeper).dispatch(0xc0007ad040)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:324 +0x63
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(topicSleeper).dispatch-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:319 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005aa3c0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*asyncSleeper).newTopicSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:319 +0x1aa

goroutine 2486 [chan receive, 1 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch(0xc000920420)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:439 +0xcc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0007225b0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*asyncSleeper).newRoomsSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x1c2

goroutine 1056 [select, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperSleeper).run(0xc0003dff20)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:615 +0x1e5
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeperSleeper).run-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:565 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005aa970)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*asyncSleeper).newsleeperSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:565 +0x195

goroutine 1073 [chan receive, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).newsleeperSleeper.func1()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:569 +0x7c
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0003f4bc0)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(asyncSleeper).newsleeperSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:568 +0x1f5

goroutine 1061 [chan receive, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver(0xc0002b7080)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:683 +0xdc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(sleeper).responseReceiver-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc0005c0710)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*sleeper).Open.func1
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/sleeper.go:150 +0x809

goroutine 1806 [chan receive, 6 minutes]:
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch(0xc00069fe60)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:439 +0xcc
github.com/1234567890123/ship/github.com/BruceLee/vishnu.(RoomsSleeper).dispatch-fm()
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x2a
github.com/1234567890123/ship/github.com/BruceLee/vishnu.withRecover(0xc000722590)
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/utils.go:45 +0x43
created by github.com/1234567890123/ship/github.com/BruceLee/vishnu.(*asyncSleeper).newRoomsSleeper
/var/jenkins_home/workspace/XXXX/src/github.com/1234567890123/ship/github.com/BruceLee/vishnu/async_sleeper.go:425 +0x1c2

rax 0x2000
rbx 0x7f95d827c350
rcx 0x8
rdx 0x440
rdi 0x4553c0
rsi 0x137ddc0
rbp 0xc000399f70
rsp 0xc000399f30
r8 0x1
r9 0x11
r10 0x7f95d827c350
r11 0x7fffffffffffff
r12 0x0
r13 0x49
r14 0x49
r15 0x49
rip 0x420c60
rflags 0x10206
cs 0x33
fs 0x0
gs 0x0

vishwaspai on 22 Jan 2019

I have to agree that a SIGILL that has occurred once is most likely memory corruption. SIGILL does not indicate a transient condition.

ianlancetaylor on 22 Jan 2019

Text/code section can not be corrupted by user code, right? Are you suggesting that there must be hardware bit flip type corruption which caused SIGILL ?

vishwaspai on 22 Jan 2019

Yes, I am suggesting a hardware problem.

Of course if it happens on a different machine then my suggestion is almost certainly wrong. But a single occurrence of SIGILL on a single machine is likely a hardware problem. They do happen.

ianlancetaylor on 22 Jan 2019

👍4

Thanks. I've asked the hosting provider if they provide any guarantees on hardware corruption (using ECC or other mechanisms).

vishwaspai on 22 Jan 2019

Hosting provider is yet to get back to us on the memory corruption.

But this time a different program running in the same environment (different H/W node and VM) has crashed with SIGSEGV around the same location. If this is same as the previous problem, then it is not pointing towards hardware corruption.

Stack trace:

2019-01-23T09:04:07.309761319Z fatal error: unexpected signal during runtime execution
2019-01-23T09:04:07.3173336Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x2000 pc=0x420c60]
2019-01-23T09:04:07.317392724Z
2019-01-23T09:04:07.317424651Z runtime stack:
2019-01-23T09:04:07.317428766Z runtime.throw(0xcb85af, 0x2a)
2019-01-23T09:04:07.317432608Z /usr/local/go1.11.4/go/src/runtime/panic.go:608 +0x72
2019-01-23T09:04:07.317437176Z runtime.sigpanic()
2019-01-23T09:04:07.317456298Z /usr/local/go1.11.4/go/src/runtime/signal_unix.go:374 +0x2f2
2019-01-23T09:04:07.317503889Z runtime.deductSweepCredit(0x2000, 0x0)
2019-01-23T09:04:07.317508757Z /usr/local/go1.11.4/go/src/runtime/mgcsweep.go:390
2019-01-23T09:04:07.317512666Z runtime.(mcentral).cacheSpan(0x13107f8, 0xc000686c10)
2019-01-23T09:04:07.317542341Z /usr/local/go1.11.4/go/src/runtime/mcentral.go:43 +0x60
2019-01-23T09:04:07.317556883Z runtime.(mcache).refill(0x7fc90a4956c0, 0xc000686b38)
2019-01-23T09:04:07.317565158Z /usr/local/go1.11.4/go/src/runtime/mcache.go:122 +0x95
2019-01-23T09:04:07.317576046Z runtime.(*mcache).nextFree.func1()
2019-01-23T09:04:07.317580455Z /usr/local/go1.11.4/go/src/runtime/malloc.go:749 +0x32
2019-01-23T09:04:07.317584669Z runtime.systemstack(0x0)
2019-01-23T09:04:07.317588461Z /usr/local/go1.11.4/go/src/runtime/asm_amd64.s:351 +0x66
2019-01-23T09:04:07.317592646Z runtime.mstart()
2019-01-23T09:04:07.317600498Z /usr/local/go1.11.4/go/src/runtime/proc.go:1229

Assembly at PC:

(gdb) disassemble 0x420c60
Dump of assembler code for function runtime.deductSweepCredit:
0x0000000000420c60 <+0>: mov %fs:0xfffffffffffffff8,%rcx
0x0000000000420c69 <+9>: cmp 0x10(%rcx),%rsp
0x0000000000420c6d <+13>: jbe 0x420dc8
0x0000000000420c73 <+19>: sub $0x20,%rsp
0x0000000000420c77 <+23>: mov %rbp,0x18(%rsp)
0x0000000000420c7c <+28>: lea 0x18(%rsp),%rbp
0x0000000000420c81 <+33>: movsd 0xeeeaef(%rip),%xmm0 # 0x130f778
0x0000000000420c89 <+41>: xorps %xmm1,%xmm1
0x0000000000420c8c <+44>: ucomisd %xmm1,%xmm0
0x0000000000420c90 <+48>: jne 0x420c98
0x0000000000420c92 <+50>: jnp 0x420dbe
0x0000000000420c98 <+56>: lea 0xef4f71(%rip),%rax # 0x1315c10
0x0000000000420c9f <+63>: cmpb $0x0,(%rax)
0x0000000000420ca2 <+66>: jne 0x420daa
0x0000000000420ca8 <+72>: mov 0x28(%rsp),%rcx
0x0000000000420cad <+77>: mov 0x30(%rsp),%rdx
0x0000000000420cb2 <+82>: jmp 0x420d1c
0x0000000000420cb4 <+84>: lea 0xef4f55(%rip),%rax # 0x1315c10
0x0000000000420cbb <+91>: mov 0x28(%rsp),%rcx
0x0000000000420cc0 <+96>: mov 0x30(%rsp),%rdx
0x0000000000420cc5 <+101>: mov 0x8(%rsp),%rbx
0x0000000000420cca <+106>: mov 0x10(%rsp),%rsi
0x0000000000420ccf <+111>: xorps %xmm1,%xmm1

Full stack trace attached below.
sigsegv.log

vishwaspai on 23 Jan 2019

SIGILL at the same location on a different Node+VM happened again.

2019-01-23T12:16:06.202968107Z SIGILL: illegal instruction
2019-01-23T12:16:06.203027783Z PC=0x420c60 m=7 sigcode=2
2019-01-23T12:16:06.203034227Z
2019-01-23T12:16:06.203038628Z goroutine 0 [idle]:
2019-01-23T12:16:06.203155238Z runtime.deductSweepCredit(0x2000, 0x0)
2019-01-23T12:16:06.203162347Z /usr/local/go1.11.4/go/src/runtime/mgcsweep.go:390
2019-01-23T12:16:06.203167258Z runtime.(mcentral).cacheSpan(0x1386598, 0xc000684b68)
2019-01-23T12:16:06.20317127Z /usr/local/go1.11.4/go/src/runtime/mcentral.go:43 +0x60
2019-01-23T12:16:06.203175828Z runtime.(mcache).refill(0x7f73c182e6c0, 0x45bb0e)
2019-01-23T12:16:06.203180122Z /usr/local/go1.11.4/go/src/runtime/mcache.go:122 +0x95
2019-01-23T12:16:06.203184492Z runtime.(*mcache).nextFree.func1()
2019-01-23T12:16:06.203220704Z /usr/local/go1.11.4/go/src/runtime/malloc.go:749 +0x32
2019-01-23T12:16:06.203226099Z runtime.systemstack(0x0)
2019-01-23T12:16:06.203230572Z /usr/local/go1.11.4/go/src/runtime/asm_amd64.s:351 +0x66
2019-01-23T12:16:06.203235035Z runtime.mstart()
2019-01-23T12:16:06.203239356Z /usr/local/go1.11.4/go/src/runtime/proc.go:1229
..
..
2019-01-23T12:16:06.311989625Z
2019-01-23T12:16:06.311991885Z rax 0x2000
2019-01-23T12:16:06.311994209Z rbx 0x7f73bf5ca898
2019-01-23T12:16:06.311996577Z rcx 0x7
2019-01-23T12:16:06.311998853Z rdx 0x380
2019-01-23T12:16:06.31202658Z rdi 0x4553c0
2019-01-23T12:16:06.312029347Z rsi 0x137ee00
2019-01-23T12:16:06.312031667Z rbp 0xc0000c1f70
2019-01-23T12:16:06.312033982Z rsp 0xc0000c1f30
2019-01-23T12:16:06.312036301Z r8 0x0
2019-01-23T12:16:06.312038613Z r9 0xe
2019-01-23T12:16:06.312040859Z r10 0x7f73bf5ca898
2019-01-23T12:16:06.312043194Z r11 0x0
2019-01-23T12:16:06.312045797Z r12 0x40
2019-01-23T12:16:06.312048176Z r13 0x40
2019-01-23T12:16:06.312050418Z r14 0xc000684fc8
2019-01-23T12:16:06.312052699Z r15 0x0
2019-01-23T12:16:06.31205496Z rip 0x420c60
2019-01-23T12:16:06.312057312Z rflags 0x10206
2019-01-23T12:16:06.312059595Z cs 0x33
2019-01-23T12:16:06.312061877Z fs 0x0
2019-01-23T12:16:06.312064173Z gs 0x0

vishwaspai on 23 Jan 2019

Thanks, that's interesting. I have no idea what could cause this. The instruction looks fine, and presumably the same instruction is executed many times in different functions, so why a SIGILL?

Is there anything in the kernel logs?

The sigcode value of 2 is ILL_ILLOPN which means "illegal operand." But looking at the kernel code it seems that is always the sigcode value for SIGILL so maybe that doesn't mean anything.

In the stack trace deductSweepCredit is called from cacheSpan, which starts with the exact same instruction. It seems implausible that %fs would have changed or that the memory mappings would have changed. The stack pointer is not at a page boundary. The SIGSEGV refers to address 0x2000, which doesn't make much sense.

You said it was a static Go binary. Is there any C code in the program, or is it pure Go?

Approximately how often does this happen? That is, you've seen it twice or maybe three times; how many times is your program running in that time period?

I gather your program is running inside a VM? Do you have any information about that?

ianlancetaylor on 23 Jan 2019

Thanks for looking into this.

Can not find anything interesting in kernel log.
Seen this problem 3 times in 2 days. 2 of those with the same program running in different dockers (pods on VM in kubernetes). Once (sigsegv) in a different program.
All binaries are 64bit X86 Linux Pure Go. (no C/Cgo involved)

vishwaspai on 23 Jan 2019

3 times in 2 days, but approximately how many instances of the program were run during those 2 days?

ianlancetaylor on 23 Jan 2019

CC @aclements

ianlancetaylor on 23 Jan 2019

3 times in 2 days, but approximately how many instances of the program were run during those 2 days?

2 SIGILL from 4 instances and 1 SIGSEGV from 2 instances.
I just had another SIGSEGV but stack trace is different (but around runtime). I'm not adding this here as that might not be relevant for this issue.
I've now started suspecting hardware again. Last 2 instances have happened on the same hardware node. I'm asking hosting provider to check that specific hardware.

vishwaspai on 23 Jan 2019

Hosting provider is still investigating the issue. We are still using the VM so that they can debug the issue.
We had another instance of this issue (SIGSEGV). What I'm unable to comprehend is the fact that even if this is hardware issue , why does this happen exactly at this point in this specific routine.
This time it was completely different program that crashed (on the same VM). This is just filebeat go client with very little code change from us.

2019-01-25T09:20:06.904573082Z fatal error: unexpected signal during runtime execution
2019-01-25T09:20:06.906516185Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x2000 pc=0x420c14]
2019-01-25T09:20:06.906580802Z
2019-01-25T09:20:06.906585345Z runtime stack:
2019-01-25T09:20:06.9068671Z runtime.throw(0xf52299, 0x2a)
2019-01-25T09:20:07.004213396Z /usr/local/go1.11.4/go/src/runtime/panic.go:608 +0x72
2019-01-25T09:20:07.004427053Z runtime.sigpanic()
2019-01-25T09:20:07.004431906Z /usr/local/go1.11.4/go/src/runtime/signal_unix.go:374 +0x2f2
2019-01-25T09:20:07.004434998Z runtime.deductSweepCredit(0x2000, 0x0)
2019-01-25T09:20:07.004437681Z /usr/local/go1.11.4/go/src/runtime/mgcsweep.go:390 +0x4
2019-01-25T09:20:07.00444066Z runtime.(mcentral).cacheSpan(0x18e1018, 0xc00058e900)
2019-01-25T09:20:07.004443314Z /usr/local/go1.11.4/go/src/runtime/mcentral.go:43 +0x60
2019-01-25T09:20:07.004445959Z runtime.(mcache).refill(0x7f225e8ae6c0, 0xc000001608)
2019-01-25T09:20:07.004448507Z /usr/local/go1.11.4/go/src/runtime/mcache.go:122 +0x95
2019-01-25T09:20:07.004451169Z runtime.(*mcache).nextFree.func1()
2019-01-25T09:20:07.004454107Z /usr/local/go1.11.4/go/src/runtime/malloc.go:749 +0x32
2019-01-25T09:20:07.004456776Z runtime.systemstack(0x0)
2019-01-25T09:20:07.004459596Z /usr/local/go1.11.4/go/src/runtime/asm_amd64.s:351 +0x66
2019-01-25T09:20:07.004462483Z runtime.mstart()
2019-01-25T09:20:07.004500865Z /usr/local/go1.11.4/go/src/runtime/proc.go:1229

vishwaspai on 25 Jan 2019

No significant update from hosting provider yet. Several crashes have happened in the past few days. One being "kubectl" command (which is not built or modified by us).

Latest instance is failure of "kubectl" command (trace below).

2019-01-28T02:48:05.901242067Z SIGILL: illegal instruction
2019-01-28T02:48:05.901333098Z PC=0x420d36 m=6 sigcode=2
2019-01-28T02:48:05.90133851Z
2019-01-28T02:48:05.901342404Z goroutine 0 [idle]:
2019-01-28T02:48:05.901346133Z runtime.(mheap).alloc_m(0x25676e0, 0x1, 0x1c, 0x7fb8afe69000)
2019-01-28T02:48:05.901350095Z /usr/local/go/src/runtime/mheap.go:536 +0x6
2019-01-28T02:48:05.901354404Z runtime.(mheap).alloc.func1()
2019-01-28T02:48:05.901360696Z /usr/local/go/src/runtime/mheap.go:627 +0x4b
2019-01-28T02:48:05.901364515Z runtime.systemstack(0xc42005be90)
2019-01-28T02:48:05.901368395Z /usr/local/go/src/runtime/asm_amd64.s:343 +0xab
2019-01-28T02:48:05.901769977Z runtime.(mheap).alloc(0x25676e0, 0x1, 0x1000000001c, 0x7fb8aff29718)
2019-01-28T02:48:05.901790953Z /usr/local/go/src/runtime/mheap.go:628 +0xa0
2019-01-28T02:48:05.901795287Z runtime.(mcentral).grow(0x2569650, 0x0)
2019-01-28T02:48:05.901799256Z /usr/local/go/src/runtime/mcentral.go:212 +0x8e
2019-01-28T02:48:05.901803089Z runtime.(mcentral).cacheSpan(0x2569650, 0x7fb8aff29bd8)
2019-01-28T02:48:05.901806835Z /usr/local/go/src/runtime/mcentral.go:93 +0x12a
2019-01-28T02:48:05.90181091Z runtime.(mcache).refill(0x7fb8b01694b0, 0x7fb80000001c, 0x7fb8aff29bd8)
2019-01-28T02:48:05.901814819Z /usr/local/go/src/runtime/mcache.go:122 +0xa0
2019-01-28T02:48:05.901836021Z runtime.(*mcache).nextFree.func1()
2019-01-28T02:48:05.901839737Z /usr/local/go/src/runtime/malloc.go:538 +0x32
2019-01-28T02:48:05.901843404Z runtime.systemstack(0xc420018000)
2019-01-28T02:48:05.901846845Z /usr/local/go/src/runtime/asm_amd64.s:327 +0x79
2019-01-28T02:48:05.901850514Z runtime.mstart()
2019-01-28T02:48:05.901853923Z /usr/local/go/src/runtime/proc.go:1132

vishwaspai on 29 Jan 2019

Occurrence of these issues stopped after change of hardware node on which these crashes were happening. Hosting provider has not provided any technical details around what might have caused the issue (ticket still pending with them).

This golang git issue can be closed.

/close

vishwaspai on 14 Feb 2019

Thanks for following up.

Sorry about your hardware.

ianlancetaylor on 14 Feb 2019

Was this page helpful?

0 / 5 - 0 ratings

Related issues

x/tools/cmd/stringer: cannot import vendored dependencies

stub42 · 3Comments

x/tools/cmd/gorename: gorename fails with on cgo-using files

ashb · 3Comments

cmd/compile: testing/quick misbehaves on Nexus 9 linux/arm64

rsc · 3Comments

all: T.FailNow used in goroutines in standard library tests

dominikh · 3Comments

proposal: cmd/vet: vet should warn when time.Time type (or types embed it) is used as map keys.

go101 · 3Comments

Go: runtime: SIGILL: illegal instruction from runtime.deductSweepCredit

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

Most helpful comment

All 19 comments

Related issues

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?