Cadvisor: fatal error: unexpected signal during runtime execution
Docker: Docker version 1.11.2, build b9f10c9
cAdvisor: 0.24.0
Every so often when creating a cAdvisor container, the following panic occurs:
user@host:~$ docker logs cadvisor
I0927 18:58:03.959701 1 storagedriver.go:50] Caching stats in memory for 2m0s
I0927 18:58:03.960076 1 manager.go:140] cAdvisor running in container: "/docker/fd5991c3273f0a2c05b16e236e381404545782caecd1b3075e3b4b5018b6e35e"
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x63 pc=0x7f77cca691ed]
runtime stack:
runtime.throw(0xfdb2e0, 0x2a)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigpanic_unix.go:12 +0x5a
goroutine 26 [syscall, locked to thread]:
runtime.cgocall(0xaba660, 0xc820031bd8, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/cgocall.go:123 +0x11b fp=0xc820031b78 sp=0xc820031b48
net._C2func_getaddrinfo(0x7f77bc0008c0, 0x0, 0xc820314300, 0xc82006c0a8, 0x0, 0x0, 0x0)
??:0 +0x55 fp=0xc820031bd8 sp=0xc820031b78
net.cgoLookupIPCNAME(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1070c00)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:111 +0x448 fp=0xc820031d50 sp=0xc820031bd8
net.cgoLookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:163 +0x56 fp=0xc820031da8 sp=0xc820031d50
net.lookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup_unix.go:67 +0x94 fp=0xc820031e18 sp=0xc820031da8
net.glob.func16(0x1070c30, 0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/hook.go:10 +0x4d fp=0xc820031e58 sp=0xc820031e18
net.lookupIPDeadline.func1(0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:106 +0x71 fp=0xc820031ed8 sp=0xc820031e58
internal/singleflight.(*Group).doCall(0x16439c0, 0xc8202fc410, 0xec0d10, 0x9, 0xc8202c08e0)
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:93 +0x2c fp=0xc820031f88 sp=0xc820031ed8
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820031f90 sp=0xc820031f88
created by internal/singleflight.(*Group).DoChan
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:86 +0x3ee
goroutine 1 [select]:
net.lookupIPDeadline(0xec0d10, 0x9, 0xecf7cb83e, 0xb260ae, 0x1644940, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:109 +0x6a6
net.internetAddrList(0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7cb83e, 0xb260ae, 0x1644940, 0x0, 0x0, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/ipsock.go:252 +0x6ee
net.resolveAddrList(0xea1cb8, 0x4, 0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7cb83e, 0xb260ae, 0x1644940, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:158 +0x466
net.(*Dialer).Dial(0xc8202935b0, 0xea6418, 0x3, 0xec0d10, 0xf, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:216 +0x124
net.DialTimeout(0xea6418, 0x3, 0xec0d10, 0xf, 0x77359400, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:200 +0xa3
github.com/google/cadvisor/container/rkt.Client.func1()
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:44 +0x59
sync.(*Once).Do(0x166a218, 0x106fd68)
/home/stclair/.gvm/gos/go1.6.3/src/sync/once.go:44 +0xe4
github.com/google/cadvisor/container/rkt.Client(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:79 +0x47
github.com/google/cadvisor/container/rkt.RktPath(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:85 +0x48
github.com/google/cadvisor/manager.New(0xc82022c100, 0x7f77ce16f958, 0x166a170, 0xdf8475800, 0x7f77ce16b501, 0xc8203552f0, 0xc820355440, 0x0, 0x0, 0x0, ...)
/home/stclair/go/src/github.com/google/cadvisor/manager/manager.go:146 +0x341
main.main()
/home/stclair/go/src/github.com/google/cadvisor/cadvisor.go:127 +0x654
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1
goroutine 3 [syscall]:
os/signal.signal_recv(0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:28 +0x37
goroutine 4 [chan receive]:
github.com/google/cadvisor/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x1644ca0)
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:882 +0x67
created by github.com/google/cadvisor/vendor/github.com/golang/glog.init.1
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:410 +0x297
goroutine 24 [select]:
net/http.(*persistConn).writeLoop(0xc820120ea0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1280 +0x472
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:861 +0x10cb
goroutine 23 [IO wait]:
net.runtime_pollWait(0x7f77ce170a98, 0x72, 0xc82019b000)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82035ff00, 0x72, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82035ff00, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82035fea0, 0xc82019b000, 0x1000, 0x1000, 0x0, 0x7f77ce167028, 0xc8200660a0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82006c080, 0xc82019b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f77ce170b58, 0xc82006c080, 0xc820120f08, 0xc82019b000, 0x1000, 0x1000, 0x409fb3, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1690 +0x67
net/http.(*noteEOFReader).Read(0xc8202c04c0, 0xc82019b000, 0x1000, 0x1000, 0xc82002fd1d, 0x0, 0x0)
<autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820064ba0)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820064ba0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc820120ea0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1076 +0x177
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:860 +0x10a6
I0927 18:58:04.287426 1 storagedriver.go:50] Caching stats in memory for 2m0s
I0927 18:58:04.287794 1 manager.go:140] cAdvisor running in container: "/docker/fd5991c3273f0a2c05b16e236e381404545782caecd1b3075e3b4b5018b6e35e"
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x63 pc=0x7fe297ae61ed]
runtime stack:
runtime.throw(0xfdb2e0, 0x2a)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigpanic_unix.go:12 +0x5a
goroutine 36 [syscall, locked to thread]:
runtime.cgocall(0xaba660, 0xc820031bd8, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/cgocall.go:123 +0x11b fp=0xc820031b78 sp=0xc820031b48
net._C2func_getaddrinfo(0x7fe2900008c0, 0x0, 0xc820325a70, 0xc82010c090, 0x0, 0x0, 0x0)
??:0 +0x55 fp=0xc820031bd8 sp=0xc820031b78
net.cgoLookupIPCNAME(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1070c00)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:111 +0x448 fp=0xc820031d50 sp=0xc820031bd8
net.cgoLookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:163 +0x56 fp=0xc820031da8 sp=0xc820031d50
net.lookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup_unix.go:67 +0x94 fp=0xc820031e18 sp=0xc820031da8
net.glob.func16(0x1070c30, 0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/hook.go:10 +0x4d fp=0xc820031e58 sp=0xc820031e18
net.lookupIPDeadline.func1(0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:106 +0x71 fp=0xc820031ed8 sp=0xc820031e58
internal/singleflight.(*Group).doCall(0x16439c0, 0xc8202f2370, 0xec0d10, 0x9, 0xc820361b00)
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:93 +0x2c fp=0xc820031f88 sp=0xc820031ed8
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820031f90 sp=0xc820031f88
created by internal/singleflight.(*Group).DoChan
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:86 +0x3ee
goroutine 1 [select]:
net.lookupIPDeadline(0xec0d10, 0x9, 0xecf7cb83e, 0x1443d926, 0x1644940, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:109 +0x6a6
net.internetAddrList(0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7cb83e, 0x1443d926, 0x1644940, 0x0, 0x0, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/ipsock.go:252 +0x6ee
net.resolveAddrList(0xea1cb8, 0x4, 0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7cb83e, 0x1443d926, 0x1644940, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:158 +0x466
net.(*Dialer).Dial(0xc8202e15b0, 0xea6418, 0x3, 0xec0d10, 0xf, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:216 +0x124
net.DialTimeout(0xea6418, 0x3, 0xec0d10, 0xf, 0x77359400, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:200 +0xa3
github.com/google/cadvisor/container/rkt.Client.func1()
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:44 +0x59
sync.(*Once).Do(0x166a218, 0x106fd68)
/home/stclair/.gvm/gos/go1.6.3/src/sync/once.go:44 +0xe4
github.com/google/cadvisor/container/rkt.Client(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:79 +0x47
github.com/google/cadvisor/container/rkt.RktPath(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:85 +0x48
github.com/google/cadvisor/manager.New(0xc820329c40, 0x7fe29d2bb4b0, 0x166a170, 0xdf8475800, 0x7fe29e2f1501, 0xc820339e00, 0xc820339ef0, 0x0, 0x0, 0x0, ...)
/home/stclair/go/src/github.com/google/cadvisor/manager/manager.go:146 +0x341
main.main()
/home/stclair/go/src/github.com/google/cadvisor/cadvisor.go:127 +0x654
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1
goroutine 18 [syscall]:
os/signal.signal_recv(0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:28 +0x37
goroutine 19 [chan receive]:
github.com/google/cadvisor/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x1644ca0)
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:882 +0x67
created by github.com/google/cadvisor/vendor/github.com/golang/glog.init.1
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:410 +0x297
goroutine 35 [runnable]:
github.com/google/cadvisor/vendor/github.com/docker/engine-api/client/transport/cancellable.Do.func3(0x7fe29d2ba240, 0xc8201084f0, 0xc820335cb0, 0xc820112de0)
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/docker/engine-api/client/transport/cancellable/cancellable.go:77
created by github.com/google/cadvisor/vendor/github.com/docker/engine-api/client/transport/cancellable.Do
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/docker/engine-api/client/transport/cancellable/cancellable.go:84 +0x336
goroutine 31 [IO wait]:
net.runtime_pollWait(0x7fe29d2bc5f0, 0x72, 0xc8202ac000)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820373c60, 0x72, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820373c60, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820373c00, 0xc8202ac000, 0x1000, 0x1000, 0x0, 0x7fe29e2ed050, 0xc820012120)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82010c058, 0xc8202ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fe29d2bc6b0, 0xc82010c058, 0xc8201150a8, 0xc8202ac000, 0x1000, 0x1000, 0x409fb3, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1690 +0x67
net/http.(*noteEOFReader).Read(0xc820361140, 0xc8202ac000, 0x1000, 0x1000, 0xc82002fd1d, 0x0, 0x0)
<autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc8201129c0)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8201129c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc820115040)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1076 +0x177
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:860 +0x10a6
goroutine 32 [select]:
net/http.(*persistConn).writeLoop(0xc820115040)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1280 +0x472
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:861 +0x10cb
I0927 18:58:04.713375 1 storagedriver.go:50] Caching stats in memory for 2m0s
I0927 18:58:04.713771 1 manager.go:140] cAdvisor running in container: "/docker/fd5991c3273f0a2c05b16e236e381404545782caecd1b3075e3b4b5018b6e35e"
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x63 pc=0x7fee9ddfa1ed]
runtime stack:
runtime.throw(0xfdb2e0, 0x2a)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigpanic_unix.go:12 +0x5a
goroutine 10 [syscall, locked to thread]:
runtime.cgocall(0xaba660, 0xc820031bd8, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/cgocall.go:123 +0x11b fp=0xc820031b78 sp=0xc820031b48
net._C2func_getaddrinfo(0x7fee900008c0, 0x0, 0xc8202e06c0, 0xc820020058, 0x0, 0x0, 0x0)
??:0 +0x55 fp=0xc820031bd8 sp=0xc820031b78
net.cgoLookupIPCNAME(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1070c00)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:111 +0x448 fp=0xc820031d50 sp=0xc820031bd8
net.cgoLookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:163 +0x56 fp=0xc820031da8 sp=0xc820031d50
net.lookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup_unix.go:67 +0x94 fp=0xc820031e18 sp=0xc820031da8
net.glob.func16(0x1070c30, 0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/hook.go:10 +0x4d fp=0xc820031e58 sp=0xc820031e18
net.lookupIPDeadline.func1(0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:106 +0x71 fp=0xc820031ed8 sp=0xc820031e58
internal/singleflight.(*Group).doCall(0x16439c0, 0xc8202f3f90, 0xec0d10, 0x9, 0xc8203339c0)
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:93 +0x2c fp=0xc820031f88 sp=0xc820031ed8
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820031f90 sp=0xc820031f88
created by internal/singleflight.(*Group).DoChan
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:86 +0x3ee
goroutine 1 [select]:
net.lookupIPDeadline(0xec0d10, 0x9, 0xecf7cb83e, 0x2f517cc4, 0x1644940, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:109 +0x6a6
net.internetAddrList(0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7cb83e, 0x2f517cc4, 0x1644940, 0x0, 0x0, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/ipsock.go:252 +0x6ee
net.resolveAddrList(0xea1cb8, 0x4, 0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7cb83e, 0x2f517cc4, 0x1644940, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:158 +0x466
net.(*Dialer).Dial(0xc8203c95b0, 0xea6418, 0x3, 0xec0d10, 0xf, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:216 +0x124
net.DialTimeout(0xea6418, 0x3, 0xec0d10, 0xf, 0x77359400, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:200 +0xa3
github.com/google/cadvisor/container/rkt.Client.func1()
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:44 +0x59
sync.(*Once).Do(0x166a218, 0x106fd68)
/home/stclair/.gvm/gos/go1.6.3/src/sync/once.go:44 +0xe4
github.com/google/cadvisor/container/rkt.Client(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:79 +0x47
github.com/google/cadvisor/container/rkt.RktPath(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:85 +0x48
github.com/google/cadvisor/manager.New(0xc82030bd00, 0x7feea0d44958, 0x166a170, 0xdf8475800, 0x7feea0d3e501, 0xc82030da10, 0xc82030db30, 0x0, 0x0, 0x0, ...)
/home/stclair/go/src/github.com/google/cadvisor/manager/manager.go:146 +0x341
main.main()
/home/stclair/go/src/github.com/google/cadvisor/cadvisor.go:127 +0x654
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1
goroutine 18 [syscall]:
os/signal.signal_recv(0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:28 +0x37
goroutine 19 [chan receive]:
github.com/google/cadvisor/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x1644ca0)
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:882 +0x67
created by github.com/google/cadvisor/vendor/github.com/golang/glog.init.1
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:410 +0x297
goroutine 29 [IO wait]:
net.runtime_pollWait(0x7fee9f504158, 0x72, 0xc820154000)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8202d6610, 0x72, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8202d6610, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8202d65b0, 0xc820154000, 0x1000, 0x1000, 0x0, 0x7feea0d3a050, 0xc820012120)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820114000, 0xc820154000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fee9f504218, 0xc820114000, 0xc820112888, 0xc820154000, 0x1000, 0x1000, 0x409fb3, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1690 +0x67
net/http.(*noteEOFReader).Read(0xc820280b00, 0xc820154000, 0x1000, 0x1000, 0xc82002fd1d, 0x0, 0x0)
<autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820056ae0)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820056ae0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc820112820)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1076 +0x177
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:860 +0x10a6
goroutine 30 [select]:
net/http.(*persistConn).writeLoop(0xc820112820)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1280 +0x472
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:861 +0x10cb
I0927 18:58:05.399579 1 storagedriver.go:50] Caching stats in memory for 2m0s
I0927 18:58:05.399783 1 manager.go:140] cAdvisor running in container: "/docker/fd5991c3273f0a2c05b16e236e381404545782caecd1b3075e3b4b5018b6e35e"
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x63 pc=0x7fdf7faf51ed]
runtime stack:
runtime.throw(0xfdb2e0, 0x2a)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigpanic_unix.go:12 +0x5a
goroutine 26 [syscall, locked to thread]:
runtime.cgocall(0xaba660, 0xc820257bd8, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/cgocall.go:123 +0x11b fp=0xc820257b78 sp=0xc820257b48
net._C2func_getaddrinfo(0x7fdf700008c0, 0x0, 0xc820311ef0, 0xc820020058, 0x0, 0x0, 0x0)
??:0 +0x55 fp=0xc820257bd8 sp=0xc820257b78
net.cgoLookupIPCNAME(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1070c00)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:111 +0x448 fp=0xc820257d50 sp=0xc820257bd8
net.cgoLookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7fdf819fee90)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:163 +0x56 fp=0xc820257da8 sp=0xc820257d50
net.lookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup_unix.go:67 +0x94 fp=0xc820257e18 sp=0xc820257da8
net.glob.func16(0x1070c30, 0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/hook.go:10 +0x4d fp=0xc820257e58 sp=0xc820257e18
net.lookupIPDeadline.func1(0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:106 +0x71 fp=0xc820257ed8 sp=0xc820257e58
internal/singleflight.(*Group).doCall(0x16439c0, 0xc82000e6e0, 0xec0d10, 0x9, 0xc82026e7c0)
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:93 +0x2c fp=0xc820257f88 sp=0xc820257ed8
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820257f90 sp=0xc820257f88
created by internal/singleflight.(*Group).DoChan
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:86 +0x3ee
goroutine 1 [select]:
net.lookupIPDeadline(0xec0d10, 0x9, 0xecf7cb83f, 0x1b0cc5b6, 0x1644940, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:109 +0x6a6
net.internetAddrList(0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7cb83f, 0x1b0cc5b6, 0x1644940, 0x0, 0x0, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/ipsock.go:252 +0x6ee
net.resolveAddrList(0xea1cb8, 0x4, 0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7cb83f, 0x1b0cc5b6, 0x1644940, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:158 +0x466
net.(*Dialer).Dial(0xc8203875b0, 0xea6418, 0x3, 0xec0d10, 0xf, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:216 +0x124
net.DialTimeout(0xea6418, 0x3, 0xec0d10, 0xf, 0x77359400, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:200 +0xa3
github.com/google/cadvisor/container/rkt.Client.func1()
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:44 +0x59
sync.(*Once).Do(0x166a218, 0x106fd68)
/home/stclair/.gvm/gos/go1.6.3/src/sync/once.go:44 +0xe4
github.com/google/cadvisor/container/rkt.Client(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:79 +0x47
github.com/google/cadvisor/container/rkt.RktPath(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:85 +0x48
github.com/google/cadvisor/manager.New(0xc82031bec0, 0x7fdf819fdc18, 0x166a170, 0xdf8475800, 0x7fdf819f7501, 0xc820356e70, 0xc820356f60, 0x0, 0x0, 0x0, ...)
/home/stclair/go/src/github.com/google/cadvisor/manager/manager.go:146 +0x341
main.main()
/home/stclair/go/src/github.com/google/cadvisor/cadvisor.go:127 +0x654
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1
goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:28 +0x37
goroutine 7 [chan receive]:
github.com/google/cadvisor/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x1644ca0)
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:882 +0x67
created by github.com/google/cadvisor/vendor/github.com/golang/glog.init.1
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:410 +0x297
goroutine 13 [IO wait]:
net.runtime_pollWait(0x7fdf819fed58, 0x72, 0xc8201c3000)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8202fc290, 0x72, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8202fc290, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8202fc230, 0xc8201c3000, 0x1000, 0x1000, 0x0, 0x7fdf819f3050, 0xc820012120)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82010c338, 0xc8201c3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fdf819fee18, 0xc82010c338, 0xc820058618, 0xc8201c3000, 0x1000, 0x1000, 0x409fb3, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1690 +0x67
net/http.(*noteEOFReader).Read(0xc82026e620, 0xc8201c3000, 0x1000, 0x1000, 0xc82002fd1d, 0x0, 0x0)
<autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820056d20)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820056d20, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8200585b0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1076 +0x177
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:860 +0x10a6
goroutine 14 [select]:
net/http.(*persistConn).writeLoop(0xc8200585b0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1280 +0x472
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:861 +0x10cb
I0927 18:58:06.470446 1 storagedriver.go:50] Caching stats in memory for 2m0s
I0927 18:58:06.470620 1 manager.go:140] cAdvisor running in container: "/docker/fd5991c3273f0a2c05b16e236e381404545782caecd1b3075e3b4b5018b6e35e"
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x63 pc=0x7fede4e4c1ed]
runtime stack:
runtime.throw(0xfdb2e0, 0x2a)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigpanic_unix.go:12 +0x5a
goroutine 27 [syscall, locked to thread]:
runtime.cgocall(0xaba660, 0xc820032bd8, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/cgocall.go:123 +0x11b fp=0xc820032b78 sp=0xc820032b48
net._C2func_getaddrinfo(0x7fedd80008c0, 0x0, 0xc8202e2d50, 0xc8201142d8, 0x0, 0x0, 0x0)
??:0 +0x55 fp=0xc820032bd8 sp=0xc820032b78
net.cgoLookupIPCNAME(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1070c00)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:111 +0x448 fp=0xc820032d50 sp=0xc820032bd8
net.cgoLookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x28646c6968632d68)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:163 +0x56 fp=0xc820032da8 sp=0xc820032d50
net.lookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup_unix.go:67 +0x94 fp=0xc820032e18 sp=0xc820032da8
net.glob.func16(0x1070c30, 0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/hook.go:10 +0x4d fp=0xc820032e58 sp=0xc820032e18
net.lookupIPDeadline.func1(0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:106 +0x71 fp=0xc820032ed8 sp=0xc820032e58
internal/singleflight.(*Group).doCall(0x16439c0, 0xc82000e8c0, 0xec0d10, 0x9, 0xc820282a60)
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:93 +0x2c fp=0xc820032f88 sp=0xc820032ed8
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820032f90 sp=0xc820032f88
created by internal/singleflight.(*Group).DoChan
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:86 +0x3ee
goroutine 1 [select]:
net.lookupIPDeadline(0xec0d10, 0x9, 0xecf7cb840, 0x1ff26135, 0x1644940, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:109 +0x6a6
net.internetAddrList(0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7cb840, 0x1ff26135, 0x1644940, 0x0, 0x0, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/ipsock.go:252 +0x6ee
net.resolveAddrList(0xea1cb8, 0x4, 0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7cb840, 0x1ff26135, 0x1644940, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:158 +0x466
net.(*Dialer).Dial(0xc8202c15b0, 0xea6418, 0x3, 0xec0d10, 0xf, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:216 +0x124
net.DialTimeout(0xea6418, 0x3, 0xec0d10, 0xf, 0x77359400, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:200 +0xa3
github.com/google/cadvisor/container/rkt.Client.func1()
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:44 +0x59
sync.(*Once).Do(0x166a218, 0x106fd68)
/home/stclair/.gvm/gos/go1.6.3/src/sync/once.go:44 +0xe4
github.com/google/cadvisor/container/rkt.Client(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:79 +0x47
github.com/google/cadvisor/container/rkt.RktPath(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:85 +0x48
github.com/google/cadvisor/manager.New(0xc820208f00, 0x7fede5d4c758, 0x166a170, 0xdf8475800, 0x7fede758f501, 0xc82033ed50, 0xc82033ee70, 0x0, 0x0, 0x0, ...)
/home/stclair/go/src/github.com/google/cadvisor/manager/manager.go:146 +0x341
main.main()
/home/stclair/go/src/github.com/google/cadvisor/cadvisor.go:127 +0x654
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1
goroutine 18 [syscall]:
os/signal.signal_recv(0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:28 +0x37
goroutine 19 [chan receive]:
github.com/google/cadvisor/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x1644ca0)
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:882 +0x67
created by github.com/google/cadvisor/vendor/github.com/golang/glog.init.1
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:410 +0x297
goroutine 24 [IO wait]:
net.runtime_pollWait(0x7fede5d4d898, 0x72, 0xc820152000)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82034bf00, 0x72, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82034bf00, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82034bea0, 0xc820152000, 0x1000, 0x1000, 0x0, 0x7fede758b050, 0xc820012120)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8201142b0, 0xc820152000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fede5d4d958, 0xc8201142b0, 0xc8202d4d68, 0xc820152000, 0x1000, 0x1000, 0x409fb3, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1690 +0x67
net/http.(*noteEOFReader).Read(0xc820282820, 0xc820152000, 0x1000, 0x1000, 0xc82002fd1d, 0x0, 0x0)
<autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820056a80)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820056a80, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8202d4d00)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1076 +0x177
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:860 +0x10a6
goroutine 25 [select]:
net/http.(*persistConn).writeLoop(0xc8202d4d00)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1280 +0x472
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:861 +0x10cb
goroutine 26 [runnable]:
github.com/google/cadvisor/vendor/github.com/docker/engine-api/client/transport/cancellable.Do.func3(0x7fede5d4c500, 0xc8200132d0, 0xc8201135c0, 0xc820056c60)
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/docker/engine-api/client/transport/cancellable/cancellable.go:77
created by github.com/google/cadvisor/vendor/github.com/docker/engine-api/client/transport/cancellable.Do
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/docker/engine-api/client/transport/cancellable/cancellable.go:84 +0x336
I0927 18:58:08.323716 1 storagedriver.go:50] Caching stats in memory for 2m0s
I0927 18:58:08.323954 1 manager.go:140] cAdvisor running in container: "/docker/fd5991c3273f0a2c05b16e236e381404545782caecd1b3075e3b4b5018b6e35e"
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x63 pc=0x7f44e13821ed]
runtime stack:
runtime.throw(0xfdb2e0, 0x2a)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigpanic_unix.go:12 +0x5a
goroutine 15 [syscall, locked to thread]:
runtime.cgocall(0xaba660, 0xc82002cbd8, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/cgocall.go:123 +0x11b fp=0xc82002cb78 sp=0xc82002cb48
net._C2func_getaddrinfo(0x7f44d40008c0, 0x0, 0xc8203265a0, 0xc820020058, 0x0, 0x0, 0x0)
??:0 +0x55 fp=0xc82002cbd8 sp=0xc82002cb78
net.cgoLookupIPCNAME(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1070c00)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:111 +0x448 fp=0xc82002cd50 sp=0xc82002cbd8
net.cgoLookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:163 +0x56 fp=0xc82002cda8 sp=0xc82002cd50
net.lookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup_unix.go:67 +0x94 fp=0xc82002ce18 sp=0xc82002cda8
net.glob.func16(0x1070c30, 0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/hook.go:10 +0x4d fp=0xc82002ce58 sp=0xc82002ce18
net.lookupIPDeadline.func1(0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:106 +0x71 fp=0xc82002ced8 sp=0xc82002ce58
internal/singleflight.(*Group).doCall(0x16439c0, 0xc8202f4000, 0xec0d10, 0x9, 0xc82032c4a0)
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:93 +0x2c fp=0xc82002cf88 sp=0xc82002ced8
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc82002cf90 sp=0xc82002cf88
created by internal/singleflight.(*Group).DoChan
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:86 +0x3ee
goroutine 1 [select]:
net.lookupIPDeadline(0xec0d10, 0x9, 0xecf7cb842, 0x16de91f2, 0x1644940, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:109 +0x6a6
net.internetAddrList(0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7cb842, 0x16de91f2, 0x1644940, 0x0, 0x0, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/ipsock.go:252 +0x6ee
net.resolveAddrList(0xea1cb8, 0x4, 0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7cb842, 0x16de91f2, 0x1644940, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:158 +0x466
net.(*Dialer).Dial(0xc82015f5b0, 0xea6418, 0x3, 0xec0d10, 0xf, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:216 +0x124
net.DialTimeout(0xea6418, 0x3, 0xec0d10, 0xf, 0x77359400, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:200 +0xa3
github.com/google/cadvisor/container/rkt.Client.func1()
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:44 +0x59
sync.(*Once).Do(0x166a218, 0x106fd68)
/home/stclair/.gvm/gos/go1.6.3/src/sync/once.go:44 +0xe4
github.com/google/cadvisor/container/rkt.Client(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:79 +0x47
github.com/google/cadvisor/container/rkt.RktPath(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:85 +0x48
github.com/google/cadvisor/manager.New(0xc820320000, 0x7f44e3a88f98, 0x166a170, 0xdf8475800, 0x7f44e3a85501, 0xc820333620, 0xc820333710, 0x0, 0x0, 0x0, ...)
/home/stclair/go/src/github.com/google/cadvisor/manager/manager.go:146 +0x341
main.main()
/home/stclair/go/src/github.com/google/cadvisor/cadvisor.go:127 +0x654
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1
goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:28 +0x37
goroutine 7 [chan receive]:
github.com/google/cadvisor/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x1644ca0)
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:882 +0x67
created by github.com/google/cadvisor/vendor/github.com/golang/glog.init.1
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:410 +0x297
goroutine 12 [IO wait]:
net.runtime_pollWait(0x7f44e3a8a0d8, 0x72, 0xc82018f000)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820362df0, 0x72, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820362df0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820362d90, 0xc82018f000, 0x1000, 0x1000, 0x0, 0x7f44e3a81050, 0xc820012120)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82012a058, 0xc82018f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f44e3a8a198, 0xc82012a058, 0xc8201230a8, 0xc82018f000, 0x1000, 0x1000, 0x409fb3, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1690 +0x67
net/http.(*noteEOFReader).Read(0xc820359b80, 0xc82018f000, 0x1000, 0x1000, 0xc82002fd1d, 0x0, 0x0)
<autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820126b40)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820126b40, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc820123040)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1076 +0x177
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:860 +0x10a6
goroutine 13 [select]:
net/http.(*persistConn).writeLoop(0xc820123040)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1280 +0x472
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:861 +0x10cb
I0927 18:58:11.764057 1 storagedriver.go:50] Caching stats in memory for 2m0s
I0927 18:58:11.764897 1 manager.go:140] cAdvisor running in container: "/docker/fd5991c3273f0a2c05b16e236e381404545782caecd1b3075e3b4b5018b6e35e"
W0927 18:58:11.852700 1 manager.go:148] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused
I0927 18:58:11.904924 1 fs.go:116] Filesystem partitions: map[/dev/vda:{mountpoint:/var/lib/docker/aufs major:253 minor:0 fsType:ext4 blockSize:0}]
I0927 18:58:11.907257 1 info.go:47] Couldn't collect info from any of the files in "/rootfs/etc/machine-id,/var/lib/dbus/machine-id"
I0927 18:58:11.907323 1 manager.go:195] Machine: {NumCores:2 CpuFrequency:2493748 MemoryCapacity:6259445760 MachineID: SystemUUID:A33BE379-9CC3-4EA2-9A8B-CDD0E6ED138C BootID:b72ecac0-0504-4462-9317-a84caa98c2d2 Filesystems:[{Device:/dev/vda Capacity:31572619264 Type:vfs Inodes:1966080 HasInodes:true} {Device:none Capacity:31572619264 Type:vfs Inodes:1966080 HasInodes:true}] DiskMap:map[253:0:{Name:vda Major:253 Minor:0 Size:32212254720 Scheduler:none} 253:16:{Name:vdb Major:253 Minor:16 Size:67108864 Scheduler:none}] NetworkDevices:[{Name:br-0c84a425725d MacAddress:02:42:f3:8b:5f:35 Speed:0 Mtu:1500} {Name:br-e3a5fd02d063 MacAddress:02:42:42:f8:13:7e Speed:0 Mtu:1500} {Name:eth0 MacAddress:74:db:d1:a0:9d:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:6259445760 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:1 Memory:0 Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I0927 18:58:11.908004 1 manager.go:201] Version: {KernelVersion:3.13.0-24-generic ContainerOsVersion:Alpine Linux v3.4 DockerVersion:1.11.2 CadvisorVersion:v0.24.0 CadvisorRevision:0cdf491}
I0927 18:58:11.961965 1 factory.go:295] Registering Docker factory
W0927 18:58:11.962020 1 manager.go:244] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused
I0927 18:58:11.962046 1 factory.go:54] Registering systemd factory
I0927 18:58:11.962212 1 factory.go:86] Registering Raw factory
I0927 18:58:11.962370 1 manager.go:1082] Started watching for new ooms in manager
W0927 18:58:11.962571 1 manager.go:272] Could not configure a source for OOM detection, disabling OOM events: unable to find any kernel log file available from our set: [/var/log/kern.log /var/log/messages /var/log/syslog]
I0927 18:58:11.963428 1 manager.go:285] Starting recovery of all containers
I0927 18:58:11.989164 1 manager.go:290] Recovery completed
I0927 18:58:12.004511 1 cadvisor.go:157] Starting cAdvisor version: v0.24.0-0cdf491 on port 8080
Output of docker inspect cadvisor:
user@host:~$ docker inspect cadvisor
[
{
"Id": "541dd6ba397ae87f23a88c0c554808105aa829a3d0bccb6bf4d1b81e40aa2665",
"Created": "2016-09-27T19:45:55.924846092Z",
"Path": "/usr/bin/cadvisor",
"Args": [
"-logtostderr",
"--docker_only"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 11531,
"ExitCode": 0,
"Error": "",
"StartedAt": "2016-09-27T19:46:00.071536827Z",
"FinishedAt": "2016-09-27T19:45:58.401842412Z"
},
"Image": "sha256:106e303be3a42f4b896da911ff5d78700be093d8a74b8f3018245a6e46f91b38",
"ResolvConfPath": "/var/lib/docker/containers/541dd6ba397ae87f23a88c0c554808105aa829a3d0bccb6bf4d1b81e40aa2665/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/541dd6ba397ae87f23a88c0c554808105aa829a3d0bccb6bf4d1b81e40aa2665/hostname",
"HostsPath": "/var/lib/docker/containers/541dd6ba397ae87f23a88c0c554808105aa829a3d0bccb6bf4d1b81e40aa2665/hosts",
"LogPath": "/var/lib/docker/containers/541dd6ba397ae87f23a88c0c554808105aa829a3d0bccb6bf4d1b81e40aa2665/541dd6ba397ae87f23a88c0c554808105aa829a3d0bccb6bf4d1b81e40aa2665-json.log",
"Name": "/cadvisor",
"RestartCount": 5,
"Driver": "aufs",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/:/rootfs:ro",
"/var/run:/var/run:rw",
"/sys:/sys:ro",
"/var/lib/docker/:/var/lib/docker:ro"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {
"max-file": "10",
"max-size": "10m"
}
},
"NetworkMode": "host",
"PortBindings": {
"8080/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "8080"
}
]
},
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"StorageOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"BlkioIOps": 0,
"BlkioBps": 0,
"SandboxSize": 0
},
"GraphDriver": {
"Name": "aufs",
"Data": null
},
"Mounts": [
{
"Source": "/sys",
"Destination": "/sys",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Source": "/var/lib/docker",
"Destination": "/var/lib/docker",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Source": "/",
"Destination": "/rootfs",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Source": "/var/run",
"Destination": "/var/run",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "REDACTED",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"8080/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"GLIBC_VERSION=2.23-r3"
],
"Cmd": [
"--docker_only"
],
"Image": "REDACTED/myproject/cadvisor:v0.24.0",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/usr/bin/cadvisor",
"-logtostderr"
],
"OnBuild": null,
"Labels": {}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "1eca19db05e184e8eef98b87fd2dd2c8debb3bedb3ee8657def6b9acde2a29d8",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/default",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"host": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "3b3ac0f48bdf2083f56b35d5c29716de10f22d33c919b6a3df4bd862a243c6d1",
"EndpointID": "ed99644f52a86afa7db5d1b1f7ae788b2ba99eb1b6e132952cacf17a5a0aeedd",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": ""
}
}
}
}
]
This seems to be unique to cAdvisor 0.24.0, because we had not seen this in the 0.23.x series.
amcrn
All 30 comments
The log reports that this is a fatal error, but it looks like cAdvisor still starts up. Are you actually seeing any issues from this, other than the scary log message?
It looks like a segfault from deep in the network package (probably something wrong with the glibc version we patch into alpine), but if the panic is recovered somewhere, it should only affect rkt, which it doesn't look like you're running with.
Also, what OS / distro are you running on? I couldn't reproduce on my local workstation.
timstclair
on 27 Sep 2016
The log reports that this is a fatal error, but it looks like cAdvisor still starts up. Are you actually seeing any issues from this, other than the scary log message?
I assumed this was because we have the Docker restart policy set to always, which at some point, it _does_ run. Might have been a poor assumption, apologies. Let me confirm this is the case.
Also, what OS / distro are you running on? I couldn't reproduce on my local workstation.
Ubuntu 14.04 LTS w/ Kernel at 3.13.0-24-generic.
amcrn
on 28 Sep 2016
@timstclair I confirmed that the retries are from cAdvisor and not due to our Docker restart policy. So it definitely recovers, but these retries account for almost 10 seconds added to startup:
I0927 18:58:03.959701 1 storagedriver.go:50] Caching stats in memory for 2m0s
...
I0927 18:58:12.004511 1 cadvisor.go:157] Starting cAdvisor version: v0.24.0-0cdf491 on port 8080
This came to our attention because we have health checks on the cAdvisor container after we start it that were failing (We hit /api/ looking for 'Supported API versions', 3 retries, exponential backoff starting at 1 second).
10 seconds is pretty significant in terms of container startup, so https://github.com/google/cadvisor/issues/1483#issuecomment-250032379 would definitely be a welcome addition :)
amcrn
on 29 Sep 2016
I'd be surprised if that error was contributing more than a couple seconds (though I could be wrong). Do you have a lot of containers running? If there are a lot of containers running on the system, cAdvisor can take a little while to load them before it starts.
timstclair
on 29 Sep 2016
@timstclair I don't see the same 10 second delay when using cAdvisor 0.23.8
Using cAdvisor 0.23.8, the logs after a docker restart cadvisor:
I0930 02:01:28.321492 1 manager.go:1062] Exiting thread watching subcontainers
I0930 02:01:28.321706 1 manager.go:347] Exiting global housekeeping thread
I0930 02:01:28.321744 1 cadvisor.go:185] Exiting given signal: terminated
I0930 02:01:28.548682 1 storagedriver.go:50] Caching stats in memory for 2m0s
I0930 02:01:28.548903 1 manager.go:138] cAdvisor running in container: "/docker/d5612ca931170142e0dc73207b3a33627b225511fef2864835f3e0f84e195f2b"
W0930 02:01:28.555835 1 manager.go:146] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused
I0930 02:01:28.565094 1 fs.go:139] Filesystem partitions: map[/dev/vdb:{mountpoint:/rootfs/mnt major:253 minor:16 fsType: blockSize:0} /dev/vda:{mountpoint:/var/lib/docker/aufs major:253 minor:0 fsType:ext4 blockSize:0}]
I0930 02:01:28.627006 1 info.go:47] Couldn't collect info from any of the files in "/rootfs/etc/machine-id,/var/lib/dbus/machine-id"
I0930 02:01:28.627087 1 manager.go:192] Machine: {NumCores:16 CpuFrequency:2593748 MemoryCapacity:126749863936 MachineID: SystemUUID:49A2923F-CAD1-4203-A4A2-BBBC04508309 BootID:51251646-7302-45c1-a004-6c828c5c5aea Filesystems:[{Device:none Capacity:31572619264 Type:vfs Inodes:1966080} {Device:/dev/vda Capacity:31572619264 Type:vfs Inodes:1966080} {Device:/dev/vdb Capacity:845381279744 Type:vfs Inodes:52428800}] DiskMap:map[253:0:{Name:vda Major:253 Minor:0 Size:32212254720 Scheduler:none} 253:16:{Name:vdb Major:253 Minor:16 Size:858993459200 Scheduler:none} 253:32:{Name:vdc Major:253 Minor:32 Size:67108864 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:74:db:d1:a0:a4:a3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:7845466112 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:1 Memory:7926992896 Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:2 Memory:7926988800 Cores:[{Id:0 Threads:[2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:3 Memory:7926992896 Cores:[{Id:0 Threads:[3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:4 Memory:7926988800 Cores:[{Id:0 Threads:[4] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:5 Memory:7926992896 Cores:[{Id:0 Threads:[5] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:6 Memory:7926988800 Cores:[{Id:0 Threads:[6] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:7 Memory:7926992896 Cores:[{Id:0 Threads:[7] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:8 Memory:7926984704 Cores:[{Id:0 Threads:[8] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:9 Memory:7926992896 Cores:[{Id:0 Threads:[9] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:10 Memory:7926988800 Cores:[{Id:0 Threads:[10] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:11 Memory:7926992896 Cores:[{Id:0 Threads:[11] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:12 Memory:7926988800 Cores:[{Id:0 Threads:[12] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:13 Memory:7926992896 Cores:[{Id:0 Threads:[13] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:14 Memory:7926988800 Cores:[{Id:0 Threads:[14] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:15 Memory:7926530048 Cores:[{Id:0 Threads:[15] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I0930 02:01:28.627909 1 manager.go:198] Version: {KernelVersion:3.13.0-24-generic ContainerOsVersion:Alpine Linux v3.2 DockerVersion:1.11.2 CadvisorVersion:0.23.8 CadvisorRevision:5c5b2b8}
I0930 02:01:28.637104 1 factory.go:228] Registering Docker factory
E0930 02:01:28.637203 1 manager.go:240] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused
I0930 02:01:28.637227 1 factory.go:54] Registering systemd factory
I0930 02:01:28.641716 1 factory.go:86] Registering Raw factory
I0930 02:01:28.646170 1 manager.go:1072] Started watching for new ooms in manager
W0930 02:01:28.646346 1 manager.go:268] Could not configure a source for OOM detection, disabling OOM events: unable to find any kernel log file available from our set: [/var/log/kern.log /var/log/messages /var/log/syslog]
I0930 02:01:28.647367 1 manager.go:281] Starting recovery of all containers
I0930 02:01:28.675280 1 manager.go:286] Recovery completed
I0930 02:01:28.706153 1 cadvisor.go:151] Starting cAdvisor version: 0.23.8-5c5b2b8 on port 8080
Do you have a lot of containers running?
Only 3 containers on this machine.
Another update: I can't cause the delay/fault behavior in cAdvisor 0.24.0 consistently. It happens on some restarts, but not on others :/
amcrn
on 30 Sep 2016
Hi, i see the same, running the binary direct on CoreOS 1122.2.0, and cadvisor does not start
cadvisor -logtostderr
I0930 08:02:45.269108 47423 storagedriver.go:50] Caching stats in memory for 2m0s
I0930 08:02:45.269314 47423 manager.go:140] cAdvisor running in container: "/user.slice/user-0.slice/session-c43.scope"
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x0 pc=0x0]
runtime stack:
runtime.throw(0xfdb2e0, 0x2a)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigpanic_unix.go:12 +0x5a
goroutine 59 [syscall, locked to thread]:
runtime.cgocall(0xaba660, 0xc820051bd8, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/cgocall.go:123 +0x11b fp=0xc820051b78 sp=0xc820051b48
net._C2func_getaddrinfo(0x7fa0400008c0, 0x0, 0xc8202c0450, 0xc820358218, 0x0, 0x0, 0x0)
??:0 +0x55 fp=0xc820051bd8 sp=0xc820051b78
net.cgoLookupIPCNAME(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:111 +0x448 fp=0xc820051d50 sp=0xc820051bd8
net.cgoLookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:163 +0x56 fp=0xc820051da8 sp=0xc820051d50
net.lookupIP(0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup_unix.go:67 +0x94 fp=0xc820051e18 sp=0xc820051da8
net.glob.func16(0x1070c30, 0xec0d10, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/hook.go:10 +0x4d fp=0xc820051e58 sp=0xc820051e18
net.lookupIPDeadline.func1(0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:106 +0x71 fp=0xc820051ed8 sp=0xc820051e58
internal/singleflight.(*Group).doCall(0x16439c0, 0xc82031c320, 0xec0d10, 0x9, 0xc82039f820)
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:93 +0x2c fp=0xc820051f88 sp=0xc820051ed8
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820051f90 sp=0xc820051f88
created by internal/singleflight.(*Group).DoChan
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:86 +0x3ee
goroutine 1 [select]:
net.lookupIPDeadline(0xec0d10, 0x9, 0xecf7ff707, 0x1e461e5e, 0x1644940, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:109 +0x6a6
net.internetAddrList(0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7ff707, 0x1e461e5e, 0x1644940, 0x0, 0x0, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/ipsock.go:252 +0x6ee
net.resolveAddrList(0xea1cb8, 0x4, 0xea6418, 0x3, 0xec0d10, 0xf, 0xecf7ff707, 0xc81e461e5e, 0x1644940, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:158 +0x466
net.(*Dialer).Dial(0xc8203055b0, 0xea6418, 0x3, 0xec0d10, 0xf, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:216 +0x124
net.DialTimeout(0xea6418, 0x3, 0xec0d10, 0xf, 0x77359400, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:200 +0xa3
github.com/google/cadvisor/container/rkt.Client.func1()
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:44 +0x59
sync.(*Once).Do(0x166a218, 0x106fd68)
/home/stclair/.gvm/gos/go1.6.3/src/sync/once.go:44 +0xe4
github.com/google/cadvisor/container/rkt.Client(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:79 +0x47
github.com/google/cadvisor/container/rkt.RktPath(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:85 +0x48
github.com/google/cadvisor/manager.New(0xc82034f980, 0x7fa04d04c118, 0x166a170, 0xdf8475800, 0x7fa04c008501, 0xc8203b7140, 0xc8203b74a0, 0x0, 0x0, 0x0, ...)
/home/stclair/go/src/github.com/google/cadvisor/manager/manager.go:146 +0x341
main.main()
/home/stclair/go/src/github.com/google/cadvisor/cadvisor.go:127 +0x654
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1
goroutine 3 [syscall]:
os/signal.signal_recv(0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:28 +0x37
goroutine 4 [chan receive]:
github.com/google/cadvisor/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x1644ca0)
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:882 +0x67
created by github.com/google/cadvisor/vendor/github.com/golang/glog.init.1
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:410 +0x297
goroutine 56 [IO wait]:
net.runtime_pollWait(0x7fa04d04d258, 0x72, 0xc820540000)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8203b81b0, 0x72, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8203b81b0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8203b8150, 0xc820540000, 0x1000, 0x1000, 0x0, 0x7fa04c004028, 0xc8200880a0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820358120, 0xc820540000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fa04d04d318, 0xc820358120, 0xc820364a28, 0xc820540000, 0x1000, 0x1000, 0x409fb3, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1690 +0x67
net/http.(*noteEOFReader).Read(0xc82039f660, 0xc820540000, 0x1000, 0x1000, 0xc82004fd1d, 0x0, 0x0)
<autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820016840)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820016840, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8203649c0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1076 +0x177
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:860 +0x10a6
goroutine 57 [select]:
net/http.(*persistConn).writeLoop(0xc8203649c0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1280 +0x472
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:861 +0x10cb
f0
on 30 Sep 2016
Version 23.02 does work
f0
on 30 Sep 2016
hm this still happens, any idea?
f0
on 23 Oct 2016
This also happens while running 0.24.1 standalone on Centos 7.1.1503, kernel 3.10.0-229.11.1.el7.x86_64. Works with 0.23.8.
carlpett
on 24 Oct 2016
Also 0.24.0-alpha1 and 0.24.0 have the same issue.
carlpett
on 24 Oct 2016
Interestingly, it works on 0.24.1 if I run cadvisor as a systemd service, but not if I run it from the commandline... In both cases, it is running as root, same cmdline (-logtostderr).
carlpett
on 24 Oct 2016
Ok, correction - it seems pretty random if it works or not. 30% of our (identical) machines it works, the rest fail with fatal error: unexpected signal during runtime execution.
Possibly there is some race condition?
carlpett
on 24 Oct 2016
@carlpett I think you're onto something, because we don't consistently see this either.
amcrn
on 2 Nov 2016
Looks like the same issue as https://github.com/prometheus/alertmanager/issues/267, which includes a possible fix.
timstclair
on 2 Nov 2016
@timstclair You mean building with netgo?
carlpett
on 2 Nov 2016
@timstclair @carlpett @amcrn building with -tags netgo fix the problem
f0
on 8 Nov 2016
🎉2
given the above, shouldn't the official cadvisor docker images now be built with the netgo tags?
amcrn
on 8 Nov 2016
any news here?
f0
on 29 Nov 2016
We are seeing this issue occur for random users of cAdvisor out in the wild. Even on very common linux distros like Ubuntu 14.04 LTS. I can also confirm that adding -tags netgo to the build command fixes the problem.
ezeev
on 1 Feb 2017
I'm having a similar issue for 2 days now.
cAdvisor worked at first but after the second docker-compose down and then docker-compose up it stopped working.
Below some info about the environment
root@vlslpbstc74qat/opt/data #>lsb_release -a
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: OracleServer
Description: Oracle Linux Server release 6.8
Release: 6.8
Codename: n/a
root@vlslpbstc74qat/opt/data #>sestatus
SELinux status: disabled
root@vlslpbstc74qat/opt/data #>docker --version
Docker version 1.12.6, build 1512168
root@vlslpbstc74qat/opt/data #>uname -a
Linux vlslpbstc74qat 4.1.12-37.6.3.el6uek.x86_64 #2 SMP Fri Aug 12 15:36:40 PDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Below the docker-compose file
# Basic version obtained from https://github.com/vegasbrianc/prometheus/blob/version-2/docker-compose.yml
version: '2'
volumes:
prometheus_data: {}
grafana_data: {}
networks:
front-tier:
driver: bridge
back-tier:
driver: bridge
services:
prometheus:
image: prom/prometheus
container_name: prometheus
volumes:
- ./prometheus/:/etc/prometheus/
- prometheus_data:/prometheus
command:
- '-config.file=/etc/prometheus/prometheus.yml'
- '-storage.local.path=/prometheus'
- '-alertmanager.url=http://alertmanager:9093'
expose:
- 9090
ports:
- 9090:9090
links:
- cadvisor:cadvisor
- alertmanager:alertmanager
depends_on:
- cadvisor
networks:
- back-tier
node-exporter:
image: prom/node-exporter
expose:
- 9100
networks:
- back-tier
alertmanager:
image: prom/alertmanager
ports:
- 9093:9093
volumes:
- ./alertmanager/:/etc/alertmanager/
networks:
- back-tier
command:
- '-config.file=/etc/alertmanager/config.yml'
- '-storage.path=/alertmanager'
cadvisor:
image: google/cadvisor
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
expose:
- 8080
networks:
- back-tier
grafana:
image: grafana/grafana
depends_on:
- prometheus
ports:
- 3000:3000
volumes:
- grafana_data:/var/lib/grafana
env_file:
- config.monitoring
networks:
- back-tier
- front-tier
Below logs being generated
I0209 16:23:45.332220 1 storagedriver.go:50] Caching stats in memory for 2m0s
I0209 16:23:45.332831 1 manager.go:140] cAdvisor running in container: "/docker/2a45e5c517393b3e52e9949cf5b5101f959d5ba1e147146df0ff6abf9044ab71"
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x63 pc=0x7f78d0ccf1ed]
runtime stack:
runtime.throw(0xfdc760, 0x2a)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigpanic_unix.go:12 +0x5a
goroutine 28 [syscall, locked to thread]:
runtime.cgocall(0xabb6c0, 0xc820030bd8, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/cgocall.go:123 +0x11b fp=0xc820030b78 sp=0xc820030b48
net._C2func_getaddrinfo(0x7f78c40008c0, 0x0, 0xc8202ca810, 0xc82010c430, 0x0, 0x0, 0x0)
??:0 +0x55 fp=0xc820030bd8 sp=0xc820030b78
net.cgoLookupIPCNAME(0xec20a0, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1072200)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:111 +0x448 fp=0xc820030d50 sp=0xc820030bd8
net.cgoLookupIP(0xec20a0, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2c307c7c722b7c7c)
/home/stclair/.gvm/gos/go1.6.3/src/net/cgo_unix.go:163 +0x56 fp=0xc820030da8 sp=0xc820030d50
net.lookupIP(0xec20a0, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup_unix.go:67 +0x94 fp=0xc820030e18 sp=0xc820030da8
net.glob.func16(0x1072270, 0xec20a0, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/hook.go:10 +0x4d fp=0xc820030e58 sp=0xc820030e18
net.lookupIPDeadline.func1(0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:106 +0x71 fp=0xc820030ed8 sp=0xc820030e58
internal/singleflight.(*Group).doCall(0x16459c0, 0xc82000e960, 0xec20a0, 0x9, 0xc8201bf6e0)
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:93 +0x2c fp=0xc820030f88 sp=0xc820030ed8
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820030f90 sp=0xc820030f88
created by internal/singleflight.(*Group).DoChan
/home/stclair/.gvm/gos/go1.6.3/src/internal/singleflight/singleflight.go:86 +0x3ee
goroutine 1 [select]:
net.lookupIPDeadline(0xec20a0, 0x9, 0xed02e8e93, 0x14c4c7f6, 0x1646980, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/lookup.go:109 +0x6a6
net.internetAddrList(0xea77a8, 0x3, 0xec20a0, 0xf, 0xed02e8e93, 0x14c4c7f6, 0x1646980, 0x0, 0x0, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/ipsock.go:252 +0x6ee
net.resolveAddrList(0xea3050, 0x4, 0xea77a8, 0x3, 0xec20a0, 0xf, 0xed02e8e93, 0x14c4c7f6, 0x1646980, 0x0, ...)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:158 +0x466
net.(*Dialer).Dial(0xc8203815b0, 0xea77a8, 0x3, 0xec20a0, 0xf, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:216 +0x124
net.DialTimeout(0xea77a8, 0x3, 0xec20a0, 0xf, 0x77359400, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/dial.go:200 +0xa3
github.com/google/cadvisor/container/rkt.Client.func1()
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:44 +0x59
sync.(*Once).Do(0x166c260, 0x10713a8)
/home/stclair/.gvm/gos/go1.6.3/src/sync/once.go:44 +0xe4
github.com/google/cadvisor/container/rkt.Client(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:79 +0x47
github.com/google/cadvisor/container/rkt.RktPath(0x0, 0x0, 0x0, 0x0)
/home/stclair/go/src/github.com/google/cadvisor/container/rkt/client.go:85 +0x48
github.com/google/cadvisor/manager.New(0xc8202e1c40, 0x7f78d0491230, 0x166c1b8, 0xdf8475800, 0x7f78d33d2501, 0xc820303620, 0xc8203037a0, 0x0, 0x0, 0x0, ...)
/home/stclair/go/src/github.com/google/cadvisor/manager/manager.go:146 +0x341
main.main()
/home/stclair/go/src/github.com/google/cadvisor/cadvisor.go:127 +0x654
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/home/stclair/.gvm/gos/go1.6.3/src/runtime/asm_amd64.s:1998 +0x1
goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/home/stclair/.gvm/gos/go1.6.3/src/os/signal/signal_unix.go:28 +0x37
goroutine 7 [chan receive]:
github.com/google/cadvisor/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x1646ce0)
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:882 +0x67
created by github.com/google/cadvisor/vendor/github.com/golang/glog.init.1
/home/stclair/go/src/github.com/google/cadvisor/vendor/github.com/golang/glog/glog.go:410 +0x297
goroutine 25 [IO wait]:
net.runtime_pollWait(0x7f78d0492370, 0x72, 0xc820296000)
/home/stclair/.gvm/gos/go1.6.3/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8203485a0, 0x72, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8203485a0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820348540, 0xc820296000, 0x1000, 0x1000, 0x0, 0x7f78d33ce050, 0xc820010120)
/home/stclair/.gvm/gos/go1.6.3/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82010c408, 0xc820296000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f78d0492430, 0xc82010c408, 0xc820058618, 0xc820296000, 0x1000, 0x1000, 0x409fb3, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1690 +0x67
net/http.(*noteEOFReader).Read(0xc8201bf540, 0xc820296000, 0x1000, 0x1000, 0xc82002dd1d, 0x0, 0x0)
<autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820054a80)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820054a80, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/stclair/.gvm/gos/go1.6.3/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8200585b0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1076 +0x177
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:860 +0x10a6
goroutine 26 [select]:
net/http.(*persistConn).writeLoop(0xc8200585b0)
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:1280 +0x472
created by net/http.(*Transport).dialConn
/home/stclair/.gvm/gos/go1.6.3/src/net/http/transport.go:861 +0x10cb
Below the output of docker inspect
[
{
"Id": "2a45e5c517393b3e52e9949cf5b5101f959d5ba1e147146df0ff6abf9044ab71",
"Created": "2017-02-09T16:23:44.377391743Z",
"Path": "/usr/bin/cadvisor",
"Args": [
"-logtostderr"
],
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 2,
"Error": "",
"StartedAt": "2017-02-09T16:23:45.14984008Z",
"FinishedAt": "2017-02-09T16:23:45.395502216Z"
},
"Image": "sha256:cc8254dd08c6ff3e6521fab4fdd9cc361c704f2fa51007ce230035f0b391f118",
"ResolvConfPath": "/opt/data/docker/containers/2a45e5c517393b3e52e9949cf5b5101f959d5ba1e147146df0ff6abf9044ab71/resolv.conf",
"HostnamePath": "/opt/data/docker/containers/2a45e5c517393b3e52e9949cf5b5101f959d5ba1e147146df0ff6abf9044ab71/hostname",
"HostsPath": "/opt/data/docker/containers/2a45e5c517393b3e52e9949cf5b5101f959d5ba1e147146df0ff6abf9044ab71/hosts",
"LogPath": "/opt/data/docker/containers/2a45e5c517393b3e52e9949cf5b5101f959d5ba1e147146df0ff6abf9044ab71/2a45e5c517393b3e52e9949cf5b5101f959d5ba1e147146df0ff6abf9044ab71-json.log",
"Name": "/prometheusgrafanacadvisor_cadvisor_1",
"RestartCount": 0,
"Driver": "overlay",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/var/lib/docker:/var/lib/docker:ro",
"/var/run:/var/run:rw",
"/:/rootfs:ro",
"/sys:/sys:ro"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "prometheusgrafanacadvisor_back-tier",
"PortBindings": {},
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": [],
"CapAdd": null,
"CapDrop": null,
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Name": "overlay",
"Data": {
"LowerDir": "/opt/data/docker/overlay/ca76231f9ea9e8cc08b09af8384ad1f6cc08b47862a8f356ec5c975d7a6f4d8d/root",
"MergedDir": "/opt/data/docker/overlay/4da26ef7ae1677f874beadad70109cfb3c99c9ec7d70013cb42d0441309006c3/merged",
"UpperDir": "/opt/data/docker/overlay/4da26ef7ae1677f874beadad70109cfb3c99c9ec7d70013cb42d0441309006c3/upper",
"WorkDir": "/opt/data/docker/overlay/4da26ef7ae1677f874beadad70109cfb3c99c9ec7d70013cb42d0441309006c3/work"
}
},
"Mounts": [
{
"Source": "/var/run",
"Destination": "/var/run",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
},
{
"Source": "/",
"Destination": "/rootfs",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Source": "/sys",
"Destination": "/sys",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Source": "/var/lib/docker",
"Destination": "/var/lib/docker",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "2a45e5c51739",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"8080/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"GLIBC_VERSION=2.23-r3"
],
"Cmd": null,
"Image": "google/cadvisor",
"Volumes": {
"/rootfs": {},
"/sys": {},
"/var/lib/docker": {},
"/var/run": {}
},
"WorkingDir": "",
"Entrypoint": [
"/usr/bin/cadvisor",
"-logtostderr"
],
"OnBuild": null,
"Labels": {
"com.docker.compose.config-hash": "3cf5b2f074b7bfade6be11280fee739ca25eea79ce857e7a44df90b4afc16db4",
"com.docker.compose.container-number": "1",
"com.docker.compose.oneoff": "False",
"com.docker.compose.project": "prometheusgrafanacadvisor",
"com.docker.compose.service": "cadvisor",
"com.docker.compose.version": "1.10.0"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "aff03cb51ad4391084c4a2969d0900b8392a100754621f08f85a31520b8ff7fc",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": null,
"SandboxKey": "/var/run/docker/netns/aff03cb51ad4",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"prometheusgrafanacadvisor_back-tier": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"2a45e5c51739",
"cadvisor"
],
"NetworkID": "ab4e4b94a2abb0c37e55ab9603d2b69c2349d02df6e431c27acc728b457ecb9f",
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": ""
}
}
}
}
]
rodrigomalara
on 9 Feb 2017
same cadvisor issue happening on CentOS Linux release 7.3.1611 vagrant host.
bkc1
on 17 Feb 2017
I hit the same problem, Red Hat Enterprise Linux Server release 7.3.
I have tried the pre-compiled 0.24.1 binary and also build version 0.24.2 myself (on Arch Linux with go1.8 amd64).
Interestingly, when running cadvisor inside gdb, it does not crash. Also, when running the official docker image it works too.
cadvisor -port 9870 -docker_only
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x63 pc=0x7f1512b8660d]
runtime stack:
runtime.throw(0xc92d96, 0x2a)
/usr/lib/go/src/runtime/panic.go:596 +0x95
runtime.sigpanic()
/usr/lib/go/src/runtime/signal_unix.go:274 +0x2db
goroutine 11 [syscall, locked to thread]:
runtime.cgocall(0xa17dc0, 0xc4200225d8, 0xc91fa6)
/usr/lib/go/src/runtime/cgocall.go:131 +0xe2 fp=0xc420022598 sp=0xc420022558
net._C2func_getaddrinfo(0x7f150c0008c0, 0x0, 0xc420344ba0, 0xc42000e040, 0x0, 0x0, 0x0)
net/_obj/_cgo_gotypes.go:86 +0x68 fp=0xc4200225d8 sp=0xc420022598
net.cgoLookupIPCNAME.func2(0x7f150c0008c0, 0x0, 0xc420344ba0, 0xc42000e040, 0xc7dec9, 0x9, 0xc420300590)
/usr/lib/go/src/net/cgo_unix.go:151 +0x171 fp=0xc420022638 sp=0xc4200225d8
net.cgoLookupIPCNAME(0xc7dec9, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/go/src/net/cgo_unix.go:151 +0x1bd fp=0xc420022738 sp=0xc420022638
net.cgoIPLookup(0xc42013a2a0, 0xc7dec9, 0x9)
/usr/lib/go/src/net/cgo_unix.go:203 +0x4d fp=0xc4200227c8 sp=0xc420022738
runtime.goexit()
/usr/lib/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4200227d0 sp=0xc4200227c8
created by net.cgoLookupIP
/usr/lib/go/src/net/cgo_unix.go:213 +0xb4
goroutine 1 [select]:
net.(*Resolver).LookupIPAddr(0x12eb6e5, 0x1235e20, 0xc42013a1e0, 0xc7dec9, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/go/src/net/lookup.go:165 +0x9b1
net.(*Resolver).internetAddrList(0x12eb6e5, 0x1235e20, 0xc42013a1e0, 0xc737c5, 0x3, 0xc7dec9, 0xf, 0x0, 0x0, 0xc42040d590, ...)
/usr/lib/go/src/net/ipsock.go:245 +0x4cc
net.(*Resolver).resolveAddrList(0x12eb6e5, 0x1235e20, 0xc42013a1e0, 0xc73d2a, 0x4, 0xc737c5, 0x3, 0xc7dec9, 0xf, 0x0, ...)
/usr/lib/go/src/net/dial.go:188 +0x6b3
...
avdv
on 21 Mar 2017
The same error happens with version 0.25.0.
But, I have recompiled cadvisor dynamically on the host system and now it works. :tada:
ldd cadvisor
linux-vdso.so.1 => (0x00007ffd69996000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffbc58e9000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007ffbc56e5000)
libc.so.6 => /lib64/libc.so.6 (0x00007ffbc5323000)
/lib64/ld-linux-x86-64.so.2 (0x00007ffbc5b12000)
Use the official image and install findutils to run normally
dockerfiles
google-cadvisor:v0.25.0
MAINTAINER pro zpang
# time
RUN apk add --update tzdata && rm -rf /var/cache/apk/*
RUN cp /usr/share/zoneinfo/Hongkong /etc/localtime
# add gnu find
RUN apk add --update findutils && rm -rf /var/cache/apk/*
pangzheng
on 21 Apr 2017
+1 fixed by adding "-tags netgo" on ubuntu 16.04 with go1.8.1
WIZARD-CXY
on 9 May 2017
+1 fixed by adding "-tags netgo" on ubuntu 16.04 with go1.8.1
Confirmed. Setting GODEBUG=netdns=go in the envirnoment to force using the Go resolver at runtime (without recompiliation) has the same effect:
# ./cadvisor -version
cAdvisor version v0.25.0 (17543be)
# GODEBUG=netdns=cgo ./cadvisor -docker_only -port 9999
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x63 pc=0x7f56933a760d]
runtime stack:
runtime.throw(0xcdd37b, 0x2a)
/usr/local/go/src/runtime/panic.go:566 +0x95
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:12 +0x2cc
goroutine 30 [syscall, locked to thread]:
[...]
# GODEBUG=netdns=go ./cadvisor -docker_only -port 9999
E0510 08:33:17.417989 3391 factory.go:305] devicemapper filesystem stats will not be reported: usage of thin_ls is disabled to preserve iops
👍4
I have the same issue on Fedora 25 with running
$ sudo docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --name=cadvisor --rm --privileged=true google/cadvisor:latest
container is able to start without --volume=/var/run:/var/run:rw as mentioned here: https://github.com/google/cadvisor/issues/1646#issuecomment-298817234
nenadalm
on 22 May 2017
Same problem on "CentOS Linux release 7.3.1611 (Core) 3.10.0-514.21.2.el7.x86_64" with cAdvisor v0.26.1 binary.
Confirmed, with "GODEBUG=netdns=go ./cadvisor" run without any issue.
mikolatero
on 27 Jun 2017
👍1
\followup: I have just upgrade to cadvisor 0.26.1 and are seeing this issue now also when running cadvisor on the host machine from systemd.
manager.go:143] cAdvisor running in container: "/system.slice/cadvisor.service"
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x63 pc=0x7fefcca4260d]
...
Mitigated by using Environment='GODEBUG=netdns=go' in the Service section of the systemd unit..
avdv
on 16 Aug 2017
😕1
I have the same issue on Ubuntu 18.0
-> solved by updating cadvisor
Arezki1995
on 23 Feb 2019
Was this page helpful?
0 / 5 - 0 ratings
Related issues
terrpan
路
5Comments
krutsko
路
4Comments
wololock
路
5Comments
mpas
路
3Comments
hridyeshpant
路
7Comments
Most helpful comment
Confirmed. Setting
GODEBUG=netdns=goin the envirnoment to force using the Go resolver at runtime (without recompiliation) has the same effect: