Rkt: api-service panics

Created on 24 Aug 2016 · 4Comments · Source: rkt/rkt

Environment

Replace this with the output of:

rkt Version: 1.13.0
appc Version: 0.8.6
Go Version: go1.6.3
Go OS/Arch: linux/amd64
Features: -TPM +SDJOURNAL
--
Linux 4.7.1 x86_64
--
NAME=Buildroot
VERSION=2016.08-rc2
ID=buildroot
VERSION_ID=2016.08-rc2
PRETTY_NAME="Buildroot 2016.08-rc2"
--
systemd 231
-PAM -AUDIT -SELINUX -IMA -APPARMOR -SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN

What did you do?

systemctl status api-service

What did you expect to see?

a running api-service

What did you see instead?

Aug 24 15:43:51 minikubeVM rkt[2126]: api-service: failed to get the PID for pod "9b867c20-dce8-4c58-a267-dfcb5f826744": no such file or directory
Aug 24 15:44:19 minikubeVM rkt[2126]: api-service: failed to get the PID for pod "500f84f8-afea-4873-b9b4-de9b28cf1e2a": no such file or directory
Aug 24 15:44:19 minikubeVM rkt[2126]: api-service: failed to get the PID for pod "500f84f8-afea-4873-b9b4-de9b28cf1e2a": no such file or directory
Aug 24 15:49:08 minikubeVM rkt[2126]: api-service: failed to get the PID for pod "386e9839-a11b-45ab-9bc3-d23af5009a80": no such file or directory
Aug 24 15:49:50 minikubeVM rkt[2126]: api-service: failed to get the PID for pod "5c280acf-2bae-4a69-83cb-8315f3aca068": no such file or directory
Aug 24 15:58:04 minikubeVM rkt[2126]: api-service: failed to get the PID for pod "ed28fbf7-34ca-4283-98c0-f5d42017ee28": no such file or directory
Aug 24 16:09:20 minikubeVM rkt[2126]: panic: runtime error: index out of range
Aug 24 16:09:20 minikubeVM rkt[2126]: goroutine 4537 [running]:
Aug 24 16:09:20 minikubeVM rkt[2126]: panic(0xe4a740, 0xc820030010)
Aug 24 16:09:20 minikubeVM rkt[2126]:         /usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
Aug 24 16:09:20 minikubeVM rkt[2126]: main.getChildPID(0xd09, 0x0, 0x0, 0x0)
Aug 24 16:09:20 minikubeVM rkt[2126]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/rkt/pods.go:877 +0xe0e
Aug 24 16:09:20 minikubeVM rkt[2126]: main.(*pod).getContainerPID1(0xc8203d6f00, 0x0, 0x0, 0x0)
Aug 24 16:09:20 minikubeVM rkt[2126]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/rkt/pods.go:929 +0x132
Aug 24 16:09:20 minikubeVM rkt[2126]: main.fillPodDetails(0xc8203d67d0, 0xc8203d6f00, 0xc82050ee60)
Aug 24 16:09:20 minikubeVM rkt[2126]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/rkt/api_service.go:561 +0x7f6
Aug 24 16:09:20 minikubeVM rkt[2126]: main.(*v1AlphaAPIServer).ListPods.func1(0xc8203d6f00)
Aug 24 16:09:20 minikubeVM rkt[2126]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/rkt/api_service.go:613 +0x6f
Aug 24 16:09:20 minikubeVM rkt[2126]: main.walkPods(0x1e, 0xc820475a10, 0x0, 0x0)
Aug 24 16:09:20 minikubeVM rkt[2126]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/rkt/pods.go:145 +0x82b
Aug 24 16:09:20 minikubeVM rkt[2126]: main.(*v1AlphaAPIServer).ListPods(0xc8202c3540, 0x7f2cec1245d0, 0xc820294660, 0xc82000d160, 0xc82040a480, 0x0, 0x0)
Aug 24 16:09:20 minikubeVM rkt[2126]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/rkt/api_service.go:610 +0x78
Aug 24 16:09:20 minikubeVM rkt[2126]: github.com/coreos/rkt/api/v1alpha._PublicAPI_ListPods_Handler(0xf33640, 0xc8202c3540, 0x7f2cec1245d0, 0xc820294660, 0xc82040a4c0, 0x0, 0x0, 0x0, 0x0, 0x0)
Aug 24 16:09:20 minikubeVM rkt[2126]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/api/v1alpha/api.pb.go:1041 +0x168
Aug 24 16:09:20 minikubeVM rkt[2126]: github.com/coreos/rkt/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc8203dad80, 0x7f2cec124390, 0xc82041abd0, 0xc820096000, 0xc82028b230, 0x1698418, 0xc820294630, 0x0, 0x0)
Aug 24 16:09:20 minikubeVM rkt[2126]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/vendor/google.golang.org/grpc/server.go:580 +0xe9e
Aug 24 16:09:20 minikubeVM rkt[2126]: github.com/coreos/rkt/vendor/google.golang.org/grpc.(*Server).handleStream(0xc8203dad80, 0x7f2cec124390, 0xc82041abd0, 0xc820096000, 0xc820294630)
Aug 24 16:09:20 minikubeVM rkt[2126]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/vendor/google.golang.org/grpc/server.go:737 +0x109d
Aug 24 16:09:20 minikubeVM rkt[2126]: github.com/coreos/rkt/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc8202c4410, 0xc8203dad80, 0x7f2cec124390, 0xc82041abd0, 0xc820096000)
Aug 24 16:09:20 minikubeVM rkt[2126]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/vendor/google.golang.org/grpc/server.go:402 +0xa0
Aug 24 16:09:20 minikubeVM rkt[2126]: created by github.com/coreos/rkt/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
Aug 24 16:09:20 minikubeVM rkt[2126]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/vendor/google.golang.org/grpc/server.go:403 +0x9a
Aug 24 16:09:20 minikubeVM systemd[1]: rkt-api.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

componenapi server kinbug prioritP1 required-brktnetes technologkubernetes

Source

s-urbaniak

Most helpful comment

I'm pretty sure this is actually a bug in the go standard library (or its documentation). The important details:

1) The documentation for file.Readdir says if Readdir returns an empty slice, it will return a non-nil error explaining why
2) We call fdir.Readdir(1) and then carefully error check... and then assume the slice is of length at least 1. According to the documentation this assumption is correct
3) The standard library documentation lied to us!

I was able to write a very simple reproduction of this bug.

We should be able to work around it trivially by just checking the length of fi and skipping if it's empty in our code. I'll make a patch for that and also report this upstream after a second pair of eyes double checks I'm not crazy. I also skimmed the file_unix.go code's implementation of readdir and didn't see it setting err for this case (and errors are not set for IsNotExist so a few IsNotExist later and you break).