kubectl panic on pod delete

[rednatto]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): panic, SIGSEGV

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT ?

Kubernetes version (use kubectl version):

• Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T21:12:46Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
• Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T20:55:30Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Minikube (minikube version: v0.24.1 (also seems to be same at v0.25.0)) ( --kubernetes-version v1.9.0 --bootstrapper kubeadm --vm-driver virtualbox )
• OS (e.g. from /etc/os-release): macOS High Sierra (Version 10.13.2)
• Kernel (e.g. uname -a): Darwin A966.local 17.3.0 Darwin Kernel Version 17.3.0: Thu Nov 9 18:09:22 PST 2017; root:xnu-4570.31.3~1/RELEASE_X86_64 x86_64
• Install tools: brew?
• Others: Docker for mac running?

What happened:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x21b8f83]

What you expected to happen:
No panic.

How to reproduce it (as minimally and precisely as possible):
Get ngnix-deployment.yaml from https://raw.githubusercontent.com/kubernetes/website/master/docs/concepts/workloads/controllers/nginx-deployment.yaml .

Create the deployment:
λ kubectl create -f nginx-deployment.yaml

deployment "nginx-deployment" created

Delete a pod from the deployment (either one or all gives the same result):
λ kubectl delete pod nginx-deployment-6c54bd5869-2lshn

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x21b8f83]

goroutine 1 [running]:
k8s.io/kubernetes/pkg/kubectl.ReaperFor(0x0, 0x0, 0x2370237, 0x3, 0x0, 0x0, 0x0, 0xc420a094a0, 0x13fdaa2, 0xbe9284d8ac70ccf0)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/delete.go:82 +0x1373
k8s.io/kubernetes/pkg/kubectl/cmd/util.(*ring1Factory).Reaper(0xc420751200, 0xc4204f79d0, 0x3666b40, 0x1a0003825900, 0xb8, 0xb8)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/factory_object_mapping.go:295 +0x151
k8s.io/kubernetes/pkg/kubectl/cmd/util.(*factory).Reaper(0xc420751230, 0xc4204f79d0, 0x3825900, 0x3e2b6c8, 0x0, 0x2678340)
        <autogenerated>:1 +0x47
k8s.io/kubernetes/pkg/kubectl/cmd.ReapResult.func1(0xc4204f7b20, 0x0, 0x0, 0x3815c00, 0x2853435)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:250 +0xe9
k8s.io/kubernetes/pkg/kubectl/resource.ContinueOnErrorVisitor.Visit.func1(0xc4204f7b20, 0x0, 0x0, 0x0, 0x0)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:396 +0x164
k8s.io/kubernetes/pkg/kubectl/resource.DecoratedVisitor.Visit.func1(0xc4204f7b20, 0x0, 0x0, 0x7, 0xa)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:372 +0xe7
k8s.io/kubernetes/pkg/kubectl/resource.FlattenListVisitor.Visit.func1(0xc4204f7b20, 0x0, 0x0, 0xc42096d140, 0x414e798)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:433 +0x4fe
k8s.io/kubernetes/pkg/kubectl/resource.(*Info).Visit(0xc4204f7b20, 0xc42096d140, 0x0, 0x10103e7)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:105 +0x42
k8s.io/kubernetes/pkg/kubectl/resource.VisitorList.Visit(0xc4201f2350, 0x1, 0x1, 0xc42096d140, 0x1, 0xc42096d140)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:234 +0x63
k8s.io/kubernetes/pkg/kubectl/resource.(*VisitorList).Visit(0xc420450900, 0xc42096d140, 0x3e2b6c8, 0x0)
        <autogenerated>:1 +0x58
k8s.io/kubernetes/pkg/kubectl/resource.FlattenListVisitor.Visit(0x365a640, 0xc420450900, 0xc4203516c0, 0xc4207864c0, 0x60000000001, 0xc4207864c0)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:428 +0x9e
k8s.io/kubernetes/pkg/kubectl/resource.(*FlattenListVisitor).Visit(0xc420450920, 0xc4207864c0, 0x28, 0x3825900)
        <autogenerated>:1 +0x58
k8s.io/kubernetes/pkg/kubectl/resource.DecoratedVisitor.Visit(0x365a5c0, 0xc420450920, 0xc4201f2360, 0x2, 0x2, 0xc420450980, 0x1, 0xc420450980)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:363 +0x9b
k8s.io/kubernetes/pkg/kubectl/resource.(*DecoratedVisitor).Visit(0xc42096d110, 0xc420450980, 0xc4201f2370, 0xc420a09b60)
        <autogenerated>:1 +0x62
k8s.io/kubernetes/pkg/kubectl/resource.ContinueOnErrorVisitor.Visit(0x365a4c0, 0xc42096d110, 0xc4204f7b90, 0x0, 0x2)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:391 +0xe4
k8s.io/kubernetes/pkg/kubectl/resource.(*ContinueOnErrorVisitor).Visit(0xc4201f2370, 0xc4204f7b90, 0x1010b48, 0x70)
        <autogenerated>:1 +0x4f
k8s.io/kubernetes/pkg/kubectl/resource.(*Result).Visit(0xc4202ae600, 0xc4204f7b90, 0xc42095bd10, 0x3679b80)```
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/result.go:98 +0x62
k8s.io/kubernetes/pkg/kubectl/cmd.ReapResult(0xc4202ae600, 0x3686980, 0xc420751230, 0x3659480, 0xc42000c018, 0xc420130001, 0x0, 0xffffffffffffffff, 0xc4209a0000, 0x3679b80, ...)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:245 +0x16c
k8s.io/kubernetes/pkg/kubectl/cmd.(*DeleteOptions).RunDelete(0xc4207104d0, 0xc420290480, 0x0)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:235 +0xd2
k8s.io/kubernetes/pkg/kubectl/cmd.NewCmdDelete.func1(0xc420290480, 0xc42013d4c0, 0x2, 0x2)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:142 +0x178
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc420290480, 0xc42013d300, 0x2, 0x2, 0xc420290480, 0xc42013d300)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x234
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420396900, 0x8000104, 0x0, 0xffffffffffffffff)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x2fe
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420396900, 0xc420751230, 0x3659440)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b
k8s.io/kubernetes/cmd/kubectl/app.Run(0x0, 0x0)
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:41 +0xd5
main.main()
        /private/tmp/kubernetes-cli-20180118-36889-kx51hi/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:27 +0x26

Anything else we need to know:
kubectl describe pod nginx-deployment-6c54bd5869-2lshn
Gives correct result.

Any deployment gives the same result.

[rednatto]

Could this be a minikube issue - should I post it there aswell/instead?

[adohe-zz]

@rednatto I would try to reproduce this and see what's going on~

[decoursin]

I'm having the same issue, but I'm using minikube version 25 and Linux Mint, otherwise I have the same client and server kube versions.

[rednatto]

Running against another, non-minikube, server seems to work for me.

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T21:12:46Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Against a cluster running on a Ubuntu16 desktop with kubeadm (kubeadm version: &version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:40:06Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
) and kubelet (Kubernetes v1.9.1) and weave.

So this might be an issue with minikube only (added https://github.com/kubernetes/minikube/issues/2504 there)?

[rednatto]

Possibly related information:

Creating a service, and then trying to fetch it gives different results when using svc vs service, against the broken cluster, but seems to work against other cluster?

kubectl create service nodeport nginx --tcp=8080

service "nginx" created

kubectl get svc nginx

the server doesn't have a resource type "svc"

kubectl get service nginx

NAME      TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
nginx     NodePort   10.98.56.135   <none>        8080:31786/TCP   18s

[adohe-zz]

svc should be just an alias of service, it should get the same result as kubectl get service ***, but what you mean against a broken cluster? @rednatto

[rednatto]

@adohe What I mean is that svc was not recognized, but service was, when running commands against the minikube cluster where pod delete panicked; and that it worked when running against another, non-minikube, cluster - where the delete did _not_ panic.

Sorry to be unclear, I had written the same things in the minikube-issue and did not keep track of what I had written.
This missing alias thing seems to cooccur with the panicking.

[aheart]

I have experienced this panic when accidentally running kubectl delete on worker node.

$ kubectl delete pod foo
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x15b6963]

goroutine 1 [running]:
k8s.io/kubernetes/pkg/kubectl.ReaperFor(0x0, 0x0, 0x176df37, 0x3, 0x0, 0x0, 0x0, 0xc4207d74a0, 0x7f58d2, 0xbea0b274755dfeff)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/delete.go:82 +0x1373
k8s.io/kubernetes/pkg/kubectl/cmd/util.(ring1Factory).Reaper(0xc420202930, 0xc420a9b810, 0x2a4f8c0, 0x1a0002c0ec40, 0xb8, 0xb8)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/factory_object_mapping.go:295 +0x151
k8s.io/kubernetes/pkg/kubectl/cmd/util.(factory).Reaper(0xc420202a20, 0xc420a9b810, 0x2c0ec40, 0x7f41ec67f000, 0x0, 0x1a75bc0)
:1 +0x47
k8s.io/kubernetes/pkg/kubectl/cmd.ReapResult.func1(0xc420a9b960, 0x0, 0x0, 0x2bfe940, 0x1c50a1e)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:250 +0xe9
k8s.io/kubernetes/pkg/kubectl/resource.ContinueOnErrorVisitor.Visit.func1(0xc420a9b960, 0x0, 0x0, 0x0, 0x0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:396 +0x164
k8s.io/kubernetes/pkg/kubectl/resource.DecoratedVisitor.Visit.func1(0xc420a9b960, 0x0, 0x0, 0x7, 0xa)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:372 +0xe7
k8s.io/kubernetes/pkg/kubectl/resource.FlattenListVisitor.Visit.func1(0xc420a9b960, 0x0, 0x0, 0xc420ad7b30, 0x7f41ec6319a8)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:433 +0x4fe
k8s.io/kubernetes/pkg/kubectl/resource.(Info).Visit(0xc420a9b960, 0xc420ad7b30, 0x0, 0x4105e7)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:105 +0x42
k8s.io/kubernetes/pkg/kubectl/resource.VisitorList.Visit(0xc4201d7460, 0x1, 0x1, 0xc420ad7b30, 0x1, 0xc420ad7b30)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:234 +0x63
k8s.io/kubernetes/pkg/kubectl/resource.(VisitorList).Visit(0xc420ae7820, 0xc420ad7b30, 0x7f41ec67f000, 0x0)
:1 +0x58
k8s.io/kubernetes/pkg/kubectl/resource.FlattenListVisitor.Visit(0x2a43400, 0xc420ae7820, 0xc420a8ec00, 0xc420a8f1c0, 0x60000000001, 0xc420a8f1c0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:428 +0x9e
k8s.io/kubernetes/pkg/kubectl/resource.(FlattenListVisitor).Visit(0xc420ae7840, 0xc420a8f1c0, 0x28, 0x2c0ec40)
:1 +0x58
k8s.io/kubernetes/pkg/kubectl/resource.DecoratedVisitor.Visit(0x2a43380, 0xc420ae7840, 0xc4201d7470, 0x2, 0x2, 0xc420ae78a0, 0xc4201d7401, 0xc420ae78a0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:363 +0x9b
k8s.io/kubernetes/pkg/kubectl/resource.(DecoratedVisitor).Visit(0xc420ad7b00, 0xc420ae78a0, 0xc4201d7480, 0xc4207d7b60)
:1 +0x62
k8s.io/kubernetes/pkg/kubectl/resource.ContinueOnErrorVisitor.Visit(0x2a43280, 0xc420ad7b00, 0xc420a9b9d0, 0x0, 0x2)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:391 +0xe4
k8s.io/kubernetes/pkg/kubectl/resource.(ContinueOnErrorVisitor).Visit(0xc4201d7480, 0xc420a9b9d0, 0x410d48, 0x70)
:1 +0x4f
k8s.io/kubernetes/pkg/kubectl/resource.(Result).Visit(0xc4202ce800, 0xc420a9b9d0, 0xc420a611d0, 0x2a62840)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/result.go:98 +0x62
k8s.io/kubernetes/pkg/kubectl/cmd.ReapResult(0xc4202ce800, 0x2a6f640, 0xc420202a20, 0x2a42480, 0xc42000c018, 0xc420390001, 0x0, 0xffffffffffffffff, 0xc420a90000, 0x2a62840, ...)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:245 +0x16c
k8s.io/kubernetes/pkg/kubectl/cmd.(DeleteOptions).RunDelete(0xc4207f44d0, 0xc42027c000, 0x0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:235 +0xd2
k8s.io/kubernetes/pkg/kubectl/cmd.NewCmdDelete.func1(0xc42027c000, 0xc42039e260, 0x2, 0x2)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:142 +0x178
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(Command).execute(0xc42027c000, 0xc42039e0c0, 0x2, 0x2, 0xc42027c000, 0xc42039e0c0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x234
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(Command).ExecuteC(0xc42021c480, 0x8000104, 0x0, 0xffffffffffffffff)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x2fe
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(Command).Execute(0xc42021c480, 0xc420202a20, 0x2a42440)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b
k8s.io/kubernetes/cmd/kubectl/app.Run(0x0, 0x0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:41 +0xd5
main.main()
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:27 +0x26

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):
  NAME="Ubuntu"
  VERSION="16.04.3 LTS (Xenial Xerus)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 16.04.3 LTS"
  VERSION_ID="16.04"
  HOME_URL="http://www.ubuntu.com/"
  SUPPORT_URL="http://help.ubuntu.com/"
  BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
  VERSION_CODENAME=xenial
  UBUNTU_CODENAME=xenial
• Kernel (e.g. uname -a):
  Linux stg-docker-004 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
  KTHW (custom)

[rsalmond]

I am seeing both of these behaviours in a non minikube environment.

$ kubectl version 
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl get po
the server doesn't have a resource type "po"

$ kubectl get svc
the server doesn't have a resource type "svc"

$ kubectl get cm
the server doesn't have a resource type "cm"

$ kubectl get pods
NAME                                                      READY     STATUS    RESTARTS   AGE
bb                                                        1/1       Running   0          1h
ideal-oyster-openvpn-578459dfb4-hx7pf                     1/1       Running   0          1h
my-nginx-nginx-ingress-controller-7d85c5f755-27hbw        1/1       Running   0          8m
my-nginx-nginx-ingress-default-backend-849598fdfb-lw54r   1/1       Running   0          11m

$ kubectl get services
NAME                                     TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ideal-oyster-openvpn                     NodePort       10.110.191.222   <none>        443:32304/TCP                1h
kubernetes                               ClusterIP      10.96.0.1        <none>        443/TCP                      1h
my-nginx-nginx-ingress-controller        LoadBalancer   10.98.68.201     <pending>     80:30883/TCP,443:30272/TCP   11m
my-nginx-nginx-ingress-default-backend   ClusterIP      10.98.149.221    <none>        80/TCP                       11m

$ kubectl get configmaps
NAME                                DATA      AGE
ideal-oyster-openvpn                4         1h
ingress-controller-leader-nginx     0         10m
my-nginx-nginx-ingress-controller   1         11m
vpn-ingress                         1         4m

$ kubectl delete pod my-nginx-nginx-ingress-controller-7d85c5f755-27hbw
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x15b6963]

goroutine 1 [running]:
k8s.io/kubernetes/pkg/kubectl.ReaperFor(0x0, 0x0, 0x176df37, 0x3, 0x0, 0x0, 0x0, 0xc4207894a0, 0x7f58d2, 0xbea13804927ac51a)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/delete.go:82 +0x1373
k8s.io/kubernetes/pkg/kubectl/cmd/util.(*ring1Factory).Reaper(0xc420441140, 0xc42032b880, 0x2a4f8c0, 0x1a0002c0ec40, 0xb8, 0xb8)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/factory_object_mapping.go:295 +0x151
k8s.io/kubernetes/pkg/kubectl/cmd/util.(*factory).Reaper(0xc420441260, 0xc42032b880, 0x2c0ec40, 0x7fab586e86c8, 0x0, 0x1a75bc0)
    <autogenerated>:1 +0x47
k8s.io/kubernetes/pkg/kubectl/cmd.ReapResult.func1(0xc42032bc70, 0x0, 0x0, 0x2bfe940, 0x1c50a1e)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:250 +0xe9
k8s.io/kubernetes/pkg/kubectl/resource.ContinueOnErrorVisitor.Visit.func1(0xc42032bc70, 0x0, 0x0, 0x0, 0x0)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:396 +0x164
k8s.io/kubernetes/pkg/kubectl/resource.DecoratedVisitor.Visit.func1(0xc42032bc70, 0x0, 0x0, 0x7, 0xa)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:372 +0xe7
k8s.io/kubernetes/pkg/kubectl/resource.FlattenListVisitor.Visit.func1(0xc42032bc70, 0x0, 0x0, 0xc420991800, 0x7fab5844db50)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:433 +0x4fe
k8s.io/kubernetes/pkg/kubectl/resource.(*Info).Visit(0xc42032bc70, 0xc420991800, 0x0, 0x4105e7)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:105 +0x42
k8s.io/kubernetes/pkg/kubectl/resource.VisitorList.Visit(0xc4202c3570, 0x1, 0x1, 0xc420991800, 0x1, 0xc420991800)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:234 +0x63
k8s.io/kubernetes/pkg/kubectl/resource.(*VisitorList).Visit(0xc4209a7180, 0xc420991800, 0x7fab586e86c8, 0x0)
    <autogenerated>:1 +0x58
k8s.io/kubernetes/pkg/kubectl/resource.FlattenListVisitor.Visit(0x2a43400, 0xc4209a7180, 0xc42034f780, 0xc42057a600, 0x60000000001, 0xc42057a600)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:428 +0x9e
k8s.io/kubernetes/pkg/kubectl/resource.(*FlattenListVisitor).Visit(0xc4209a71a0, 0xc42057a600, 0x28, 0x2c0ec40)
    <autogenerated>:1 +0x58
k8s.io/kubernetes/pkg/kubectl/resource.DecoratedVisitor.Visit(0x2a43380, 0xc4209a71a0, 0xc4202c3580, 0x2, 0x2, 0xc4209a7200, 0xc4202c3501, 0xc4209a7200)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:363 +0x9b
k8s.io/kubernetes/pkg/kubectl/resource.(*DecoratedVisitor).Visit(0xc4209917d0, 0xc4209a7200, 0xc4202c35a0, 0xc420789b60)
    <autogenerated>:1 +0x62
k8s.io/kubernetes/pkg/kubectl/resource.ContinueOnErrorVisitor.Visit(0x2a43280, 0xc4209917d0, 0xc42032bce0, 0x0, 0x2)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:391 +0xe4
k8s.io/kubernetes/pkg/kubectl/resource.(*ContinueOnErrorVisitor).Visit(0xc4202c35a0, 0xc42032bce0, 0x410d48, 0x70)
    <autogenerated>:1 +0x4f
k8s.io/kubernetes/pkg/kubectl/resource.(*Result).Visit(0xc42026c780, 0xc42032bce0, 0xc420903590, 0x2a62840)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/result.go:98 +0x62
k8s.io/kubernetes/pkg/kubectl/cmd.ReapResult(0xc42026c780, 0x2a6f640, 0xc420441260, 0x2a42480, 0xc42000c018, 0xc4201b0001, 0x0, 0xffffffffffffffff, 0xc420950000, 0x2a62840, ...)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:245 +0x16c
k8s.io/kubernetes/pkg/kubectl/cmd.(*DeleteOptions).RunDelete(0xc4206f7ef0, 0xc42035cfc0, 0x0)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:235 +0xd2
k8s.io/kubernetes/pkg/kubectl/cmd.NewCmdDelete.func1(0xc42035cfc0, 0xc4201b3720, 0x2, 0x2)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:142 +0x178
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc42035cfc0, 0xc4201b3580, 0x2, 0x2, 0xc42035cfc0, 0xc4201b3580)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x234
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc42036efc0, 0x5000107, 0x0, 0xffffffffffffffff)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x2fe
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(0xc42036efc0, 0xc420441260, 0x2a42440)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b
k8s.io/kubernetes/cmd/kubectl/app.Run(0x0, 0x0)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:41 +0xd5
main.main()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:27 +0x26

I am able to replicate with ease if anyone needs any more information.

[Liujingfang1]

I'd like to investigate on this issue. @rsalmond Can you share how you created my-nginx-nginx-ingress-controller in the above example?

[rsalmond]

@Liujingfang1 for sure, that was created using the stable nginx ingress controller helm chart.

helm install stable/nginx-ingress --name my-nginx --set rbac.create=true --set controller.hostNetwork=true

[decoursin]

It's probably not the reason, but it could be a result of running out of disk space.

[Liujingfang1]

@rsalmond Thank you for your help. I used the same environment and running the same chart as you listed, but couldn't reproduce the error. The reason might be out of disk space as @decoursin mentioned. I checked my nodes and none of them are out of disk.

[Liujingfang1]

Based on the stacktrace, the error is from this part of code in k8s.io/kubernetes/pkg/kubectl/cmd/util/factory_object_mapping.go

clientset, clientsetErr := f.clientAccessFactory.ClientSet()
reaper, reaperErr := kubectl.ReaperFor(mapping.GroupVersionKind.GroupKind(), clientset)

If clientset is nil, then we will see this segmentation error. The fix would be to add a check on clientsetErr:

    clientset, clientsetErr := f.clientAccessFactory.ClientSet()
    if clientsetErr != nil {
        return nil, clientsetErr
    }
    reaper, reaperErr := kubectl.ReaperFor(mapping.GroupVersionKind.GroupKind(), clientset)

[jirkadanek]

I got the same stacktrace as reproduced before while deleting pods, probably due to networking issues.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"archive", BuildDate:"1970-01-01T00:00:01Z", GoVersion:"go1.10", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server 10.37.144.42:6443 was refused - did you specify the right host or port?

[MartinX3]

Same error here

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6", GitCommit:"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2", GitTreeState:"clean", BuildDate:"2017-06-16T18:34:20Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

[rishabp]

Hi, i am working on a project to create a declarative application in
printers...I dont have any idea how to approach this...my job depends on
this...i searched on the internet and found your project...guys can you
please help me out for my project or any documents that i can study dor
declarative application...
Regards
Rishab Patnaik

On Sun, Apr 1, 2018, 6:32 PM Martin Dünkelmann [e-mail]
wrote:

Same error here
Server Version is 1.9.6
Client Version is 1.9.6

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubectl/issues/226#issuecomment-377785401,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbcxyYS_1mcKh1lvkhip_bUMFe6Vs5TRks5tkNAAgaJpZM4RsiGi
.

[jirkadanek]

@rishabp please do not hijack bug tickets in an issue tracker. https://paste.kde.org/pmh0qaunn

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close