Calico: Hit error connecting to datastore - x509: cannot validate certificate

Created on 14 Jun 2018  路  20Comments  路  Source: projectcalico/calico


Calico pod fails to start - continually in CrashLoopBackOff.
Log for calico-node pod indicates certificate error:
2018-06-14 16:28:51.092 [INFO][9] startup.go 317: Hit error connecting to datastore - retry error=Get https://10.96.0.1:443/api/v1/nodes/foo: x509: cannot validate certificate for 10.96.0.1 because it doesn't contain any IP SANs

Expected Behavior

Calico pod should start and kube-dns should start

Current Behavior

Calico pod fails to start - continually in CrashLoopBackOff

Steps to Reproduce (for bugs)

  1. kubeadm init
  2. kubectl apply -f rbac-kdd.yaml
  3. kubectl apply -f calico.yaml

Context

user001@k8s-master001:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system calico-node-jm695 1/2 Running 0 57s 10.127.156.171 k8s-master001
kube-system kube-apiserver-k8s-master001 1/1 Running 0 29m 10.127.156.171 k8s-master001
kube-system kube-controller-manager-k8s-master001 1/1 Running 0 29m 10.127.156.171 k8s-master001
kube-system kube-dns-86f4d74b45-s56kj 0/3 ContainerCreating 0 29m k8s-master001
kube-system kube-proxy-jbpvp 1/1 Running 0 29m 10.127.156.171 k8s-master001
kube-system kube-scheduler-k8s-master001 1/1 Running 0 39m 10.127.156.171 k8s-master001

user001@k8s-master001:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system calico-node-jm695 1/2 Running 1 1m 10.127.156.171 k8s-master001
kube-system kube-apiserver-k8s-master001 1/1 Running 0 30m 10.127.156.171 k8s-master001
kube-system kube-controller-manager-k8s-master001 1/1 Running 0 30m 10.127.156.171 k8s-master001
kube-system kube-dns-86f4d74b45-s56kj 0/3 ContainerCreating 0 29m k8s-master001
kube-system kube-proxy-jbpvp 1/1 Running 0 29m 10.127.156.171 k8s-master001
kube-system kube-scheduler-k8s-master001 1/1 Running 0 39m 10.127.156.171 k8s-master001

user001@k8s-master001:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system calico-node-jm695 1/2 CrashLoopBackOff 5 6m 10.127.156.171 k8s-master001
kube-system kube-apiserver-k8s-master001 1/1 Running 0 35m 10.127.156.171 k8s-master001
kube-system kube-controller-manager-k8s-master001 1/1 Running 0 35m 10.127.156.171 k8s-master001
kube-system kube-dns-86f4d74b45-s56kj 0/3 ContainerCreating 0 34m k8s-master001
kube-system kube-proxy-jbpvp 1/1 Running 0 34m 10.127.156.171 k8s-master001
kube-system kube-scheduler-k8s-master001 1/1 Running 0 44m 10.127.156.171 k8s-master001

user001@k8s-master001:~$ kubectl logs calico-node-jm695 -n kube-system calico-node
2018-06-14 16:28:16.495 [INFO][9] startup.go 251: Early log level set to info
2018-06-14 16:28:16.495 [INFO][9] startup.go 267: Using NODENAME environment for node name
2018-06-14 16:28:16.495 [INFO][9] startup.go 279: Determined node name: k8s-master001
2018-06-14 16:28:16.496 [INFO][9] startup.go 302: Checking datastore connection
2018-06-14 16:28:16.816 [INFO][9] startup.go 317: Hit error connecting to datastore - retry error=Get https://10.96.0.1:443/api/v1/nodes/foo: x509: cannot validate certificate for 10.96.0.1 because it doesn't contain any IP SANs

user001@k8s-master001:~$ kubectl logs calico-node-jm695 -n kube-system install-cni
ls: /calico-secrets: No such file or directory
Wrote Calico CNI binaries to /host/opt/cni/bin
CNI plugin version: v3.1.3
/host/secondary-bin-dir is non-writeable, skipping
CNI config: {
"name": "k8s-pod-network",
"cniVersion": "0.3.0",
"plugins": [
{
"type": "calico",
"log_level": "info",
"datastore_type": "kubernetes",
"nodename": "k8s-master001",
"mtu": 1500,
"ipam": {
"type": "host-local",
"subnet": "usePodCidr"
},
"policy": {
"type": "k8s"
},
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
}
},
{
"type": "portmap",
"snat": true,
"capabilities": {"portMappings": true}
}
]
}
Created CNI config 10-calico.conflist
Done configuring CNI. Sleep=true

Environment

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

$ docker images | grep calico
quay.io/calico/node v3.1.3 7eca10056c8e 2 weeks ago 248 MB
quay.io/calico/typha v0.7.4 c8f53c1b7957 2 weeks ago 56.9 MB
quay.io/calico/cni v3.1.3 9f355e076ea7 2 weeks ago 68.8 MB

NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"

kinsupport
Source
picture bladerunner512

Most helpful comment

I think the 10.96.0.1:443 is the kubernetes service endpoint which will be NAT'ed to the real IP for the apiserver and the correct port (6443).

I don't think this is a Calico issue at this point. I think you should look at the kubeadm issues to see if there are any similar reports with kube-dns. If I'm misunderstanding something and you believe this is something Calico related please let me know.

picture tmjd on 19 Jul 2018
👍2

All 20 comments

Weird. The certificate Calico uses in KDD mode is provided by Kubernetes. If that cert doesn't have IP SAN's, then that would suggest that k8s isn't generating the certs properly.

Which version of kubeadm is this?

ozdanborne picture ozdanborne on 14 Jun 2018

v1.10.0 amd64:
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

bladerunner512 picture bladerunner512 on 14 Jun 2018

Issue most likely due to ETCD RBAC - followed these instructions which required additional image 'calico-kube-controllers' in a different yaml deployment file :
https://docs.projectcalico.org/v3.1/reference/advanced/etcd-rbac/kubernetes

New YAML has calico-kube-controllers image that fails - I am also unable to delete previous deployment even after 'kubeadm reset':

$ kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system calico-kube-controllers-98989846-p6vp6 0/1 Error 1 25s 10.127.156.171 k8s-master001
kube-system calico-node-ctnth 1/2 CrashLoopBackOff 1 26s 10.127.156.171 k8s-master001
kube-system kube-apiserver-k8s-master001 1/1 Running 0 1h 10.127.156.171 k8s-master001
kube-system kube-controller-manager-k8s-master001 1/1 Running 0 1h 10.127.156.171 k8s-master001
kube-system kube-dns-86f4d74b45-zqqms 0/3 ContainerCreating 0 59m k8s-master001
kube-system kube-proxy-t44lp 1/1 Running 0 1h 10.127.156.171 k8s-master001
kube-system kube-scheduler-k8s-master001 1/1 Running 0 1h 10.127.156.171 k8s-master001
user001@k8s-master001:~$ kubectl logs calico-kube-controllers-98989846-p6vp6 --namespace=kube-system
2018-06-14 20:21:50.436 [INFO][1] main.go 69: Loaded configuration from environment config=&config.Config{LogLevel:"info", ReconcilerPeriod:"5m", CompactionPeriod:"10m", EnabledControllers:"policy,profile,workloadendpoint,node", WorkloadEndpointWorkers:1, ProfileWorkers:1, PolicyWorkers:1, NodeWorkers:1, Kubeconfig:""}

bladerunner512 picture bladerunner512 on 14 Jun 2018

After deleting previous deployment, new deployment still fails:

user001@k8s-master001:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                     READY     STATUS              RESTARTS   AGE       IP               NODE
kube-system   calico-kube-controllers-98989846-p6txc   0/1       CrashLoopBackOff    9          26m       10.127.156.171   k8s-master001
kube-system   calico-node-9v5ql                        1/2       CrashLoopBackOff    9          26m       10.127.156.171   k8s-master001
kube-system   kube-apiserver-k8s-master001             1/1       Running             0          17h       10.127.156.171   k8s-master001
kube-system   kube-controller-manager-k8s-master001    1/1       Running             0          17h       10.127.156.171   k8s-master001
kube-system   kube-dns-86f4d74b45-p4wmx                0/3       ContainerCreating   0          17h       <none>           k8s-master001
kube-system   kube-proxy-5vwbj                         1/1       Running             0          17h       10.127.156.171   k8s-master001
kube-system   kube-scheduler-k8s-master001             1/1       Running             0          17h       10.127.156.171   k8s-master001

$ sudo journalctl -xeu kubelet
Jun 15 11:06:33 k8s-master001 kubelet[37312]: E0615 11:06:33.003060   37312 cni.go:227] Error while adding to cni network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:33 k8s-master001 kubelet[37312]: E0615 11:06:33.155787   37312 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:33 k8s-master001 kubelet[37312]: E0615 11:06:33.155843   37312 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:33 k8s-master001 kubelet[37312]: E0615 11:06:33.155859   37312 kuberuntime_manager.go:646] createPodSandbox for pod "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:33 k8s-master001 kubelet[37312]: E0615 11:06:33.155922   37312 pod_workers.go:186] Error syncing pod b89917c4-7020-11e8-bc72-00530c0067fa ("kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)"), skipping: failed to "CreatePodSandbox" for "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"kube-dns-86f4d74b45-p4wmx_kube-system\" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/"
Jun 15 11:06:33 k8s-master001 kubelet[37312]: W0615 11:06:33.360743   37312 pod_container_deletor.go:77] Container "c0584c1647994a11a4149ccaf6c60273169079a3d2e12608ab8c33c0b3636a77" not found in pod's containers
Jun 15 11:06:34 k8s-master001 kubelet[37312]: E0615 11:06:34.006015   37312 cni.go:259] Error adding network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:34 k8s-master001 kubelet[37312]: E0615 11:06:34.006046   37312 cni.go:227] Error while adding to cni network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:34 k8s-master001 kubelet[37312]: E0615 11:06:34.143879   37312 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:34 k8s-master001 kubelet[37312]: E0615 11:06:34.143931   37312 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:34 k8s-master001 kubelet[37312]: E0615 11:06:34.143946   37312 kuberuntime_manager.go:646] createPodSandbox for pod "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:34 k8s-master001 kubelet[37312]: E0615 11:06:34.144010   37312 pod_workers.go:186] Error syncing pod b89917c4-7020-11e8-bc72-00530c0067fa ("kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)"), skipping: failed to "CreatePodSandbox" for "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"kube-dns-86f4d74b45-p4wmx_kube-system\" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/"
Jun 15 11:06:34 k8s-master001 kubelet[37312]: W0615 11:06:34.397102   37312 pod_container_deletor.go:77] Container "cd5e5dfb424e6238a7a04fe10d2fd30c43a7314d692e57556f4b0c8c27fd8205" not found in pod's containers
Jun 15 11:06:35 k8s-master001 kubelet[37312]: E0615 11:06:35.062056   37312 cni.go:259] Error adding network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:35 k8s-master001 kubelet[37312]: E0615 11:06:35.062088   37312 cni.go:227] Error while adding to cni network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:35 k8s-master001 kubelet[37312]: E0615 11:06:35.208665   37312 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:35 k8s-master001 kubelet[37312]: E0615 11:06:35.208722   37312 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:35 k8s-master001 kubelet[37312]: E0615 11:06:35.208747   37312 kuberuntime_manager.go:646] createPodSandbox for pod "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:35 k8s-master001 kubelet[37312]: E0615 11:06:35.208822   37312 pod_workers.go:186] Error syncing pod b89917c4-7020-11e8-bc72-00530c0067fa ("kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)"), skipping: failed to "CreatePodSandbox" for "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"kube-dns-86f4d74b45-p4wmx_kube-system\" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/"
Jun 15 11:06:35 k8s-master001 kubelet[37312]: W0615 11:06:35.433754   37312 pod_container_deletor.go:77] Container "f716a178301c7a3c1166b5909c5d0710d16010775934966f9af6ae1cc0997437" not found in pod's containers
Jun 15 11:06:36 k8s-master001 kubelet[37312]: E0615 11:06:36.097776   37312 cni.go:259] Error adding network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:36 k8s-master001 kubelet[37312]: E0615 11:06:36.097816   37312 cni.go:227] Error while adding to cni network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:36 k8s-master001 kubelet[37312]: E0615 11:06:36.239976   37312 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:36 k8s-master001 kubelet[37312]: E0615 11:06:36.240031   37312 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:36 k8s-master001 kubelet[37312]: E0615 11:06:36.240046   37312 kuberuntime_manager.go:646] createPodSandbox for pod "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:36 k8s-master001 kubelet[37312]: E0615 11:06:36.240108   37312 pod_workers.go:186] Error syncing pod b89917c4-7020-11e8-bc72-00530c0067fa ("kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)"), skipping: failed to "CreatePodSandbox" for "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"kube-dns-86f4d74b45-p4wmx_kube-system\" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/"
Jun 15 11:06:36 k8s-master001 kubelet[37312]: W0615 11:06:36.466865   37312 pod_container_deletor.go:77] Container "1991683c522ae5457137fcb784604056c1dd20797718b11ced4b73fb6a37f572" not found in pod's containers
Jun 15 11:06:37 k8s-master001 kubelet[37312]: E0615 11:06:37.118546   37312 cni.go:259] Error adding network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:37 k8s-master001 kubelet[37312]: E0615 11:06:37.118581   37312 cni.go:227] Error while adding to cni network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:37 k8s-master001 kubelet[37312]: E0615 11:06:37.252341   37312 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:37 k8s-master001 kubelet[37312]: E0615 11:06:37.252396   37312 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:37 k8s-master001 kubelet[37312]: E0615 11:06:37.252411   37312 kuberuntime_manager.go:646] createPodSandbox for pod "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:37 k8s-master001 kubelet[37312]: E0615 11:06:37.252471   37312 pod_workers.go:186] Error syncing pod b89917c4-7020-11e8-bc72-00530c0067fa ("kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)"), skipping: failed to "CreatePodSandbox" for "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"kube-dns-86f4d74b45-p4wmx_kube-system\" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/"
Jun 15 11:06:37 k8s-master001 kubelet[37312]: W0615 11:06:37.506774   37312 pod_container_deletor.go:77] Container "0d8ef6a6d027487cd0c348fb1e17781fda8ca3637a38abc81b1e22808be04fdd" not found in pod's containers
Jun 15 11:06:38 k8s-master001 kubelet[37312]: E0615 11:06:38.168332   37312 cni.go:259] Error adding network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:38 k8s-master001 kubelet[37312]: E0615 11:06:38.168361   37312 cni.go:227] Error while adding to cni network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:38 k8s-master001 kubelet[37312]: E0615 11:06:38.340172   37312 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:38 k8s-master001 kubelet[37312]: E0615 11:06:38.340231   37312 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:38 k8s-master001 kubelet[37312]: E0615 11:06:38.340246   37312 kuberuntime_manager.go:646] createPodSandbox for pod "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-p4wmx_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Jun 15 11:06:38 k8s-master001 kubelet[37312]: E0615 11:06:38.340311   37312 pod_workers.go:186] Error syncing pod b89917c4-7020-11e8-bc72-00530c0067fa ("kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)"), skipping: failed to "CreatePodSandbox" for "kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-dns-86f4d74b45-p4wmx_kube-system(b89917c4-7020-11e8-bc72-00530c0067fa)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"kube-dns-86f4d74b45-p4wmx_kube-system\" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/"
Jun 15 11:06:38 k8s-master001 kubelet[37312]: I0615 11:06:38.421909   37312 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/b3da23e8-70ad-11e8-bc72-00530c0067fa-lib-modules") pod "calico-node-9v5ql" (UID: "b3da23e8-70ad-11e8-bc72-00530c0067fa")
Jun 15 11:06:38 k8s-master001 kubelet[37312]: I0615 11:06:38.421950   37312 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "cni-net-dir" (UniqueName: "kubernetes.io/host-path/b3da23e8-70ad-11e8-bc72-00530c0067fa-cni-net-dir") pod "calico-node-9v5ql" (UID: "b3da23e8-70ad-11e8-bc72-00530c0067fa")
Jun 15 11:06:38 k8s-master001 kubelet[37312]: I0615 11:06:38.421974   37312 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "calico-kube-controllers-token-gc2vw" (UniqueName: "kubernetes.io/secret/b3dcd454-70ad-11e8-bc72-00530c0067fa-calico-kube-controllers-token-gc2vw") pod "calico-kube-controllers-98989846-p6txc" (UID: "b3dcd454-70ad-11e8-bc72-00530c0067fa")
Jun 15 11:06:38 k8s-master001 kubelet[37312]: I0615 11:06:38.422018   37312 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "var-run-calico" (UniqueName: "kubernetes.io/host-path/b3da23e8-70ad-11e8-bc72-00530c0067fa-var-run-calico") pod "calico-node-9v5ql" (UID: "b3da23e8-70ad-11e8-bc72-00530c0067fa")
Jun 15 11:06:38 k8s-master001 kubelet[37312]: I0615 11:06:38.422051   37312 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "cni-bin-dir" (UniqueName: "kubernetes.io/host-path/b3da23e8-70ad-11e8-bc72-00530c0067fa-cni-bin-dir") pod "calico-node-9v5ql" (UID: "b3da23e8-70ad-11e8-bc72-00530c0067fa")
Jun 15 11:06:38 k8s-master001 kubelet[37312]: I0615 11:06:38.422081   37312 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/secret/b3da23e8-70ad-11e8-bc72-00530c0067fa-etcd-certs") pod "calico-node-9v5ql" (UID: "b3da23e8-70ad-11e8-bc72-00530c0067fa")
Jun 15 11:06:38 k8s-master001 kubelet[37312]: I0615 11:06:38.422111   37312 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "calico-node-token-5rzzh" (UniqueName: "kubernetes.io/secret/b3da23e8-70ad-11e8-bc72-00530c0067fa-calico-node-token-5rzzh") pod "calico-node-9v5ql" (UID: "b3da23e8-70ad-11e8-bc72-00530c0067fa")
Jun 15 11:06:38 k8s-master001 kubelet[37312]: I0615 11:06:38.422140   37312 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/secret/b3dcd454-70ad-11e8-bc72-00530c0067fa-etcd-certs") pod "calico-kube-controllers-98989846-p6txc" (UID: "b3dcd454-70ad-11e8-bc72-00530c0067fa")
Jun 15 11:06:38 k8s-master001 kubelet[37312]: I0615 11:06:38.422178   37312 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "var-lib-calico" (UniqueName: "kubernetes.io/host-path/b3da23e8-70ad-11e8-bc72-00530c0067fa-var-lib-calico") pod "calico-node-9v5ql" (UID: "b3da23e8-70ad-11e8-bc72-00530c0067fa")
bladerunner512 picture bladerunner512 on 15 Jun 2018

Jun 15 11:06:38 k8s-master001 kubelet[37312]: E0615 11:06:38.168332 37312 cni.go:259] Error adding network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

This looks like you're missing the /var/lib/calico mount into calico/node. Where did you get the calico.yaml file? Is it hand edited?

That mount was added in v3.1, so if you tried to use a v3.0 manifest with v3.1 images it likely won't work.

caseydavenport picture caseydavenport on 15 Jun 2018

Used the link to 3.1.3:
https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/calico.yaml

Followed instructions for etcd-rbac on:
https://docs.projectcalico.org/v3.1/reference/advanced/etcd-rbac/kubernetes

edited calico.yaml as instructed:
uncommented:
etcd_ca: "/calico-secrets/etcd-ca"
etcd_cert: "/calico-secrets/etcd-cert"
etcd_key: "/calico-secrets/etcd-key"

and added base64 encoded secrets
etcd-key: LS0tLS1CRU...
etcd-cert: LS0tLS1CRUdJ...
etcd-ca: LS0tLS1CRUdJTiBDRVJ...

bladerunner512 picture bladerunner512 on 15 Jun 2018

/var/lib/calico directory created but empty
/opt/cni/bin has calico and calico-ipam
/var/run/calico created but empty
/etc/cni/net.d has calico-tls dir (w/etcd-ca,cert,key) and files: 10-calico.conflist, calico-kubeconfig

bladerunner512 picture bladerunner512 on 15 Jun 2018

looks like /var/lib/calico is mounted in calico.yaml file:

volumeMounts:
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- mountPath: /var/run/calico
name: var-run-calico
readOnly: false
- mountPath: /var/lib/calico
name: var-lib-calico
readOnly: false
- mountPath: /calico-secrets
name: etcd-certs

volumes:
    # Used by calico/node.
    - name: lib-modules
      hostPath:
        path: /lib/modules
    - name: var-run-calico
      hostPath:
        path: /var/run/calico
    - name: var-lib-calico
      hostPath:
        path: /var/lib/calico
    # Used to install CNI.
    - name: cni-bin-dir
      hostPath:
        path: /opt/cni/bin
    - name: cni-net-dir
      hostPath:
        path: /etc/cni/net.d

kubelet has cni args:
--cni-conf-dir=/etc/cni/net.d
--cni-bin-dir=/opt/cni/bin

bladerunner512 picture bladerunner512 on 15 Jun 2018

ran container quay.io/calico/node:v3.1.3 so I could look inside and did not see /var/lib/calico directory?
ran container quay.io/calico/cni:v3.1.3 so I could look inside and did not see /var/lib/calico directory?

Does an entrypoint script create /var/lib/calico directory? What is supposed to be in this directory?

bladerunner512 picture bladerunner512 on 15 Jun 2018

I see a little bit of conflicting information here so I want to ask a high level question. Are you trying to use Kubernetes as the datastore for Calico or etcd? I'm asking because in your original issue you had "datastore_type": "kubernetes" which indicates that Calico would be using kubernetes for the datastore but then several of your follow up comments are referring to configuring etcd certs and link to etcd manifests and you've got calico-kube-controllers running which point to using etcd as the datastore for Calico.

As for your question about /var/lib/calico, I believe when calico/node runs that it should create a file in /var/lib/calico. On any host running calico/node you should be able to look in /var/lib/calico and see the file (I think it is called nodename, but I'm not sure about that).

tmjd picture tmjd on 21 Jun 2018

Calico should be using etcd - "datastore_type": "kubernetes" appears in original post before trying to use etcd certs specified in calico.yaml:

https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/calico.yaml

I never get calico/node running (always get 'CrashLoopBackOff') so that is probably why nothing in /var/lib/calico.

bladerunner512 picture bladerunner512 on 21 Jun 2018

Is calico-node still getting the error Hit error connecting to datastore - retry error=Get https://10.96.0.1:443/api/v1/nodes/foo: x509: cannot validate certificate for 10.96.0.1 because it doesn't contain any IP SANs? Have you ensured that the certificate you are using does contain IP SANS, specifically including 10.96.0.1?

tmjd picture tmjd on 21 Jun 2018
👍1

No, after adding etcd cert info in calico.yaml link mentioned above, I get the "Error adding network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container" message:

kubelet[76951]: W0621 14:29:06.275642 76951 pod_container_deletor.go:77] Container "33a7916c02468c5277760333d807c056256ff950de3f33062de2bda2321ce969" not found in pod's cont
kubelet[76951]: E0621 14:29:07.169786 76951 cni.go:259] Error adding network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container
kubelet[76951]: E0621 14:29:07.169842 76951 cni.go:227] Error while adding to cni network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/no
kubelet[76951]: E0621 14:29:07.479151 76951 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set
kubelet[76951]: E0621 14:29:07.479222 76951 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-86f4d74b45-ljsn6_kube-system(ce87d291-732c-11e8-8244-00530c0067fa)"
kubelet[76951]: E0621 14:29:07.479239 76951 kuberuntime_manager.go:646] createPodSandbox for pod "kube-dns-86f4d74b45-ljsn6_kube-system(ce87d291-732c-11e8-8244-00530c0067fa)
kubelet[76951]: E0621 14:29:07.479312 76951 pod_workers.go:186] Error syncing pod ce87d291-732c-11e8-8244-00530c0067fa ("kube-dns-86f4d74b45-ljsn6_kube-system(ce87d291-732c-

bladerunner512 picture bladerunner512 on 21 Jun 2018

Added verbose output, if that helps:

$ sudo journalctl -xeu kubelet -o verbose
_TRANSPORT=stdout
PRIORITY=6
SYSLOG_FACILITY=3
_UID=0
_GID=0
_CAP_EFFECTIVE=3fffffffff
_SYSTEMD_SLICE=system.slice
_BOOT_ID=285ed22a66274c91b19c75239eb4f77b
_MACHINE_ID=cf6230dc06da44828f9c81b651465e19
_HOSTNAME=k8s-master001
SYSLOG_IDENTIFIER=kubelet
_PID=76951
_COMM=kubelet
_EXE=/usr/bin/kubelet
_CMDLINE=/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --
_SYSTEMD_CGROUP=/system.slice/kubelet.service
_SYSTEMD_UNIT=kubelet.service
MESSAGE=E0621 14:36:14.470358 76951 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-86f4d74b45-ljsn6_kube-system(ce87d291-732c-11e8-8244-00530c0067fa)" failed: rpc error: code = Unknown
Thu 2018-06-21 14:36:14.470781 EDT [s=ae85e0a969024703945bc1d773040b1c;i=8b777e;b=285ed22a66274c91b19c75239eb4f77b;m=467ebfaa50;t=56f2b30acf27d;x=cecdc244bbaaad12]
_TRANSPORT=stdout
PRIORITY=6
SYSLOG_FACILITY=3
_UID=0
_GID=0
_CAP_EFFECTIVE=3fffffffff
_SYSTEMD_SLICE=system.slice
_BOOT_ID=285ed22a66274c91b19c75239eb4f77b
_MACHINE_ID=cf6230dc06da44828f9c81b651465e19
_HOSTNAME=k8s-master001
SYSLOG_IDENTIFIER=kubelet
_PID=76951
_COMM=kubelet
_EXE=/usr/bin/kubelet
_CMDLINE=/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --
_SYSTEMD_CGROUP=/system.slice/kubelet.service
_SYSTEMD_UNIT=kubelet.service
MESSAGE=E0621 14:36:14.470377 76951 kuberuntime_manager.go:646] createPodSandbox for pod "kube-dns-86f4d74b45-ljsn6_kube-system(ce87d291-732c-11e8-8244-00530c0067fa)" failed: rpc error: code = Unknown
Thu 2018-06-21 14:36:14.470900 EDT [s=ae85e0a969024703945bc1d773040b1c;i=8b777f;b=285ed22a66274c91b19c75239eb4f77b;m=467ebfaac7;t=56f2b30acf2f4;x=cc6d6ae4d5e9e6d6]
_TRANSPORT=stdout
PRIORITY=6
SYSLOG_FACILITY=3
_UID=0
_GID=0
_CAP_EFFECTIVE=3fffffffff
_SYSTEMD_SLICE=system.slice
_BOOT_ID=285ed22a66274c91b19c75239eb4f77b
_MACHINE_ID=cf6230dc06da44828f9c81b651465e19
_HOSTNAME=k8s-master001
SYSLOG_IDENTIFIER=kubelet
_PID=76951
_COMM=kubelet
_EXE=/usr/bin/kubelet
_CMDLINE=/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --
_SYSTEMD_CGROUP=/system.slice/kubelet.service
_SYSTEMD_UNIT=kubelet.service
MESSAGE=E0621 14:36:14.470432 76951 pod_workers.go:186] Error syncing pod ce87d291-732c-11e8-8244-00530c0067fa ("kube-dns-86f4d74b45-ljsn6_kube-system(ce87d291-732c-11e8-8244-00530c0067fa)"), skipping

bladerunner512 picture bladerunner512 on 21 Jun 2018

You said that calico-node is still in CrashLoopBackoff, the error from the kubelet is still expected if calico-node is crashing. You should concentrate on fixing whatever is preventing calico-node from starting before looking into the kubelet errors.
What do the logs from calico-node report now?

tmjd picture tmjd on 21 Jun 2018

Not much info from calico-node, but keeps restarting:

user001@k8s-master001:~$ kubectl logs calico-node-zdnlt -n kube-system calico-node
2018-06-22 15:43:56.694 [INFO][9] startup.go 251: Early log level set to info
2018-06-22 15:43:56.694 [INFO][9] startup.go 271: Using HOSTNAME environment (lowercase) for node name
2018-06-22 15:43:56.695 [INFO][9] startup.go 279: Determined node name: k8s-master001
2018-06-22 15:43:56.715 [INFO][9] startup.go 101: Skipping datastore connection test

user001@k8s-master001:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system calico-node-zdnlt 1/2 Running 13 18h 10.127.156.171 k8s-master001
kube-system kube-apiserver-k8s-master001 1/1 Running 0 33m 10.127.156.171 k8s-master001
kube-system kube-controller-manager-k8s-master001 1/1 Running 0 33m 10.127.156.171 k8s-master001
kube-system kube-dns-86f4d74b45-5bhkl 0/3 ContainerCreating 0 32m k8s-master001
kube-system kube-proxy-r2wfl 1/1 Running 0 18h 10.127.156.171 k8s-master001
kube-system kube-scheduler-k8s-master001 1/1 Running 0 18h 10.127.156.171 k8s-master001

Eventually enters 'CrashLoopBackOff', but unable to get log from calico-node at that point:

user001@k8s-master001:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system calico-node-zdnlt 1/2 CrashLoopBackOff 15 18h 10.127.156.171 k8s-master001
kube-system kube-apiserver-k8s-master001 1/1 Running 0 42m 10.127.156.171 k8s-master001
kube-system kube-controller-manager-k8s-master001 1/1 Running 0 42m 10.127.156.171 k8s-master001
kube-system kube-dns-86f4d74b45-5bhkl 0/3 ContainerCreating 0 42m k8s-master001
kube-system kube-proxy-r2wfl 1/1 Running 0 18h 10.127.156.171 k8s-master001
kube-system kube-scheduler-k8s-master001 1/1 Running 0 18h 10.127.156.171 k8s-master001

bladerunner512 picture bladerunner512 on 25 Jun 2018

After updating ConfigMap to point to certs I get a little further:

changed:
"etcd_endpoints": "__ETCD_ENDPOINTS__",
"etcd_key_file": "__ETCD_KEY_FILE__",
"etcd_cert_file": "__ETCD_CERT_FILE__",
"etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",

to:
"etcd_endpoints": "https://12.127.136.176:2379,https://12.127.136.177:2379,https://12.127.136.178:2379",
"etcd_key_file": "/etc/kubernetes/pki/etcd/client-key.pem",
"etcd_cert_file": "/etc/kubernetes/pki/etcd/client.pem",
"etcd_ca_cert_file": "/etc/kubernetes/pki/etcd/ca.pem",

I get NodeLost for kube-proxy pod and calico-node pod:
$ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-98989846-z5zbv 1/1 Running 0 7m
calico-node-hgm25 2/2 Running 0 6m
calico-node-mj9s6 0/2 NodeLost 0 6m
kube-apiserver-master001 1/1 Running 0 1h
kube-controller-manager-master001 1/1 Running 0 1h
kube-dns-86f4d74b45-dxpwq 0/3 ContainerCreating 0 1h
kube-proxy-f6w6h 1/1 NodeLost 0 1h
kube-proxy-n69nd 1/1 Running 0 1h
kube-scheduler-dcmidavgcc1624 1/1 Running 0 1h

Can't get logs for NodeLost pods and other logs do not indicate errors?

bladerunner512 picture bladerunner512 on 13 Jul 2018

NodeLost is probably from the kubelet on a K8s node not checking in with the apiserver. You'll probably have to figure out what is going wrong on that host to fix those.

For kube-dns you should do kubectl describe -n kube-system pod kube-dns-86f4d74b45-dxpwq to maybe see why it isn't starting. You can also look at the kubelet log on the host where it is trying to be started.

tmjd picture tmjd on 16 Jul 2018

It looks like everything is running now except kube-dns. Getting x509 error:
Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: cannot validate certificate for 10.96.0.1 because it doesn't contain any IP SANs.

Output from kubeadm init indicates that apiserver certs are correct:
...
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [node1624 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 xx.xxx.xxx.174 127.0.0.1]
...

Shouldn't api server port be 6443 not 443?
Get https://10.96.0.1:443/api/v1/services?resourceVersion=0

$ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-98989846-25bql 1/1 Running 0 2m
calico-node-zfbzc 2/2 Running 0 55s
kube-apiserver-node1624 1/1 Running 0 18h
kube-controller-manager-node1624 1/1 Running 0 18h
kube-dns-86f4d74b45-m8bqx 1/3 CrashLoopBackOff 614 18h
kube-proxy-ccvsq 1/1 Running 0 18h
kube-scheduler-node1624 1/1 Running 0 18h

$ kubectl -n kube-system logs kube-dns-86f4d74b45-m8bqx kubedns
I0717 14:38:39.174531 1 server.go:201] Starting SkyDNS server (0.0.0.0:10053)
I0717 14:38:39.174756 1 server.go:220] Skydns metrics enabled (/metrics:10055)
I0717 14:38:39.174768 1 dns.go:146] Starting endpointsController
I0717 14:38:39.174772 1 dns.go:149] Starting serviceController
I0717 14:38:39.176242 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0717 14:38:39.176270 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
E0717 14:38:39.499135 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: x509: cannot validate certificate for 10.96.0.1 because it doesn't contain any IP SANs
E0717 14:38:39.508016 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: x509: cannot validate certificate for 10.96.0.1 because it doesn't contain any IP SANs
I0717 14:38:39.674970 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0717 14:38:40.175009 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0717 14:38:40.674950 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...

bladerunner512 picture bladerunner512 on 17 Jul 2018

I think the 10.96.0.1:443 is the kubernetes service endpoint which will be NAT'ed to the real IP for the apiserver and the correct port (6443).

I don't think this is a Calico issue at this point. I think you should look at the kubeadm issues to see if there are any similar reports with kube-dns. If I'm misunderstanding something and you believe this is something Calico related please let me know.

tmjd picture tmjd on 19 Jul 2018
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

caseydavenport picture caseydavenport  路  6Comments
mindw picture mindw  路  4Comments
lwr20 picture lwr20  路  5Comments
cann0nf0dder picture cann0nf0dder  路  5Comments
mrsherlock88 picture mrsherlock88  路  3Comments