Flannel: Stuck at kube-flannel-ds Init:0/1 and kube-proxy containercreating

[armanriazi]

@kubernetes/sig-contributor-experience-test-failures
@kubernetes/sig-onprem
@kubernetes/sig-network

When I run kubectl get pods -o wide -n kube-system i get this statues:

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
coredns-757f4547d9-g6hd8 0/1 ContainerCreating 0 47m k8s-master
coredns-757f4547d9-xvzgx 1/1 Running 0 53m 10.244.1.50 k8s-node-01
etcd-k8s-master 1/1 Running 5 3h35m 172.18.3.9 k8s-master
kube-apiserver-k8s-master 1/1 Running 5 3h35m 172.18.3.9 k8s-master
kube-controller-manager-k8s-master 1/1 Running 5 139m 172.18.3.9 k8s-master
kube-flannel-ds-amd64-4kxqv 1/1 Running 0 55m 172.18.3.15 k8s-node-01

*kube-flannel-ds-amd64-8qlg9 0/1 Init:0/1 0 54m 172.18.3.9 k8s-master

kube-flannel-ds-amd64-vtbbm 1/1 Running 0 55m 172.18.3.16 k8s-node-02
kube-proxy-dpc99 1/1 Running 0 54m 172.18.3.16 k8s-node-02

*kube-proxy-f9fgh 0/1 ContainerCreating 0 54m 172.18.3.9 k8s-master

kube-proxy-pvflv 1/1 Running 0 54m 172.18.3.15 k8s-node-01
kube-scheduler-k8s-master 1/1 Running 5 3h35m 172.18.3.9 k8s-master
kubernetes-dashboard-7459f9d896-g8zf5 1/1 Running 1 91m 10.244.2.161 k8s-node-02

traced:

root@k8s-master:~# kubectl describe pods kube-flannel-ds-amd64-8qlg9 -n kube-system
Name: kube-flannel-ds-amd64-8qlg9
Namespace: kube-system
Priority: 0
PriorityClassName:
Node: k8s-master/172.18.3.9
Start Time: Tue, 01 Jan 2019 14:16:38 +0330
Labels: app=flannel
controller-revision-hash=6697bf5fc6
pod-template-generation=1
tier=node
Annotations:
Status: Pending
IP: 172.18.3.9
Controlled By: DaemonSet/kube-flannel-ds-amd64
Init Containers:
install-cni:
Container ID:
Image: quay.io/coreos/flannel:v0.10.0-amd64
Image ID:
Port:
Host Port:
Command:
cp
Args:
-f
/etc/kube-flannel/cni-conf.json
/etc/cni/net.d/10-flannel.conflist
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
Mounts:
/etc/cni/net.d from cni (rw)
/etc/kube-flannel/ from flannel-cfg (rw)
/var/run/secrets/kubernetes.io/serviceaccount from flannel-token-7mm6h (ro)
Containers:
kube-flannel:
Container ID:
Image: quay.io/coreos/flannel:v0.10.0-amd64
Image ID:
Port:
Host Port:
Command:
/opt/bin/flanneld
Args:
--ip-masq
--kube-subnet-mgr
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 100m
memory: 50Mi
Requests:
cpu: 100m
memory: 50Mi
Environment:
POD_NAME: kube-flannel-ds-amd64-8qlg9 (v1:metadata.name)
POD_NAMESPACE: kube-system (v1:metadata.namespace)
Mounts:
/etc/kube-flannel/ from flannel-cfg (rw)
/run from run (rw)
/var/run/secrets/kubernetes.io/serviceaccount from flannel-token-7mm6h (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
run:
Type: HostPath (bare host directory volume)
Path: /run
HostPathType:
cni:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
flannel-cfg:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-flannel-cfg
Optional: false
flannel-token-7mm6h:
Type: Secret (a volume populated by a Secret)
SecretName: flannel-token-7mm6h
Optional: false
QoS Class: Guaranteed
Node-Selectors: beta.kubernetes.io/arch=amd64
Tolerations: :NoSchedule
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/network-unavailable:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unschedulable:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 89m default-scheduler Successfully assigned kube-system/kube-flannel-ds-amd64-8qlg9 to k8s-master
Normal Pulled 89m kubelet, k8s-master Container image "quay.io/coreos/flannel:v0.10.0-amd64" already present on machine
Normal Pulled 51m kubelet, k8s-master Container image "quay.io/coreos/flannel:v0.10.0-amd64" already present on machine

kubectl logs --namespace kube-system kube-flannel-ds-amd64-8qlg9 -c kube-flannel
Error from server (BadRequest): container "kube-flannel" in pod "kube-flannel-ds-amd64-8qlg9" is waiting to start: PodInitializing

root@k8s-master:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.4", GitCommit:"f49fa022dbe63faafd0da106ef7e05a29721d3f1", GitTreeState:"clean", BuildDate:"2018-12-14T06:59:37Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

for another pods:

root@k8s-master:~# kubectl describe pods coredns-757f4547d9-rtzpl -n kube-system
Name: coredns-757f4547d9-rtzpl
Namespace: kube-system
Priority: 0
PriorityClassName:
Node: k8s-master/172.18.3.9
Start Time: Tue, 01 Jan 2019 16:01:02 +0330
Labels: k8s-app=kube-dns
pod-template-hash=757f4547d9
Annotations:
Status: Pending
IP:
Controlled By: ReplicaSet/coredns-757f4547d9
Containers:
coredns:
Container ID:
Image: 172.18.3.9:5000/coredns:1.2.2
Image ID:
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Environment:
Mounts:
/etc/coredns from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from coredns-token-2wqfh (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
coredns-token-2wqfh:
Type: Secret (a volume populated by a Secret)
SecretName: coredns-token-2wqfh
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m35s default-scheduler Successfully assigned kube-system/coredns-757f4547d9-rtzpl to k8s-master
Warning FailedCreatePodSandBox 2m33s kubelet, k8s-master Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9b2694bdaf2816f9b41432e26c950704e950b87f15af5a8267fc98e65769ad14" network for pod "coredns-757f4547d9-rtzpl": NetworkPlugin cni failed to set up pod "coredns-757f4547d9-rtzpl_kube-system" network: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 2m32s kubelet, k8s-master Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "edb799c77003405599ff6e84ab2d1968ec282e1d25cfca4c61a60ec7ef9405c9" network for pod "coredns-757f4547d9-rtzpl": NetworkPlugin cni failed to set up pod "coredns-757f4547d9-rtzpl_kube-system" network: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 2m30s kubelet, k8s-master Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "101110b6e4d6617c29641c581e153f12af456ddadde36dcd4181d0e0426e4f5d" network for pod "coredns-757f4547d9-rtzpl": NetworkPlugin cni failed to set up pod "coredns-757f4547d9-rtzpl_kube-system" network: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 2m28s kubelet, k8s-master Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f5a649b91857bd34f13bbef850e5b072390a58f2d7ef3e2c2969cd725efbd00c" network for pod "coredns-757f4547d9-rtzpl": NetworkPlugin cni failed to set up pod "coredns-757f4547d9-rtzpl_kube-system" network: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 2m26s kubelet, k8s-master Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a56513b40003f81faccf2100b7f3777c9e1dbf6a1481e31f568f27c135ae81b9" network for pod "coredns-757f4547d9-rtzpl": NetworkPlugin cni failed to set up pod "coredns-757f4547d9-rtzpl_kube-system" network: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 2m25s kubelet, k8s-master Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e8dcbf391c856e7fbf2a2a307ef94e871eafd10603eda738d9858c128dae385e" network for pod "coredns-757f4547d9-rtzpl": NetworkPlugin cni failed to set up pod "coredns-757f4547d9-rtzpl_kube-system" network: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 2m23s kubelet, k8s-master Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "74e92773bb17b01b72baa1e622d844554929df217237978ec9e6081f7330d25c" network for pod "coredns-757f4547d9-rtzpl": NetworkPlugin cni failed to set up pod "coredns-757f4547d9-rtzpl_kube-system" network: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 2m20s kubelet, k8s-master Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ed92b2652cee21d11bd3f80fa2317cc732c9acbe45ccdc72223d7f5755c59b0b" network for pod "coredns-757f4547d9-rtzpl": NetworkPlugin cni failed to set up pod "coredns-757f4547d9-rtzpl_kube-system" network: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 2m18s kubelet, k8s-master Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ef4c4c0b9ff7d752802c3e7f9a52eccd9081fbe762a95b378a874a2d0d11531a" network for pod "coredns-757f4547d9-rtzpl": NetworkPlugin cni failed to set up pod "coredns-757f4547d9-rtzpl_kube-system" network: open /run/flannel/subnet.env: no such file or directory
Normal SandboxChanged 2m12s (x12 over 2m33s) kubelet, k8s-master Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 2m11s (x4 over 2m17s) kubelet, k8s-master (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "51ea1d68bebc10c873ac67a1462549129fa9a2bf3d2fd0b2250900eb1125b4e4" network for pod "coredns-757f4547d9-rtzpl": NetworkPlugin cni failed to set up pod "coredns-757f4547d9-rtzpl_kube-system" network: open /run/flannel/subnet.env: no such file or directory

even I deleted two pods but it no change anything

[armanriazi]

I applied it again with this config but anything didn't change :

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: 172.18.3.9:5000/coreos/flannel:v0.10.0-amd64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: 172.18.3.9:5000/coreos/flannel:v0.10.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-arm64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: arm64
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: 172.18.3.9:5000/coreos/flannel:v0.10.0-arm64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: 172.18.3.9:5000/coreos/flannel:v0.10.0-arm64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-arm
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: arm
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: 172.18.3.9:5000/coreos/flannel:v0.10.0-arm
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: 172.18.3.9:5000/coreos/flannel:v0.10.0-arm
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-ppc64le
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: ppc64le
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: 172.18.3.9:5000/coreos/flannel:v0.10.0-ppc64le
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: 172.18.3.9:5000/coreos/flannel:v0.10.0-ppc64le
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-s390x
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: s390x
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: 172.18.3.9:5000/coreos/flannel:v0.10.0-s390x
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: 172.18.3.9:5000/coreos/flannel:v0.10.0-s390x
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

[michaelfig]

I had a similar problem that was caused by updating kubelets to 1.13 without properly running kubeadm upgrade plan and kubeadm upgrade apply....

Reverting to 1.12, deleting the flannel daemonset and then doing the upgrade and applying the flannel yaml made it all work again.

Hope this helps someone,
Michael.