Flannel: network not ready after `kubectl apply -f kube-flannel.yaml` in v1.16 cluster

Created on 19 Sep 2019 · 25Comments · Source: coreos/flannel

Expected Behavior

Container network should get ready after kubectl apply -f kube-flannel.yaml

Current Behavior

After creating a v1.16 cluster by kubeadm init and installing flannel cni by kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/32a765fd19ba45b387fdc5e3812c41fff47cfd55/Documentation/kube-flannel.yml, the network still does not get ready and coredns remain pending.

Possible Solution

Steps to Reproduce (for bugs)

Create a v1.16 cluster with kubeadm
kubectl apply -f kube-flannel.yaml
3.
4.

Context

Your Environment

Flannel version: v0.11.0
Backend used (e.g. vxlan or udp):
Etcd version:
Kubernetes version (if used): v1.16.0
Operating System and version: ubuntu 18.04
Link to your project (optional):

Source

sshukun

👍4

Most helpful comment

I was able to get nodes to come up as ready by adding:

      "cniVersion":"0.3.1",

Under this line: https://github.com/coreos/flannel/blob/d893bcbfe6b04791054aea6c7569dea4080cc289/Documentation/kube-flannel.yml#L108

When I do a kubectl create -f the-above-flannel.yml

dougbtv on 19 Sep 2019

👍32 🚀5 ❤4 😄3 👎3 👀2 😕2 🎉2

All 25 comments

Same behaviour here, the only difference is the underlying OS:
CentOS Linux release 7.6.1810 (Core)

foofoo-2 on 19 Sep 2019

👍1

Same behavior here

On CentOS 7 Linux node1 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13 22:55:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
[root@node1 ~]#

journalctl -f -u kubelet.service :

Sep 19 04:21:44 node1 kubelet[10636]: W0919 04:21:44.389281   10636 cni.go:202] Error validating CNI config &{cbr0  false [0xc001613540 0xc0016135c0] [123 10 32 32 34 110 97 109 101 34 58 32 34 99 98 114 48 34 44 10 32 32 34 112 108 117 103 105 110 115 34 58 32 91 10 32 32 32 32 123 10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 102 108 97 110 110 101 108 34 44 10 32 32 32 32 32 32 34 100 101 108 101 103 97 116 101 34 58 32 123 10 32 32 32 32 32 32 32 32 34 104 97 105 114 112 105 110 77 111 100 101 34 58 32 116 114 117 101 44 10 32 32 32 32 32 32 32 32 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 32 116 114 117 101 10 32 32 32 32 32 32 125 10 32 32 32 32 125 44 10 32 32 32 32 123 10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 112 111 114 116 109 97 112 34 44 10 32 32 32 32 32 32 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 32 123 10 32 32 32 32 32 32 32 32 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 32 116 114 117 101 10 32 32 32 32 32 32 125 10 32 32 32 32 125 10 32 32 93 10 125 10]}: [plugin flannel does not support config version ""]
Sep 19 04:21:44 node1 kubelet[10636]: W0919 04:21:44.389341   10636 cni.go:237] Unable to update cni config: no valid networks found in /etc/cni/net.d
Sep 19 04:21:45 node1 kubelet[10636]: E0919 04:21:45.583516   10636 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

There was message [plugin flannel does not support config version ""

lff0305 on 19 Sep 2019

I also encounter same error

Linux hrshl 5.3.0-1.el7.elrepo.x86_64 #1 SMP Mon Sep 16 08:44:46 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

[root@hrsh ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                              READY   STATUS              RESTARTS   AGE
kube-system   coredns-5644d7b6d9-hq7qp          0/1     ContainerCreating   0          96m
kube-system   coredns-5644d7b6d9-wq2c7          0/1     ContainerCreating   0          96m
kube-system   etcd-hrsh                      1/1     Running             4          95m
kube-system   kube-apiserver-hrsh            1/1     Running             4          96m
kube-system   kube-controller-manager-hrsh   1/1     Running             5          96m
kube-system   kube-flannel-ds-amd64-p4nwk       1/1     Running             1          15m
kube-system   kube-proxy-pfh9d                  1/1     Running             4          96m
kube-system   kube-scheduler-hrsh            1/1     Running             5          96m

[root@hrsh ~]# kubectl describe pod coredns-5644d7b6d9-hq7qp -n kube-system
Name:                 coredns-5644d7b6d9-hq7qp
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 hrsh/xxx.xxx.xxx.84
Start Time:           Thu, 19 Sep 2019 16:32:38 +0530
Labels:               k8s-app=kube-dns
                      pod-template-hash=5644d7b6d9
Annotations:          <none>
Status:               Pending
IP:                   
IPs:                  <none>
Controlled By:        ReplicaSet/coredns-5644d7b6d9
Containers:
  coredns:
    Container ID:  
    Image:         k8s.gcr.io/coredns:1.6.2
    Image ID:      
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-8hbwc (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-8hbwc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-8hbwc
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason           Age                    From              Message
  ----     ------           ----                   ----              -------
  Warning  NetworkNotReady  20m (x1955 over 85m)   kubelet, hrsh  network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
  Warning  NetworkNotReady  6m59s (x307 over 17m)  kubelet, hrsh  network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
  Warning  NetworkNotReady  60s (x25 over 109s)    kubelet, hrsh  network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

[root@hrsh ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

[root@hrsh ~]# systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Thu 2019-09-19 18:05:59 IST; 5min ago
     Docs: https://kubernetes.io/docs/
 Main PID: 1009 (kubelet)
    Tasks: 22
   Memory: 124.5M
   CGroup: /system.slice/kubelet.service
           └─1009 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1

Sep 19 18:11:05 hrsh kubelet[1009]: E0919 18:11:05.452153    1009 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:06 hrsh kubelet[1009]: E0919 18:11:06.096629    1009 pod_workers.go:191] Error syncing pod 1d7bbe8c-3736-43e1-93f6-bf3c043195ab ("coredns-5644d7b6d9-wq2c7_kube-system(1d7bbe8c-3736-43e1-93f6-bf3c043195ab)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:06 hrsh kubelet[1009]: E0919 18:11:06.096870    1009 pod_workers.go:191] Error syncing pod f2ca516a-166f-48da-b54b-6cc8d6750469 ("coredns-5644d7b6d9-hq7qp_kube-system(f2ca516a-166f-48da-b54b-6cc8d6750469)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:08 hrsh kubelet[1009]: E0919 18:11:08.096630    1009 pod_workers.go:191] Error syncing pod 1d7bbe8c-3736-43e1-93f6-bf3c043195ab ("coredns-5644d7b6d9-wq2c7_kube-system(1d7bbe8c-3736-43e1-93f6-bf3c043195ab)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:08 hrsh kubelet[1009]: E0919 18:11:08.096904    1009 pod_workers.go:191] Error syncing pod f2ca516a-166f-48da-b54b-6cc8d6750469 ("coredns-5644d7b6d9-hq7qp_kube-system(f2ca516a-166f-48da-b54b-6cc8d6750469)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:08 hrsh kubelet[1009]: W0919 18:11:08.850500    1009 cni.go:202] Error validating CNI config &{cbr0  false [0xc001701fe0 0xc00171c060] [123 10 32 32 34 110 97 109 101 34 58 32 34 99 98 114 48 34 44 10 32 32 34 112 108 117 103 105 110 115 34 58 32 91 10 32 32 32 32 123 10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 102 108 97 110 110 101 108 34 44 10 32 32 32 32 32 32 34 100 101 108 101 103 97 116 101 34 58 32 123 10 32 32 32 32 32 32 32 32 34 104 97 105 114 112 105 110 77 111 100 101 34 58 32 116 114 117 101 44 10 32 32 32 32 32 32 32 32 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 32 116 114 117 101 10 32 32 32 32 32 32 125 10 32 32 32 32 125 44 10 32 32 32 32 123 10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 112 111 114 116 109 97 112 34 44 10 32 32 32 32 32 32 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 32 123 10 32 32 32 32 32 32 32 32 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 32 116 114 117 101 10 32 32 32 32 32 32 125 10 32 32 32 32 125 10 32 32 93 10 125 10]}: [plugin flannel does not support config version ""]
Sep 19 18:11:08 hrsh kubelet[1009]: W0919 18:11:08.850567    1009 cni.go:237] Unable to update cni config: no valid networks found in /etc/cni/net.d
Sep 19 18:11:10 hrsh kubelet[1009]: E0919 18:11:10.096604    1009 pod_workers.go:191] Error syncing pod f2ca516a-166f-48da-b54b-6cc8d6750469 ("coredns-5644d7b6d9-hq7qp_kube-system(f2ca516a-166f-48da-b54b-6cc8d6750469)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:10 hrsh kubelet[1009]: E0919 18:11:10.096765    1009 pod_workers.go:191] Error syncing pod 1d7bbe8c-3736-43e1-93f6-bf3c043195ab ("coredns-5644d7b6d9-wq2c7_kube-system(1d7bbe8c-3736-43e1-93f6-bf3c043195ab)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:10 hrsh kubelet[1009]: E0919 18:11:10.474639    1009 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

[root@hrsh ~]# docker image list
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-apiserver            v1.16.0             b305571ca60a        22 hours ago        217MB
k8s.gcr.io/kube-controller-manager   v1.16.0             06a629a7e51c        22 hours ago        163MB
k8s.gcr.io/kube-proxy                v1.16.0             c21b0c7400f9        22 hours ago        86.1MB
k8s.gcr.io/kube-scheduler            v1.16.0             301ddc62b80b        22 hours ago        87.3MB
k8s.gcr.io/etcd                      3.3.15-0            b2756210eeab        2 weeks ago         247MB
k8s.gcr.io/coredns                   1.6.2               bf261d157914        5 weeks ago         44.1MB
quay.io/coreos/flannel               v0.11.0-amd64       ff281650a721        7 months ago        52.6MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        21 months ago       742kB

drkhosla on 19 Sep 2019

same here ubuntu 1.18, docker version 18.06.3

tobiasvdp on 19 Sep 2019

Same behaviour here with Debian Buster (10.1)
Commenting for updates

smsimeonov on 19 Sep 2019

Same here as tobiasvdp. Latest version of Flannel and nodes are telling me Not Ready the container network is not working. This occurred after updating to version 1.16 of kubernetes. Luckily it's dev so we can just hold off until a fix is applied.

sirdarkat on 19 Sep 2019

Same here with Ubuntu 18.04. Also node selector namespace is beta.kubernetes.io which was deprecated and now kubernetes.io. I replaced it in yaml but still no network.

schizofreindly on 19 Sep 2019

I was able to get nodes to come up as ready by adding:

      "cniVersion":"0.3.1",

Under this line: https://github.com/coreos/flannel/blob/d893bcbfe6b04791054aea6c7569dea4080cc289/Documentation/kube-flannel.yml#L108

When I do a kubectl create -f the-above-flannel.yml

dougbtv on 19 Sep 2019

👍32 🚀5 ❤4 😄3 👎3 👀2 😕2 🎉2

I was able to get nodes to come up as ready by adding:
      "cniVersion":"0.3.1",
Under this line:

https://github.com/coreos/flannel/blob/d893bcbfe6b04791054aea6c7569dea4080cc289/Documentation/kube-flannel.yml#L108

When I do a kubectl create -f the-above-flannel.yml

Works for me. How did you know ?

schizofreindly on 20 Sep 2019

😕2 👍2 🚀1 🎉1

@dougbtv, works for me to.

jar3b on 20 Sep 2019

@dougbtv Awesome find, this worked for me as well.

If you can give a response on why this works I would love to know why. Thanks

sirdarkat on 20 Sep 2019

It seems like in v1.16, kubelet will validate cni config file and if the cniVersion is not supported, an error will be returned.
https://github.com/kubernetes/kubernetes/pull/80482
Currently cni-conf.json data in the configmap of kube-flannel.yaml lacks the cniVersion field, so an error like plugin flannel does not support config version "" is returned.

Seems we need to add supported cniVersion to kube-flannel.yaml.

sshukun on 20 Sep 2019

👍3

1181 helps with this

sshukun on 20 Sep 2019

1181 helps with this

this does work on my bare-metal clusters with 1.16.0.
have to insert "cniVersion": "0.2.0" to /etc/cni/net.d/10-flannel.conflist on every node !!

rayyen on 20 Sep 2019

👍3 🎉1

Thanks @dougbtv, works for me to.

drkhosla on 20 Sep 2019

Got the problem that can not find cp command in the PATH
Host OS is Ubuntu 18.04

ghost on 21 Sep 2019

Same here on a raspberry pi 4 - arm 64

nbroeking on 23 Sep 2019

This created some iptables rules that prevented ssh from working.

AHassanSOS on 26 Sep 2019

@schizofreindly - pure gold. thank you. wasted a day trying all sorts of things before I stumbled across this post.

sonishi85 on 26 Sep 2019

👍1

Thanks @schizofreindly That works for me.

oomichi on 26 Sep 2019

Fixed by #1174

arasananbu on 27 Sep 2019

Closing as per above comment.

rajatchopra on 27 Sep 2019

work for me,

CentOS Linux release 7.7.1908 (Core)

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

isnuryusuf on 2 Oct 2019

This is really helpful. I think flannel should come out with a revision or put it in the docs under kubernetes 1.16 support in order for users to notice this issue.
“Lessons learned from upgrading kubernetes offline to 1.16” https://link.medium.com/j6FVuVBxv1

yonahd on 11 Nov 2019

1181 helps with this

this does work on my bare-metal clusters with 1.16.0.
have to insert "cniVersion": "0.2.0" to /etc/cni/net.d/10-flannel.conflist on every node !!

Added "cniVersion":"0.3.1" to /etc/cni/net.d/10-flannel.conflist and got it working

{
  "name": "cbr0",
  "cniVersion":"0.3.1",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}