Kubeadm: [BUG REPORT] cni config uninitialized after updating to v1.16.0

[lukaszgryglicki]

What keywords did you search in kubeadm issues before filing this one?

cni error, nodes not ready

Versions

kubeadm version (use kubeadm version): kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:34:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: bare metal 4-node packet.net servers.
• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

• Kernel (e.g. uname -a): Linux node-0.k8s.devstats.cncf.io01 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Others:

What happened?

After updating from v14.2 to v15.4 all was fine (I've updated kubeadm to v1.15.4 in this step), then after update from v15.4 to v1.16.0 nodes became NotReady, after describe I've seen: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized, then after running journalctl -xeu kubelet I saw: plugin flannel does not support config version "", then I've found this: https://stackoverflow.com/questions/58037620/how-to-fix-flannel-cni-plugin-error-plugin-flannel-does-not-support-config-ve.

Finally, the solution was adding "cniVersion": "0.2.0", to /etc/cni/net.d/10-flannel.conflist file.

What you expected to happen?

Smooth update without downtime.

How to reproduce it (as minimally and precisely as possible)?

Try to update K8s cluster from v15.4 to v1.16 using flannel CNI network plugin.

Anything else we need to know?

I've found the solution as described above, just wanted to report this so. you can fix kubeadm.

[renilrandy]

I had faced the same issue while doing kubeadm setup with Flannel CNI...later installed caclio ...will again try to setup with flannel CNI.
Thanks for sharing the information @lukaszgryglicki

[lukaszgryglicki]

I've found elsewhere that "cniVersion": "0.3.1" also works and lists newer version. Updated my config.

[manukoli1986]

I have also figured out by adding "cniVersion": "0.2.0" version in flannel.yaml file or 10-flannel.config file and it worked as expected

{
"cniVersion": "0.2.0",
"name": "cbr0",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"cniVersion": "0.2.0",
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}

[varunvvv]

@lukaszgryglicki "cniVersion": "0.2.0" & "cniVersion": "0.3.1" both versions it works right ?

[varunvvv]

I have also figured out by adding "cniVersion": "0.2.0" version and it worked as expected.

Could you paste complete 10-flannel.conflist config file

[lukaszgryglicki]

Yes, both work, pasted /etc/cni/net.d/10-flannel.conflist file:

{
  "name": "cbr0",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

[varunvvv]

@lukaszgryglicki Thanks for u r info ....Due to Flannel issue m using Calico now ...will re-setup with Flannel 👍

[wangzhipeng]

kubectl apply -f https://raw.githubusercontent.com/wangzhipeng/flannel/master/Documentation/kube-flannel.yml

[lukaszgryglicki]

Yeah, it contains: "cniVersion": "0.3.1".

[varunvvv]

Hi lukaszgryglicki
once i restart my docker and kubelet services my nodes become notready state in /etc/cni/net.d/10-flannel.conflist file i found "cniVersion": "0.3.1" entry was missing.
is there any permanent fix for this issue.

[neolit123]

not a kubeadm bug, please report to the CNI plugin maintainers.
/close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

not a kubeadm bug, please report to the CNI plugin maintainers.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[lukaszgryglicki]

Yeah, after restarting kubelet and/or docker file is overwritten and fix is gone :(