Amazon-eks-ami: AMI 1.13 is failing with No networks found in /etc/cni/net.d

[safiqueahemad]

What happened:
We tired AMI 1.12 so far and it was working well with no issue. Recent upgrade to 1.13 doesn't bring up the kubelet and other services like cni up well.
error message from kubelet:
Jul 29 22:57:22 ip-10-195-18-60 kubelet: W0729 22:57:22.250285 4916 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 29 22:57:22 ip-10-195-18-60 kubelet: E0729 22:57:22.250429 4916 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 29 22:57:27 ip-10-195-18-60 kubelet: W0729 22:57:27.251256 4916 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 29 22:57:27 ip-10-195-18-60 kubelet: E0729 22:57:27.251388 4916 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:
amazon linux

• AWS Region:
  us-east-1
• Instance Type(s):
  m5.xlarge
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion):
  eks.2
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version):
  1.12
• AMI Version:
  1.13
• Kernel (e.g. uname -a):
  Linux ip-10-195-18-60.ts-qa.cloud 4.14.133-113.105.amzn2.x86_64 #1 SMP Wed Jul 10 16:57:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Release information (run cat /etc/eks/release on a node):

BASE_AMI_ID="ami-052b172b0a9552df4"
BUILD_TIME="Tue Jul 23 04:08:48 UTC 2019"
BUILD_KERNEL="4.14.123-111.109.amzn2.x86_64"
ARCH="x86_64"

[mnejat]

I am experiencing the same issue. The command "kubectl get pods --all-namespaces"
shows STATUS as "ContainerCreating" or "Pending". Prior to 1.13 has been working fine.

[safiqueahemad]

I got this issue fixed. The problem was EKS cluster control and data plan was 1.12. I migrated control and data plan on 1.13 fixed the issue.

[SkullyKiddo]

I'm facing the same issue:

Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

AWS Region:
us-east-1
Instance Type(s):
t3.large
EKS Platform version (use aws eks describe-cluster --name eks.2
Kubernetes version (use aws eks describe-cluster --name 1.13
AMI Version:
1.13

$ cat /etc/eks/release
BASE_AMI_ID="ami-052b172b0a9552df4"
BUILD_TIME="Mon Jul  1 21:38:37 UTC 2019"
BUILD_KERNEL="4.14.123-111.109.amzn2.x86_64"
ARCH="x86_64"

Kernel (e.g. uname -a):
Linux ip-172-31-7-80.ec2.internal 4.14.128-112.105.amzn2.x86_64 #1 SMP Wed Jun 19 16:53:40 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[whereisaaron]

Hi guys, not sure if this is the issue, but did you also upgrade your CNI networking and coredns services along with your nodes and control plane upgrades?
https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html
https://docs.aws.amazon.com/eks/latest/userguide/coredns.html

[branhap]

I experienced this issue as well. However, our issue was that the AWS EKS upgrade had failed, and wasn't very clear that it had failed, so we had continued onto our other steps of the upgrade.

We retried the upgrade to 1.13 and it was successful the second time.

[cpanato]

same issue for me.
control plane aws reported successful. the worker nodes are in not ready state due this issue.

any advice on how to fix it?

[xdays]

you need upgrade aws CNI plugin, but it's bad that cni break the whole cluster.

https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html

[zhangyelong]

If the only solution is to upgrade the aws CNI plugin by kubectl, what if a node is in NotReady status? How can I use kubectl apply to upgrade CNI plugin on the NotReady node :(

[mogren]

@zhangyelong The CNI is needed to make the node ready. You can update the CNI independently of the other components. Just make sure that kube-proxy starts correctly, or the CNI won't be able to talk to the API server.