amazon-vpc-cni-k8s 🚀 - Pods stuck in ContainerCreating due to CNI Failing to Assing IP to Container Until aws-node ...

@druidsbane can you log onto this node and run /opt/cni/bin/aws-cni-support.sh and send me ([email protected]) /var/log/aws-routed-eni/aws-cni-support.tar.gz? thanks

liwenwu-amazon on 10 Apr 2018

@druidsbane from the log, it looks like you hit issue #18. here is a snips of logs ipamd.log.2018-04-10-17:

2018-04-10T17:59:17Z [INFO] Send AddNetworkReply: IPv4Addr , DeviceNumber: 0, err: datastore: no available IP addresses
2018-04-10T17:59:17Z [INFO] Received AddNetwork for NS /proc/5025/ns/net, Pod clamav-bb688497d-hrrln, NameSpace default, Container b30274f12c192d65d0e716b8b3178221cfdeaa8f116fc505c9916d02a883775d, ifname eth0
2018-04-10T17:59:17Z [DEBUG] AssignIPv4Address: IP address pool stats: total:18, assigned 18
2018-04-10T17:59:17Z [DEBUG] AssignPodIPv4Address, skip ENI eni-c10169f7 that do not have available addresses
2018-04-10T17:59:17Z [DEBUG] AssignPodIPv4Address, skip ENI eni-ad147c9b that do not have available addresses
2018-04-10T17:59:17Z [INFO] DataStore has no available IP addresses
2018-04-10T17:59:17Z [INFO] Send AddNetworkReply: IPv4Addr , DeviceNumber: 0, err: datastore: no available IP addresses
2018-04-10T17:59:17Z [INFO] Received AddNetwork for NS /proc/5140/ns/net, Pod lasso-engine-api-67484c8f8-thtlc, NameSpace default, Container 0e8ad12946302e7bbca9d97aa160ce3b0a97466d5c33052d34a73123dcc60d2b, ifname eth0
2018-04-10T17:59:17Z [DEBUG] AssignIPv4Address: IP address pool stats: total:18, assigned 18
2018-04-10T17:59:17Z [DEBUG] AssignPodIPv4Address, skip ENI eni-c10169f7 that do not have available addresses
2018-04-10T17:59:17Z [DEBUG] AssignPodIPv4Address, skip ENI eni-ad147c9b that do not have available addresses
2018-04-10T17:59:17Z [INFO] DataStore has no available IP addresses
2018-04-10T17:59:17Z [INFO] Send AddNetworkReply: IPv4Addr , DeviceNumber: 0, err: datastore: no available IP addresses
2018-04-10T17:59:17Z [INFO] Received AddNetwork for NS /proc/5104/ns/net, Pod frontend-merge-request-nginx-84959cf4bc-cpvw7, NameSpace default, Container e5b38c10611d8a5eedbf4156a374a21ed74dd467ba861e10ded526ed627c446d, ifname eth0
2018-04-10T17:59:17Z [DEBUG] AssignIPv4Address: IP address pool stats: total:18, assigned 18
2018-04-10T17:59:17Z [DEBUG] AssignPodIPv4Address, skip ENI eni-c10169f7 that do not have available addresses
2018-04-10T17:59:17Z [DEBUG] AssignPodIPv4Address, skip ENI eni-ad147c9b that do not have available addresses
2018-04-10T17:59:17Z [INFO] DataStore has no available IP addresses

we have a PR (#31) and proposal #26 to address this .

liwenwu-amazon on 10 Apr 2018

🎉3

Great, interesting to see the details on how to solve this. I'll follow those two and hopefully they get merged soon so we can test before they're GA!

druidsbane on 11 Apr 2018

Here is the root cause of this problem:
when following 2 events happens almost simultaneously:

Release an IP address for a deleted Pod ( kubelet ->(delete) -> CNI ->L-IPAM)
Allocate an unused IP address for a new Pod (kubelet ->(add) -> CNI -> L-IPAM)

L-IPAM can reassign the IP address which was just released immediately to a new Pod, instead of obeying the pod cooling period design. This can cause CNI fail to setup routing table for the newly added Pod. When this failure happens, kubelet will NOT invoke a delete Pod CNI call. Since CNI never release this IP back in the case of failure, this IP is leaked.

Here is the snips from plug.log.xxx

 // kubelet -> CNI to delete a Pod
2018-04-11T20:08:01Z [INFO] Received CNI del request: ContainerID(1eb7ad292495633a6a68876a741604164321bab048552302856f1a53ab9aeb74) Netns(/proc/17283/ns/net) IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=venn-frontend-v2-7f6c64df86-5vl2v;K8S_POD_INFRA_CONTAINER_ID=1eb7ad292495633a6a68876a741604164321bab048552302856f1a53ab9aeb74) Path(/opt/aws-cni/bin:/opt/cni/bin) argsStdinData({"cniVersion":"","name":"aws-cni","type":"aws-cni","vethPrefix":"eni"})
2018-04-11T20:08:01Z [INFO] Delete toContainer rule for 192.168.109.167/32

// kubelet -> cni to add a new Pod
2018-04-11T20:08:01Z [INFO] Received CNI add request: ContainerID(86e77ed0a6917d1401695b4f8281ad23384630d9e9a3a679fbd8703f4579336f) Netns(/proc/20315/ns/net) IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=site-monitor-7b58cf584b-l2cfm;K8S_POD_INFRA_CONTAINER_ID=86e77ed0a6917d1401695b4f8281ad23384630d9e9a3a679fbd8703f4579336f) Path(/opt/aws-cni/bin:/opt/cni/bin) argsStdinData({"cniVersion":"","name":"aws-cni","type":"aws-cni","vethPrefix":"eni"})
2018-04-11T20:08:01Z [INFO] Received add network response for pod site-monitor-7b58cf584b-l2cfm namespace default container 86e77ed0a6917d1401695b4f8281ad23384630d9e9a3a679fbd8703f4579336f: 192.168.109.167, table 0
2018-04-11T20:08:01Z [ERROR] Failed SetupPodNetwork for pod site-monitor-7b58cf584b-l2cfm namespace default container 86e77ed0a6917d1401695b4f8281ad23384630d9e9a3a679fbd8703f4579336f: setup NS network: failed to add host route: file exists

liwenwu-amazon on 16 Apr 2018

👍1

I'm having a similar problem with a pod not coming up with the same reason FailedCreatePodSandBox.

E0418 19:09:56.012799    3852 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "fluentd-pxks9_kube-system" network: add cmd: failed to assign an IP address to container
Apr 18 19:09:56 ip-172-20-19-97.us-west-2.compute.internal kubelet[3852]: E0418 19:09:56.012841    3852 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "fluentd-pxks9_kube-system(873acabd-4339-11e8-a92c-0609dc29c0b0)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "fluentd-pxks9_kube-system" network: add cmd:
Apr 18 19:09:56 ip-172-20-19-97.us-west-2.compute.internal kubelet[3852]: E0418 19:09:56.012855    3852 kuberuntime_manager.go:647] createPodSandbox for pod "fluentd-pxks9_kube-system(873acabd-4339-11e8-a92c-0609dc29c0b0)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "fluentd-pxks9_kube-system" network: add cmd
Apr 18 19:09:56 ip-172-20-19-97.us-west-2.compute.internal kubelet[3852]: E0418 19:09:56.012904    3852 pod_workers.go:186] Error syncing pod 873acabd-4339-11e8-a92c-0609dc29c0b0 ("fluentd-pxks9_kube-system(873acabd-4339-11e8-a92c-0609dc29c0b0)"), skipping: failed to "CreatePodSandbox" for "fluentd-pxks9_kube-system(873acabd-4339-11e8-a92c-0609d
Apr 18 19:09:56 ip-172-20-19-97.us-west-2.compute.internal kubelet[3852]: W0418 19:09:56.398522    3852 pod_container_deletor.go:77] Container "4ababa2ccded34032f99ee3e06cd0452a37ecc91539f8e33d8180f1e52e7de41" not found in pod's containers
Apr 18 19:09:57 ip-172-20-19-97.us-west-2.compute.internal kubelet[3852]: E0418 19:09:57.029012    3852 cni.go:259] Error adding network: add cmd: failed to assign an IP address to container
Apr 18 19:09:57 ip-172-20-19-97.us-west-2.compute.internal kubelet[3852]: E0418 19:09:57.029040    3852 cni.go:227] Error while adding to cni network: add cmd: failed to assign an IP address to container
Apr 18 19:09:57 ip-172-20-19-97.us-west-2.compute.internal kubelet[3852]: E0418 19:09:57.130779    3852 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "fluentd-pxks9_kube-system" network: add cmd: failed to assign an IP address to container
Apr 18 19:09:57 ip-172-20-19-97.us-west-2.compute.internal kubelet[3852]: E0418 19:09:57.130822    3852 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "fluentd-pxks9_kube-system(873acabd-4339-11e8-a92c-0609dc29c0b0)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "fluentd-pxks9_kube-system" network: add cmd:
Apr 18 19:09:57 ip-172-20-19-97.us-west-2.compute.internal kubelet[3852]: E0418 19:09:57.130836    3852 kuberuntime_manager.go:647] createPodSandbox for pod "fluentd-pxks9_kube-system(873acabd-4339-11e8-a92c-0609dc29c0b0)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "fluentd-pxks9_kube-system" network: add cmd
Apr 18 19:09:57 ip-172-20-19-97.us-west-2.compute.internal kubelet[3852]: E0418 19:09:57.130886    3852 pod_workers.go:186] Error syncing pod 873acabd-4339-11e8-a92c-0609dc29c0b0 ("fluentd-pxks9_kube-system(873acabd-4339-11e8-a92c-0609dc29c0b0)"), skipping: failed to "CreatePodSandbox" for "fluentd-pxks9_kube-system(873acabd-4339-11e8-a92c-0609d
Apr 18 19:09:57 ip-172-20-19-97.us-west-2.compute.internal kubelet[3852]: W0418 19:09:57.457326    3852 pod_container_deletor.go:77] Container "70d490486d4b9d1d8e8550b26f03801b465623b28ea302371da845972801ab2a" not found in pod's containers

@liwenwu-amazon I can provide the same logs if you wish. Thank you!

incognick on 18 Apr 2018

@incognick can you log onto this node and run /opt/cni/bin/aws-cni-support.sh and send me ([email protected]) /var/log/aws-routed-eni/aws-cni-support.tar.gz? thanks

liwenwu-amazon on 18 Apr 2018

I also got stuck in ContainerCreating:
Warning FailedCreatePodSandBox 16m (x58161 over 17h) kubelet, ip-172-x.x.x.us-west-2.compute.internal Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "wordpress-xxxx-vmtkg_default" network: add cmd: failed to assign an IP address to container

And the fix was to reload "aws-node" pod on the Node which stopped issuing IP's

edwize on 25 Apr 2018

👍1

PR #62 should fix this issue. Please re-open if the issues happens again

liwenwu-amazon on 12 May 2018

Having similar failures on AWS EKS.

stafot on 20 Jun 2018

👍116

I also have this on EKS. Very simply situation (1 cluster node, 1 simple deployment). Just scale deployment from 1 replica to 20 and I get pods that are stuck as ContainerCreating:

your-name-app1-7d8f55b547-2k7kh                             0/1       ContainerCreating   0          3m
your-name-app1-7d8f55b547-4m849                             1/1       Running             0          3m
your-name-app1-7d8f55b547-7wr5g                             0/1       ContainerCreating   0          3m
your-name-app1-7d8f55b547-7xg6j                             0/1       ContainerCreating   0          3m
your-name-app1-7d8f55b547-b97k7                             0/1       ContainerCreating   0          3m
your-name-app1-7d8f55b547-bh7r9                             0/1       ContainerCreating   0          3m
your-name-app1-7d8f55b547-bs4mj                             1/1       Running             0          3m
your-name-app1-7d8f55b547-bvqcb                             1/1       Running             0          3m

Events from the pod:

Events:
  Type     Reason                  Age               From                                 Message
  ----     ------                  ----              ----                                 -------
  Normal   Scheduled               3m                default-scheduler                    Successfully assigned your-name-app1-7d8f55b547-7xg6j to ip-10-0-2-206.ec2.internal
  Normal   SuccessfulMountVolume   3m                kubelet, ip-10-0-2-206.ec2.internal  MountVolume.SetUp succeeded for volume "default-token-4z94m"
  Warning  FailedCreatePodSandBox  3m (x12 over 3m)  kubelet, ip-10-0-2-206.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "your-name-app1-7d8f55b547-7xg6j_default" network: add cmd: failed to assign an IP address to container
  Normal   SandboxChanged          3m (x12 over 3m)  kubelet, ip-10-0-2-206.ec2.internal  Pod sandbox changed, it will be killed and re-created.

max-rocket-internet on 20 Jun 2018

👍9

@liwenwu-amazon can we reopen this issue?

max-rocket-internet on 20 Jun 2018

👍6

@max-rocket-internet can you collect node level debugging info by running

/opt/cni/bin/aws-cni-support.sh

#collect
/var/log/aws-routed-eni/aws-cni-support.tar.gz

you can send it to me ([email protected]) or attach it to this issue.

In addition, i have few more questions

what's the instance type of this node?
do you see these 1st time you scale up? How long have you wait ? Right now, if your instance requires more ENIs, the ipamD will need to call EC2 API to allocate and attach ENI to your instance. This can take sometime depends how busy is your VPC.

thanks,

liwenwu-amazon on 20 Jun 2018

👍1

@liwenwu-amazon
At first thanks for your answer.

We are using right 4->5 t2.medium instances with eks-worker-v20 ami. (Is not the production environment)
The issue seems to triggered by pods raising by CronJob resources. (but when starts happenning affects of course all kinds of pods)
I don't feel comfortable to share all the logs you asked for because inside them there are many critical application info (volume mounts, pods, containers used etc.) I found though some errors on logs which are probably related.

2018-06-21T07:59:05Z [INFO] DataStore has no available IP addresses
2018-06-21T07:59:05Z [INFO] Send AddNetworkReply: IPv4Addr , DeviceNumber: 0, err: datastore: no available IP addresses
......
2018-06-21T07:59:22Z [DEBUG] IP pool stats: total = 15, used = 14, c.currentMaxAddrsPerENI = 6, c.maxAddrsPerENI = 6```


2018-06-21T07:59:22Z [DEBUG] Start increasing IP Pool size
2018-06-21T07:59:22Z [DEBUG] Skipping increase IPPOOL due to max ENI already attached to the instance : 3

Thanks in advance for any answer.

stafot on 21 Jun 2018

Hi @liwenwu-amazon

I think I've made some mistake somewhere in the Terraform code. Our TF code is largely based on this but I've built a second cluster based on this module and it does not have this problem.

Both clusters are using same subnets, AMI and t2.medium node instance type with a single node.

I see this in the /var/log/aws-routed-eni/ipamd.log.* log files:

2018-06-21T14:31:50Z [DEBUG] AssignIPv4Address: IP address pool stats: total:15, assigned 15
2018-06-21T14:31:50Z [DEBUG] AssignPodIPv4Address, skip ENI eni-79cb08e4 that does not have available addresses
2018-06-21T14:31:50Z [DEBUG] AssignPodIPv4Address, skip ENI eni-294083b4 that does not have available addresses
2018-06-21T14:31:50Z [DEBUG] AssignPodIPv4Address, skip ENI eni-bde82b20 that does not have available addresses
2018-06-21T14:31:50Z [INFO] DataStore has no available IP addresses

To reproduce it, I just create the most basic deployment with 1 replica. Then scale the deployment to 30 replicas and then it happens.

Even if it is a configuration error on my part, it might be nice to know what the problem is since it seems a few people hit this error and can't work out where things went wrong.

do you see these 1st time you scale up?

Yes.

How long have you wait ?

Maximum 60 seconds.

I've emailed you the logs.

max-rocket-internet on 21 Jun 2018

👍2

2018-06-21T14:31:48Z [DEBUG] Skipping increase IPPOOL due to max ENI already attached to the instance : 3

BTW I know t2.medium is limited to 3 network interfaces but I still can't see how one cluster doesn't have this error but the other does when they are both using a single t2.medium node.

max-rocket-internet on 21 Jun 2018

@max-rocket-internet the way you described seems like something is wrong on this example userdata. Our cluster TF code has it also as "base" with adjustments based on our custom needs.

stafot on 21 Jun 2018

@stafot I thought this too so I checked the userdata from both clusters and they are the same except for sed -i s,MAX_PODS,100,g. Once cluster is set to 17 and the other is set to 100. So perhaps not userdata issue.

max-rocket-internet on 21 Jun 2018

@max-rocket-internet I have 100 too on mine while the example has 20. So if only this is the difference probably something else is the root cause.
Thanks for sharing your thoughts again.

stafot on 21 Jun 2018

@max-rocket-internet is it possible that you are running into issue #18 ?
few more questions:

what's the version of cni? you can find out by
- in "ipamd.log.xxx" search for "Starting L-IPAMD x.x.x ..."
- or just kubectl get ds aws-node -n kube-system -o yaml

also can you find out how many pods have been scheduled to run on this node

kubectl get pod --all-namespaces -o wide | grep <node-ip> | wc -l

liwenwu-amazon on 21 Jun 2018

@liwenwu-amazon cni version : 602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni:1.0.0
falures happen in all nodes 3 nodes run 9 pods and 2 nodes 11

stafot on 21 Jun 2018

@stafot few more questions:

can you check ipamD datastore by running, and do u see any discrepancy there?

curl http://localhost:61678/v1/pods | python -m json.tool

if you restart ipamdD, do u see problem goes away?
also can you share or send me([email protected]) the ipamD metrics ?

curl http://localhost:61678/metrics

liwenwu-amazon on 21 Jun 2018

🎉1

Thanks for the reply @liwenwu-amazon

is it possible that you are running into issue #18 ?

Yes, it looks very similar or identical but I don't know the cause so can't tell for sure.

what's the version of cni?

We are using eks-worker-v20 AMI (ami-dea4d5a1). The container image is 602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni:1.0.0

also can you find out how many pods have been scheduled to run on this node

33 but only 17 Running, the rest are ContainerCreating

max-rocket-internet on 22 Jun 2018

@liwenwu-amazon we should be able to run more than 17 pods on a t2.medium instance, right? I noticed that the other TF module has a hard coded list of pod limits per instance type:
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/local.tf#L9

max-rocket-internet on 22 Jun 2018

❤4

@liwenwu-amazon @max-rocket-internet seems that I have the same problem because with15 pods runs fine but after this start misbehaving and stuck.

stafot on 22 Jun 2018

As described in the design proposal, amazon-vpc-cni-k8s uses Secondary IP for Pods. And there is a limit based on the instance type.
In your example, a t2.medium instance can have a maximum of 3 ENIs and each ENI can have up to 6 IP addresses. amazon-vpc-cni-k8s CNI reserves 1 IP address for each ENI. So, it can gives out maximum of 15 IP addresses. Since each node always have aws-node, kube-proxy, so, MAX_POD is set 17 for t2.medium

liwenwu-amazon on 22 Jun 2018

👍5 ❤1

Thank you both @liwenwu-amazon @max-rocket-internet for helping clarify which is exactly the issue.

stafot on 23 Jun 2018

OK understood. Thanks @liwenwu-amazon

max-rocket-internet on 25 Jun 2018

I'm still seeing this issue, even after setting a max-pod limit via (sed -i s,MAX_PODS,${max_pod_count},g /etc/systemd/system/kubelet.service in the instance _userdata_).

It happens randomly but at least once per day in a 3 node test cluster.

The resolution: once I deleted aws-node on the host from the kube-system namespace all the stuck containers came up.

Yes, this workaround worked for me as well.

To reproduce:

We're using t2.medium instance types with max-pods set to 17. With the cluster autoscaler installed.
Start a bunch of nodes: kubectl run autoscaler-test --image=nginx --replicas=56
Wait a few minutes.
Check pods: kubectl get pods -o wide. Notice some are stuck in state ContainerCreating. If not, clean up with kubectl delete deployment autoscaler-test, wait 8 hours and try again.
Check one of these pods: kubectl describe pod autoscaler-test-b5d56b77-ptzxs
> Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "xxx" network: add cmd: failed to assign an IP address to container

tomfotherby on 5 Jul 2018

@tomfotherby , is it possible for you to install cni_metrics_helper.yaml and gather following information before you start kubectl run autoscaler-test --image=nginx --replicas=56 can you gather cluster level IP addresses stats. They are in cni_metrics_helper logs:

kubectl logs cni-metrics-helper-xxx -n kube-system

...
I0706 00:03:30.536559       7 metrics.go:340] Produce COUNTER metrics: ipamdErr, value: 0.000000
I0706 00:03:30.536566       7 metrics.go:350] Produce GAUGE metrics: ipamdActionInProgress, value: 0.000000
I0706 00:03:30.536571       7 metrics.go:350] Produce GAUGE metrics: assignIPAddresses, value: 23.000000
I0706 00:03:30.536576       7 metrics.go:350] Produce GAUGE metrics: eniAllocated, value: 8.000000
I0706 00:03:30.536580       7 metrics.go:350] Produce GAUGE metrics: eniMaxAvailable, value: 15.000000
I0706 00:03:30.536584       7 metrics.go:350] Produce GAUGE metrics: maxIPAddresses, value: 735.000000
I0706 00:03:30.536589       7 metrics.go:350] Produce GAUGE metrics: totalIPAddresses, value: 387.000000

liwenwu-amazon on 6 Jul 2018

@tomfotherby , to debug why container stucked in ContainerCreating, can u gather node level debugging info, for example:

kubectl get pod -o wide 
NAME                            READY     STATUS    RESTARTS   AGE       IP                NODE
worker-hello-5974f49799-5dpcg   1/1       ContainerCreating   0          4d        192.168.145.162   ip-192-168-190-66.us-west-2.compute.internal <--- node IP
worker-hello-5974f49799-85t2l   1/1       Running   0          4d        192.168.153.80    ip-192-168-190-66.us-west-2.compute.internal
worker-hello-5974f49799-9m2bh   1/1       Running   0          4d        192.168.94.54     ip-192-168-97-112.us-west-2.compute.internal

then ssh into the problem node (e.g. ip-192-168-190-66.us-west-2.compute.internal)
you can run /opt/cni/bin/aws-cni-support.sh which collect cni support bundle at /var/log/aws-routed-eni/aws-cni-support.tar.gz

liwenwu-amazon on 6 Jul 2018

@liwenwu-amazon - Thanks for your help. When installing the cni_metrics_helper.yaml I tripped up on the problem:

$ kubectl -f https://raw.githubusercontent.com/liwenwu-amazon/amazon-vpc-cni-k8s-1/metrics1/misc/cni_metrics_helper.yaml

$ kubectl get pod -o wide -n=kube-system cni-metrics-helper-56485f8b9d-dptz4
NAME                                  READY     STATUS              RESTARTS   AGE       IP        NODE
cni-metrics-helper-56485f8b9d-dptz4   0/1       ContainerCreating   0          2m        <none>    ip-10-56-0-151.ec2.internal

$ kubectl describe pod -n=kube-system cni-metrics-helper-56485f8b9d-dptz4
  ...
  Warning  FailedCreatePodSandBox  3m (x12 over 3m)  kubelet, ip-10-56-0-151.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "cni-metrics-helper-56485f8b9d-dptz4_kube-system" network: add cmd: failed to assign an IP address to container
  Normal   SandboxChanged          3m (x12 over 3m)  kubelet, ip-10-56-0-151.ec2.internal  Pod sandbox changed, it will be killed and re-created.

The node is running 11 pods (out of 17 allowed):

$ kubectl describe node ip-10-56-0-151.ec2.internal
...
Allocatable:
 cpu:                2
 ephemeral-storage:  19316009748
 hugepages-2Mi:      0
 memory:             3939308Ki
 pods:               17
...
Non-terminated Pods:         (11 in total)
  Namespace                  Name                                           CPU Requests  CPU Limits  Memory Requests  Memory Limits
 ...

I can see from the web-console there are all IPs used up:

Private IPs: 10.56.0.151, 10.56.0.99, 10.56.0.134
Secondary private IPs: 10.56.0.216, 10.56.0.249, 10.56.0.139, 10.56.0.18, 10.56.0.115, 10.56.0.186, 10.56.0.107, 10.56.0.161, 10.56.0.33, 10.56.0.197, 10.56.0.207, 10.56.0.226, 10.56.0.195, 10.56.0.100, 10.56.0.103

So now I can send you debugging info.

$ ssh -i eks.pem [email protected]
$ sudo /opt/cni/bin/aws-cni-support.sh
$ ls -lh /var/log/aws-routed-eni/aws-cni-support.tar.gz
-rw-r--r-- 1 root root 11M Jul  6 14:42 /var/log/aws-routed-eni/aws-cni-support.tar.gz

I'll email that file to [email protected] now... done.

tomfotherby on 6 Jul 2018

From looking at the logs from /opt/cni/bin/aws-cni-support.sh, I can see the problem might be related to our CronJob resource. We run a CronJob every minute( i.e. schedule: "* * * * *") and usually the pod that is created each minute finishes after 30s and is in the Completed state until the next CronJob runs. I think the AWS CNI plugin is giving each CronJob pod a IP address but not reclaiming it back after the Pod is removed by the next run.

For reference, this is a extract from our CronJob resource:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: pph-notifications-crons
  labels:
    tier: console
    version: blah
spec:
 schedule: "* * * * *"
 concurrencyPolicy: Forbid
 successfulJobsHistoryLimit: 1
 failedJobsHistoryLimit: 1
 startingDeadlineSeconds: 100
 jobTemplate:
   spec:
     template:
       spec:
...

tomfotherby on 6 Jul 2018

@tomfotherby today, when an IP address is returned to ipamD, ipamD keeps a 30 seconds cooling period (addressCoolingPeriod data_store.go ). In another word, when a cronjob is terminated, its IP address won't get re-assigned to another job until 30 seconds later.

liwenwu-amazon on 6 Jul 2018

@tomfotherby thank you for collecting /var/log/aws-routed-eni/aws-cni-support.tar.gz. From it, I found one CNI bug that whenever CNI fails to setup network stack for a Pod, it does NOT release pod's IP back to ipamD datastore. Here is error msgs in plug-xxx.log

grep "Failed to setup NS network failed to Statfs" plugin*
plugin.log.2018-07-05-17:2018-07-05T17:47:10Z [ERROR] Failed to setup NS network failed to Statfs "/proc/10617/ns/net": no such file or directory
plugin.log.2018-07-05-18:2018-07-05T18:24:08Z [ERROR] Failed to setup NS network failed to Statfs "/proc/12979/ns/net": no such file or directory

liwenwu-amazon on 8 Jul 2018

👍5

Good job @liwenwu-amazon . I'm glad to have helped.

tomfotherby on 8 Jul 2018

👍1

Is there a workaround to release the addresses from IPAMD we can use until AWS release a new node AMI with this fix included?

max-rocket-internet on 20 Jul 2018

👍3

Restart daemonset's pod aws-node on specific node which is stuck and not providing IPs. If any is available will get released.

stafot on 20 Jul 2018

👍2 👎1

What did I wrong when executing /opt/cni/bin/aws-cni-support.sh returns curl: (7) Failed to connect to localhost port 61678: Connection refused?

kube-dns and kubernetes-dashboard pods are stating: Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-64b69465b4-t759p_kube-system" network: rpc error: code = Unavailable desc = grpc: the connection is unavailable.

EKS installed successfully and the t2.medium worker node is in state Ready.

andrejmaya on 29 Jul 2018

I am hitting this problem every week now as we have quite a volatile environment.

I see v1.1.0 is released just now. How long until we have a new worker AMI? Is this the repo to watch for a new release?
https://github.com/awslabs/amazon-eks-ami

Thanks.

max-rocket-internet on 30 Jul 2018

👍1

@max-rocket-internet a work around is a daily cron to handle it. Since we did it our cluster facing no issues with IPs. But I agree is unfortunate after this long the fix is applied no new eks worker has been created. But I believe is beyond this repository. We might create a ticket in the repo you mentioned referring to this conversation.

stafot on 30 Jul 2018

@max-rocket-interne: There is no new worker AMI. New EKS cluster automatically uses CNI 1.1. For existing cluster, it can be upgraded to CNI 1.1 by following https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html

liwenwu-amazon on 30 Jul 2018

🎉4

Ah yes, of course. I forgot that it's just a daemonset:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/master/config/v1.1/aws-k8s-cni.yaml

max-rocket-internet on 30 Jul 2018

Got it worked with https://github.com/terraform-aws-modules/terraform-aws-eks

this is the terraform template I used: https://gist.github.com/andrejmaya/8c79c21d6ffa1624638545a0a2db0404

andrejmaya on 30 Jul 2018

@liwenwu-amazon I am still seeing this issue:

Events:
  Type     Reason                  Age                From                                  Message
  ----     ------                  ----               ----                                  -------
  Normal   Scheduled               7m                 default-scheduler                     Successfully assigned db-access1 to ip-10-0-99-99.ec2.internal
  Normal   SuccessfulMountVolume   7m                 kubelet, ip-10-0-99-99.ec2.internal  MountVolume.SetUp succeeded for volume "default-token-xxxx"
  Warning  FailedCreatePodSandBox  7m                 kubelet, ip-10-0-99-99.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "db-access1_default" network: add command: failed to setup network: setup NS network: failed to add host route: file exists
  Warning  FailedCreatePodSandBox  7m (x11 over 7m)   kubelet, ip-10-0-99-99.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "db-access1_default" network: add cmd: failed to assign an IP address to container
  Normal   SandboxChanged          2m (x272 over 7m)  kubelet, ip-10-0-99-99.ec2.internal  Pod sandbox changed, it will be killed and re-created.

Using image 602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni:1.1.0

I had to kill the aws-node pod. I also collected aws-cni-support.tar.gz on this node. Do you want me to email it to you?

Can we reopen this issue? I think people in Slack are also still seeing this problem.

max-rocket-internet on 16 Aug 2018

Maybe we are tracking this problem in https://github.com/aws/amazon-vpc-cni-k8s/issues/123 ?

max-rocket-internet on 16 Aug 2018

Ran into this same problem, no pods were able to start when using t3.xlarge worker nodes. Switched back to t2 and everything worked as expected.

Bfoster-melrok on 24 Aug 2018

@Bfoster-melrok which version of this were you running?

Check with kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

This is believed to be fixed in 1.1

I personally haven't ran into this since patching

stickyd on 24 Aug 2018

@stickyd I did try updating to 1.1 first and it did not resolve the issue.

Bfoster-melrok on 24 Aug 2018

@Bfoster-melrok t3 is pretty new type. Might has a support issue? I am interested on this topic as t3 looks promising for our k8s experiments. On our cluster creation through terraform there is a map that correlates types with number of ips available. My main assumption that t3 types are missing from this map in your case, and this is the reason of failure, because as you said with t2 work.

stafot on 25 Aug 2018

https://github.com/awslabs/amazon-eks-ami/issues/27
@Bfoster-melrok seems not supported yet.

stafot on 29 Aug 2018

I seem to be hitting this on new r5.2xlarge instances as well. The kube-dns pod won't start and using describe I get:

Events:
  Type     Reason                  Age                   From                               Message
  ----     ------                  ----                  ----                               -------
  Normal   SandboxChanged          7m (x347027 over 4d)  kubelet, ip-10-0-3-5.ec2.internal  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  2m (x346552 over 4d)  kubelet, ip-10-0-3-5.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-64b69465b4-shsxm_kube-system" network: add cmd: failed to assign an IP address to container

Running:

kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

Gives:

amazon-k8s-cni:1.1.0

geerlingguy on 12 Sep 2018

It looks like r5.2xlarge is not yet supported?

2018-09-12T14:33:35Z [INFO] DataStore has no available IP addresses
2018-09-12T14:33:35Z [INFO] Send AddNetworkReply: IPv4Addr , DeviceNumber: 0, err: datastore: no available IP addresses
2018-09-12T14:33:35Z [ERROR] Failed to get eni IP limit due to unknown instance type r5.2xlarge
2018-09-12T14:33:35Z [INFO] Failed to retrieve ENI IP limit: vpc ip resource(eni ip limit): unknown instance type
2018-09-12T14:33:35Z [DEBUG] IP pool stats: total = 0, used = 0, c.currentMaxAddrsPerENI = 1, c.maxAddrsPerENI = 1
2018-09-12T14:33:35Z [DEBUG] Start increasing IP Pool size
2018-09-12T14:33:35Z [ERROR] Failed to get eni limit due to unknown instance type r5.2xlarge

(Found by monitoring logs as mentioned in: https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/troubleshooting.md)

Added https://github.com/awslabs/amazon-eks-ami/issues/47 for R5 support. Will stick with R4 instances for now :(

geerlingguy on 12 Sep 2018

I'm seeing a version of this, running 1.2.1, on r4.8xlarge nodes. One node seems to come up in this state and can't start new pods (we have a daemonset that deploys periodically as a canary).

ts4z on 28 Nov 2018

Im still facing this issue. We are using version 1.2.1, on t2.mediums . This issue is still open?

frederiksf on 3 Jan 2019

Same issue on t3.medium.

kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni:1.2.1

Can we reopen this issue?

michalpiasecki1 on 3 Jan 2019

👍5

IIRC in our case, when we ran into this, AWS was having some issues -- re:Invent was going on, and a lot of things were odd -- so it may happen when AWS just isn't giving out IP addresses for whatever reason.

ts4z on 4 Jan 2019

Same issue here on t3.medium in "eu-central-1". I have updated to amazon-k8s-cni:1.3.0 following these instructions but it didn't help.

Pod is still failing with this error:

 (..) NetworkPlugin cni failed to set up pod (...) "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused

PierreR on 4 Jan 2019

Just ran into this issue today. Tried to upgrade our nodes from t2.medium to t3.mediums. When I switched back to t2.mediums the issue went away.

rdavison on 5 Jan 2019

This just started happening for me. I haven't changed anything in cluster (such as node type) lately.

UPDATE: Disregard. Updated to amazon-k8s-cni:1.3.0 and that fixed my issue.

rkarroll on 7 Jan 2019

🎉1

FIX -
Had the same problem with
EKS K8s 1.11.5
t2.medium worker nodes
amazon-k8s-cni:1.2.1
found by kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

FIXED it by upgrading to latest amazon-k8s-cni - instructions found here -
found this - https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html

pkelleratwork on 7 Jan 2019

👍4

Look like the issue is not resolved, guys.

I'm just trying to make use of EKS. Getting the same problem with both 1.2.1 and 1.3 CNI versions. To reinstall the nodes, I scale the Auto Scaling Group down to 0 and back to 8. As a result - less than a half of nodes are getting broken, so that I need to shutdown those.

megastallman on 22 Jan 2019

👍3

We're having the same issue. We're running the latest version of the aws-cni 1.3.

damascenorakuten on 24 Jan 2019

👍4

Same here.

rahulwa on 25 Jan 2019

@damascenorakuten did you terminate the instances after upgrading to 1.3?

SoMuchToGrok on 25 Jan 2019

I hit this problem with t3.medium kubernetes 1.11 eks.1

https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html
Terminate all t3 nodes manually

Solved. Thanks!

sturfee-petrl on 6 Feb 2019

❤3 🎉1

I have a cluster at AWS deployed with kops (all latest versions) and also had to terminate the failing instance so kops gets me a new one.

pulpbill on 7 Feb 2019

I have tried to upgrade kops cluster from 1.10.12 to 1.11.6 and got the same errors.

NAMESPACE      NAME                                                        READY   STATUS              RESTARTS   AGE
istio-system   grafana-5c7c959bbb-wmkkl                                    0/1     ContainerCreating   0          8m
istio-system   istio-citadel-6f54c84866-xws2k                              0/1     ContainerCreating   0          8m
istio-system   istio-galley-75449cdf45-lgl8q                               0/1     ContainerCreating   0          8m
istio-system   istio-ingress-757d69b6-2c2x6                                0/1     ContainerCreating   0          8m
istio-system   istio-pilot-7db87986b4-8zfhn                                0/2     ContainerCreating   0          8m
istio-system   istio-pilot-7db87986b4-pchss                                0/2     ContainerCreating   0          8m
istio-system   istio-policy-6494d6c876-b44lt                               0/2     ContainerCreating   0          8m
istio-system   istio-sidecar-injector-766d8b6987-cjwmb                     0/1     ContainerCreating   0          8m
istio-system   istio-telemetry-5b679cb7fb-l6lnh                            0/2     ContainerCreating   0          8m
istio-system   istio-tracing-69df49f997-tffxf                              0/1     ContainerCreating   0          8m
istio-system   kiali-55bcf9f584-7rjpz                                      0/1     ContainerCreating   0          8m
istio-system   prometheus-584d94b46c-wh4n2                                 0/1     ContainerCreating   0          8m
istio-system   servicegraph-7d995777dd-6ndhj                               0/1     ContainerCreating   0          8m
kube-system    aws-node-8fq62                                              1/1     Running             0          1d
kube-system    aws-node-fsw64                                              1/1     Running             5          5m
kube-system    aws-node-k7kng                                              1/1     Running             0          1d
kube-system    aws-node-tjp2r                                              1/1     Running             0          1d

plugin.log

2019-02-08T16:00:00Z [INFO] Starting CNI Plugin v1.3.0  ...
2019-02-08T16:00:00Z [INFO] Received CNI del request: ContainerID(8590367c4f8930504d6e9f5deae91b475e11c46ed380e858634bc6bfcf8a051b) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=istio-system;K8S_POD_NAME=istio-galley-75449cdf45-lgl8q;K8S_POD_INFRA_CONTAINER_ID=8590367c4f8930504d6e9f5deae91b475e11c46ed380e858634bc6bfcf8a051b) Path(/opt/cni/bin/) argsStdinData({"cniVersion":"","name":"aws-cni","type":"aws-cni","vethPrefix":"eni"})
2019-02-08T16:00:00Z [ERROR] Error received from DelNetwork grpc call for pod istio-galley-75449cdf45-lgl8q namespace istio-system container 8590367c4f8930504d6e9f5deae91b475e11c46ed380e858634bc6bfcf8a051b: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
2019-02-08T16:00:01Z [INFO] Starting CNI Plugin v1.3.0  ...
2019-02-08T16:00:01Z [INFO] Received CNI del request: ContainerID(f7a0bd40e817ea32ee5be84db87fb57152bf4186ef6f7c23b4f8a999d6845231) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=logging;K8S_POD_NAME=fluentd-elasticsearch-bjsm7;K8S_POD_INFRA_CONTAINER_ID=f7a0bd40e817ea32ee5be84db87fb57152bf4186ef6f7c23b4f8a999d6845231) Path(/opt/cni/bin/) argsStdinData({"cniVersion":"","name":"aws-cni","type":"aws-cni","vethPrefix":"eni"})
2019-02-08T16:00:01Z [INFO] Starting CNI Plugin v1.3.0  ...
2019-02-08T16:00:01Z [INFO] Received CNI del request: ContainerID(e97a26377b073d4f510a73795e12da42e2ca33aa9edd773183488c9db7c8ce57) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=external-dns-58d5c9fb9-bqtcb;K8S_POD_INFRA_CONTAINER_ID=e97a26377b073d4f510a73795e12da42e2ca33aa9edd773183488c9db7c8ce57) Path(/opt/cni/bin/) argsStdinData({"cniVersion":"","name":"aws-cni","type":"aws-cni","vethPrefix":"eni"})
2019-02-08T16:00:01Z [INFO] Starting CNI Plugin v1.3.0  ...
2019-02-08T16:00:01Z [INFO] Received CNI del request: ContainerID(f9f82023e086fb1344424eb83b47a31403ed52be96676b559572ec44a8879174) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=coredns-6ffb8b9799-lxj8q;K8S_POD_INFRA_CONTAINER_ID=f9f82023e086fb1344424eb83b47a31403ed52be96676b559572ec44a8879174) Path(/opt/cni/bin/) argsStdinData({"cniVersion":"","name":"aws-cni","type":"aws-cni","vethPrefix":"eni"})
2019-02-08T16:00:01Z [ERROR] Error received from DelNetwork grpc call for pod coredns-6ffb8b9799-lxj8q namespace kube-system container f9f82023e086fb1344424eb83b47a31403ed52be96676b559572ec44a8879174: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
2019-02-08T16:00:02Z [INFO] Starting CNI Plugin v1.3.0  ...

nisan270390 on 8 Feb 2019

I'm experiencing this issue today on 2/12/2019 with a m5a.large instance type running on EKS v1.11

MorrisMatrix on 12 Feb 2019

Also seeing this with an R5.xlarge -- tiller pod gets wedged in container create. The cluster was a 1.10 EKS cluster that was updated to 1.11 using the "upgrade" process and was still running 1.1.0 of the cni. Used the upgrade process to move to 1.3 and now all is well. Be nice if the official upgrade process for EKS made sure the overall infrastructure was up-to-date as well.

sfitts on 12 Feb 2019

I'm seeing this same issue on a c5n.4xlarge instance. CoreDNS is stuck as ContainerCreating because cmd: failed to assign an IP address to container. I am using kubernetes version 1.11 on EKS with amazon-k8s-cni:v1.3.2. Everything is fine on t3.xlarge but we need c5n.4xlarge for the networking.

edfungus on 27 Feb 2019

👍3

@edfungus Can you log onto this node and run /opt/cni/bin/aws-cni-support.sh and send me ([email protected]) /var/log/aws-routed-eni/aws-cni-support.tar.gz? I'd like see the logs and also try and reproduce the issue.

mogren on 27 Feb 2019

I, like @MorrisMatrix, am getting this with a new node group of m5a.large instances. I have a cluster that at normal size is 18 nodes, so right now it is at 36 as I try to migrate the workloads onto the new node group. I am on the latest CNI version and AMI version for k8s 1.10.11.

But the failed to assign an IP address to container is only happening on 1 of the 18 nodes in the new node group.

duro on 28 Feb 2019

@mogren it might also be worth noting that I do have a couple of deamonsets running on the cluster and it is those pods that run into the problem. Could there be some kind of race condition with when the deamonsets are trying to boot up along side aws-node?

duro on 28 Feb 2019

I can confirm @sturfee-petrl's solution worked for me. Thanks !

Mokto on 28 Feb 2019

I'm running into the same issue on my t3.medium instances.. I've triple checked my cluster's CNI version is amazon-k8s-cni:v1.3.2

describing one of the pods, I get this

Events:
  Type     Reason                  Age               From                                      Message
  ----     ------                  ----              ----                                      -------
  Normal   Scheduled               6m                default-scheduler                         Successfully assigned default/redis-master-2mqbp to ip-192-168-172-104.ec2.internal
  Warning  FailedCreatePodSandBox  6m                kubelet, ip-192-168-172-104.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "bac4d2c0522a72c16b9303cea1ea2d07314f9cc9c8dc27abeaa35f0bfe2ad87d" network for pod "redis-master-2mqbp": NetworkPlugin cni failed to set up pod "redis-master-2mqbp_default" network: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused", failed to clean up sandbox container "bac4d2c0522a72c16b9303cea1ea2d07314f9cc9c8dc27abeaa35f0bfe2ad87d" network for pod "redis-master-2mqbp": NetworkPlugin cni failed to teardown pod "redis-master-2mqbp_default" network: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"]
  Normal   SandboxChanged          1m (x25 over 6m)  kubelet, ip-192-168-172-104.ec2.internal  Pod sandbox changed, it will be killed and re-created.

and if I run /opt/cni/bin/aws-cni-support.sh on a node, I get

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 61678: Connection refused

any ideas?

elvisnovoa on 3 Mar 2019

👍1

Withdrawing this. In our case it was due to IP pool exhaustion. I have provisioned worker nodes in a new autoscaling group and increased the /26 ranges to /24 and we are up and running.

rust84 on 4 Mar 2019

👍1

Same issue with a kops-managed k8s 1.11.8 cluster, and the v1.3.2 addon. Had to reboot the instance on which all the pods were stuck in ContainerCreating.

tsuna on 7 Mar 2019

Just wondering why is this not re-opened or has someone opened a new bug? This has happened to my cluster with m5.xlarge 2 worker nodes twice in one month. I updated CNI to 1.3.2, terminated nodes, but still had the same issue where nodes were struck with below error in ContainerCreating status

We were hitting the IP limits when this issue started, but then we brought back the cluster to use 1/3rd of the IP pool but still the below error is observed

Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "somepod-6f4b649bff-85j4z_dev" network: add cmd: failed to assign an IP address to container

rizwan-kh on 7 Mar 2019

Yeah I don't understand how people use this either... I hit the bug again within 12h of rebooting the aforementioned instance.

tsuna on 7 Mar 2019

@geerlingguy

...and c5n.4xlarge too

aws-routed-eni/ipamd.log.2019-03-08-17:2019-03-08T17:49:33Z [ERROR] Failed to get eni IP limit due to unknown instance type c5n.4xlarge
aws-routed-eni/ipamd.log.2019-03-08-17:2019-03-08T17:49:33Z [INFO] Failed to retrieve ENI IP limit: vpc ip resource(eni ip limit): unknown instance type
aws-routed-eni/ipamd.log.2019-03-08-17:2019-03-08T17:49:33Z [ERROR] Failed to get eni limit due to unknown instance type c5n.4xlarge

aaomoware on 8 Mar 2019

Seen on c5n.18xlarge

Warning FailedCreatePodSandBox 2m21s (x4 over 2m24s) kubelet, {IP_ADDR}.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "561e6fa1249f2f5b86dc28fa6752536f3d8cab284a5f0abb5133769403e26db5" network for pod "{POD_NAME}": NetworkPlugin cni failed to set up pod "{POD_NAME}" network: add cmd: failed to assign an IP address to container

joegoldbeck on 8 Mar 2019

Happens here on almost any m5 instance. Mostly on m5.4xl

igorshapiro on 13 Mar 2019

Same here with the latest 1.3.2 version on a c4.2xlarge node.
I have this in the logs:


$ kubectl -n kube-system logs -f aws-node-2crkd
=====Starting installing AWS-CNI =========
=====Starting amazon-k8s-agent ===========
ERROR: logging before flag.Parse: W0316 00:22:03.222787      13 client_config.go:533] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ERROR: logging before flag.Parse: W0316 13:49:19.817306      13 reflector.go:341] github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi/discovery.go:271: watch of *v1.Pod ended with: too old resource version: 9207 (86953)

krish7919 on 16 Mar 2019

Hi,

I think there are a mix of problems mentioned in this issue that now are tracked in newer open issues. For example, all c5n family issues are because the 1.3.2 does not include that instance type. The initial bugs in this issue was fixed by Liwen, but there are still related issues with IP and ENI allocation and deletions tracked in #69, #123, #330 and #294.

Thanks for reporting issues!

mogren on 16 Mar 2019

👍1

Also #344, where the issue isn't related to c5 family instances but the symptoms look similar.

tsuna on 17 Mar 2019

It looks like m5ad family is also has this issue

mrshpaks on 3 Apr 2019

please reopen - hitting this too on EKS1.11.9 with AMI amazon-eks-node-1.11-v20190329 (ami-0e82e73403dd69fa3) and cni 1.3.2 on instance type m4.2xlarge

$ kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni:v1.3.2

flegler-zz on 3 Apr 2019

👍1

Yes, the m5ad family has been added to the master branch and will be in the v1.4.0 release, planned to be out soon.

mogren on 3 Apr 2019

👍2

for those still facing issues with c5n instance family, and are using AWS EKS, you can upgrade to the latest by apply the current release of CNI

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/master/config/v1.3/aws-k8s-cni.yaml

see documentation here

https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html

for CNI version v1.3.3 to work on c5n, or any instance family, the eni-max-pods.txt file needs updating with the instance type and Mapping number. the mapping numbers for the c5n families is listed below:

c5n.large 27
c5n.xlarge 56
c5n.2xlarge 56
c5n.4xlarge 232
c5n.9xlarge 232

to work out your own instance family type mapping numbers see here

and the calculation is:

number of ENI * (number of IPv4 per ENI - 1)  + 2

example of c5n.2xlarge mapping number workout will be

4×(15−1) = 56

to automate the updating of the eni-max-pods.txt file echo can be passed to user_data

locals {
  wn_userdata = <<USERDATA
#!/bin/bash
echo "c5n.large 27
c5n.xlarge 56
c5n.2xlarge 56
c5n.4xlarge 232
c5n.9xlarge 232" >> /etc/eks/eni-max-pods.txt
/etc/eks/bootstrap.sh --apiserver-endpoint '${data.consul_keys.tf.var.eks_cluster_endpoint}' --b64-cluster-ca '${data.consul_keys.tf.var.eks_cluster_certificate}' '${var.eks_cluster_name}-${terraform.workspace}'
USERDATA
}

aaomoware on 4 Apr 2019

👍5

Im getting the same issue with m4.large. Why are more than 20 pods getting scheduled on my m4.large? Then it runs out of IP addresses and I see the same issue.

romilpunetha on 22 Apr 2019

👍2

Have the same issue using
v1.3.3
In which version is fixed?

admssa on 3 May 2019

I think v1.4.0 has the fix.

tsuna on 3 May 2019

It still happens sometimes (I'm using t2.medium).

NetworkPlugin cni failed to set up pod "container" network: add cmd: failed to assign an IP address to container

okgolove on 7 May 2019

Happens with t3a also, adding it to eni-max-pods.txt didn't help... t3 works fine.

denis111 on 9 May 2019

@denis111 The issue is the same, I just added t3a to the CNI in #459, that should be in the v1.5.0 release. Might be worth making a v1.4.2 with just the new instance types though...

mogren on 9 May 2019

We are having a similar issue (on a C5.2xlarge) but running /opt/cni/bin/aws-cni-support.sh does not create /var/log/aws-routed-eni/aws-cni-support.tar.gz.

I get this output on the command line:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 584 100 584 0 0 570k 0 --:--:-- --:--:-- --:--:-- 570k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2 100 2 0 0 2000 0 --:--:-- --:--:-- --:--:-- 2000% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 106 100 106 0 0 103k 0 --:--:-- --:--:-- --:--:-- 103k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 83 100 83 0 0 83000 0 --:--:-- --:--:-- --:--:-- 83000
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 28 100 28 0 0 28000 0 --:--:-- --:--:-- --:--:-- 28000
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3850k 100 3850k 0 0 51.5M 0 --:--:-- --:--:-- --:--:-- 50.8M
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 10255: Connection refused

The EC2 appears to have 4 ENI attached and no secondary IPs so it doesn't like like exactly the same issue.

K8 error:
Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "56e19b4b62a6e29a38b0f5521253d9b4b8a2a55be4b00c932d0d95a413a355b5" network for pod "sonata-batchproc-6b98bbd67d-jr5s7": NetworkPlugin cni failed to set up pod "sonata-batchproc-6b98bbd67d-jr5s7_sonata" network: add cmd: failed to assign an IP address to container

kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni:v1.3.2

ChrisMullard on 12 Jun 2019

Just had the same issue now with t3a.medium spot instances. @max-rocket-internet I am using your EKS module v5.0.0.

sc250024 on 24 Jun 2019

If anyone comes here because he has an issue with the new t3a instances do this:
check the current CNI plugin version:
kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2
If it's less than 1.5, upgrade:
kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/master/config/v1.5/aws-k8s-cni.yaml

This solved it for me

Jonathan-7 on 24 Jun 2019

👍10 🎉4

$ kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni:v1.4.1

@Jonathan-7 Yep, that solved it, thank you! Looks like the AMIs have an old version of the CNI plugin installed. There's a corresponding issue in the AWS EKS AMI repo: https://github.com/awslabs/amazon-eks-ami/issues/262

sc250024 on 24 Jun 2019

@Jonathan-7 Thanks. This solves the problem.

I noticed the default CNI version comes with EKS 1.13 is CNI1.4.1

Update: Problem still exists...

left4taco on 25 Jun 2019

I changed the CNI to amazon-k8s-cni:v1.5.0 using the @Jonathan-7 comment. However, I am getting the same error of network: add cmd: failed to assign an IP address to container with a m5a.2xlarge spot node. Is there also a possibility that there is an AWS limit that is needed to be increased. I have 47 Network Interfaces in use. Thanks

imriss on 28 Jun 2019

I'm still getting the same error as well. c5.xlarge and c5.2xlarge spot instances. I'm also wondering about an AWS limit since we never ran into this issue until pushing our cluster to semi-reasonable scale. Everything works fine at <30 nodes, but now with ~130, I have almost 2000 containers stuck. It is also very possible I'm over-provisioning the nodes, but in looking at the describe node of some of them, they _look_ like they have capacity...

wreed4 on 29 Jun 2019

@wreed4 EKS uses an eni per pod. So there is a limit in how many pods you can have on a node as you can only attach so many ENIs to them, depending on the instance type. Also, every pod uses up a VPC IP, so you can run out of IPs fast depending on the CIDR range. Also remember load balancers and other such resources use up IPs as well.
IMO this is one of the worst decisions made by AWS because it limits your cluster in ways it doesn't needs to, instead of having a pod network namespace like most other services out there...

Jonathan-7 on 29 Jun 2019

@Jonathan-7 I _think_ I'm well under that limit. I have a c5.2xlarge doing this right now with 26 pods on it. According to the network interface page, that instance type should have 4*15-1 ENIs. so 59. Why is this happening with only 26 pods???

Also when I describe the nodes, you can clearly see the pod capacity and it is set correctly. The number of pods on that node is way under that limit.

EDIT: I also should be _nowhere near_ my VPC IP limit. I'm only trying to run ~1500 containers and my subnets (2 AZs) should each have ~30,000 available IPs. (CIDR /17)

wreed4 on 29 Jun 2019

Okay, I think I see what's happening... I looked through the cni logs on the aws-node pod on one of the nodes with issues. It seems like it's not adding an ENI. So for c5.xlarges, they're supposed to have 15 addresses per ENI, and 4 available ENIs. However, looking through the logs, it says there are only 14 available IPs (15 - the host IP), and then that it failed to assign an IP. It looks like it never tries adding beyond the one ENI.

It would make sense at this point, if I was being limited on the region/account/vpc to a certain number of ENIs. So I went to look at that. There are a ton of ENIs, but only a small percentage are "in use". Many are "Available" and not attached to anything. Shouldn't the CNI grab and attach another ENI if there are so many available??

wreed4 on 29 Jun 2019

@wreed4 Hmm, the CNI should definitely get more ENIs. Do you have any custom configuration? Like MAX_ENI or WARM_IP_TARGETset?

mogren on 1 Jul 2019

No, it seems ultimately it was the ENI limit on the account. I am still confused as to how that displayed in the UI though, as it really looked like there were plenty of unused ENIs.

wreed4 on 1 Jul 2019

On r5.large I'm also seeing the same issue. Is there a way to force pods to fail instead of hanging when this error occurs? Or are there any other workarounds anyone has found?

ianCambrio on 1 Jul 2019

I've just encountered this problem on eks 1.13.7 with cni 1.5.0 on a 3 node test cluster. It may or may not be related to the fact I am terminating the nodes for testing whenever I make user-data changes.

3x m5.xlarge workers reside in 3 small subnets in 3 AZs, all /26, so that might pose unique ip assignment challenges.

Interestingly, deleting the aws-node pod that was failing to assign ips did not help.
In the console, I saw a few Available ENIs and 4 active ones.
Note: Only 1 of them had "aws-K8S" in description.

I deleted ALL of the Available ENIs. A moment later there were:

6 Active ENIs total
3 of which had aws-K8S in description
0 Available ones
And the issue was gone.

I'll be testing this further, so I would be able to provide logs, but this was an unpleasant surprise as I thought by 1.5.0 such issues have been wiped out.

Note: beside external SNAT, I'm not touching any variables of the CNI.

krzysztof-bronk on 4 Jul 2019

👍6

It seems to me that when nodes terminate, the ENIs they were using do not get cleaned up. We're using spot instances and yesterday there was a ton of churn with instances getting reclaimed and new ones coming up getting added to the cluster. At any given few minute period, we'd be gaining or losing between 10 and 100 nodes most of the day. Throughout the day, our unused ENIs went through the roof. I had to write a script to delete all "available" and run it constantly throughout the day. As long as we had ENI headroom, things went pretty well, nodes came up and pods got scheduled, but those ENIs were killer.

So basically my solution has been to write a small script to loop through and delete all available ENIs. It seems they'll never get used and will just hang around forever. The aws_node pods will create more.

#!/usr/bin/env python3


import click
import boto3
import botocore.exceptions as exp
import time

@click.command()
@click.option('--dry-run', is_flag=True)
def main(dry_run):
    client = boto3.client('ec2')
    interfaces = client.describe_network_interfaces(
            Filters= [
                {
                    'Name': 'status',
                    'Values': [
                        'available',
                        ]
                }
            ]
        )

    ids = [ i['NetworkInterfaceId'] for i in interfaces["NetworkInterfaces"] ]
    click.secho(f"Deleting {len(ids)} enis", fg="green")
    if not dry_run:
        click.confirm("Continue?", abort=True)

    with click.progressbar(ids) as bar:
        for eni in bar:
            try:
                client.delete_network_interface(
                        NetworkInterfaceId=eni,
                        DryRun=dry_run,
                        )
            except client.exceptions.ClientError as e:
                print("exception:", e)
                time.sleep(0.5)


if __name__ == "__main__":
    main()

Here's my script if anyone wants it. requires python3.6+, click, and awscli/boto3/botocore. The exception handling isn't perfect, obviously, but it was just to get around the api rate limiting. Other accounts may have other needs and you may need to sleep every iteration of the loop

wreed4 on 11 Jul 2019

👍3

I've just encountered this problem after upgrading the cluster to 1.13 in EKS. Deleting the stuck pod fixes the problem (as in, the "new" pod will be created successfully). I'm using m5.large, and the cluster is provisioned with https://github.com/terraform-aws-modules/terraform-aws-eks.

EDIT: I believe my issue is related to #318. I've patched the daemonset to use a newer version of CNI, will see if that helps.

MrSaints on 12 Jul 2019

👍4

I have the exact situation as @krzysztof-bronk. Using CNI 1.5.0 does not fix.

caiconkhicon on 31 Jul 2019

👍3

Currently encountering this problem with an EKS cluster version 1.12, instances are t3.large with 26 pods. I updated the CNI to 1.15, was previously 1.14.

It should be able to go up to 35 but it sent this error: NetworkPlugin cni failed to set up pod.

Will investigate as well, it seems like this issue has been there for some time and I'm not sure about the answer?

oceaneLonneux on 5 Aug 2019

@MrSaints

I've just encountered this problem after upgrading the cluster to 1.13 in EKS. Deleting the stuck pod fixes the problem (as in, the "new" pod will be created successfully). I'm using m5.large, and the cluster is provisioned with https://github.com/terraform-aws-modules/terraform-aws-eks.

EDIT: I believe my issue is related to #318. I've patched the daemonset to use a newer version of CNI, will see if that helps.

Using EKS 1.11 with cni 1.2.1, can confirm that deleting the pod and recreating is a workaround.

amine250 on 28 Aug 2019

👍1

Following up on my post earlier, I can easily replicate this issue even with aws-cni 1.5.1

I have a 1 node cluster, and every time I terminate the instance from the UI so that ASG can spin up a replacement, the ENI is left dangling as Available, with IPs attached, and never deleted as far as I can tell.
So sooner or later you will exhaust IPs and then pods will fail to be started.

krzysztof-bronk on 3 Sep 2019

👍1

Face the same problem:

kubelet, ip-10-1-30-135.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "xxx" network for pod "xxx": NetworkPlugin cni failed to set up pod "xxxxx" network: add cmd: failed to assign an IP address to container

amazon-k8s-cni:v1.5.4
EKS: v1.14.7-eks-e9b1d0

1 single t3.small instance. Expected pod number is 11. When the 11th pod created, above error occurred.

zjuxxd on 12 Oct 2019

@zjuxxd If you only have one node, were two CoreDNS pods running on that node? Also, what were your settings?

mogren on 14 Oct 2019

Events:
  Type     Reason                  Age                 From                                    Message
  ----     ------                  ----                ----                                    -------
  Normal   Scheduled               28s                 default-scheduler                       Successfully assigned x-x/x-x-x-xx-fdlnd to ip-10-13-xx-xx.ec2.internal
  Warning  FailedCreatePodSandBox  27s                 kubelet, ip-10-13-xx-xx.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "xxxxxxxxxx" network for pod "x-x-x-5b44869477-fdlnd": NetworkPlugin cni failed to set up pod "x-x-x-x-x-x" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  26s                 kubelet, ip-10-13-xx-xx.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "xxxxxxxxxxx" network for pod "x-x-x-x-fdlnd": NetworkPlugin cni failed to set up pod "x-x-x-x-fdlnd_uber-portal" network: add cmd: failed to assign an IP address to container

$ kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni:v1.5.4

+1

fmb-chin on 24 Oct 2019

Hello, @fmb-chin, in v1.5/aws-k8s-cni.yaml I still see amazon-k8s-cni:v1.5.3 image. Is the 1.5.4 already fixed? And if so, is it already released? Thank you

pdostal on 29 Oct 2019

@pdostal
No, still failing in 1.5.4.

My problem happened after I changed instance type to t3.medium.

I changed instance type back to t3.large, no error.

(I use branch release-1.5.4 to upgrade to 1.5.4)

fmb-chin on 30 Oct 2019

👍1

Hello @druidsbane, this issue is probably active again - a lot of people are reporting so.

pdostal on 1 Nov 2019

👍26 ❤1

@fmb-chin Same here, switching to t3.large did the job for us.

Edit: After checking AWS documentation, it seems that there is an IP limit per instance: 18 for t3.medium, 36 for t3.large

https://docs.aws.amazon.com/fr_fr/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

neverdane on 14 Nov 2019

I’ve been haunted by this issue for a longer period. I want to add that our root issue was a too limited CIDR in the attached Subnet the Pods were scheduled in. It simply ran out of available IPs.

schumacher-m on 14 Nov 2019

@MrSaints
I've just encountered this problem after upgrading the cluster to 1.13 in EKS. Deleting the stuck pod fixes the problem (as in, the "new" pod will be created successfully). I'm using m5.large, and the cluster is provisioned with https://github.com/terraform-aws-modules/terraform-aws-eks.
EDIT: I believe my issue is related to #318. I've patched the daemonset to use a newer version of CNI, will see if that helps.

Using EKS 1.11 with cni 1.2.1, can confirm that deleting the pod and recreating is a workaround.

same happened with me today

ChaturvediSulabh on 19 Nov 2019

I've solved for the moment by upgrading cni to v1.5.5 and modifying the deamonset adding the environment variable 'WARM_IP_TARGET' set to value relative to this table ( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html ) to force ip addresses / network interface reservation.

cni configuration doc: https://docs.aws.amazon.com/eks/latest/userguide/cni-env-vars.html

Edit: The configuration just has delayed the problem, still happening when scaling up/down both pods and nodes

GaruGaru on 20 Nov 2019

👍1

@GaruGaru Glad you found a work-around, even if it will use a lot of IPs. You could also set WARM_ENI_TARGET to be the maximum number of ENIs for the instance type, that would also allocate all available IPs.

mogren on 20 Nov 2019

Also seeing this issue.
EKS 1.13
amazon-k8s-cni:v1.5.5 (upgraded in an attempt to solve this issue)
instance type: m5.large (27 IPs per node available)

We have enough IPs in our subnet CIDR, and comparing this https://docs.aws.amazon.com/fr_fr/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI to the number of pods we're running we should have more than 100 IPs available for new pods accross our cluster

Tried:
restarting the pods stuck in "ContainerCreating"
restarting "aws-node" pods
upgrading CNI
setting WARM_ENI_TARGET=3
setting WARM_IP_TARGET=27
Draining all the worker nodes one by one

We encountered this issue after creating and deleting a large number of namespaces & pods and it hasn't gone away since, it seems like some IPs are just not getting released back

At the moment we are simply not able to create new pods in our cluster.

jurajpetrik on 22 Nov 2019

Why is this issue closed when a lot of people keep reporting problems? For the past few days I've been experimenting with EKS cluster creation. I'm using terraform, actually a terraform module similar to the popular community module.
What I've observed:

Creating clusters below version 1.14 have no problems with worker nodes being in a "Ready" state. I'm using the latest CNI version: amazon-k8s-cni:v1.5.5
BUT,
No matter what I try when creating 1.14 version clusters, worker nodes are in the "NotReady" state even though I've applied the aws-auth-cm.yaml configmap and the latest CNI version. Upon closer look (kubectl describe node <node_name>) I see an error that the CNI is uninitialized, also when I look at the running pods (kubectl get pods -n kube-system) I can see the core-dns pods being in a "pending" state and the aws-node pods crashing every few seconds.
I've then taken some steps to see if I could fix it:
a) downgraded the CNI version to 1.5.3 - this resulted in nodes getting to "Ready" state but this didn't fix the problem, the core-dns pods were now in "ContainerCreating" status constantly and aws-node pods had the same behaviour. Upgrading the CNI back to 1.5.5 didn't change anything.
b) Next what I tried was to create a 1.13 cluster first with nodes using a 1.14 kubernetes AMI. The nodes didn't have any problems joining the cluster and were ready. I then upgraded the cluster version and this resulted in a working 1.14 cluster with the nodes joined and being ready. - HOWEVER, if I increased the number of nodes in an auto scaling group, the new nodes had the same old problems of not being ready no matter what I tried.

To sum up, I've decided to use a 1.13 cluster in which I see no problems with nodes using a 1.14 AMI in hopes of fixing this problem in the near future.

Erokos on 8 Dec 2019

👍1

Same issue witth EKS 1.14 and amazon-k8s-cni:v1.5.3 with multiple t3.medium ec2.

descampsk on 11 Dec 2019

Seeing this with EKS 1.13 and AWS VPC CNI v1.5.3 when using c4 family instance types.

iliasbertsimas on 11 Dec 2019

For anyone running EKS, you need to update the CNI plugin to the latest version (currently 1.5.5). As per AWS docs: Amazon EKS does not automatically upgrade the CNI plugin on your cluster when new versions are released. So this issue will reappear periodically.

To find out your current version: kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

To update, follow the instructions at https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html

Credit goes to @pkelleratwork and their Jan 7 answer in this issue: https://github.com/aws/amazon-vpc-cni-k8s/issues/59#issuecomment-451989505

fubar on 18 Dec 2019

👍2

This issue should be re-opened.
kubernetes: v1.15.3
cni version: amazon-k8s-cni:v1.5.5
cni ENVs:

- name: AWS_VPC_K8S_CNI_LOGLEVEL
          value: DEBUG
        - name: MY_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
          value: "true"
        - name: ENI_CONFIG_LABEL_DEF
          value: failure-domain.beta.kubernetes.io/zone

We came across this while using t2.medium instances - coredns pods are stuck in ContainerCreating:

ip-10-128-30-89 ~ # kubectl get po --all-namespaces -o wide | grep ip-10-128-30-89.ec2.internal
kube-system   aws-encryption-provider-ip-10-128-30-89.ec2.internal    1/1     Running             0          40h   10.128.30.89    ip-10-128-30-89.ec2.internal    <none>           <none>
kube-system   aws-node-bl5p2                                          1/1     Running             1          40h   10.128.30.89    ip-10-128-30-89.ec2.internal    <none>           <none>
kube-system   coredns-5c98db65d4-4h9q2                                0/1     ContainerCreating   0          22h   <none>          ip-10-128-30-89.ec2.internal    <none>           <none>
kube-system   kube-apiserver-ip-10-128-30-89.ec2.internal             1/1     Running             0          40h   10.128.30.89    ip-10-128-30-89.ec2.internal    <none>           <none>
kube-system   kube-controller-manager-ip-10-128-30-89.ec2.internal    1/1     Running             0          40h   10.128.30.89    ip-10-128-30-89.ec2.internal    <none>           <none>
kube-system   kube-proxy-8zmt5                                        1/1     Running             0          40h   10.128.30.89    ip-10-128-30-89.ec2.internal    <none>           <none>
kube-system   kube-scheduler-ip-10-128-30-89.ec2.internal             1/1     Running             0          40h   10.128.30.89    ip-10-128-30-89.ec2.internal    <none>           <none>

notice that the ENI has a maximum of 6 IP addresses, on the 7th allocation (coredns), it does not automatically use private IPs associated to other attached ENIs.

The logs are littered with:

2019-12-20T20:22:39.824Z [WARN] UnassignPodIPv4Address: Failed to find pod coredns-5c98db65d4-4h9q2 namespace kube-system Container bd2790bb1074ac3948eb9a703db35777f29547f8d7c835647cec0d56fddaea05
2019-12-20T20:22:39.824Z [DEBUG]        UnassignPodIPv4Address: IP address pool stats: total:0, assigned 0, pod(Name: coredns-5c98db65d4-4h9q2, Namespace: kube-system, Container )
2019-12-20T20:22:39.824Z [WARN] UnassignPodIPv4Address: Failed to find pod coredns-5c98db65d4-4h9q2 namespace kube-system Container
2019-12-20T20:22:39.824Z [INFO] Send DelNetworkReply: IPv4Addr , DeviceNumber: 0, err: datastore: unknown pod
2019-12-20T20:22:40.137Z [INFO] Received AddNetwork for NS /proc/20665/ns/net, Pod coredns-5c98db65d4-4h9q2, NameSpace kube-system, Container 4b0f732f38db52e525c6f7167f6b44763905083c39f895cb310e13ec8642478d, ifname eth0
2019-12-20T20:22:40.137Z [DEBUG]        AssignIPv4Address: IP address pool stats: total: 0, assigned 0
2019-12-20T20:22:40.137Z [DEBUG]        AssignPodIPv4Address: Skip ENI eni-0ace80db803f1bedd that does not have available addresses
2019-12-20T20:22:40.137Z [ERROR]        DataStore has no available IP addresses
2019-12-20T20:22:40.137Z [DEBUG]        VPC CIDR 10.128.0.0/16
2019-12-20T20:22:40.137Z [INFO] Send AddNetworkReply: IPv4Addr , DeviceNumber: 0, err: assignPodIPv4AddressUnsafe: no available IP addresses
2019-12-20T20:22:40.163Z [INFO] Received DelNetwork for IP <nil>, Pod coredns-5c98db65d4-4h9q2, Namespace kube-system, Container 4b0f732f38db52e525c6f7167f6b44763905083c39f895cb310e13ec8642478d
2019-12-20T20:22:40.163Z [DEBUG]        UnassignPodIPv4Address: IP address pool stats: total:0, assigned 0, pod(Name: coredns-5c98db65d4-4h9q2, Namespace: kube-system, Container 4b0f732f38db52e525c6f7167f6b44763905083c39f895cb310e13ec8642478d)
2019-12-20T20:22:40.163Z [WARN] UnassignPodIPv4Address: Failed to find pod coredns-5c98db65d4-4h9q2 namespace kube-system Container 4b0f732f38db52e525c6f7167f6b44763905083c39f895cb310e13ec8642478d
2019-12-20T20:22:40.163Z [DEBUG]        UnassignPodIPv4Address: IP address pool stats: total:0, assigned 0, pod(Name: coredns-5c98db65d4-4h9q2, Namespace: kube-system, Container )
2019-12-20T20:22:40.163Z [WARN] UnassignPodIPv4Address: Failed to find pod coredns-5c98db65d4-4h9q2 namespace kube-system Container
2019-12-20T20:22:40.163Z [INFO] Send DelNetworkReply: IPv4Addr , DeviceNumber: 0, err: datastore: unknown pod
2019-12-20T20:22:40.859Z [DEBUG]        IP pool stats: total = 0, used = 0, c.maxIPsPerENI = 5
2019-12-20T20:22:40.859Z [DEBUG]        IP pool is too low: available (0) < ENI target (1) * addrsPerENI (5)
2019-12-20T20:22:40.859Z [DEBUG]        Starting to increase IP pool size
2019-12-20T20:22:40.859Z [DEBUG]        Skip the primary ENI for need IP check
2019-12-20T20:22:40.859Z [INFO] ipamd: using custom network config: [sg-0d27151dbb3b5b230], subnet-011ffa6e63cbd065c
2019-12-20T20:22:40.859Z [DEBUG]        Found security-group id: sg-0d27151dbb3b5b230
2019-12-20T20:22:40.859Z [INFO] createENI: use custom network config, &[0xc0003eaa00], subnet-011ffa6e63cbd065c
2019-12-20T20:22:40.885Z [INFO] Received DelNetwork for IP <nil>, Pod coredns-5c98db65d4-4h9q2, Namespace kube-system, Container 4b0f732f38db52e525c6f7167f6b44763905083c39f895cb310e13ec8642478d
2019-12-20T20:22:40.885Z [DEBUG]        UnassignPodIPv4Address: IP address pool stats: total:0, assigned 0, pod(Name: coredns-5c98db65d4-4h9q2, Namespace: kube-system, Container 4b0f732f38db52e525c6f7167f6b44763905083c39f895cb310e13ec8642478d)
2019-12-20T20:22:40.885Z [WARN] UnassignPodIPv4Address: Failed to find pod coredns-5c98db65d4-4h9q2 namespace kube-system Container 4b0f732f38db52e525c6f7167f6b44763905083c39f895cb310e13ec8642478d
2019-12-20T20:22:40.885Z [DEBUG]        UnassignPodIPv4Address: IP address pool stats: total:0, assigned 0, pod(Name: coredns-5c98db65d4-4h9q2, Namespace: kube-system, Container )
2019-12-20T20:22:40.885Z [WARN] UnassignPodIPv4Address: Failed to find pod coredns-5c98db65d4-4h9q2 namespace kube-system Container
2019-12-20T20:22:40.885Z [INFO] Send DelNetworkReply: IPv4Addr , DeviceNumber: 0, err: datastore: unknown pod
2019-12-20T20:22:41.004Z [INFO] Created a new ENI: eni-0bed5114b18f865e6
2019-12-20T20:22:41.004Z [DEBUG]        Trying to tag newly created ENI: key=node.k8s.amazonaws.com/instance_id, value=i-0a6a78862de42c867
2019-12-20T20:22:41.110Z [DEBUG]        Successfully tagged ENI: eni-0bed5114b18f865e6
2019-12-20T20:22:41.212Z [DEBUG]        Discovered device number is used: 0
2019-12-20T20:22:41.212Z [DEBUG]        Discovered device number is used: 1
2019-12-20T20:22:41.212Z [DEBUG]        Discovered device number is used: 2
2019-12-20T20:22:41.212Z [DEBUG]        Found a free device number: 3
2019-12-20T20:22:41.234Z [INFO] Received AddNetwork for NS /proc/20821/ns/net, Pod coredns-5c98db65d4-4h9q2, NameSpace kube-system, Container a63be77de55eed688d433be1b3002647b5a6ee288a797a6370e98c786e35c085, ifname eth0
2019-12-20T20:22:41.234Z [DEBUG]        AssignIPv4Address: IP address pool stats: total: 0, assigned 0
2019-12-20T20:22:41.234Z [DEBUG]        AssignPodIPv4Address: Skip ENI eni-0ace80db803f1bedd that does not have available addresses
2019-12-20T20:22:41.234Z [ERROR]        DataStore has no available IP addresses
2019-12-20T20:22:41.234Z [DEBUG]        VPC CIDR 10.128.0.0/16
2019-12-20T20:22:41.234Z [INFO] Send AddNetworkReply: IPv4Addr , DeviceNumber: 0, err: assignPodIPv4AddressUnsafe: no available IP addresses
2019-12-20T20:22:41.276Z [INFO] Received DelNetwork for IP <nil>, Pod coredns-5c98db65d4-4h9q2, Namespace kube-system, Container a63be77de55eed688d433be1b3002647b5a6ee288a797a6370e98c786e35c085
2019-12-20T20:22:41.276Z [DEBUG]        UnassignPodIPv4Address: IP address pool stats: total:0, assigned 0, pod(Name: coredns-5c98db65d4-4h9q2, Namespace: kube-system, Container a63be77de55eed688d433be1b3002647b5a6ee288a797a6370e98c786e35c085)
2019-12-20T20:22:41.276Z [WARN] UnassignPodIPv4Address: Failed to find pod coredns-5c98db65d4-4h9q2 namespace kube-system Container a63be77de55eed688d433be1b3002647b5a6ee288a797a6370e98c786e35c085
2019-12-20T20:22:41.276Z [DEBUG]        UnassignPodIPv4Address: IP address pool stats: total:0, assigned 0, pod(Name: coredns-5c98db65d4-4h9q2, Namespace: kube-system, Container )
2019-12-20T20:22:41.276Z [WARN] UnassignPodIPv4Address: Failed to find pod coredns-5c98db65d4-4h9q2 namespace kube-system Container
2019-12-20T20:22:41.276Z [INFO] Send DelNetworkReply: IPv4Addr , DeviceNumber: 0, err: datastore: unknown pod
2019-12-20T20:22:41.858Z [INFO] Exceeded instance ENI attachment limit: 2
2019-12-20T20:22:41.858Z [ERROR]        Failed to attach ENI eni-0bed5114b18f865e6: AttachmentLimitExceeded: Interface count 4 exceeds the limit for t2.medium
        status code: 400, request id: 86c8556c-a16e-41ed-bd76-4048f32709ad
2019-12-20T20:22:41.858Z [DEBUG]        Trying to delete ENI: eni-0bed5114b18f865e6

The ENIs are properly attached to the instance:

After deleting the aws-node pod manually, we see that its now correctly assigning ip addresses from another ENI:

ip-10-128-32-105 ~ # kubectl delete  po aws-node-bl5p2 -n kube-system e
pod "aws-node-bl5p2" deleted

ip-10-128-32-105 ~ # kubectl get po -n kube-system -o wide | grep ip-10-128-30-89.ec2.internal
aws-encryption-provider-ip-10-128-30-89.ec2.internal    1/1     Running   0          40h     10.128.30.89     ip-10-128-30-89.ec2.internal    <none>           <none>
aws-node-xlc67                                          1/1     Running   0          2m32s   10.128.30.89     ip-10-128-30-89.ec2.internal    <none>           <none>
coredns-5c98db65d4-4h9q2                                1/1     Running   0          22h     10.128.152.246   ip-10-128-30-89.ec2.internal    <none>           <none>
kube-apiserver-ip-10-128-30-89.ec2.internal             1/1     Running   0          40h     10.128.30.89     ip-10-128-30-89.ec2.internal    <none>           <none>
kube-controller-manager-ip-10-128-30-89.ec2.internal    1/1     Running   0          40h     10.128.30.89     ip-10-128-30-89.ec2.internal    <none>           <none>
kube-proxy-8zmt5                                        1/1     Running   0          40h     10.128.30.89     ip-10-128-30-89.ec2.internal    <none>           <none>
kube-scheduler-ip-10-128-30-89.ec2.internal             1/1     Running   0          40h     10.128.30.89     ip-10-128-30-89.ec2.internal    <none>           <none>

adnankobir on 21 Dec 2019

👍4

For anyone running EKS, you need to update the CNI plugin to the latest version (currently 1.5.5). As per AWS docs: Amazon EKS does not automatically upgrade the CNI plugin on your cluster when new versions are released. So this issue will reappear periodically.

As I've written above, I've used the latest CNI version and it didn't fix the problem when a 1.14 EKS cluster gets creates.

Erokos on 22 Dec 2019

@Erokos @adnankobir I'm hesitant to re-open an issue that was closed >18 months ago, but I'll do so anyway. I acknowledge that you and others are experiencing unacceptable levels of annoyance dealing with this issue. I'm wondering whether it's worth opening a new issue versus re-opening this old one (which contains links to docs that don't exist and comments about things that no longer exist in the modern CNI plugin).

Questions for you to narrow down this bug:

1) @adnankobir I notice you are using custom networking. Are you able by any chance to tell if this bug occurs when you do not use custom networking? @Erokos: are you using custom networking?

2) @adnankobir and @Erokos: have you noticed that there are specific instance types that this bug occurs with but not with other instance types? I'm wondering if perhaps there is an error in the calculated number of ENIs and/or IPs per ENI for some instance types?

@adnankobir and @Erokos: have you noticed that this issue only happens when spinning up/down many pods at once? Or does the issue occur also for "normal operations" where the rate of pods being created/deleted is low?

Thanks very much for your help folks!
-jay

jaypipes on 31 Dec 2019

❤3

@jaypipes appreciate you re-opening this issue. I ran the tests you requested to try and narrow it down:

@adnankobir I notice you are using custom networking. Are you able by any chance to tell if this bug occurs when you do not use custom networking? @Erokos: are you using custom networking?

It looks like this may be the underlying issue - without custom networking:

$ kubectl --kubeconfig kubeconfig get po -o wide --all-namespaces | grep ip-10-128-7-97.ec2.internal
default       busybox-sleep-20                                        1/1     Running   0          5m      10.128.5.143     ip-10-128-7-97.ec2.internal     <none>           <none>
default       busybox-sleep-21                                        1/1     Running   0          4m52s   10.128.6.48      ip-10-128-7-97.ec2.internal     <none>           <none>
default       busybox-sleep-22                                        1/1     Running   0          4m46s   10.128.7.177     ip-10-128-7-97.ec2.internal     <none>           <none>
default       busybox-sleep-23                                        1/1     Running   0          4m37s   10.128.6.146     ip-10-128-7-97.ec2.internal     <none>           <none>
default       busybox-sleep-24                                        1/1     Running   0          5m8s    10.128.5.87      ip-10-128-7-97.ec2.internal     <none>           <none>
default       busybox-sleep-25                                        1/1     Running   0          4m10s   10.128.5.55      ip-10-128-7-97.ec2.internal     <none>           <none>
default       busybox-sleep-26                                        1/1     Running   0          4m2s    10.128.7.250     ip-10-128-7-97.ec2.internal     <none>           <none>
default       busybox-sleep-27                                        1/1     Running   0          3m55s   10.128.7.61      ip-10-128-7-97.ec2.internal     <none>           <none>
default       busybox-sleep-28                                        1/1     Running   0          3m36s   10.128.7.161     ip-10-128-7-97.ec2.internal     <none>           <none>
default       busybox-sleep-29                                        1/1     Running   0          3m23s   10.128.7.119     ip-10-128-7-97.ec2.internal     <none>           <none>
default       busybox-sleep-30                                        1/1     Running   0          3m15s   10.128.7.107     ip-10-128-7-97.ec2.internal     <none>           <none>
default       busybox-sleep-31                                        1/1     Running   0          2m36s   10.128.5.118     ip-10-128-7-97.ec2.internal     <none>           <none>
default       busybox-sleep-32                                        1/1     Running   0          2m19s   10.128.6.107     ip-10-128-7-97.ec2.internal     <none>           <none>
default       busybox-sleep-33                                        1/1     Running   0          2m3s    10.128.5.30      ip-10-128-7-97.ec2.internal     <none>           <none>
kube-system   aws-node-krlgw                                          1/1     Running   0          7m30s   10.128.7.97      ip-10-128-7-97.ec2.internal     <none>           <none>
kube-system   kube-proxy-mtv2g                                        1/1     Running   0          7m30s   10.128.7.97      ip-10-128-7-97.ec2.internal     <none>           <none>
kube-system   kube2iam-ws6xq                                          1/1     Running   0          7m      10.128.7.97      ip-10-128-7-97.ec2.internal     <none>           <none>

getting the full 17 pod allocation for t2.medium (I have a pod limit set to 17 for this node, this includes the DS that have hostNetwork=true)

For the custom networking I have the following:

        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
          value: "true"
        - name: ENI_CONFIG_LABEL_DEF
          value: failure-domain.beta.kubernetes.io/zone

and for ENI config:

$ kubectl --kubeconfig kubeconfig get eniconfig -n kube-system
NAME         AGE
us-east-1a   33d
us-east-1b   33d
us-east-1c   33d

and for each - the associated subnet in that AZ:

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: us-east-1x
spec:
  securityGroups:
  - sg-xxx
  subnet: subnet-xxx

Please validate if this is valid configuration pattern. I have validated that the subnets belong to the correct AZ.

The remaining tests have custom networking enabled.

@adnankobir and @Erokos: have you noticed that there are specific instance types that this bug occurs with but not with other instance types? I'm wondering if perhaps there is an error in the calculated number of ENIs and/or IPs per ENI for some instance types?

I changed the instance type to a t2.small and was able to schedule 9 pods without issues:

t2.small
---------
Non-terminated Pods:          (9 in total)
  Namespace                   Name                                CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                ------------  ----------  ---------------  -------------  ---
  default                     busybox-sleep                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         3m37s
  default                     busybox-sleep-1                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         2m11s
  default                     busybox-sleep-2                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         29s
  default                     busybox-sleep-3                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         17s
  default                     efs-provisioner-787dd44fcb-2s5fq    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d18h
  kube-system                 aws-node-vhpxh                      10m (1%)      0 (0%)      0 (0%)           0 (0%)         17d
  kube-system                 coredns-5c98db65d4-v89h2            100m (10%)    0 (0%)      70Mi (3%)        170Mi (8%)     3d21h
  kube-system                 kube-proxy-ckbf6                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         17d
  kube-system                 kube2iam-sgs66                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         17d

However; One thing to note here is though it looks like pods with hostNetwork: true counts against the ip addresses available per instance even though the pods would share the primary ENI ip address.

Is my assumption/observation correct here?

running the pods on a t2.medium, I hit network: add cmd: failed to assign an IP address to container on the 14th pod / 12th ip assignment (3 pods are running hostNetwork:true):

t2.medium
----------
$ kubectl --kubeconfig kubeconfig get po --all-namespaces -o wide | grep ip-10-128-4-14.ec2.internal
default       busybox-sleep-0                                         1/1     Running             0          104m    10.128.175.159   ip-10-128-4-14.ec2.internal     <none>           <none>
default       busybox-sleep-1                                         1/1     Running             0          104m    10.128.153.166   ip-10-128-4-14.ec2.internal     <none>           <none>
default       busybox-sleep-10                                        1/1     Running             0          106m    10.128.163.214   ip-10-128-4-14.ec2.internal     <none>           <none>
default       busybox-sleep-2                                         1/1     Running             0          99m     10.128.178.141   ip-10-128-4-14.ec2.internal     <none>           <none>
default       busybox-sleep-3                                         0/1     ContainerCreating   0          99m     <none>           ip-10-128-4-14.ec2.internal     <none>           <none>
default       busybox-sleep-4                                         1/1     Running             0          107m    10.128.129.154   ip-10-128-4-14.ec2.internal     <none>           <none>
default       busybox-sleep-5                                         1/1     Running             0          107m    10.128.160.55    ip-10-128-4-14.ec2.internal     <none>           <none>
default       busybox-sleep-6                                         1/1     Running             0          107m    10.128.176.248   ip-10-128-4-14.ec2.internal     <none>           <none>
default       busybox-sleep-7                                         1/1     Running             0          106m    10.128.159.56    ip-10-128-4-14.ec2.internal     <none>           <none>
default       busybox-sleep-8                                         1/1     Running             0          106m    10.128.146.111   ip-10-128-4-14.ec2.internal     <none>           <none>
default       busybox-sleep-9                                         1/1     Running             0          106m    10.128.143.21    ip-10-128-4-14.ec2.internal     <none>           <none>
kube-system   aws-node-lg8wg                                          1/1     Running             1          108m    10.128.4.14      ip-10-128-4-14.ec2.internal     <none>           <none>
kube-system   kube-proxy-vblmq                                        1/1     Running             0          108m    10.128.4.14      ip-10-128-4-14.ec2.internal     <none>           <none>
kube-system   kube2iam-l8c8g                                          1/1     Running             0          107m    10.128.4.14      ip-10-128-4-14.ec2.internal     <none>           <none>

According to the ip allocation formula: (the number of ENIs for the instance type × (the number of IPs per ENI - 1)) + 2 - for t2.medium we should have (3 x (6-1)) +2 = 17 addresses.

After inspecting the ENIs on the node, I observed the following:

primary ENI does not have any secondary IPs attached
The remaining ENIs, only the secondary private IPs are allocated to pods

This provides exactly 11 addresses and explains why the 12th assignment fails:
1 Primary IP - primary ENI
5 Secondary IPs - second ENI
5 Secondary IPs - third ENI

Please clarify if this is intended behaviour.

@adnankobir and @Erokos: have you noticed that this issue only happens when spinning up/down many pods at once? Or does the issue occur also for "normal operations" where the rate of pods being created/deleted is low?

The cluster I'm currently working with currently has no traffic and this issue is occurring when dealing with single pods.

adnankobir on 6 Jan 2020

@adnankobir Thanks for investigating this. When you use custom networking, you will not be able to use any secondary IPs from the primary ENI. From the documentation:

Enabling this feature effectively removes an available elastic network interface (and all of its available IP addresses for pods) from each worker node that uses it. The primary network interface for the worker node is not used for pod placement when this feature is enabled. You should choose larger instance types with more available elastic network interfaces if you choose to enable this feature.

The issue here is that the --max-pods setting comes from the EKS AMI, not the CNI, and the way to override it is to set the USE_MAX_PODS env var to "false" and then set KUBELET_EXTRA_ARGS="--max-pods=XX" to override the value for the instance type from the eni-max-pods.txt file, in your case 12 for t2.medium instead of the default 17.

See https://github.com/awsdocs/amazon-eks-user-guide/pull/72 for some more background.

mogren on 6 Jan 2020

thanks for the quick response @mogren . Quick followup question -

When you use custom networking, you will not be able to use any secondary IPs from the primary ENI.

That would leave us with 2 ENIs and we should be able to allocate the full 12 addresses correct? At the moment, it looks like only 10 addresses are allocated. The Private IPs of the non-primary ENIs are not being allocated, only the secondary Private IPs are correctly being allocated to pods. Is this to be expected?

adnankobir on 6 Jan 2020

@adnankobir Yes, the formula is +2 because aws-node (the CNI pod) and kube-proxy use host networking, but are still counted against the max-pods limit.

The CNI will only use the secondary IPs for your pods, so 10 would be available for your pods in this case. The reason for this might not be valid any more, but there used to be problems using the primary IP of the ENI for pods on some distros.

mogren on 6 Jan 2020

👍2

@mogren the primary reason we wanted to separate subnet was to take advantage of a different SG for pods, is it possible to have pods use a different SG without having to use different subnets? I've tried adding the same subnets used by worker nodes in the custom ENIConfig, but it looks like doing so causes the same limitation on the number of pods.

adnankobir on 7 Jan 2020

@adnankobir Yes, since security groups are per ENI, as soon as AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG is set to true, the secondary IPs on the first ENI can no longer be used for pods. The subnets are not the issue.

mogren on 7 Jan 2020

@adnankobir In case my previous reply wasn't clear enough, the issue is that in order to have different security groups for the nodes and the pods, we can't use any secondary IPs from the first ENI, because that's where the node IP is. That's the reason you get less available IPs.

mogren on 8 Jan 2020

👍2

As soon as i updated the AWS EKS CNI version from 1.5.0 to 1.5.5 it worked for me.
Referral: https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html

rohannayakwadi on 23 Jan 2020

Upgrading to 1.5.7 did not resolve this for me. This seems like an issue still.

prestonvanloon on 22 Apr 2020

@prestonvanloon running EKS? Check if your worker nodes have reached their limit of the number of pods they can run:
https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt
https://medium.com/faun/aws-eks-and-pods-sizing-per-node-considerations-964b08dcfad3

fubar on 22 Apr 2020

@fubar we were having issues with a single EKS node. It was a t3.large that was only a few hours old. I ended up draining it and terminating it. Pods were able to be scheduled on other nodes without issue.

prestonvanloon on 22 Apr 2020

This also happened again on our cluster, using CNI version of 1.5.5. Restarting the node doesn't fix the error. We have to create a new node and shutdown the old one.

left4taco on 22 Apr 2020

This is still happening. why is this issue being clossed without proper resolution or fix?

santhoshm153 on 15 May 2020

👍24

Recently we faced this issue on t3a.large instance running eks 1.16 and cni:v1.6.3, issue got resolved when cluster-autoscaler launched new instances.
Error message in events log

FailedCreatePodSandBox pod/app-web-45454-fpk9k Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7e0fsdfdsf4fe6sdfsdasdasdsadsadasd" network for pod "app-web-45454-fpk9k": networkPlugin cni failed to set up pod "app-web-45454-fpk9k" network: add cmd: failed to assign an IP address to container

shyamjos on 23 Jul 2020

I had a very similar problem, I managed to solve it by updating Wave Net.

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

rexrod on 3 Aug 2020

👎8 👍1

issue still happening in amazon-k8s-cni:v1.7.5

tried many ways, nothing seems to work
Issue is only solved by manually creating new nodes and terminating old ones

This is a major bug , proper resolution/fix is needed

siddharthshah3030 on 10 Oct 2020

👍3

Hi @siddharthshah3030

Can you please kindly share the below information -

Steps to repro
Cluster ARN
Is this with Custom networking
How many pods you have on this instance and the type of instance?
logs - sudo bash /opt/cni/bin/aws-cni-support.sh
Kubectl describe o/p of aws-node

You can either email it to me [email protected] or open a support ticket and let me know the support ticket ID. We will look into the issue asap.

Reopening the issue to investigate further.

Thanks.

jayanthvn on 10 Oct 2020

👍2

@jayanthvn

I am not able to reproduce it right now

Also, Since it's production I can't do much testing there
Will update when issue happens again

Thanks

siddharthshah3030 on 14 Oct 2020

👍3

@jayanthvn Just chiming in to say we're another user also experiencing this.

It has happened on 2 of our 5 EKS clusters (all in separate AWS accounts), 784 times in the last 7 days (according to metrics). Most of the time it only reports once on a container before resolving, but 138 times it has reported a second time, so far we have not gone beyond 4 reports on the one container before it is resolved.

While I can't answer all of the questions above at the moment I can answer these:

I'm unsure of exact steps to reproduce but it's likely as pods are scaling up and down; there is also one CronJob that it occasionally reports on (but not always)
All 5 of our clusters are in ap-southeast-2 and the two experiencing this issue are suffixed -test and -internal (in different accounts)
No, with AWS VPC CNI v1.6.3 (and all clusters are on k8s 1.17 eks.2)
Instance type is t3.medium; number of pods change from time to time but we generally fill it up to the max pods possible for the instance type

5 & 6. I will try to remember to get these next time it happens

tdmalone on 18 Oct 2020

Hi @tdmalone

Sorry to know you are experiencing this often. Kindly share me the logs and cluster ARN once you hit the issue next time.

Cluster ARN
logs - sudo bash /opt/cni/bin/aws-cni-support.sh
Kubectl describe o/p of aws-node

Thank you!

jayanthvn on 18 Oct 2020

I'm also experiencing this issue w/ an EKS cluster running 1.18 w/ CNI version 1.7.5. I compiled some of relevant logs here: https://gist.github.com/dallasmarlow/445c926ea15d0dba71725a44bb9295f0

dallasmarlow on 14 Nov 2020

Hi @dallasmarlow

Based on the logs you have attached, I see there are only 3 IPs attached to one of the ENI and other ENI has no secondary IPs. Can you please check on the EC2 console for the instance if the ENIs and secondary IPs are attached? Also can you please share the instance id?

{"level":"info","ts":"2020-11-14T17:02:45.211Z","caller":"ipamd/ipamd.go:773","msg":"Added ENI(eni-0c06846cc125fe51a)'s IP 100.64.3.113 to datastore"}
{"level":"info","ts":"2020-11-14T17:02:45.211Z","caller":"ipamd/ipamd.go:773","msg":"Added ENI(eni-0c06846cc125fe51a)'s IP 100.64.1.101 to datastore"}
{"level":"info","ts":"2020-11-14T17:02:45.211Z","caller":"ipamd/ipamd.go:773","msg":"Added ENI(eni-0c06846cc125fe51a)'s IP 100.64.7.253 to datastore"}
{"level":"debug","ts":"2020-11-14T17:02:45.211Z","caller":"ipamd/ipamd.go:763","msg":"IP Address Pool stats: total: 3, assigned: 0"}
{"level":"debug","ts":"2020-11-14T17:02:45.211Z","caller":"ipamd/ipamd.go:628","msg":"Successfully increased IP pool, total: 3, used: 0"}
{"level":"debug","ts":"2020-11-14T17:02:45.211Z","caller":"ipamd/ipamd.go:641","msg":"IP pool stats: total = 3, used = 0, c.maxIPsPerENI = 3"}
{"level":"debug","ts":"2020-11-14T17:02:47.049Z","caller":"sdk/informer-sync.go:88","msg":"Handle ENIConfig Add/Update: us-east-2a, [sg-0bca3189f9aded496], subnet-0bc1fa1b8ce84159d"}
{"level":"debug","ts":"2020-11-14T17:02:47.050Z","caller":"sdk/informer-sync.go:88","msg":"Handle ENIConfig Add/Update: us-east-2b, [sg-0bca3189f9aded496], subnet-02d3c9b96d47a66d7"}
{"level":"debug","ts":"2020-11-14T17:02:47.051Z","caller":"sdk/informer-sync.go:88","msg":"Handle corev1.Node: ip-10-200-100-10.us-east-2.compute.internal, map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true], map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3a.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-10-200-100-10.us-east-2.compute.internal kubernetes.io/os:linux node.kubernetes.io/instance-type:t3a.small topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a vpc.amazonaws.com/eniConfig:us-east-2a]"}
{"level":"debug","ts":"2020-11-14T17:02:47.878Z","caller":"sdk/informer-sync.go:88","msg":"Handle corev1.Node: ip-10-200-100-10.us-east-2.compute.internal, map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true], map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3a.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-10-200-100-10.us-east-2.compute.internal kubernetes.io/os:linux node.kubernetes.io/instance-type:t3a.small topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a vpc.amazonaws.com/eniConfig:us-east-2a]"}
{"level":"debug","ts":"2020-11-14T17:02:47.893Z","caller":"sdk/informer-sync.go:88","msg":"Handle corev1.Node: ip-10-200-100-10.us-east-2.compute.internal, map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true], map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3a.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-10-200-100-10.us-east-2.compute.internal kubernetes.io/os:linux node.kubernetes.io/instance-type:t3a.small topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a vpc.amazonaws.com/eniConfig:us-east-2a]"}
{"level":"info","ts":"2020-11-14T17:02:51.333Z","caller":"rpc/rpc.pb.go:486","msg":"Received AddNetwork for NS /proc/5333/ns/net, Sandbox a1391ea1df4a7d46bde8b5e1a0ff7c22d6f2d1f0d77b528cee1d502fe720fe68, ifname eth0"}
{"level":"debug","ts":"2020-11-14T17:02:51.333Z","caller":"rpc/rpc.pb.go:486","msg":"AddNetworkRequest: ClientVersion:\"v1.7.5\" K8S_POD_NAME:\"coredns-66bc8b7b7b-hlzsh\" K8S_POD_NAMESPACE:\"kube-system\" K8S_POD_INFRA_CONTAINER_ID:\"a1391ea1df4a7d46bde8b5e1a0ff7c22d6f2d1f0d77b528cee1d502fe720fe68\" ContainerID:\"a1391ea1df4a7d46bde8b5e1a0ff7c22d6f2d1f0d77b528cee1d502fe720fe68\" IfName:\"eth0\" NetworkName:\"aws-cni\" Netns:\"/proc/5333/ns/net\" "}
{"level":"debug","ts":"2020-11-14T17:02:51.333Z","caller":"ipamd/rpc_handler.go:142","msg":"AssignIPv4Address: IP address pool stats: total: 3, assigned 0"}
{"level":"debug","ts":"2020-11-14T17:02:51.334Z","caller":"ipamd/rpc_handler.go:142","msg":"AssignPodIPv4Address: ENI eni-0a62d2dbdda3de253 does not have available addresses"}
{"level":"info","ts":"2020-11-14T17:02:51.334Z","caller":"datastore/data_store.go:499","msg":"AssignPodIPv4Address: Assign IP 100.64.3.113 to sandbox aws-cni/a1391ea1df4a7d46bde8b5e1a0ff7c22d6f2d1f0d77b528cee1d502fe720fe68/eth0"}
{"level":"debug","ts":"2020-11-14T17:02:51.334Z","caller":"rpc/rpc.pb.go:486","msg":"VPC CIDR 10.200.100.0/26"}
{"level":"debug","ts":"2020-11-14T17:02:51.334Z","caller":"rpc/rpc.pb.go:486","msg":"VPC CIDR 100.64.0.0/20"}
{"level":"info","ts":"2020-11-14T17:02:51.334Z","caller":"rpc/rpc.pb.go:486","msg":"Send AddNetworkReply: IPv4Addr 100.64.3.113, DeviceNumber: 1, err: <nil>"}
{"level":"info","ts":"2020-11-14T17:02:51.584Z","caller":"rpc/rpc.pb.go:486","msg":"Received AddNetwork for NS /proc/5385/ns/net, Sandbox a2872b49cd4affdb306b84a948affc73b5498858fa12a00df971b067c7e89ce1, ifname eth0"}
{"level":"debug","ts":"2020-11-14T17:02:51.584Z","caller":"rpc/rpc.pb.go:486","msg":"AddNetworkRequest: ClientVersion:\"v1.7.5\" K8S_POD_NAME:\"metrics-server-7949d47784-wzmv4\" K8S_POD_NAMESPACE:\"kube-system\" K8S_POD_INFRA_CONTAINER_ID:\"a2872b49cd4affdb306b84a948affc73b5498858fa12a00df971b067c7e89ce1\" ContainerID:\"a2872b49cd4affdb306b84a948affc73b5498858fa12a00df971b067c7e89ce1\" IfName:\"eth0\" NetworkName:\"aws-cni\" Netns:\"/proc/5385/ns/net\" "}
{"level":"debug","ts":"2020-11-14T17:02:51.584Z","caller":"ipamd/rpc_handler.go:142","msg":"AssignIPv4Address: IP address pool stats: total: 3, assigned 1"}
{"level":"debug","ts":"2020-11-14T17:02:51.584Z","caller":"ipamd/rpc_handler.go:142","msg":"AssignPodIPv4Address: ENI eni-0a62d2dbdda3de253 does not have available addresses"}
{"level":"info","ts":"2020-11-14T17:02:51.584Z","caller":"datastore/data_store.go:499","msg":"AssignPodIPv4Address: Assign IP 100.64.1.101 to sandbox aws-cni/a2872b49cd4affdb306b84a948affc73b5498858fa12a00df971b067c7e89ce1/eth0"}
{"level":"debug","ts":"2020-11-14T17:02:51.584Z","caller":"rpc/rpc.pb.go:486","msg":"VPC CIDR 10.200.100.0/26"}
{"level":"debug","ts":"2020-11-14T17:02:51.584Z","caller":"rpc/rpc.pb.go:486","msg":"VPC CIDR 100.64.0.0/20"}
{"level":"info","ts":"2020-11-14T17:02:51.584Z","caller":"rpc/rpc.pb.go:486","msg":"Send AddNetworkReply: IPv4Addr 100.64.1.101, DeviceNumber: 1, err: <nil>"}

jayanthvn on 14 Nov 2020

Hello @jayanthvn the instance ID for this kubelet host is i-01d5a3fe17d09bc48. The instance does shows that it has 3 secondary ip addresses attached to the ENI currently from the subnet configured via the ENIConfig CRD for this topology.kubernetes.io/zone / AZ. The subnet (subnet-0bc1fa1b8ce84159d) shows that there are 2039 free addresses left.

dallasmarlow on 14 Nov 2020

Will check the instance id and get back to you.

VPC CIDR -

{"level":"debug","ts":"2020-11-14T17:02:51.334Z","caller":"rpc/rpc.pb.go:486","msg":"VPC CIDR 100.64.0.0/20"}

ENI CIDR -

"level":"debug","ts":"2020-11-14T17:02:45.207Z","caller":"awsutils/awsutils.go:569","msg":"Found ENI: eni-0c06846cc125fe51a, MAC 02:88:46:38:7d:3c, device 1"}
{"level":"debug","ts":"2020-11-14T17:02:45.207Z","caller":"awsutils/awsutils.go:589","msg":"Found CIDR 100.64.0.0/21 for ENI 02:88:46:38:7d:3c"}
{"level":"debug","ts":"2020-11-14T17:02:45.208Z","caller":"awsutils/awsutils.go:589","msg":"Found IP addresses [100.64.7.96 100.64.3.113 100.64.1.101 100.64.7.253] on ENI 02:88:46:38:7d:3c"}

jayanthvn on 14 Nov 2020

@jayanthvn that instance was scaled down by an ASG, but it appears as though i-0b2609c9deb3a523c is in the same state and I updated the ASG to retain the instance. I have observed that roughly 50% of the kubelet hosts (using the latest EKS AMI and t3a.small instances) come online with this issue after scaling several nodes and testing.

https://gist.github.com/dallasmarlow/0dd2aecc0d16b4aad4dc74f22b3de03d

dallasmarlow on 14 Nov 2020

I have the same issue right now.
Cluster version 1.18
amazon-k8s-cni-init:v1.7.5
amazon-k8s-cni:v1.7.5

sarbajitdutta on 19 Nov 2020

👍2

@sarbajitdutta can you provide some logs from a host stuck in this state?
@jayanthvn did you ever have a chance to look into this issue?

dallasmarlow on 20 Nov 2020

Hi @dallasmarlow and @sarbajitdutta

Can you please share the steps to repro and collect the logs using this - sudo bash /opt/cni/bin/aws-cni-support.sh and email it to me [email protected]?

Thank you!

jayanthvn on 20 Nov 2020

@dallasmarlow - Never mind I found the logs attached here https://gist.github.com/dallasmarlow/0dd2aecc0d16b4aad4dc74f22b3de03d.

jayanthvn on 20 Nov 2020

Hi @dallasmarlow

You are using t3a.small instance with custom networking. T3a.small (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html) has 2 ENIs and 4 IPs per ENI. With custom networking primary ENI will not be used. So you will have only one ENI and 3 IPs since the first IP out of 4 is reserved. Hence you will have only 3 IPs and can schedule only 3 pods.

{"level":"debug","ts":"2020-11-14T21:44:52.264Z","caller":"ipamd/ipamd.go:852","msg":"IP pool stats: total = 3, used = 1, c.maxIPsPerENI = 3"}
{"level":"debug","ts":"2020-11-14T21:44:52.264Z","caller":"ipamd/ipamd.go:482","msg":"IP pool is too low: available (2) < ENI target (1) * addrsPerENI (3)"}
{"level":"debug","ts":"2020-11-14T21:44:52.264Z","caller":"ipamd/ipamd.go:483","msg":"Starting to increase IP pool size"}
{"level":"debug","ts":"2020-11-14T21:44:52.264Z","caller":"ipamd/ipamd.go:709","msg":"Skip the primary ENI for need IP check"}
{"level":"debug","ts":"2020-11-14T21:44:52.264Z","caller":"ipamd/ipamd.go:483","msg":"Skipping ENI allocation as the max ENI limit of 2 is already reached (accounting for 0 unmanaged ENIs and 0 trunk ENIs)"}

Also have you set the max pods value [Section 7 here - https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html] . Can you please share me the kubelet max pods value configured on the instance?

cat /etc/kubernetes/kubelet/kubelet-config.json | grep max
  "maxPods": 29

This should be the math to set the max pods -

maxPods = (numInterfaces - 1) * (maxIpv4PerInterface - 1) + 2 

Subtracting 1 from numInterfaces - Primary ENI is not used with custom networking
Subtracting 1 from maxIpv4PerInterface - Primary IP is reserved
Adding 2 as a constant to account for aws-node and kube-proxy as both use hostNetwork.

jayanthvn on 20 Nov 2020

@jayanthvn Thank you for looking into this, the kubelet hosts are configured to accept a maximum of 8 pods which I now see will not work for this instance type due to the ENI limitations you described. It looks like the reason that sometimes this works and other times it didn't was due to a race condition w/ the other pods being scheduled (which for my testing cluster includes 2 coredns pods and a metrics-server pod). I will update the max pods configuration to 5 pods for this instance type and try again.

dallasmarlow on 20 Nov 2020

👍1

Thanks for confirming @dallasmarlow :)

jayanthvn on 20 Nov 2020

👍1

Hey @jayanthvn I've run aws-cni-support.sh and opened ticket 7782332001 for this, thanks!

lowjoel on 24 Dec 2020

Thanks @lowjoel , we will look into it. Do you have managed addons?

jayanthvn on 24 Dec 2020

@jayanthvn AFAIK no - I didn't know that existed until I searched for the announcement after seeing your comment 😂

lowjoel on 25 Dec 2020

Amazon-vpc-cni-k8s: Pods stuck in ContainerCreating due to CNI Failing to Assing IP to Container Until aws-node is deleted

Most helpful comment

All 169 comments

Related issues