Amazon-vpc-cni-k8s: Remove hard restrictions of no of pods per instance type

[alok87]

What?

Request - Will it be possible to remove the restrictions on no of pods for an instance type. Currently there is a restriction

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

[EmiiKhaos]

Not a maintainer, but as I can see, no. The ENI/IP address limits per instance types are hard limits from AWS EC2 itself. So it's good to have the number of pods restricted as quota limit on the node too if this is already done. Otherwise you would be stuck in error loops. If the assignable IP addresses on a node are exhausted, the k8s scheduler shouldn't be allowed to schedule a new pod to the node.

[alok87]

@EmiiKhaos So the user will be limited to run n no of pods in a node - which is not right. In the other CNIs no restrictions are there and the user is free to run as many pods they want

[EmiiKhaos]

Yes, if you choose small instance sizes you're very restricted to run n number of pods given on the ENI/IP address limits of AWS EC2. That is right for this CNI, because that's the hardware restriction of the implementation.
Other CNI use primary overlay networks, which don't have this hardware restriction and thus the whole CIDR space available. Excellently described in the EKS network foundations.

Edit: k8s has a pod limit per node of 100 max hard too.

[alok87]

@EmiiKhaos 100 is not a hard limit, its a default value AFAI, and it is configurable using kops
cool i will read about it, the pods can be very small in resources and can be more than n based on how the user wish to use it. But the AWS restrictions of 50 in m4.xlarge is very restrictive.

[dbenhur]

@alok87 This CNI uses EC2 VPC networking natively and is subject to those limits. If you need greater pod density than supported, you can use an alternative CNI implementation with an overlay network. You will trade some performance for pod density, but that may be the right answer for you.

There is ongoing work to increase interface and IP address density/instance, and when it's ready there will be a new CNI supporting it, but the current limits are what we must live with for this implementation.