Amazon-vpc-cni-k8s: Large clusters/VPCs with multiple clusters can hit describeNetworkInterface rate limits

Created on 13 Sep 2019 · 4Comments · Source: aws/amazon-vpc-cni-k8s

I have a few large testing clusters (in the same VPC, ~500 nodes). At a certain point I noticed that we were being rate limited on describeNetworkInterface requests in the logs of some other tooling I was using to do AWS things. In any case, I suspect the majority of these requests are actually coming from the VPC CNI. Please let me know if this is not correct.

Opening this issue just to ask:

does the CNI make more describeNetworkInterface calls than it needs to?
I see there is some caching, but could there be... more? :D

Any thoughts @mogren?

question

Source

rudoi

Most helpful comment

@sethp-nr Good question!

Using IMDS (Instance MetaData Service) does not count against the EC2 rate limit, but unfortunately we can't rely on it. I've explained the reason for this a bit in more detail here, but in short it can be very out of date.

For older instance types we have seen up to 30 minutes. This means we can not really trust it, and the only way to avoid assigning the same IP to more than one pod is to check with the EC2 control plane.

mogren on 13 Sep 2019

👍2

All 4 comments

Yes, you are right. It's a tough balance to not call EC2 too much, but also to quickly add ENIs when needed, and also not keep them around for too long when scaling down.

We have to describe the interfaces to see which ENIs and IPs that are attached. Using WARM_IP_TARGET will add a lot more calls and is probably not something you should enable on large clusters. Increasing WARM_ENI_TARGET from the default of 1 would reduce the calls to EC2 as well, but it comes with the cost of having more allocated ENIs and IPs in the VPC.

mogren on 13 Sep 2019

Thanks for the quick response @mogren ! @rudoi filed this issue on my behalf, but one thing I'm not sure I understand:

We have to describe the interfaces to see which ENIs and IPs that are attached.

I believe this information is also exposed through the instance's "metadata" service, is that right? Does that end up being charged against the same rate limits as the Describe* call?