Containers-roadmap: [EKS][Fargate] [request]: Increase the ulimit for EKS Fargate

[abhishek181]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Increase the ulimit hard limit for the EKS Fargate

Which service(s) is this request for?
EKS, Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Currently the hard limit for the ulimit is set to 4096 for a pod deployed in EKS Fargate. So, the soft limit of ulimit can only be set to the hard limit and it cannot be increased than the hard limit. Sometimes, there can be a requirement for higher ulimit than 4096.

Are you currently working around this issue?
No workaround as of now.

[casret]

ulimit -n unlimited also raises an error instead of going to the hard limit, which is annoying.

[booleanbetrayal]

Also experiencing this on our EKS Fargate cluster, leaving us to even wonder if this is a Fargate deal-killer. Meanwhile, it would be nice to have some sort of interface in order to take the ambiguity out of these limits; something AWS could respect - https://github.com/kubernetes/kubernetes/issues/3595

[mlanner-aws]

Just to follow up on this one, is it nofile in particular you are concerned with or are there other limits that are of concern?

[casret]

For me nofile. Just got a EMFILE error this morning in fact.

[booleanbetrayal]

@mlanner-aws - Definitely hitting nofile issues right off the bat with the 1024 soft / 4096 hard limits. Haven't had a chance to test any other constraints, but i suspect we would want to uncap memlock from the current 64k defaults as well.

[booleanbetrayal]

@mlanner-aws - Any ETA on getting this fixed in the Fargate EKS worker node AMIs? Can the nofile limit be set to the value of sysctl fs.file-max ?

[mlanner-aws]

No ETA yet, we're still discussing and prioritizing. It's unlikely we'll be able to get to this in 2020.

[booleanbetrayal]

@mlanner-aws - I want to believe that EKS Fargate is a viable product offering, but right now it feels like it isn't. I'm sure that an arbitrary hard-cap of 4096 nofile breaks significant numbers of prospective workloads right out the gate. Given the AMI history behind ulimit, I'm wondering what sort of quality control goes into the EKS Fargate product line, at this point. Finally, the fact that it may potentially take several months to update approximately 10 lines of code, makes me wonder what we'd be paying for, and whether or not it would just be reverted again in the future. If there is an escape hatch that I'm unaware of, please let me know. As it stands, we are in the process of evaluating potential alternatives.

PS - Shamelessly leveraging my Brent-powers to (hopefully) find an advocate in @brentley