Containers-roadmap: [EKS] [request]: Enable PodNodeSelector Admission Controller for EKS

[toran414]

Tell us about your request
It would be nice if it was possible to use PodNodeSelector in EKS.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
In our EKS clusters, we are looking to leverage namespaces to represent our deployment environments. In order to do this safely, we want to label different auto-scaling groups of nodes as production and non-production. We then could use have the clusterDefaultNodeSelector select the non-production nodes, and the prod namespace target production nodes.

Are you currently working around this issue?
We are currently considering 2 workarounds.

1. Having our CD pipeline inject a nodeSelector into each deployment manifest based on the namespace being deployed to.
2. Writing a MutatingAdmissionWebhook to handle node selection, but this seems like we would just be re-writing the built-in PodNodeSelector.

[whereisaaron]

The PodNodeSelector is an old-skool, compiled-in Admission Controller. A MutatingAdmissionWebhook is the more contemporary way to implement this functionality and requires no messing with the control plane configuration. So you are doing the right thing there I think. Plus your out-of-tree Admission Controller can be more flexible to your needs. Implementing this replacement would be good thing. Node selectors and thus PodNodeSelector should eventually be deprecated and replaced by affinity (once more efficient).

But if it were easy to enable PodNodeSelector right now, it would be a quicker fix for your issue.

[kinghajj]

Another solution I've considered using: give each auto scaling group a different taint. Say the current production environment is called "release 2019-05-01", then one ASG would have the taint release=2019-05-01:NoSchedule, and pods for that release have a toleration {"key": "release", "operator": "Equals", "value": "2019-05-01", "effect": "NoSchedule"}. For new staging releases, you'd make another ASG with a unique taint, and have its pods tolerate it. Thus, pods from the different releases couldn't intermingle on the same node.

[omerfsen]

So right now even with the latest EKS 1.13 eks.2 we don't have "PodNodeSelector" feature right?

[omerfsen]

My requirement is that to run pods on nodes on default namespace which has label lifecycle=Ec2Spot and run dashboard,grafana,prometheus etc.. (management stuff) on ASG labeled lifecycle=OnDemand so to achieve that

https://stackoverflow.com/questions/52487333/how-to-assign-a-namespace-to-certain-nodes

[rosscdh]

+1

[excieve]

Now that there are managed node groups in 1.14 this would be super useful to automatically restrict pods to certain groups only.

[liangrog]

For those whom still want podnodesselector functionality but can't be bothered to write the admission webhook controllers: https://github.com/liangrog/admission-webhook-server

[pierluigilenoci]

@tabern is there any hope that it will be activated on EKS soon?

[AceHack]

Yes, any update?

[AceHack]

Any update, this is no longer alpha

[tabern]

We're taking a look at this.

Update 11/11
We took a deeper look at this. Because the namespaces that are declared for this controller are statically defined in config files that are on the api server, it is not something that we can easily enable today. We're keeping this on our backlog and will continue to evaluate the feasibility of properly enabling this controller in the future.

[derhally]

Is there any other solution to this problem? We have a 3rd party helm chart that doesn't cleanly handle setting tolerations and affinity. We would need everything to run on a specific node group and this would have solved the problem