Containers-roadmap: [EKS] [Security]: Allow restricting access to EKS Kubernetes Public API Endpoint

[jrnt30]

Tell us about your request
Currently there is no way to limit the ingress to the K8 API in any way.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Generally speaking we try and restrict access to our deployed services as much as possible. K8 being a 3rd party service being hosted by AWS also means that there are potential 0-day bugs that could lead to compromises in the platform to no fault of AWS.

Allowing for explicit whitelisting of access via Security Groups would allow users to secure their endpoints as they saw fit.

Are you currently working around this issue?
We are unable to do anything at this time to secure the endpoints in a fashion we are comfortable with.

[ewbankkit]

If the EKS endpoint is exposed in-vpc like other _Interface_ VPC endpoints then security groups can be specified on that endpoint.

[sumashivaprasad]

Is there any update/progress on this issue so far? It would be great if this support is added. The alternate option which is to have a Private Access Endpoint setup for API Server which was released a month back - https://github.com/aws/containers-roadmap/issues/22 is a lot of additional work for securing access to API server.

[007]

Applying a SG to the EKS control plane doesn't do anything to restrict access to the API endpoint. That's a serious trust mismatch, if I set up my control plane to have nodes + fixed ip access and I can get to the endpoint from the open internet!

How has it been in this state for seven months or more without being addressed?

[mikestef9]

We are working on getting this feature out. When launched, you will be able to restrict traffic to your public API endpoint by applying CIDR ranges to be whitelisted

[lostick]

We are working on getting this feature out. When launched, you will be able to restrict traffic to your public API endpoint by applying CIDR ranges to be whitelisted

Will we also be able to restrict internal CIDRs if the endpoint is set as private?

[tabern]

@lostick no, this will only apply to the public endpoint at launch. Can you give us an idea of the use case for having a security group / CIDR restriction on the private endpoint within the VPC?

[whereisaaron]

@mikestef9 sounds great, does your comment mean this item should be labelled "We're working on it"?

[casey-robertson]

So much engineering time because this and DNS resolution aren't available..... This is super frustrating coming from GKE.

[mikestef9]

Hi all,

We’re getting close to rolling out this feature and want to gather feedback on the final implementation.

There will be a new API field on the cluster VPC Config object that will accept an array of IPv4 CIDR blocks (up to 40) and will be applied as a whitelist to the public endpoint only.

The value of this field will not change in response to any changes to publicEndpointAccess or privateEndpointAccess. So if you have public CIDRs set through this new field, change publicEndpointAccess to False, then change publicEndpointAccess back to True, the same CIDRs will still be applied. The EKS update-cluster-config API will accept endpointPublicAccess and this new field to be set in a single call.

By default on new clusters, if you enable the public endpoint and don't specify any CIDR blocks on cluster creation, the public endpoint is accessible from anywhere and the new API field will have a value of 0.0.0.0/0.

Interested to hear your feedback to validate that this covers the use case as initially proposed in the issue. Thanks!

-Mike

[MarcusNoble]

@mikestef9 Does this mean there wont be an option to only access the endpoint via Direct Connect?

[mikestef9]

@MarcusNoble this feature only applies to the public endpoint. The API server endpoint access configuration options won't change, and you will still have the option to disable the public endpoint so your cluster is not accessible from the internet.

[MarcusNoble]

Sorry, I thought this was #221 😆 I haven't woke up enough yet it seems.

[kkapoor1987]

@tabern Do you know have est. ETA for this this feature. I ask this since we are planning to rollout prod cluster in few weeks. We have bastion host implemented but it requires lot of work. If we can simply add SG to public endpoint it will reduce all the overhead we have to make EKS endpoint secure.

[kari-awake]

I'd like a way to create temporary whitelist. Would this be possible in your current solution? E.g. give TTL for the IP address or add "create timestamp" to the resource..

Our developers are currently using a script that gets temporary credentials to different aws accounts (and EKS). These credentials are valid only max 12hours. It would be nice if the whitelist could be cleared automatically also..

[mikestef9]

I'm happy to announce that the ability to restrict access to the EKS Kubernetes public endpoint at the network level is now generally available for all EKS clusters!

Now, when the public endpoint is enabled, you can choose to restrict access by specifying IPv4 address ranges in CIDR notation from which connection requests can be made. Any client with an IP address outside this range will not be able to connect to the public endpoint. This access control can be configured using the AWS Console, AWS SDKs, or eksctl.

Important Note
If you limit public endpoint access to specific CIDR blocks, then it is recommended that you also enable the private endpoint, or ensure that the CIDR blocks that you specify include the addresses that worker nodes and Fargate pods (if you use them) access the public endpoint from.

Learn More

• What's new post
• Documentation

[valorl]

@tabern I think this needs more clarification in docs. From your comment above, it seems like in case of private-only endpoint setup, ingress rules for the control plane would not take any effect.

However, I just found out that the default ingress rule created for control plane only allows traffic from within the VPC. I had to add an ingress rule to the control plane SG that allows 443 traffic from my on-premise IPs (using VPN/transit GW).

The docs (https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html#private-access), at least to me, make it sound like this is something that should just work OOTB, which is not the case as I just explained.

What do you think ?

EDIT: I noticed the docs are open source, so I have made awsdocs/amazon-eks-user-guide#85

[drewhemm]

I've found an issue with this. We had an existing cluster which we wanted to secure, so I added the whitelist CIDR range and then tested from a non-whitelisted IP - I was still able to access the API. We have premium AWS support so I have opened a case, which has been escalated.

Have other people who have implemented this feature _actually verified that the access is being restricted?_ Could lead to a very false sense of security otherwise...

[drewhemm]

A follow up on this: We have three EKS clusters for different environments and restrictions are only working on one of them; the output of aws eks describe-cluster confirms that the config is correct on all three.

I believe I have found the issue: Of the three clusters, only one was rebuilt _after_ this feature was released - meaning that EKS clusters built before that _silently_ fail to enforce access restrictions.

I suggest that AWS investigate why this feature does not work on older clusters and/or urgently update the documentation so that customers realise that older clusters are not protected.

[MarcusNoble]

@drewhemm It might be worth checking if the platformVersion of your clusters differ or not.

aws eks describe-cluster --name cluster-name --query cluster.platformVersion

I'm guessing the older clusters have an older version of EKS running.

[drewhemm]

@MarcusNoble Same version 1.12 / eks.7 in all cases.

[plygrnd]

@drewhemm In this comment you said you had a case open. I'll follow up with you out of band with more.

[micahhausler]

@drewhemm I'm working with @plygrnd on this one. Can you email me your case ID? [email protected]

[plygrnd]

@drewhemm Confirmed that @micahhausler and I have reviewed.