Terraform-aws-eks: Workstation cidr possibly not doing what's intended?

[bshelton229]

I'm submitting a...

• [X] bug report
• [ ] feature request
• [ ] support request
• [X] kudos, thank you, warm fuzzy

What is the current behavior?

If this is a bug, how to reproduce? Please include a code sample if relevvant.

This is much more a question than an issue. I see the workstation cidr being allowed to access 443 in the security group attached to the eks cluster. I see the same thing in the terraform eks getting started post. My assumption was this would limit access to the kubernetes api (control plane) to that cidr. It doesn't do that and these control planes are fully accessible on the internet. Was the intention to allow only that cidr to access the control plane? I really wish that were possible. I'm likely completely missing the reason to allow that ingress.

What's the expected behavior?

I expected the control plane to be limited to the IP address in the cidr. My expectation may be completely wrong in which case maybe a different variable description could help.

Are you able to fix this problem and submit a PR? Link here if you have already.

Environment details

• Affected module version:
• OS:
• Terraform version:

Any other relevant info

[brandonjbjelland]

Ouch! I'm seeing the same thing. That's definitely not as intended and your interpretation of the variable descriptions appear to match what I intended. Good catch. Let's get this fixed! Any ideas?

I reproduced this in test and manually removed both the cluster and workers' inbound security group rules and even still I'm able to connect to 443 of the cluster endpoint and run kubectl commands from separate IPs. Not expected.

For a second I considered that aws-iam-authenticator might be doing misdirection with the kubectl commands (i.e. passing them through an AWS API), but even if it did, that wouldn't mean I can telnet to 443 of the cluster URL... I'm drawing a blank on this one.

Questions/thoughts to track down:

• How does a new cluster behave when created manually through the console?
• I wonder if there's a test scenario that could cover this...
• Perhaps this is by design? I'll be spending some time with the docs to try and drill down. If anyone can repro and ping AWS support about it, that'd be kind!

[bshelton229]

I'm also going to do more research and testing, but I've heard from the #eks room in the kubernetes slack org that it's currently not possible to lock down the control plane network access. I would imagine launching the control plane in your own VPC (completely) is a future feature they're working on. I made the same assumption that putting this ingress limitation on a security group you're assigning to EKS itself would have the desired effect, but upon testing it definitely doesn't work that way.

The security groups on the control plane and worker nodes seem to interact to allow the control plane to communicate with the nodes inside their vpcs mostly. This is the best description I could find - https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html. I'm still a little fuzzy on whether or not the worker nodes communicate back out to the control plane via some kind of private connection via the ENIs the EKS security groups are attached to, or if they communicate over the public ip addresses that resolves for the control plane; the same way kubectl communicates with the control plane. Kubelet is fed a kubeconfig that looks mostly like the one on my machine, with the same api endpoint.

I'm going to ping AWS support tomorrow to get a clear answer, but so far what I've heard from folks is you cannot lock down the networking on the control plane. I'd still like a better understanding of how the communication between control plane and worker nodes is accomplished, and hopefully I can get some more details next week.

Assuming locking the control plane itself down isn't an option, I'm not totally sure how to update this module? The best thing is probably to remove the workstation_cidr functionality completely, but that wouldn't be backwards compatible.

[micahhausler]

The security group for the EKS control plane is only for outbound traffic from the Kubernetes masters to the kubelets within the VPC. It is for exec/logs/proxy/port-forward

PrivateLink support for the API server is on our roadmap.

[max-rocket-internet]

I think this was removed here: https://github.com/terraform-aws-modules/terraform-aws-eks/pull/84

So will now close 🙂