Containers-roadmap: DNS resolution for EKS Private Endpoints

[tabern]

Automatic DNS resolution for EKS private endpoint private hosted zone within VPC. This allows fully direct connect functionality from on-prem computers to Kubernetes cluster API server that is only accessible within a VPC.

Follow-up feature from https://github.com/aws/containers-roadmap/issues/22

Edit 12-13-19: This feature is now available

[rajarajanpsj]

@tabern Thanks for creating a separate issue for this. We want to see if there is a temporary way to tackle this issue assuming we definitely want to use the setting of disable public access and enable private access. At this point, even if we have to setup some extra hacks, would something like below work? Do you see any issues?

1. Have a lambda within the VPC where EKS cluster resides that will pick up the cluster endpoint hostname/IP (Is this possible? I dont see the PHZ that EKS manages, so where do we get the IP) and update a common Route53 resolver running a common VPC (assuming our datacenter DNS is setup up to forward to this common Route53 resolver which will give back the private IP of the EKS endpoint).

2. Will the above logic work? Do we have like a private ip from the VPC per subnet mapped to EKS point DNS name? This would be the case if it internally uses the private link technology right? Or is this an incorrect assumption?

3. Does the EKS private endpoint itself use NLB at the background? So i can assume that the if I map the private IP to the EKS DNS name once, then it is guaranteed not to change? or it can change anytime and we have to constantly check for that from our lamdba?

[nurus]

I wanted to add that it might be beneficial for the EKS cluster api endpoints to appear as a target for Route53 Alias records. We are able to resolve a Route53 private hosted zone over VPN but using a CNAME created in this private zone doesn't help us resolve the shadow eks.amazonaws.com domain created for the cluster. An ALIAS record would return A records to the endpoints and solve our problem.

[ikester]

@nurus did you try creating ALIAS records for the EKS endpoint on the Route53 private hosted zone?

[nurus]

@ikester, I tried to create it manually through the console but was unable to but this probably because there is no hosted zone provided for the EKS endpoint. Looking at doing this with the aws cli or terraform shows that this is a required parameter.

[rajarajanpsj]

@nurus Did it work, it will be great if you give more information on how you made it work.

[nurus]

@rajarajanpsj, unfortunately not. AWS does not provide the hosted zone of the EKS cluster api endpoint which is a required parameter to create an ALIAS record.

[kivagant-ba]

Hello. Overall the change is very nice.
What about a peered VPC? For example if a CI/CD system installed in another VPC how to access the private EKS API endpoint?

[joshkurz]

what's the concern for just making the DNS entry public? I think that would solve all these issues.

[pwdebruin1]

I agree with @joshkurz it worked for RDS, if your RDS instance is privately exposed your RDS endpoint DNS name is resolvable publicly to a private IP within your VPC.

[wico]

One way to workaround this issue is a HTTPS/CONNECT proxy (e.g. tinyproxy) running inside the VPC, which allows connecting to the kubernetes private endpoint but without asking the VPC DNS from the client side. But thats not a nice solution.

It would be really really good if the authorative DNS-servers for sk1.eu-west-1.eks.amazonaws.com would also have a knowledge about the private endpoint records and not only the VPC internal DNS.

[cansong]

We managed to solve this by using Route53 Resolver. Basically, you want an Inbound Endpoint in your EKS cluster VPC, an Outbound Endpoint in your peered VPC, and a rule associated to your peered VPC that forwards your cluster domain name requests to the Inbound Endpoint ip addresses.

Don't forget to allow UDP 53 on your cluster security group for your peered VPC outbound endpoint ip addresses, and to check your on-going Network ACL rules.

[pwdebruin1]

@cansong genius! Would be cool to get more native support but best workaround I've seen so far!

[ikester]

@cansong That's a good solution if you can have peered VPCs like that. We've been documenting that approach and will be releasing it soon to help customers for whom the multi-VPC solution works.

[wico]

For all the others who cannot do the the solution mentioned by @cansong, we really need the zones being public (or having an option to make the zone public or private), as written by https://github.com/aws/containers-roadmap/issues/221#issuecomment-479015377 :)

[IuryAlves]

For all the others who cannot do the the solution mentioned by @cansong, we really need the zones being public (or having an option to make the zone public or private), as written by #221 (comment) :)

@wico Genius!!

[robertgates55]

We're using a transit gateway to route between our VPCs. Is there a workaround using resolvers (or any workaround at all!) that we could try out?

[yrsurya]

https://aws.amazon.com/blogs/compute/enabling-dns-resolution-for-amazon-eks-cluster-endpoints/

[tabern]

All - wanted to let you know we published a blog that details how to enable resolution for private endpoints when using a peered VPC or Direct Connect: https://aws.amazon.com/blogs/compute/enabling-dns-resolution-for-amazon-eks-cluster-endpoints/

This has been developed and test by AWS solution architects and the EKS team. Let us know if it works or doesn't work for you.

We'll be keeping this issue open as we consider delivery of this feature to be when resolution works automatically for EKS clusters and the team is continuing to develop a integrated solution for private endpoint DNS resolution.

[wico]

Unfortunately (and as mentioned), the above solution eventually does not help if people/pods/... still query e.g. 8.8.8.8 (and if you don't intercept this traffic).

An example:
In EKS, the builtin-dns resolution (coredns) comes per default with a setting to proxy (forward) DNS-querys to the VPC dns and as a fallback to 8.8.8.8:

The coredns-pod has this config:

/ # cat /etc/resolv.conf
nameserver <VPC_DNS>
nameserver 8.8.8.8

That means: Even with a solution as mentioned by @tabern and even if you are inside the VPC - if you have e.g. 2 EKS clusters (cluster 1 and cluster 2) and you do operations from cluster 1 against the API of the cluster 2, you could end up with not being able to resolve the API endpoint of that cluster 2. This happens if your in-cluster coredns of cluster 1 forwards the request to 8.8.8.8 to resolve the API of the cluster 2.

Sure, the settigs for coredns could be changed to not use (remove) 8.8.8.8 (and also, the resolv.conf has options timeout:2 attempts:5) - but I just wanted to illustrate the problem with a specific example. And I hope I did not miss anything. :)

Note: The above applies to EKS kubernetes 1.11. I have not testet with 1.12 - maybe the default coredns-config is different there.

[shinka81]

Creating a conditional forwarding rule for each private API endpoint seems very burdensome, hopefully what @joshkurz suggested is being considered as that seems like the most hands off solution to me. in our case there are some controls around usage of the appliance (infoblox) where DNS is managed on prem where these forwarding rules would be programmatically created so it is difficult for us to automate that after creating these clusters.

[mvillumsen]

@tabern I just used the blog post to setup dns resolution for EKS from our on-premise network using a VPN connection and it works fine. So thanks for the guide. However, as previously mentioned it would be a lot easier if you were just able to choose whether or not you want the DNS zones to be public or private available and control the access using a security group.

[hobti01]

@tabern the proposed solution is good when it is allowed to modify the DNS resolution. Unfortunately we cannot do this due to security policies.

It would be great to have the ability to assign a custom hostname/FQDN for the API endpoint which could be managed simply via Route53. Of course the server certificate should include the custom name(s).

[chrisferry]

@tabern this doesn't help for transit gateways. When do we expect to see support for them?

[MithilShah]

One interim solution is to modify the /etc/hosts file on the developer machine to locally resolve the endpoint dns. Here's a blog that shows a shell script that can do that automatically.
http://www.studytrails.com/devops/kubernetes/local-dns-resolution-for-eks-with-private-endpoint/

[ttercero]

Thank you, very useful script.

--Tony

On Tue, May 28, 2019 at 6:51 PM MithilShah [e-mail] wrote:

One interim solution is to modify the /etc/hosts file on the developer
machine to locally resolve the endpoint dns. Here's a blog that shows a
shell script that can do that automatically.

http://www.studytrails.com/devops/kubernetes/local-dns-resolution-for-eks-with-private-endpoint/

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/aws/containers-roadmap/issues/221?email_source=notifications&email_token=AHEKZFYBF4SHWAWCJC2HK4LPXXOTHA5CNFSM4HBKIZGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWN5TFI#issuecomment-496753045,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHEKZF5AX2MKSPNT3KRNOYDPXXOTHANCNFSM4HBKIZGA
.

[newlyregistered26]

We are experiencing this issue when trying to communicate between two VPCs which have EKS clusters in them. We have one cluster which has prometheus deployed in a "services" type VPC trying to scrape metrics other workload VPCs. What we've noticed is that the cluster names are not always in the same DNS namespace, ie y4.ap-southeast-2.eks.amazonaws.com and another in sk1.ap-southeast-2.eks.amazonaws.com so we can use a workaround using a custom DNS zone with the internal API hosts files within sk1..... and associate this with the prometheus cluster in y4..... We are however unable to understand how the cluster naming works, as one environment still shares the same namespace as the services SVC which is overlapping and as such we can't associate this. There is another Kubernetes property which allows you to override specific pod DNS entries as below https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/, however we are using Prometheus Operator pattern and as such this doesn't support custom pod specs. Keenly watching this issue for updates.

[alfianabdi]

The DNS resolution using route53 resolver works for me, but it also give me additional cost.
$0.125 per eni per hour, with 2 ENIs it is more expensive than the EKS master itself. It is just for one VPC.
It would we nice if we can resolve it natively just like the RDS endpoint.

[witold-gren]

@wico Can you write how to configure proxy for EKS private endpoint?

[wico]

@witold-gren

There are different options (all are a bit hackish).

Option #1: You set up a EC2 instance running e.g. tinyproxy (inside the VPC where you have the EKS private endpoint) with the config-directive "ConnectPort 443". Then you can export the HTTPS_PROXY environment variable wherever you are and point that to the EC2-instance. After all is up, you can use your kubectl-config as usual and connect to the kubernetes-API via the HTTPS/Connect proxy. The DNS resolution (which is the issue) is done on the EC2 instance running the proxy and not on your local machine.
PRO: Easy setup
CON: Everyone has to set the HTTPS_PROXY env var all the time and depending on where/how you did that, ALL HTTPS-traffic goes through the proxy in that session/shell. :(

Option #2: You set up an ingress/nginx inside your EKS cluster and expose that with an IP and DNS entry as you do for every other service. Then you ship a kubectl-config which points to that ingress. The nginx works like a reverse-proxy and decrypts/reencrypts the traffic and forwards that to the real API.
PRO: Transparent for the users
CON: More complex to setup, and the API becomes unreachable if the cluster itself is in a broken state. Chicken egg problem. :)

Option #3: You use a AWS loadbalancer (in tcp-mode) inside the same VPC and point it to the IPs (!) of the kubernetes-API. Then you just need to point in your kubectl-config to the AWS loadbalancer.
PRO: Easy setup
CON: The AWS loadbalancer does not verify the TLS-cert of the kubernetes API. That can be a security risk.

That was a very short writeup and I wrote it from my memory. Tell me if you need more details. :)

[robertgates55]

@tabern - Is there any movement at all on this?

As @alfianabdi correctly stated, right now the workaround to allow us to _connect_ to EKS costs more than the EKS control plane itself. Which feels wrong. The other workarounds (eg /etc/hosts hacks, proxies, dns-hacks, k8s-endpoint-as-IP) all feel hacky.

[alfianabdi]

@robertgates55
What make it feels worse, is that there are rds, elasticache, elasticsearch and such, but the dns resolution works just fine. Why dont make EKS using the same concept.

[jammerful]

@tabern I'm also running into the issue, and the proposed solutions are usable for me. Is there a timeline for this issues getting resolved?

[msvechla]

Same thing here, going the route53 resolver way is very expensive with a large number of clusters. I agree making the DNS entry public would be the best solution and match other services behaviour.

[stateisbad]

I just want to echo the above comments and show my support for this. My company has a feature request open for this, and I believe others do, too. This is essentially a show-stopper for us to move to EKS from kops, and I can't really imagine it's not for many others, too. The R53 resolver method is a laughable "solution" to what seems pretty obviously to be a design oversight.

[krzysztof-bronk]

You know something is off when the DNS resolution for EKS Control Plane costs you more than the EKS Control Plane itself :)

[pulkitmehra]

We are facing similar issue. This is stopping us in moving to EKS. Do we have any cost effective solution from AWS team?

[tabern]

Hi everyone,

We’re getting close to being able to implement a permanent fix to this issue and want to gather feedback as we make a final implementation decision.

To recap, today when you disable the public endpoint for an Amazon EKS cluster, there is no way to resolve private endpoint when using a peered VPC or AWS DirectConnect. More specifically, there is no way for the client that is routing through the peered VPC to your cluster’s VPC to acquire the IP address that it needs to connect to the private endpoint.

Our current proposal is to configure the public endpoint to vend private IP addresses. This will only work when the public endpoint is disabled and the private endpoint enabled. In this case, the client attempting to connect to the private endpoint through a peered VPC or direct connect will receive a private IP from the public DNS hosted zone. Clients can then use that IP to connect to the cluster. For any client without a route to the private worker node VPC, this IP will be meaningless and they will be unable to connect.

As noted above, this is how other AWS services including RDS and ELB behave today.

It is important to understand that in this solution no cluster data is exposed. If a client does not have the permissions to access the private cluster VPC, the IP they receive will be meaningless. Without a route to the VPC, there is no way that they can ever resolve the IP address to connect to the private endpoint. Your cluster is fully secured using AWS VPC networking. For clients that can access the VPC, connecting to the private EKS cluster endpoint will be seamless and automatic.

This is a change to the way EKS behaves today so we’re interested to hear your feedback to validate that this is the right architecture for your use case. Let us know!

-- Nate

[walkafwalka]

This will only work when the public endpoint is disabled and the private endpoint enabled.

@tabern What if both the public and private endpoints are enabled? Like EC2 instances, will the private IP be preferred, even when accessing the public endpoint, if accessible by the client?

[tabern]

@walkafwalka this will only happen when just private is enabled. If the public endpoint is enabled, it will vend public IPs and this DNS resolution mechanism is not necessary.

[vascop]

this will only happen when just private is enabled. If the public endpoint is enabled, it will vend public IPs and this DNS resolution mechanism is not necessary.

@tabern I think what would be preferred, as referred to by @walkafwalka is to have EKS be consistent with most (all?) other AWS services.

1. Private VPC-resolvable-only endpoints always available for intra-VPC traffic.
2. Public endpoints only available if public access is configured.
3. Private routes preferred when accessing it from other locations within the VPC (even if public access is enabled).

This would be similar to EC2, RDS etc. It's not great to have private intra-VPC traffic to have to go outside of the VPC onto the internet only because we enabled public access.

[whereisaaron]

Thanks @tabern I have no problem with that workaround. However I know it won’t satisfy everybody, as I know a small number of our clients would not let us use a solution like that, because they view even exposing the existence of an internal IP, e.g. even in internal documentation, as an unacceptable risk.

Note that when we create non-EKS clusters with private and public endpoints, we don’t use the same domain name for both endpoints. We find it clearer for clients to specifically target an endpoint. And it avoids split-horizon DNS drama like this. Nodes are configured explicitly to only use the internal endpoint DNS name.

Personally I don’t think public-only endpoints should ever be (have been) an option. Cluster-internal communication should always be (have been) VPC-internal. EKS clusters would ideally be private-first/only, with the option to add a separately named public endpoint for external clients.

[wjam]

I'm unsure whether that proposal would work in our situation. We currently have each environment completely isolated, without any VPC peering connections for PCI-DSS reasons, with only VPC endpoints to access certain things within the environment.

Would it be possible to stick an NLB with SSL termination in front the IP address so that a VPC endpoint could be exposed for that endpoint - obviously the URL that EKS exposes as the cluster address would be different to the one you actually use...

[troyfontaine]

@tabern As previously mentioned, that would certainly work for some folks-and also mentioned for the folks with PCI it would be problematic.

What about a private EKS endpoint similar to the S3, Route 53 resolver, etc that you could set a static IP for inside another VPC? That takes the DNS off Amazon's hands and puts the customer in control as well as meet those tight control requirements for PCI-DSS and other paranoid policies?

I know on my team we'd be ecstatic with the proposed solution immediately and would want to look at a more controlled option later.

[DewinGoh]

@tabern great to hear about the solution, it works very similarly to our existing Kops setups. Is there an ETA for this feature release?

[luisdeniz]

@tabern thank you for the update. I have one question regarding the proposed solution -- with this new functionality, are the changes to the VPC DNS settings still required by EKS? Namely, I'm referring to the existing prerequisite for private endpoints:

From https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html
VPC must have enableDnsHostnames and enableDnsSupport set to true

We really hope these VPC settings are no longer required with this new feature since we currently run custom DNS servers where we have domain filtering rules. Enabling the AmazonProvidedDNS (.2) within our VPC(s) is not acceptable due to its known limitations and exploits.

[krzysztof-bronk]

@tabern This solution is just fine for our use cases.

While it's not ideal to vend private ips through public dns, as it's viewed by some as a security risk, to me it has always been security through obscurity at best.
When it comes to the issue of the DNS solution not being in line with other AWS services - well... it will be.
ELBs work the same way :)

[jclynny]

@tabern I'd like to +1 on @vascop 's comment above. At my Company we would really like to see it work like other AWS services with private access only. The solution you described above really seems like you're basically enabling private and public which is still exposing things to the Public internet which we want to avoid when possible.

[lilley2412]

@tabern I would be happy to see it work as described because it's simple and consistent with ELB DNS resolution. I have heard objection to "ip leakage" in the past on ELB's but I personally don't consider it a problem.

[mouellet-coveo]

@tabern I would also be happy with this solution even though I consider it a bit hackish.

Personally, I think the way DNS on Elastic IP works would be a better solution. I mean, EIP's DNS will resolve to the public IPs when queried from outside the VPC and will resolve the private IP when queried from within the VPC (or peered with DNS resolution enabled)

[tabern]

Thanks for all the feedback! @whereisaaron and @lilley2412 best summarized our query, that some customers will find this an unacceptable security posture. We are curious to understand more about why this is.

Our current thinking is that the private endpoint is still fully secure and unaccessible without a route to the VPC - indeed several AWS services have PrivateLink implementations with this exact same behavior.

Specific comments and reactions:

@vascop this is exactly how it will work - from within the cluster's VPC, private routes are always preferred. If public and private are enabled, then traffic from outside of the VPC (including peered VPCs) will go over the public endpoint.

@whereisaaron why do these customers consider this to be an unacceptable risk? Many other AWS services behave this same way when PrivateLink is enabled for them, for example RDS. This is exactly the concern we are trying to understand. Like you said, this is actually how our private endpoint works today, any time the private endpoint is enabled (even if public is turned on), nodes within the VPC will route through the private endpoint.

@troyfontaine the reason we've stayed away from creating Route53 resolver endpoints is expensive and we were advised to not implement this. We're working with the AWS compliance team to ensure anything we launch will allow continued compliance with PCI-DSS.

@luisdeniz No, there will be no change to this. You will still need to set VPC DNS hostnames and resolution to true. Can you open a new roadmap issue to track this request? Its not on our short term horizon but something we can investigate in the future.

@jclynny this actually will work similar to other AWS services like @krzysztof-bronk mentioned.

@mouellet-coveo Why do you think this is hackish? What you are proposing is exactly how this will work, without the ability to resolve to the public IP as there will be no public IPs vended when the public endpoint is disabled, only private IPs.

[mouellet-coveo]

@tabern Sorry if I misunderstand but I was under the impression that if you enable both Public and Private endpoint it would always resolv the public endpoint. Having the ability when both are enabled to resolv it based on your origin is probably something people might need.

Still, I'd be more than happy with the proposed solution.

[tabern]

@mouellet-coveo Glad to hear this! Generally if both public & private are enabled, worker node traffic goes through the private and other traffic through the public. We are planning https://github.com/aws/containers-roadmap/issues/108 that will allow you to limit access to the public endpoint when it is enabled as well.

[walkafwalka]

If public and private are enabled, then traffic from outside of the VPC (including peered VPCs) will go over the public endpoint.

Why would peered VPCs also go over the public endpoint? I do not believe this is the case for most other services, most notably, EC2 DNS names.

[tabern]

@walkafwalka Because of how the private endpoint is exposed within the VPC, peered VPCs are unable to get the IPs for the private endpoint. This is actually the root cause of the issue the proposed fix will solve.

[solvip]

@walkafwalka Because of how the private endpoint is exposed within the VPC, peered VPCs are unable to get the IPs for the private endpoint. This is actually the root cause of the issue the proposed fix will solve.

Now I'm confused. Are you referring to how things are now?

Just to make sure that I understand the upcoming fix correctly:
if I've got a private cluster, I'll be able to look up the private addresses of the control plane over the public internet?
And as long as I've got a route to them, I'm good to go?

[lkishalmi]

Well the proposed workaround would be ok, though I would not accept that as a final solution.

[tabern]

@solvip sorry for the confusion! Yes, today, if you are attempting to connect to an EKS cluster VPC through a peered VPC with only the private endpoint enabled, there is no way to get the IP address of the private endpoint with out doing some extra work (see above for details on the current solution).

Yes, you are 100% correct on the upcoming fix. This is exactly how the proposed solution will work.

@lkishalmi why would you not accept this as a final solution?

[whereisaaron]

@whereisaaron why do these customers consider this to be an unacceptable risk? Many other AWS services behave this same way when PrivateLink is enabled for them, for example RDS. This is exactly the concern we are trying to understand. Like you said, this is actually how our private endpoint works today, any time the private endpoint is enabled (even if public is turned on), nodes within the VPC will route through the private endpoint.

@tabern it is one of those positions that there is no point discussing 🤷‍♂ It is non-negotiable policy, and no one is interested in debating it. I can assure you that your "this IP will be meaningless and they will be unable to connect“ argument will not budge the needle 😉 I assume the basis is that attackers are expected to do their own reconnaissance to determine internal IP numbers.

I was saying that ‘if I ruled EKS’ that the private endpoint would always exist and nodes would only ever use that endpoint. The public endpoint would be an optional feature to enable, to which users could bind their own domain name and ACM certificate (bonus credit for EIP support). Nodes would never use the public API, nor need to know it’s DNS name or existence.

[alfianabdi]

@whereisaaron Aren't services like RDS also using public domain?

For me this is ok. As long as it is not routable from internet.

[whereisaaron]

@alfianabdi yes some other AWS services work that way too. I was just saying there are a small number of customers that don’t allow that, not matter how innocuous it may seem.

[lilley2412]

I think everyone has to understand that the proposed solution is the only one that is simple, robust, and solves all use cases without a) requiring the EKS team to do a major redesign or b) requiring end-users of EKS to do work or maintain additional resources.

The majority problem is peered vpc's and on-prem networks connected with vpn or tgw/dx that need kubectl access to the control plane. We can't just say that only nodes access the private endpoints; CI processes and cluster admins in corporate envs need to kubectl from other places (in my case we use CI agents in centralized VPC's to build clusters on other VPC's and deploy k8s apps, and we also have kubectl users on-prem that administer clusters).

Unless the private endpoint DNS resolution is available "everywhere", some kind of work has to be done (dns forwarding, new EKS features, etc.) to enable kubectl access, simply no way around that.

About IP leakage, if it's strictly prohibited by some corporations, then those environments are already extremely limited for AWS VPC-based resources - does that mean RDS and ELB's are prohibited? While it's unfortunate that some environments have those limitations, IMO the EKS feature shouldn't attempt to address that problem since they are already really limiting AWS adoption with those kind of rules.

[troyfontaine]

@tabern at this point I will happily accept this solution and consider this case closed. The rational behind not exposing IPs is weak and security through obscurity as demonstrated time and again is just lazy.

Is there any sort of timeline that can be provided on this solution going live (rough even)?

[robertgates55]

@tabern this works for us - great news. (it also means utterly minimal changes to existing code, which as @lilley2412 mentioned is a nice bonus)

[korend]

@tabern the proposed solution will work well for us, it's aligned with other services and simple to implement. it will also enable utilizing the same code on-prem and in the VPC.

[tmattausch]

@tabern The proposed solution works very well for us. Since this is done in the same way for example in RDS or with ELBs. tbh this is exactly the behaviour I would have expected after enabling the feature for the first time.

[tabern]

Thanks all, @whereisaaron @lilley2412 I think you've both hit the point square on here. We need to build something that will work for as many customers as possible in a simple way that doesn't require a ton of work.

As for timing, this is something the team is working on. We will make sure to update this thread ASAP when its available!

[jclynny]

@tabern Another thing I was thinking of here, if EKS allowed you to pre-fix the K8s API endpoint with something unique, it would make setting up Conditional Forwarders for each of your environments pretty easy from on-prem networks. For instance, I can't setup a conditional forwarder now that will catch and route other than like eks.amazonaws.com, but if I could put something in the name like prod-<unique string>.eks.amazonaws.com then I could resolve things pretty easily for most environments.

[dhineshbabuelango]

Hi @tabern, I tried the below link for DNS resolution from peered VPC but it is working fine.

https://aws.amazon.com/blogs/compute/enabling-dns-resolution-for-amazon-eks-cluster-endpoints/

But even If I create both inbound and outbound endpoints and its security group in the cluster VPC itself and in the rule just associated other VPCs its still working

EKS Cluster in APP VPC
Inbound endpoint and its security group in APP VPC
outbound endpoint and its security group in APP VPC

While creating Rule, just associated other VPCs, the DNS resolution is working fine. Is this a correct approach on creating the inbound and outbound rule both on the cluster residing VPC and just associate the required VPCs just in the rule

[jclynny]

@dhineshbabuelango I actually did the same thing yesterday and after some trials, I finally got on-prem resolving DNS in both VPC's, so my issues are fixed as far as that goes.

[Alien2150]

@tabern is there an ETA? Especially after this cve details have been released: https://thenewstack.io/kubernetes-billion-laughs-vulnerability-is-no-laughing-matter/.

[nodesocket]

Why isn't there an option when making the EKS API endpoint private to specify if you want to use a Route53 public DNS or Route53 private DNS endpoint? Really, there is no benefit to making the API DNS endpoint private in terms of DNS. I'd like to be able to VPN into my VPC, but then still run kubectl locally from my machine. This is currently not possible because of the DNS resolution issue.

[stefansedich]

@nodesocket reading through this issue it looks like it is being changed to be a public DNS entry to better match what other AWS services already do.

[nodesocket]

@nodesocket reading through this issue it looks like it is being changed to be a public DNS entry to better match what other AWS services already do.

Ok thanks. I really don't know why they used a private Route53 zone to begin with. The only thing I can think of is that a public Route53 zone leaks private ip's, but not a big deal if you can't access them.

[ffjia]

When will this feature be released? It's probably not perfect, but already a small win, and definitely better than playing with Route53 inbound/outbound rules.

[nodesocket]

@ffjia agree I would love to know when public DNS is available. I got private Kubernetes API endpoint working by creating a EC2 instance in the same VPC as the EKS cluster. Installing aws-cli, kubectl, helm, tiller, aws-iam-authenticator to the EC2 instance. Adding the Kubernetes config to the EC2 instance into ~/.kube/config. Then finally adding my IAM AWS credentials into the instance using aws configure. Not a fun experience.

_Note: Attaching the default created EKS IAM instance profile to the EC2 instance I created unfortunately does not work since it only has node level read permissions. Thus I was required to store my IAM credentials on the EC2 instance._

[cdenneen]

@dhineshbabuelango I actually did the same thing yesterday and after some trials, I finally got on-prem resolving DNS in both VPC's, so my issues are fixed as far as that goes.

How did you get conditional forwarders to multiple VPC's? without the domain request you had made?

[dhineshbabuelango]

@cdenneen, as long as the all the VPCs in the same account, you can just associate those VPC in the rules page.

[alfianabdi]

@dhineshbabuelango But it only works if all EKS clusters are in one VPC, right?

[dhineshbabuelango]

@alfianabdi If you have your EKS cluster in VPC A, if you want VPC B and VPC to resolve the DNS, you can create create inbound endpoint and outbound endpoint in VPC A and create a rule to map VPC B and VPC, then all the servers in VPC B and VPC C should be able to resolve the DNS. If VPC B and VPC C are in different account, you need to create outbound endpoint and rule from the account where you need to resolve the EKS DNS

[BernhardLenz]

I have a VPC in Google and a VPC in AWS, both connected through a site-to-site vpn tunnel.

The AWS VPC hosts a EKS private cluster as well as a private RDS instance. The AWS VPC has an inbound endpoint and the Google VPC has a DNS Policy with the AWS endpoint IPs as alternate DNS servers plus 8.8.8.8.

With that setup, from a VM in Google I can resolve the ("private") RDS DNS but not the private EKS DNS.
If I remove 8.8.8.8 from the Google DNS Policy the RDS lookup fails too.

This leads me to believe that RDS has a public DNS entry with my private RDS IP, whereas EKS does not seem use a public DNS entry but rather a AWS VPC private DNS entry.

I see that folks in this thread talk about creating an outbound endpoint and a rule. Is this also required when trying to use the private EKS DNS from Google? I was thinking that I only need an inbound endpoint as I'm only sending DNS queries from the Google VPC to the AWS VPC... Is that correct?

[tmattausch]

@alfianabdi you can have a look at my Architecture supporting multiple EKS clusters in multiple AWS accounts by using one central DNS account. But I really would love to see the public DNS feature as soon as possible, especially considering the high costs of route53 endpoints.

[msvechla]

Any update on this? I think this thread already shows that everyone is aware of the different workarounds, however all of them are quite expensive.

Is there any ETA on this feature?

[tdmalone]

@msvechla AWS don't share ETAs on new features. This moved to 'coming soon' on 20 Sep so I'd _guess_ - based on previous shipped features - that this could potentially just be days away.

[stefansedich]

Does anyone have any updates on the ETA for this?

[jrhoward]

why don't you do what Google do with GKE. You can add a flag using the tooling to either retrieve the internal or external endpoint address. It sets up your ~/.kube/config appropriately.

[krzysztof-bronk]

I think the mentioned change (private IPs via public DNS) has been applied already!

I just noticed it after creating a fresh cluster using the latest version.

[CEikermann]

Can confirm, DNS hostnames of EKS private endpoints are now globally resolvable to private IPs

[niroz89]

Yes we are able to resolve also ap-southeast-1.

On Thu, 12 Dec 2019, 19:04 Christian Eikermann, [e-mail]
wrote:

Can confirm, DNS hostnames of EKS Private Endpoints are now globally
resolvable to private IPs

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/aws/containers-roadmap/issues/221?email_source=notifications&email_token=ALCTG3ZJQBKA5KWGR7FMRFDQYILE3A5CNFSM4HBKIZGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGWJYWI#issuecomment-564960345,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALCTG36M6XYCABIBYSHA6JTQYILE3ANCNFSM4HBKIZGA
.

--

NOTE: Privileged/Confidential Information may be contained in this
message. If you are not the addressee indicated in this message (or
responsible for delivery of the message to such person), you may not copy
or deliver this message to anyone. In such case, you should destroy this
message and kindly notify the sender by reply email. All opinions,
conclusions and other information expressed in this message not of an
official nature shall not be deemed as given or endorsed by Singapore Press
Holdings Ltd or any of its subsidiaries unless otherwise indicated by an
authorised representative independent of this message.
Singapore Press
Holdings Ltd Co
Regn No. 198402868E

--

[itsnotapt]

I'm still not able to resolve my private endpoints. Are there any changes that need to be made for this to work? Is it only newly created clusters?

[CEikermann]

Yeah seems to work only newly created clusters

[nodesocket]

Yeah seems to work only newly created clusters

Argg that is annoying. So no way to enable on existing clusters? Seems like this should be an option you can enable in the EKS GUI.

[ac-hibbert]

I am running k8s 1.14 on eks.2 in us-east-1. This does seem to work although I also had to edit my cluster security group. The private IP was resolvable but port 443 was not reachable. I got this message first when running kubectl get pods

Unable to connect to the server: dial tcp X.X.X.X:443: i/o timeout

[krzysztof-bronk]

I guess we should wait for confirmation from @tabern, but in my case it works as I hoped it would.

[CEikermann]

I am running k8s 1.14 on eks.2 in us-east-1. This does seem to work although I also had to edit my cluster security group. The private IP was resolvable but port 443 was not reachable. I got this message first when running kubectl get pods

Unable to connect to the server: dial tcp X.X.X.X:443: i/o timeout

You need to allow the traffic in the "Cluster security group", by default its not allowed

[tabern]

We're in the process of deploying this now - which is why you may see it starting to work on some new clusters. Updates shortly!

[tabern]

I'm excited to announce that DNS resolution to the EKS private cluster endpoint is now generally available for all new or newly updated EKS clusters.

Now, when only the private endpoint is enabled, Amazon EKS automatically advertises the private IP addresses of the private endpoint from the public endpoint. Clients (such as the kubectl CLI tool) use the public endpoint as a DNS resolver to connect to the private endpoint through a peered VPC automatically. Since these are always private IPs, clients without access to the private VPC, may receive the IP, but are unable to connect to the cluster.

Private endpoint DNS resolution is available for all newly created Amazon EKS clusters today.
Over the coming month, EKS will update all existing clusters to have automatic private endpoint DNS resolution without requiring any action. We'll update this post as this happens.

Learn More

• What's new post
• Documentation

[KIVagant]

[dmateos]

Still having issues resolving the private endpoint on a few of my clusters, while others are working as expected.

Interestingly its the 1.14 clusters that don't work and the 1.12 that do, but this could just be a coincidence.

[tabern]

@dmateos we're still backfilling across the fleet for existing clusters. Have you tried enabling then disabling the public endpoint? This would force the update for those clusters.

[johnjeffers]

@tabern is there any way to tell (platform version?) whether your cluster has been updated to support this?

[tabern]

It should be automatically enabled with eks.7. However, you can enable it for _any_ cluster immediately by toggling the public/private endpoint settings (turn on public endpoint then disable).

[vikas027]

Thanks @tabern . Just tested out the same, works beautifully.

[vikas027]

Hey @tabern , Is there a way to force an existing cluster to use the latest platform version eks.7?

[tabern]

No, there is not. You'll need to create a new cluster or wait for the cluster to be updated.

[rishabh1635]

I have one question to access EKS Cluster with different VPC network.
I have created one EKS cluster with one VPC-A network and also have another VPC-B where our jenkins server is running and I want to access EKS via jenkins for CICD build process and our jenkins server is running in different VPC-B but here the problems is jenkins doesn’t have public ip and it's running on private subnet with private ip so when I am adding private IP on EKS network to access the Kubernetes endpoint, it is not taking because its a private ip.
so I have done the VPC peering on both side and has opened all port for jenkins server but I am still not able to access eks with private ip is any know any clue

[valorl]

@rishabh1635 Have you made sure the EKS (master) security group allows 443 traffic from that VPC ?

[KIVagant]

@rishabh1635 and also it's not clear if all necessary route tables are configured between the peered networks and if there are some ACL rules. Please double-check all common potential issues first.

[vivekanandg]

This seems to work if "only private" endpoints are enabled and not if both public and private are enabled.

My usecase:
Enable both private and public (a set of whitelisted ips) endpoints.
Access the API server from a peered VPC.

In this case the dns resolution doesn't seem to work from the peered VPC. However it works when I disable public endpoint completely. Any workarounds? Did anyone get this working?

[rishabh1635]

@vivekanandg Is there anyway to access EKS with both endpoint that means via private and public endpoint. Because I want to deploy eks pods via jenkins server only with private endpoint and access the kubernetes cluster via public endpoint (a set of whitelisted ips).

[tabern]

@vivekanandg the assumption is that if the public endpoint is enabled, can access the cluster using it. Why do you have a requirement to access the endpoint through a peered VPC when the public is available?

[vivekanandg]

@tabern You are right, vpc peering isn't necessary.

Didn't realise that the nat gateway public ip (Elastic) could be whitelisted. It's working fine now.

[cdenneen]

@tabern I’m curious based on this scenario. If you have an account with VPC that’s completely isolated except for VPC peering. The second account has EKS with private and public. Then the first account would need the private IP for resolution since it wouldn’t have public gateway egress. Right?

[alexandrem]

I'd like to understand what happens if the selected subnet zone for the kube api endpoint private ip has an outage. Will the DNS entry resolves dynamically to another zone private ip? Is my cluster endpoint resilient to failures within a zone?

I can't find the information about that.