Containers-roadmap: CoreDNS support for ExternalName services is broken in EKS 1.11

Created on 24 Jan 2019 · 17Comments · Source: aws/containers-roadmap

Environment

Kubernetes version: 1.11
Platform version: eks.1
Cluster DNS provider: CoreDNS
CoreDNS image: 602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/coredns:v1.1.3

Problem

The version of coreDNS being deployed to 1.11 EKS clusters suffers from this issue - https://github.com/coredns/coredns/issues/2038 (repro steps available here)

This issue has been fixed in coredns 1.2.1 but EKS is still running the older version (i.e. 1.1.3) and so ExternalName services that serve as an alias for in-cluster kubernetes services in another namespace cannot currently be resolved.

This problem does not exist in 1.10 i.e. w/ kubedns

EKS

Source

bhang

👍1

Most helpful comment

Thanks for reporting this @bhang. We will get this fixed.

eswarbala on 30 Jan 2019

👍6

All 17 comments

Thanks for reporting this @bhang. We will get this fixed.

eswarbala on 30 Jan 2019

👍6

I am facing the same problem, any workaround? Maybe remove CoreDNS and come back to kube-dns again?

mogaal on 14 Feb 2019

@mogaal we just run newer version of coredns. Version 1.3.1.

spronin-aurea on 14 Feb 2019

👍2

@spronin-aurea so is it fine to use upstream releases there aren't any "compatibility" problems with eks?
/cc @eswarbala

stafot on 26 Feb 2019

@bhang You should now be able to edit your coredns deployment to new image 602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/coredns:v1.2.2. Though note that upstream tested k8s 1.11 with 1.1.3 coredns and is the recommended one for 1.11. So can you try and let us know if new one works?

shyamjvs on 7 Mar 2019

@spronin-aurea so is it fine to use upstream releases there aren't any "compatibility" problems with eks?
/cc @eswarbala

Yes, it is fine. We run it in heavy prod for some time now.

spronin-aurea on 7 Mar 2019

👍1

Ok, I believe we can mark this one fixed then. Re-open as apt.

shyamjvs on 7 Mar 2019

👍1

Yes I used also latest. Seems fine.

stafot on 7 Mar 2019

@shyamjvs Apologies for the delay. Can confirm that switching to the version you indicated solves the problem

bhang on 29 Mar 2019

eksctl 0.1.26 has a eksctl utils update-coredns command now to make updating existing clusters easy.
https://github.com/weaveworks/eksctl/pull/678
https://github.com/weaveworks/eksctl/releases/tag/0.1.26

whereisaaron on 30 Mar 2019

The 1.1.3 manifest lives at https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2019-02-11/dns.yaml

Which we updated to use 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/coredns:v1.2.2

Do we have a repository or tracking issue where we can find the approved manifest as it is updated?

StevenACoffman on 23 Apr 2019

@whereisaaron ref aws/containers-roadmap#266

StevenACoffman on 24 Apr 2019

👍1

I tried to patch my EKS 1.13 coredns deployment as follows:

kubectl patch -n kube-system deployment/coredns --type json --patch '[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"coredns/coredns:1.3.1"}]'

But none of the versions 1.3.1 and 1.4.0 worked (1.6.1 fails to start). I cannot resolve service names of type ExternalName.

The above comments read like upstream releases _should work_. Which steps did you take to fix this?

AndiDog on 5 Aug 2019

@AndiDog, I had a similar issue but doing this resolved it for me: https://github.com/coredns/coredns/issues/2038#issuecomment-465303762

PFacheris on 5 Aug 2019

I'd rather not use workarounds to fix stuff in production, especially on strongly audited systems. Would be very surprising if an application then doesn't work anymore after a cluster version upgrade.

Today, I have upgraded my EKS cluster to 1.14 including setting all the recommended image versions (core-dns etc.), but it still fails to resolve ExternalName services.

apiVersion: v1
kind: Service
metadata:
  name: exampledotcom
spec:
  type: ExternalName
  externalName: 127.0.0.1

Attempt to access from some pod fails:

$ curl exampledotcom.default.svc
curl: (6) Could not resolve host: exampledotcom.default.svc

$ curl -k https://my-non-external-svc.my-namespace.svc:443
<!doctype html>...

AndiDog on 19 Sep 2019

Per https://kubernetes.io/docs/concepts/services-networking/service/#externalname, it has to be a canonical DNS name:

Note: ExternalName accepts an IPv4 address string, but as a DNS names comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName is intended to specify a canonical DNS name. To hardcode an IP address, consider using headless Services.

ctran on 22 Sep 2019

👍1

Indeed, using an FQDN instead of an IP works! On EKS with Kubernetes 1.14 (platform version eks.1) at least, cannot test other versions right now. Maybe other people can verify if this works for them.

apiVersion: v1
kind: Service
metadata:
  name: exampledotcom
spec:
  type: ExternalName
  externalName: example.com

$ curl -sS -H 'Host: example.com' exampledotcom.default | tail

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is established to be used for illustrative examples in documents. You may use this
    domain in examples without prior coordination or asking for permission.</p>
    <p><a href="http://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>

AndiDog on 24 Sep 2019

Was this page helpful?

0 / 5 - 0 ratings