Kubeadm: Internal dns entries for externalNames stopped working in v1.13.3 ?

[sokoow]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:05:53Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  $ kubectl version Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

What happened?

At some stage in kubeadm, 2-3 versions back resolving externalNames on internal kube dns stopped working.

What you expected to happen?

This used to work pretty well.

How to reproduce it (as minimally and precisely as possible)?

Create an externalName svc, try to resolve it from any pod.

Anything else we need to know?

There are workarounds, like creating an endpoint, but it's awkward.

[bart0sh]

Create an externalName svc, try to resolve it from any pod.

can you give exact steps to reproduce the issue please?

[sokoow]

Sure:

1. Initialize the cluster using kubeadm init
2. add overlay manifest
3. add this externalName manifest:

apiVersion: v1
kind: Service
metadata:
   name: postgres1-rds
   labels:
     k8s-app: postgres1-rds
     kubernetes.io/name: "postgres1-rds"
spec:
   type: ExternalName
   externalName: 10.16.2.8
   ports:
     - port: 5432
       targetPort: 5432

1. Try to use this externalName from any other pod by calling postgres1-rds.default:5432

[bart0sh]

thanks. Can you elaborate on items 2 and 4 a bit more?

[bart0sh]

This used to work pretty well.

in which k8s release did it work for you last time?

[sokoow]

1.10.something - but maybe we'll need more samples of other people to confirm that.

[neolit123]

2. add overlay manifest

which CNI plugin are you using?
have you tried with a different one?

[sokoow]

I tried flannel and weave already, same problem.

[sokoow]

what works is this workaround that I found on internetz:

apiVersion: v1
kind: Service
metadata:
  name: postgres1-rds
  namespace: default
spec:
  clusterIP: None
  ports:
  - name: db
    port: 5432
    protocol: TCP
---
kind: Endpoints
apiVersion: v1
metadata:
  name: postgres1-rds
  namespace: default
subsets:
  - addresses:
      - ip: 10.16.10.20
    ports:
      - port: 5432
        name: db
        protocol: TCP

[neolit123]

what happens if you try using kube-dns instead of coredns?
some information how to do that in 1.13 is at the bottom of this section:
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-addon

also see ClusterConfiguration:
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1

[chenzhiwei]

ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName is intended to specify a canonical DNS name.

https://kubernetes.io/docs/concepts/services-networking/service/#externalname

[neolit123]

@chenzhiwei
could you explain what you are trying to say by quoting:
https://kubernetes.io/docs/concepts/services-networking/service/#externalname

[neolit123]

the user is claiming that it used to work.
but i'm tempted to close this one due to no response.

[aojea]

@neolit123 I think that @chenzhiwei wants to mention that´s the expected behavior, it accepts the IPv4 address as DNS name but doesn´t resolve the address

Note: ExternalName accepts an IPv4 address string, but as a DNS name comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName is intended to specify a canonical DNS name. To hardcode an IP address, consider headless services

that honestly seems a reason to close the issue, since is the expected behavior of CoreDNS. Checking the docs seems that CoreDNS is default since 1.13, that can explain why it worked before

In Kubernetes version 1.13 and later the CoreDNS feature gate is removed and CoreDNS is used by default.

[neolit123]

@aojea

it accepts the IPv4 address as DNS name but doesn´t resolve the address

ok, this makes sense.

that honestly seems a reason to close the issue, since is the expected behavior of CoreDNS. Checking the docs seems that CoreDNS is default since 1.13, that can explain why it worked before

that part that confuses me is that we defaulted CoreDNS back in kubeadm 1.11.
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#sig-cluster-lifecycle

but i'm going to close this.
thanks for the details.