Coredns: CoreDNS does not resolve short names

[marcin54321]

Hello Guys,
I've come across the following issue. I have an exemplary kubernetes service hostnames.

[root@ccm ~]# kubectl get svc -n kube-system -o wide
NAME        TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE       SELECTOR
hostnames   ClusterIP   10.100.1.23   <none>        80/TCP          20h       app=hostnames
kube-dns    ClusterIP   10.100.1.10   <none>        53/UDP,53/TCP   24d       k8s-app=kube-dns

I have 2 coredns pods.

[root@ccm ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                READY     STATUS             RESTARTS   AGE       IP             NODE            NOMINATED NODE
kube-system   coredns-8699c7dcfb-4t4qs            1/1       Running            9          20h       10.100.0.29    bm-node0.owc3   <none>
kube-system   coredns-8699c7dcfb-jd2kc            1/1       Running            7          20h       10.100.0.3     ccm.owc3        <none>

When I query a coredns pod with the FQDN about 'hostnames' service I get back the proper answer of the IP address.

bash-4.4# dig @10.100.0.29 hostnames.kube-system.svc.cluster.local +tcp

; <<>> DiG 9.12.2-P1 <<>> @10.100.0.29 hostnames.kube-system.svc.cluster.local +tcp
; (1 server found)
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10301
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;hostnames.kube-system.svc.cluster.local. IN A

;; ANSWER SECTION:
hostnames.kube-system.svc.cluster.local. 3 IN A 10.100.1.23

;; Query time: 0 msec
;; SERVER: 10.100.0.29#53(10.100.0.29)
;; WHEN: Tue Dec 18 09:17:45 UTC 2018
;; MSG SIZE  rcvd: 112

When I query the coredns pod with the short name about 'hostnames' I don't get back the IP address of the service.

bash-4.4# dig @10.100.0.29 hostnames.kube-system +tcp

; <<>> DiG 9.12.2-P1 <<>> @10.100.0.29 hostnames.kube-system +tcp
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 35658
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: accabe8ef435f368 (echoed)
;; QUESTION SECTION:
;hostnames.kube-system.     IN  A

;; Query time: 2024 msec
;; SERVER: 10.100.0.29#53(10.100.0.29)
;; WHEN: Tue Dec 18 09:18:18 UTC 2018
;; MSG SIZE  rcvd: 62

Here's the log from the coredns pod:

[root@ccm ~]# kubectl logs coredns-8699c7dcfb-4t4qs
2018-12-18T09:18:02.703Z [INFO] 10.100.0.46:46453 - 63916 "A IN hostnames.kube-system. tcp 63 false 65535" SERVFAIL qr,rd 63 12.023994907s
 [ERROR] plugin/errors: 0 hostnames.kube-system. A: unreachable backend: no upstream host
2018-12-18T09:18:02.704Z [INFO] 10.100.0.46:58489 - 11982 "A IN hostnames.kube-system. tcp 63 false 65535" SERVFAIL qr,rd 63 2.024081606s
 [ERROR] plugin/errors: 0 hostnames.kube-system. A: unreachable backend: no upstream host
2018-12-18T09:18:04.695Z [INFO] 10.100.0.46:49093 - 32274 "A IN hostnames.kube-system. tcp 63 false 65535" SERVFAIL qr,rd 63 0.000142869s
 [ERROR] plugin/errors: 0 hostnames.kube-system. A: unreachable backend: no upstream host
2018-12-18T09:18:18.975Z [INFO] 10.100.0.46:59268 - 14792 "A IN hostnames.kube-system. tcp 63 false 65535" SERVFAIL qr,rd 63 12.024694872s
 [ERROR] plugin/errors: 0 hostnames.kube-system. A: unreachable backend: no upstream host
2018-12-18T09:18:18.976Z [INFO] 10.100.0.46:58443 - 35658 "A IN hostnames.kube-system. tcp 63 false 65535" SERVFAIL qr,rd 63 2.02487151s
 [ERROR] plugin/errors: 0 hostnames.kube-system. A: unreachable backend: no upstream host

Can you please advise what needs to be configured to have coredns response with an IP address for FQDNs as well as for short names?

System info

• kubectl version

[root@ccm ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:08:19Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

• image version

Image:         gcr.io/google_containers/coredns:1.2.6

• coredns configmap

apiVersion: v1
data:
  Corefile: ".:53 {\n    log\n    errors\n    health\n    kubernetes cluster.local
    in-addr.arpa ip6.arpa {\n      pods insecure\n      upstream\n      fallthrough
    in-addr.arpa ip6.arpa\n    }\n    prometheus :9153    \n    proxy . /etc/resolv.conf\n
    \   loadbalance\n    cache 30\n    reload\n}\n"
kind: ConfigMap
metadata:
  creationTimestamp: 2018-11-23T12:58:13Z
  name: coredns
  namespace: kube-system
  resourceVersion: "2864603"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 6fd61ab4-ef1f-11e8-a638-525400727db8

[fturib]

I think you are missing the search options on your client pod
I mean options in the /etc/resolv.conf file that extend search to

• "cluster.local"
• "svc.cluster.local"

[fturib]

I found an example of resolve.conf file you should have on your client pod (the one from which your run the query).

/ # cat /etc/resolv.conf
nameserver 100.64.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5 

NOTE: if you are running your dig command from outside the cluster, then your need to specify the full domain you are searching. I mean it is expected, in the case your are out of cluster, that you query hostnames.kube-system.svc.cluster.local

[johnbelamaric]

Dig does not automatically follow the search path. You need to add the +search
option.

On Tue, Dec 18, 2018 at 8:34 AM Francois Tur [e-mail]
wrote:

I found an example of resolve.conf file you should have on your client pod
(the one from which your run the query).

/ # cat /etc/resolv.conf
nameserver 100.64.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

NOTE: if you are running your dig command from outside the cluster, then
your need to specify the full domain you are searching.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/coredns/coredns/issues/2407#issuecomment-448283757,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJB4s3uozJW-adzqM1HYom-gNT__C6iUks5u6RkngaJpZM4ZX-aU
.

[marcin54321]

Big thanks for your replies. The +search option helped when using dig. But what about other apps that are deployed on pods, do I need to ensure each of them is configured to use search option?

[chrisohaver]

do I need to ensure each of them is configured to use search option?

No, it is generally default behavior of dns clients to use the search domains. dig is an exception, since it's primarily a troubleshooting tool.