Minikube: coredns unable to resolve external DNS (NXDOMAIN)

[darkedges]

Having an issue with 1.0.0 where it cannot resolve external DNS entries, causing cert-manager to return errors as it cannot locate the domain to issue a certificate against.

By editing the configmap for coredns and updating to use

forward . 8.8.8.8:53

It can then resolve externally

Minikube version

$ minikube version
minikube version: v1.0.0

Operating System

Microsoft Windows [Version 10.0.17763.437]
(c) 2018 Microsoft Corporation. All rights reserved.

Deployed

kubectl -n kube-system edit configmap coredns

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  Corefile: |
    .:53 {
        log
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv/conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2019-04-26T20:43:46Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "5425"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: fcad6502-6863-11e9-b838-0800276aaa26

command

/ # nslookup www.news.com.au
Server:         10.96.0.10
Address:        10.96.0.10:53

** server can't find www.news.com.au: NXDOMAIN


*** Can't find www.news.com.au: No answer

Updated

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  Corefile: |
    .:53 {
        log
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . 8.8.8.8:53
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2019-04-26T20:43:46Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "5425"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: fcad6502-6863-11e9-b838-0800276aaa26

command

/ # nslookup www.news.com.au
Server:         10.96.0.10
Address:        10.96.0.10:53

Non-authoritative answer:
www.news.com.au canonical name = wildcardsan.news.com.au.edgekey.net
wildcardsan.news.com.au.edgekey.net     canonical name = e3774.b.akamaiedge.net

*** Can't find www.news.com.au: No answer

[darkedges]

Seems this is documented here https://github.com/coredns/coredns/blob/master/plugin/loop/README.md in the last section

[jaksonwkr]

I have the same problem here.

The systemd-resolved just stop working.

[darkedges]

The systemd-resolved just stop working.

Where were you seeing that?

[jaksonwkr]

Where were you seeing that?

I accessed the minikube VM and tried to ping google.com, it didn't work, then I changed the file /etc/systemd/resolved.com and added the line below

DNS = 8.8.8.8

restarted systemd-resolved and it came back to work

[jbialy]

Same result for me, I first noticed that deployments were unable to pull images. I sshed into the minikube VM and checked /etc/resolv.conf:

cat /etc/resolv.conf
nameserver 192.168.64.1

Adding:

[Resolve]
DNS=1.1.1.1

to /etc/systemd/resolved.conf and restarting systemd-resolved helped.

[rodjjo]

I created this script that is a workaround for minikube

#!/bin/bash
kubectl -n kube-system get configmap coredns -o yaml | sed 's/\/etc\/resolv.conf/8.8.8.8/gi' | kubectl apply -f -
PODNAMES=(`kubectl -n kube-system get pods -o jsonpath='{.items[*].metadata.name}'`)
for name in ${PODNAMES[@]}; do
    if echo "$name" | grep -q 'coredns-'; then
        kubectl -n kube-system delete pods "$name"
    fi
done

note: it deletes coredns pods. they will be automatically created again with the new dns configurations

[j14s]

While this does fix things, if you do a minikube stop and start, the /etc/systemd/resolved.com gets reset to default and everything is broken again. Rinse/Repeat.

[dfang]

Where were you seeing that?

I accessed the minikube VM and tried to ping google.com, it didn't work, then I changed the file /etc/systemd/resolved.com and added the line below

DNS = 8.8.8.8

restarted systemd-resolved and it came back to work

tried to change DNS to 223.5.5.5 （aliyun dns), and restarted systemd-resolved
but still failed to nslookup

nslookup baidu.com
Server:    223.5.5.5
Address 1: 223.5.5.5
nslookup: can't resolve 'baidu.com'

[fsadykov]

Hello @rodjjo,

I see your code is works for coredns. I have same problem with kube-dns. Should I just replace coredns -> kube-dns and use your script?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[achyutjhunjhunwala]

/remove-lifecycle stale

[tstromberg]

I'm closing this as it was opened a really long time ago (a year), and wasn't reproducible for me given the same information on minikube v1.9.2:

# kubectl run -it --rm --restart=Never alpine --image=alpine sh
If you don't see a command prompt, try pressing enter.
/ # nslookup www.news.com.au
Server:     10.96.0.10
Address:    10.96.0.10:53

Non-authoritative answer:
www.news.com.au canonical name = wildcardsan.news.com.au.edgekey.net
wildcardsan.news.com.au.edgekey.net canonical name = e3774.b.akamaiedge.net
Name:   e3774.b.akamaiedge.net
Address: 23.1.244.154

If you are still seeing this, /reopen with precise reproduction steps, including the minikube start command-line and output. Thank you for sharing your experience - my apologies that we did not follow-up sooner.