Minikube: Internet DNS resolving does not work on KVM

[bronger]

I observe the following with minikube 0.15.0 and 0.16.0.

If I create a minikube instance with minikube --vm-driver=kvm start and deploy a sample pod with kubectl run diagnosis --image=bronger/diagnosis, I have no Internet DNS resolving in the pod. For example,

kubectl exec -t diagnosis-3343753269-xxxxx -- ping www.heise.de

fails with

ping: unknown host

The container has only 10.0.0.10 in its resolv.conf. Pinging from the KVM machine itself works. Pinging from within containers created directly with docker on the KVM machine also works.

[bronger]

I've now observed that it is perfectly alternating: Pinging from within the pod works every other time I tear down the minikube and start it again.

[bronger]

If I remove ~/.minikube after having torn down the minikube, it is non-working every time. On the other hand, only the logs and the TLS-related things are different between calls of minikube.

[GheRivero]

I can confirm this issue. Working on it

[GheRivero]

Using virtualbox as the vm-driver doesn't have this problem. Must be kvm related

[GheRivero]

Got it!
On kvm, minikube uses the default network (usually NATted) and creates a second network: docker-machines (isolated).
The pods created in the default namespace have the resolv.conf file properly configured (nameserver 10.0.0.10)
But the pods in the kube-dns addon, have two nameserver entries, and the first one (192.168.44.1) is for the isolated address, failing any name resolution.

How to check: login into the kube-dns pods, delete the first nameserver entry in /etc/resolv.conf and restart the service.

[caiobegotti]

@GheRivero err, sorry but how exactly did you do the "and restart the service." part of it?

[caiobegotti]

For the record, at work we noticed SOMETIMES the nameservers entries in /etc/resolv.conf of the pods get reversed:

$ kubectl -n kube-system exec -it kube-dns-v20-9djv6 -- cat /etc/resolv.conf                             
Defaulting container name to kubedns.
Use 'kubectl describe pod/kube-dns-v20-9djv6' to see all of the containers in this pod.
nameserver 192.168.122.1
nameserver 192.168.42.1
options ndots:5

So nameserver 192.168.42.1 is the bad entry as per @GheRivero investigation, it is 2nd here so resolution works:

$ kubectl exec -it busybox -- nslookup google.com                                                        
Server:    10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

Name:      google.com
Address 1: 2800:3f0:4003:c01::65
Address 2: 64.233.190.139 ce-in-f139.1e100.net
Address 3: 64.233.190.101 gsapartner1.google.com
Address 4: 64.233.190.100 gsademo19.google.com
Address 5: 64.233.190.113 ce-in-f113.1e100.net
Address 6: 64.233.190.102 gsapartner2.google.com
Address 7: 64.233.190.138 ce-in-f138.1e100.net

[GheRivero]

Easiest way: kill -1 1

[caiobegotti]

Ok, so after finding out only SOMETIMES the nameserver ordering gets messed up (see above) this is the current work around if you need to keep using the KVM driver:

Put the nameservers in their right order:

kubectl --namespace=kube-system exec -ti $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | sed 's|pod/||') -- sort /etc/resolv.conf -o /etc/resolv.conf

Then restart the service (thanks @GheRivero):

kubectl --namespace=kube-system exec -ti $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | sed 's|pod/||') -- kill -1 1