Minikube: dnsmasq issues on .19.[01]

[klaus]

Minikube version (use minikube version):
minikube version: v0.19.1

Environment:

• OS (e.g. from /etc/os-release):
  debian: 9.0
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName):
  kvm
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION):
  v0.18.0
• Install tools:
  minikube start
• Others:

What happened:
dns is not / partially working. it has very slow resolve times and hangs. when i exec into any container, it is takes >30 seconds to resolve e.g. google.com (getent hosts google.com). Sometimes, I am experiencing a timeout alltogether.
sometimes, resolving just works in 'bursts' it seems.
internal resolving (cluster local addresses is also affected e.g. redis.default.svc.cluster.local.)

What you expected to happen:
well, dns should work instantly and consistently. It is needed to wire up all my services.

How to reproduce it (as minimally and precisely as possible):
I can minikube start, and then kubectl exec -ti anyofmyimages bash, getent hosts anyhost

Anything else do we need to know:
minikube logs:

Jun 08 16:40:18 minikube localkube[9761]: E0608 16:40:18.563151    9761 remote_runtime.go:163] ListPodSandbox with filter "nil" from runtime service failed: rpc error: code = 2 desc = Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Jun 08 16:40:18 minikube localkube[9761]: E0608 16:40:18.563249    9761 kuberuntime_sandbox.go:185] ListPodSandbox failed: rpc error: code = 2 desc = Cannot connect to the Docker daemon. Is the docker daemon running on this host?

strangely many restarts on kube-dns

kubectl -n kube-system get pods                                            :(
NAME                               READY     STATUS    RESTARTS   AGE
kube-addon-manager-minikube        1/1       Running   3          7h
kube-dns-196007617-8x8s7           3/3       Running   69         7h
kubernetes-dashboard-50njm         1/1       Running   3          7h
traefik-ingress-controller-405d2   1/1       Running   3          6h

Jun 08 16:40:18 minikube localkube[9761]: E0608 16:40:18.563286    9761 generic.go:198] GenericPLEG: Unable to retrieve pods: rpc error: code = 2 desc = Cannot connect to the Docker daemon. Is the docker daemon running on this host?

and strangely totally empty:

stern --all-namespaces kube-dns -c dnsmasq

others are experiencing the same: https://kubernetes.slack.com/archives/C1F5CT6Q1/p1496933436345999

Thank You!!, totally love minikube!

[r2d4]

I think that we should have a policy of sticking to the same version that GKE is pinned to. We've seen a lot of regressions in the upstream DNS addon in regards to stability.

[klaus]

@r2d4 does that hide a solution to my specific issue? like somehow using an older version for the kube-dns pod?

[r2d4]

I investigated and my test GKE cluster is using 1.14.1 - which is fairly new. It looks like we might have an issue on the minikube side regardless.

[fabiand]

Yes, I am seeing the same issue:

kube-system   kube-dns-268032401-p19s8                  3/3       Running   61         1d

and logs

I0610 19:15:06.992552       1 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --log-facility=- --server=127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000}
I0610 19:15:06.992731       1 nanny.go:86] Starting dnsmasq [-k --cache-size=1000 --log-facility=- --server=127.0.0.1#10053]
I0610 19:15:07.001148       1 nanny.go:111] 
W0610 19:15:07.001172       1 nanny.go:112] Got EOF from stdout
I0610 19:15:07.001203       1 nanny.go:108] dnsmasq[10]: started, version 2.76 cachesize 1000
I0610 19:15:07.001237       1 nanny.go:108] dnsmasq[10]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
I0610 19:15:07.001287       1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053
I0610 19:15:07.001334       1 nanny.go:108] dnsmasq[10]: reading /etc/resolv.conf
I0610 19:15:07.001368       1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053
I0610 19:15:07.001399       1 nanny.go:108] dnsmasq[10]: using nameserver 192.168.122.1#53
I0610 19:15:07.001462       1 nanny.go:108] dnsmasq[10]: read /etc/hosts - 7 addresses

[r2d4]

This should be fixed by #1577