Minikube: hyperkit conflict with local DNS daemon (dns: read udp src->dst:53: read: connection refused)

[dahaili]

Minikube version (use minikube version): v0.33.1

• OS (e.g. from /etc/os-release): macOS Mojave version 10.14.2
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): hyperkit
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): minikube-v0.33.1.iso
• Install tools: brew
• Others:

Command "minkube start --vm-driver hyperkit" fails with the following messages:

Starting local Kubernetes v1.13.2 cluster...
Starting VM...
Getting VM IP address...
E0123 16:14:34.829156 21704 start.go:243] Error parsing version semver: Version string empty
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Stopping extra container runtimes...
Starting cluster components...
E0123 16:14:37.615834 21704 start.go:376] Error starting cluster: kubeadm init:
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI

[init] Using Kubernetes version: v1.13.2
[preflight] Running pre-flight checks
[WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.7:42770->192.168.64.1:53: read: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.7:45819->192.168.64.1:53: read: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.7:46142->192.168.64.1:53: read: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.7:36363->192.168.64.1:53: read: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.7:37734->192.168.64.1:53: read: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.7:35622->192.168.64.1:53: read: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.7:58397->192.168.64.1:53: read: connection refused
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...

: Process exited with status 1

[tstromberg]

Is it possible your host has a firewall or proxy interfering? It's not clear what's going on here except:

• The VM is configured to use 192.168.64.1 as the DNS server
• 192.168.64.1 isn't configured to answer DNS queries

I'm not yet familiar with how hyperkit implements DNS, so I'm not sure what else might be going wrong. As a temporary workaround, you can try VirtualBox.

Help wanted!

Tangentially related: #3145

[slayerjain]

I'm also experiencing this issue on macOS 10.14.3, Minikube 0.33.1 with hyperkit driver

Using the virtualbox driver for now.

[jopicornell]

I'm also experiencing this issue on macOS 10.14.3, Minikube 0.35.0 with hyperkit and xhyve drivers.

With Virtualbox driver is working correctly.

[tstromberg]

My interpretation of this message is that the hyperkit DNS forwarder isn't available, possibly due to a port conflict, perhaps with another DNS daemon. If you run into this, do you mind sharing the output of:

sudo lsof -ni:53

I've tried to search for documentation on how hyperkit's DNS implementation works, but haven't come up with much: it all seems to be VPNKit magic. Anyone have any tips?

[mhumeSF]

I was running into this same issue because I was running dnsmasq. sudo brew services stop dnsmasq solved things for me.

[jopicornell]

@tstromberg Very good catch! I was running dnsmasq as well. lsof command shows dnsmasq instances. I think I will stick to default driver for the moment. Btw we should close this issue and if someone want this fixed, open it on the hyperkit driver repo.

[pfremm]

Also having this issue. I am not using dnsmasq, and I don't have anything running on port 53. Appears for me hyperkit dns server is never starting.

[MindTooth]

This seems to also occur with DNSCrypt Proxy as well. Would love for a way to make minikube work with DNSMasq, DNSCrypt Proxy, etc.

[tstromberg]

Related bug: #3036 (hyperkit + dnsmasq)

[tstromberg]

If you run into this problem, please try upgrading to the latest minikube hyperkit driver:

curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit && sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/

Then run minikube delete to remove old state. Thanks!

[MindTooth]

Initial attempt seems to work. Did get the fail to pull message, but it seems to solve itself, and it now works. 👍

Ed1t: heim did not go well. Not sure if it's related.

Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.14.0": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 192.168.65.1:53: read udp 192.168.65.3:37754->192.168.65.1:53: read: connection refused

[indrayam]

It did not work for me.

Hyperkit version:

hyperkit -v
hyperkit: 0.20180403

Homepage: https://github.com/docker/hyperkit
License: BSD

MacOS version: 10.14.5

minikube profile knative

minikube start -p knative --memory=8192 --cpus=4 \
  --kubernetes-version=v1.12.0 \
  --vm-driver=hyperkit \
  --disk-size=50g \
  --extra-config=apiserver.enable-admission-plugins="LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook"

✅  minikube profile was successfully set to knative
😄  minikube v1.1.0 on darwin (amd64)
💿  Downloading Minikube ISO ...
 131.28 MB / 131.28 MB [============================================] 100.00% 0s
🔥  Creating hyperkit VM (CPUs=4, Memory=8192MB, Disk=50000MB) ...
🐳  Configuring environment for Kubernetes v1.12.0 on Docker 18.09.6
    ▪ apiserver.enable-admission-plugins=LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook
💾  Downloading kubeadm v1.12.0
💾  Downloading kubelet v1.12.0
🚜  Pulling images ...
❌  Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.12.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.4:37133->192.168.64.1:53: read: connection refused
, error: exit status 1
: Process exited with status 1
🚀  Launching Kubernetes ...
⌛  Verifying: apiserver proxy etcd scheduler controller dns
🏄  Done! kubectl is now configured to use "knative"

When I run mk ssh and do a nslookup k8s.gcr.io, it fails.

Here's my lsof output:

sudo lsof -n -i :53

COMMAND     PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
dnscrypt- 24368 nobody   62u  IPv4 0x7507542b6aa22d77      0t0  UDP 127.0.0.1:domain
dnscrypt- 24368 nobody   63u  IPv4 0x7507542b845a12ff      0t0  TCP 127.0.0.1:domain (LISTEN)

[MindTooth]

Adding the IP to dnscrypt-proxy solved it, it seems. I feel stupid. I hope that this might help others. Also, maybe it should have been added to the hyperkit page. I'll see if I submit a PR for it. 😄

[indrayam]

Ok, so I cannot figure out how to add the IP (I am assuming Minikube IP) to dnscrypt-proxy.

Here's what I am seeing:

sudo lsof -n -i :53

COMMAND     PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
dnscrypt- 85639 nobody   62u  IPv4 0x7507542b71a3d337      0t0  UDP 127.0.0.1:domain
dnscrypt- 85639 nobody   63u  IPv4 0x7507542b5c25c5ff      0t0  TCP 127.0.0.1:domain (LISTEN)

Then I did this...

pstree | grep -i 85639
 | |   \--- 85639 nobody /opt/cisco/anyconnect/bin/dnscrypt-proxy --user nobody --local-address=127.0.0.1:53 --plugin=/opt/cisco/anyconnect/lib/libdcplugin_erc.so -d 0101A75A568AB7E5
 |       \--- 14399 anasharm grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn -i 85639

Not sure where specifically do I go to "add the IP". Also, since the dnscrypt here was installed by Cisco AnyConnect, not sure if I have a lot of wiggle room.

Thoughts?

[tstromberg]

This is still a popular support issue with no way to workaround it other than modifying the configuration of the local DNS daemon.

[tstromberg]

This issue appears to be semi-duplicate of #3036, do you mind if we move the conversation there?

Ths way we can centralize the content relating to the issue. If you feel that this issue is not in fact a duplicate, please re-open it using /reopen. If you have additional information to share, please add it to the new issue.

Thank you for reporting this!

[dahaili]

Please go ahead.

-Dahai