Minikube: hyperkit: no internet connectivity, can't pull kube-addon-manager

[douglascamata]

The exact command to reproduce the issue:

minikube start

The full output of the command that failed:

😄  minikube v1.5.2 on Darwin 10.14.3
✨  Automatically selected the 'hyperkit' driver (alternates: [virtualbox])
💾  Downloading driver docker-machine-driver-hyperkit:
    > docker-machine-driver-hyperkit.sha256: 65 B / 65 B [---] 100.00% ? p/s 0s
    > docker-machine-driver-hyperkit: 10.79 MiB / 10.79 MiB  100.00% 636.16 KiB
🔑  The 'hyperkit' driver requires elevated permissions. The following commands will be executed:

    $ sudo chown root:wheel /Users/dcamata/.minikube/bin/docker-machine-driver-hyperkit
    $ sudo chmod u+s /Users/dcamata/.minikube/bin/docker-machine-driver-hyperkit

🔥  Creating hyperkit VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...
⚠️  VM may be unable to resolve external DNS records
⚠️  VM is unable to access k8s.gcr.io, you may need to configure a proxy or set --image-repository
🐳  Preparing Kubernetes v1.16.2 on Docker '18.09.9' ...
🚜  Pulling images ...
❌  Unable to pull images, which may be OK: running cmd: "/bin/bash -c \"sudo env PATH=/var/lib/minikube/binaries/v1.16.2:$PATH kubeadm config images pull --config /var/tmp/minikube/kubeadm.yaml\"": Process exited with status 1
🚀  Launching Kubernetes ...
⌛  Waiting for: apiserverj
🏄  Done! kubectl is now configured to use "minikube"

The output of the minikube logs command:

Huge spam of this:

Nov 19 18:03:53 minikube kubelet[2721]: E1119 18:03:53.519954    2721 pod_workers.go:191] Error syncing pod c3e29047da86ce6690916750ab69c40b ("kube-addon-manager-minikube_kube-system(c3e29047da86ce6690916750ab69c40b)"), skipping: failed to "StartContainer" for "kube-addon-manager" with ImagePullBackOff: "Back-off pulling image \"k8s.gcr.io/kube-addon-manager:v9.0.2\""
Nov 19 18:04:07 minikube kubelet[2721]: E1119 18:04:07.531993    2721 remote_image.go:113] PullImage "k8s.gcr.io/kube-addon-manager:v9.0.2" from image service failed: rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:60592->192.168.64.1:53: read: connection refused
Nov 19 18:04:07 minikube kubelet[2721]: E1119 18:04:07.532050    2721 kuberuntime_image.go:50] Pull image "k8s.gcr.io/kube-addon-manager:v9.0.2" failed: rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:60592->192.168.64.1:53: read: connection refused
Nov 19 18:04:07 minikube kubelet[2721]: E1119 18:04:07.532087    2721 kuberuntime_manager.go:783] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:60592->192.168.64.1:53: read: connection refused
Nov 19 18:04:07 minikube kubelet[2721]: E1119 18:04:07.532111    2721 pod_workers.go:191] Error syncing pod c3e29047da86ce6690916750ab69c40b ("kube-addon-manager-minikube_kube-system(c3e29047da86ce6690916750ab69c40b)"), skipping: failed to "StartContainer" for "kube-addon-manager" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:60592->192.168.64.1:53: read: connection refused"
Nov 19 18:04:18 minikube kubelet[2721]: E1119 18:04:18.518574    2721 pod_workers.go:191] Error syncing pod c3e29047da86ce6690916750ab69c40b ("kube-addon-manager-minikube_kube-system(c3e29047da86ce6690916750ab69c40b)"), skipping: failed to "StartContainer" for "kube-addon-manager" with ImagePullBackOff: "Back-off pulling image \"k8s.gcr.io/kube-addon-manager:v9.0.2\""

The operating system version: macOS 10.14.3

[douglascamata]

Quick note: changing the driver from hyperkit to virtualox makes it work.

[douglascamata]

I tried SSHing into the machine and effectively it cannot even ping google.com.

[tstromberg]

https://github.com/kubernetes/minikube/pull/5935 will fix the addon-manager, but as you mentioned, the root cause is that your hyperkit installation apparently does not have any network connectivity. It is very interesting that switching to virtualbox makes it work however.

Do you have any special VPN or Docker for Desktop configuration on this host by any chance?

Thanks!

[douglascamata]

Hi @tstromberg!

Do you have any special VPN or Docker for Desktop configuration on this host by any chance?

I have a standard Docker for Mac installation, nothing special. Sometimes I use corporate VPNs (even a VPN over another VPN).

I always disconnect all the VPNs when using minikube, as I know they don't place nice together.

If there are any commands you would like me to run to give you more information I'm happy to help.

[smoya]

Same issue here.

➜ minikube start
😄  minikube v1.5.2 on Darwin 10.15.1
💾  Downloading driver docker-machine-driver-hyperkit:
    > docker-machine-driver-hyperkit.sha256: 65 B / 65 B [---] 100.00% ? p/s 0s
    > docker-machine-driver-hyperkit: 10.79 MiB / 10.79 MiB  100.00% 4.88 MiB p
🔑  The 'hyperkit' driver requires elevated permissions. The following commands will be executed:

    $ sudo chown root:wheel /Users/smoya/.minikube/bin/docker-machine-driver-hyperkit
    $ sudo chmod u+s /Users/smoya/.minikube/bin/docker-machine-driver-hyperkit


Password:
💿  Downloading VM boot image ...
    > minikube-v1.5.1.iso.sha256: 65 B / 65 B [--------------] 100.00% ? p/s 0s
    > minikube-v1.5.1.iso: 143.76 MiB / 143.76 MiB [-] 100.00% 6.37 MiB p/s 23s
🔥  Creating hyperkit VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...
⚠️  VM may be unable to resolve external DNS records
⚠️  VM is unable to access k8s.gcr.io, you may need to configure a proxy or set --image-repository
🐳  Preparing Kubernetes v1.16.2 on Docker '18.09.9' ...
💾  Downloading kubeadm v1.16.2
💾  Downloading kubelet v1.16.2
🚜  Pulling images ...
❌  Unable to pull images, which may be OK: running cmd: "/bin/bash -c \"sudo env PATH=/var/lib/minikube/binaries/v1.16.2:$PATH kubeadm config images pull --config /var/tmp/minikube/kubeadm.yaml\"": Process exited with status 1
🚀  Launching Kubernetes ...
⌛  Waiting for: apiserver
🏄  Done! kubectl is now configured to use "minikube"

➜ kubectl get pods --all-namespaces
NAMESPACE     NAME                          READY   STATUS             RESTARTS   AGE
kube-system   coredns-5644d7b6d9-2p2s9      1/1     Running            0          67s
kube-system   coredns-5644d7b6d9-4c6k7      1/1     Running            0          67s
kube-system   etcd-minikube                 1/1     Running            0          18s
kube-system   kube-addon-manager-minikube   0/1     ImagePullBackOff   0          68s
kube-system   kube-apiserver-minikube       1/1     Running            0          9s
kube-system   kube-proxy-8fsj6              1/1     Running            0          67s
kube-system   kube-scheduler-minikube       1/1     Running            0          10s

The error:

Failed to pull image "k8s.gcr.io/kube-addon-manager:v9.0.2": rpc error: code = Unknown desc = Error response from daemon
: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:58839->192.168.64.1:53: read: connection refused

[Jorijn]

I found this issue because I was experiencing the same problem. I also found the culprit:

When SSH'ing into the machine I found out it's using the host machine's IP as resolving IP. When using nslookup I first got connection refused, then connection timeout. This makes no sense, but it triggered me for something something else:

For local development I've been using dnsmasq, so in fact some other DNS server was running as opposed to the one hyperkit needs. When stopping dnsmasq, setting up a new minikube instance worked as intended. ✅

[douglascamata]

@Jorijn you think maybe then this could be related to internal DNS configuration/servers used by VPN software? I have both Cisco AnyConnect and Tunnelblick installed.

[Jorijn]

@douglascamata it might! you could try disconnecting VPN connections, quitting those programs and trying again.

[douglascamata]

@Jorijn I tried this already. I always disconnect and close all VPN software when using minikube (even before this problem), as I know they don't place nice together.

[bzub]

I see this issue as well. I believe it has something to do with the Blue Coat Unified Agent" installed on my corporate laptop. Do others having this issue see a bcua-service or symbol on their taskbar? I'm on MacOS.

For now as mentioned above I use minikube start --vm-driver=virtualbox to work around the problem.

[tycooon]

I have the same problem when connected to a corporate VPN via Tunnelblick. In case I turn it off the network inside minikube ssh is working just fine.

[fhopfensperger]

I have the same problem when connected to a corporate VPN via Cisco AnyConnect. Is there a way to configure hyperkit to use a different network connection?

[priyawadhwa]

I'm not sure how to fix the network connection issue, but the latest version of minikube no longer pulls kube-addon-manager. If you upgrade to our latest release then that should resolve a piece of this issue.

[medyagh]

@douglascamata I haven't heard from you I will close this issue, and I am sorry that you faced this isssue, please feel free to reopen this issue
I also suggest to try our new driver "docker" with latest version of minikbue which might solve your issue.

minikube start --vm-driver=docker

[nomansadiq11]

restart network adapter and its works for me