Minikube: Unable to pull images from Docker Hub after v0.23 upgrade

[bschwartz757]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug Report

Please provide the following details:

Environment:

Minikube version (use minikube version): minikube version: v0.23.0

• OS (e.g. from /etc/os-release): From sw_vers: ProductName: Mac OS X ProductVersion: 10.12.6 BuildVersion: 16G29
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName):
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): "DriverName": "xhyve"
• Install tools: Followed mac installation instructions here: used homebrew for kubectl and xhyve, and already had Docker for Mac installed. Previously-working environment used minikube v0.22.6.
• Others:
  The above can be generated in one go with the following commands (can be copied and pasted directly into your terminal):

minikube version
echo "";
echo "OS:";
cat /etc/os-release
echo "";
echo "VM driver": 
grep DriverName ~/.minikube/machines/minikube/config.json

echo "";
echo "ISO version";
grep -i ISO ~/.minikube/machines/minikube/config.json

What happened:

I have been working on a prototype project for work (using Redis-cluster in kubernetes) and had a working local env using minikube. Yesterday while running "brew update" I was prompted to install the minikube update (v0.23); around the same time I decided to uninstall Docker for Mac and re-install using homebrew (I was working on a shell script to automate setup and wanted to see if installing all deps via homebrew would work.)

After re-installing everything, I was able to start the minikube as before (although kubectl -p now shows NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE kube-system kube-addon-manager-minikube 1/1 Running 1 14m 192.168.64.5 minikube kube-system kube-dns-6fc954457d-tv729 3/3 Running 3 14m 172.17.0.3 minikube kube-system kubernetes-dashboard-lt82t 0/1 ImagePullBackOff 0 14m 172.17.0.2 minikube - the dashboard pod previously didn't have this issue and it won't go away.) This isn't an issue for me, but what IS a problem is that now every time I tried to pull a Docker image while using the minikube context it timed out with this error: $ docker pull hello-world Using default tag: latest Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

I also found that the homebrew version of Docker didn't seem to work well at all, I wasn't actually able to use my regular desktop version of Docker with it. I uninstalled that and re-installed Docker for Mac and now my desktop installation works as before - I can pull images from Docker Hub, and build containers etc.

I also tried uninstalling Minikube (there doesn't seem to be a complete Uninstall command anywhere that I can find, and I had to hunt around online to figure out how to completely uninstall) and re-installing v0.22, but still wasn't able to pull images from Docker Hub (although I could confirm that it was hitting https://registry-1.docker.io/v1/.)

After uninstalling everything again and getting back to square 1 (minikube v0.23/xhyve installed via homebrew, Docker for Mac - not thru homebrew) I'm able to use Docker just fine outside of the minikube, and start the minikube as mentioned above, but I can't pull any images (such as redis) from Docker Hub, making this setup a non-starter for me.

When I try to visit that v2 endpoint in the browser or cURL against it, I get an 'unauthorized' response - I'm signed in to Docker from both the app and the terminal.

What you expected to happen:

For my existing project to work as it did before (and be even better! :)

How to reproduce it (as minimally and precisely as possible):

Upgrade to Minikube v0.23 using homebrew, run using xhyve, and try to pull a Docker image. Possibly try with homebrew-ed Docker installation as well.

Output of minikube logs (if applicable): This is repeated many times: Nov 09 19:27:47 minikube localkube[2808]: E1109 19:27:47.014261 2808 remote_image.go:108] PullImage "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.7.0" from image service failed: rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Nov 09 19:27:47 minikube localkube[2808]: E1109 19:27:47.014339 2808 kuberuntime_image.go:50] Pull image "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.7.0" failed: rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Nov 09 19:27:47 minikube localkube[2808]: E1109 19:27:47.014408 2808 kuberuntime_manager.go:714] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Nov 09 19:27:47 minikube localkube[2808]: E1109 19:27:47.014436 2808 pod_workers.go:182] Error syncing pod 5c53be2f-c57d-11e7-a15a-86c7c338ec99 ("kubernetes-dashboard-lt82t_kube-system(5c53be2f-c57d-11e7-a15a-86c7c338ec99)"), skipping: failed to "StartContainer" for "kubernetes-dashboard" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 09 19:27:47 minikube localkube[2808]: E1109 19:27:47.423748 2808 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address

Anything else do we need to know:

Minikube has been very useful so far, I would love to get this working again! In the meantime I will be investigating Canonical's Distribution of Kubernetes, which says it can be used on Mac (although not ideal because I will have to setup LXD on a virtual machine it looks like :(

[bgehman]

Best advice (as a workaround): switch to virtualbox or vmware-fusion driver if on a Mac. Don't use the xhyve hypervisor, as it+Minikube seem to be plagued with network-related problems.

[Lookyan]

I've faced the same problem with xhyve. The reason was dnsmasq installed on the host. Check it out.
And also try this script if you use VPN: https://gist.github.com/mowings/633a16372fb30ee652336c8417091222

[zetaron]

@Lookyan did you find a workaround for having dnsmasq installed on the host?
I'm asking as I've got minikube-ingress-dns installed which requires dnsmasq :(

[neverfox]

I had the same problem and realized it was because I was still connected to my VPN rather than a "pure" internet connection.

[Lookyan]

@zetaron unfortunately no, but I think you should configure dnsmasq in right way, so it will proxy to the right DNS server. With defaults it doesn't work.

[vhosakot]

I saw the same error too when I ran docker build -t hello-node:v1 . in the tutorial https://kubernetes.io/docs/tutorials/stateless-application/hello-minikube/#create-a-docker-container-image on Mac OS X Sierra (10.12.6), Docker version 17.12.0-ce, minikube version v0.25.0, xhyve version 0.2.0 and kubernetes version v1.9.0:

$ docker build -t hello-node:v1 .
Sending build context to Docker daemon  3.072kB
Step 1/4 : FROM node:6.9.2
Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.3:37095->192.168.64.1:53: read: connection refused

Switching to virtualbox fixed this issue for me.

I stopped minikube, deleted it, started it without --vm-driver=xhyve (minikube uses virtualbox driver by default), and then docker build -t hello-node:v1 . worked fine without errors:

minikube stop
eval $(minikube docker-env -u)
minikube delete

minikube start (without --vm-driver=xhyve)

eval $(minikube docker-env)
docker build -t hello-node:v1 .

$ docker images | grep hello
hello-node                                    v1                  9cc51aa82a30        39 seconds ago      655MB

I do see the kube-dns pod running:

$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE
kube-system   kube-addon-manager-minikube             1/1       Running   0          1m
kube-system   kube-dns-54cccfbdf8-vmvjm               3/3       Running   0          1m
kube-system   kubernetes-dashboard-77d8b98585-5mbcf   1/1       Running   0          1m
kube-system   storage-provisioner                     1/1       Running   0          1m

This issue does look like an xhyve issue not seen with virtualbox. My virtualbox version is 5.2.6r120293:

$ VBoxManage --version
5.2.6r120293

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[CentUser]

Same problem with minikube v1.12.1.

I find out that minikube started docker image do not have ping and route, and thank god, it do have /bin/bash, I managed to append 8.8.8.8 nameserver to /etc/resolv.conf, it still could not connect to the public network.

But using minikube ssh, I can find out the images pulled from ali registry. Using docker pull docker.io/cilium/cilium still get a connection timeout error.

I don't have a proxy server, so I configured the host as an proxy server, started minikube with --docker-env HTTP_PROXY and --docker-env HTTPS_PROXY, emmm... there is no connection request connected to the squid proxy server.

I'm wandering whether there is something that I missed when configuring minikube with docker driver. Or there were some bugs in minikube using docker driver.