Minikube: failed to pull images: Client.Timeout exceeded while awaiting headers

[android-coco]

If this is a bug report, please include:

pod-basic git:(master) ✗ minikube start
😄 minikube v0.34.1 on darwin (amd64)
🔥 Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
📶 "minikube" IP address is 192.168.99.103
🐳 Configuring Docker as the container runtime ...
✨ Preparing Kubernetes environment ...
🚜 Pulling images required by Kubernetes v1.13.3 ...
❌ Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.13.3": output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
: Process exited with status 1
🚀 Launching Kubernetes v1.13.3 using kubeadm ...
💣 Error starting cluster: kubeadm init:
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI

[init] Using Kubernetes version: v1.13.3
[preflight] Running pre-flight checks
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...

: Process exited with status 1

😿 Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉 https://github.com/kubernetes/minikube/issues/new

[tstromberg]

This sounds like:

https://github.com/kubernetes/minikube/blob/master/docs/http_proxy.md#unable-to-pull-imagesclienttimeout-exceeded-while-awaiting-headers

For some reason, your host is blocking the VM from accessing the internet. You may be able to work around this using the --cache-images flag, but in the end, you may need to pass in some proxy information.

Is this in a corporate or home environment?

[tstromberg]

NOTE: related to #2956

[heydonovan]

I'm experiencing the same. It might be because my VPN is enabled (even though everything else works just fine). I've disabled it, and sometimes it works, sometimes it doesn't.

Any specific commands we should run to provide more information?

$ minikube start --vm-driver hyperkit
😄  minikube v0.34.1 on darwin (amd64)
🔥  Creating hyperkit VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
📶  "minikube" IP address is 192.168.64.4
🐳  Configuring Docker as the container runtime ...
✨  Preparing Kubernetes environment ...
🚜  Pulling images required by Kubernetes v1.13.3 ...
❌  Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.13.3": output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
: Process exited with status 1
🚀  Launching Kubernetes v1.13.3 using kubeadm ...

$ brew info hyperkit
hyperkit: stable 0.20180403 (bottled)

$ brew info docker-machine-driver-hyperkit
docker-machine-driver-hyperkit: stable 1.0.0 (bottled)

$ minikube logs
https://gist.github.com/heydonovan/e11f1e029dbe6f59fd51e864ddc0f72c

[toothrot]

EDIT: A reboot fixed this for me. I leaving this here below for posterity.

I experience this same issue on a fresh minikube install of a fresh debian-testing install on my laptop, installed using the 1.0.0 release image of the docker-machine-driver-kvm2 and minikube.

I started minikube using kvm2:

minikube start --vm-driver kvm2

There are no proxies configured for connecting to the internet on my machine or in the VM.

Here's an example of a failing log line:

Error syncing pod 62c04486-5be2-11e9
-be94-a81a7ae73940 ("nginx-ingress-controller-586cdc477c-rgsq4_kube-system(62c04486-5be2-11e9-be94-a81a7ae73940)"), skipping: 
failed to "StartContainer" for "nginx-ingress-controller" with ImagePullBackOff: "Back-off pulling image \"quay.io/kubernetes-
ingress-controller/nginx-ingress-controller:0.23.0\""

  Warning  Failed     6m54s (x4 over 9m31s)   kubelet, minikube  Failed to pull image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Interestingly, DNS lookups seem to work fine:

$ minikube ssh nslookup google.com
Server:    192.168.122.1
Address 1: 192.168.122.1

Name:      google.com
Address 1: 172.217.6.238 lga25s55-in-f238.1e100.net
Address 2: 2607:f8b0:4006:819::200e lga34s19-in-x0e.1e100.net

But requests do not:

$ minikube ssh "curl -v google.com"
* Failed to connect to google.com port 80: Connection timed out
curl: (7) Failed to connect to google.com port 80: Connection timed out
ssh: Process exited with status 7

docker run configured for local docker:

$ docker run hello-world

Hello from Docker!

docker run configured for minikube:

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.

The documentation linked to at https://github.com/kubernetes/minikube/issues/2956#issuecomment-471017929 did not help, as there's no proxy to configure.

What can I do to help debug?

[liu0013]

looks like I also have this issue:
😄 minikube v1.0.0 on darwin (amd64)
🤹 Downloading Kubernetes v1.14.0 images in the background ...
💡 Tip: Use 'minikube start -p ' to create a new cluster, or 'minikube delete' to delete this one.
🔄 Restarting existing virtualbox VM for "minikube" ...
⌛ Waiting for SSH access ...
📶 "minikube" IP address is 192.168.99.100
🐳 Configuring Docker as the container runtime ...
🐳 Version of container runtime is 18.06.2-ce
⌛ Waiting for image downloads to complete ...
E0422 22:06:51.975490 2478 start.go:209] Error caching images: Caching images for kubeadm: caching images: caching image /Users/aaron/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner_v1.8.1: fetching remote image: Get https://gcr.io/v2/: dial tcp 74.125.204.82:443: i/o timeout
✨ Preparing Kubernetes environment ...
❌ Unable to load cached images: loading cached images: loading image /Users/aaron/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner_v1.8.1: stat /Users/aaron/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner_v1.8.1: no such file or directory
🚜 Pulling images required by Kubernetes v1.14.0 ...
❌ Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.14.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
: Process exited with status 1
🔄 Relaunching Kubernetes v1.14.0 using kubeadm ...
⌛ Waiting for pods: apiserver proxy etcd scheduler controller dns
📯 Updating kube-proxy configuration ...
🤔 Verifying component health ......
💗 kubectl is now configured to use "minikube"
🏄 Done! Thank you for using minikube!

[tstromberg]

I believe this issue was resolved in the v1.1.0 release, by making it more resilient against network failures and proxy misconfigurations. Please try upgrading to the latest release of minikube and run minikube delete to remove the previous cluster state.

If the same issue occurs, please re-open this bug. Thank you opening this bug report, and for your patience!

[zach-source]

I ran into this problem too, but it turns out that my /etc/hosts file ownership was changed to root and my user no longer had access.

chown $USER /etc/hosts

fixed it for me.

[scholtalbers]

Encountered this error presumably after updating Docker Desktop which added an entry to /etc/hosts and changed the permissions of it? Similar to @zach-source chmod g+w /etc/hosts && minikube stop && minikube start fixed this for me.

[g-luo]

_EDIT:_ A reboot fixed this for me. I leaving this here below for posterity.

I experience this same issue on a fresh minikube install of a fresh debian-testing install on my laptop, installed using the 1.0.0 release image of the docker-machine-driver-kvm2 and minikube.

I started minikube using kvm2:

minikube start --vm-driver kvm2

There are no proxies configured for connecting to the internet on my machine or in the VM.

Here's an example of a failing log line:

Error syncing pod 62c04486-5be2-11e9
-be94-a81a7ae73940 ("nginx-ingress-controller-586cdc477c-rgsq4_kube-system(62c04486-5be2-11e9-be94-a81a7ae73940)"), skipping: 
failed to "StartContainer" for "nginx-ingress-controller" with ImagePullBackOff: "Back-off pulling image \"quay.io/kubernetes-
ingress-controller/nginx-ingress-controller:0.23.0\""

  Warning  Failed     6m54s (x4 over 9m31s)   kubelet, minikube  Failed to pull image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Interestingly, DNS lookups seem to work fine:

$ minikube ssh nslookup google.com
Server:    192.168.122.1
Address 1: 192.168.122.1

Name:      google.com
Address 1: 172.217.6.238 lga25s55-in-f238.1e100.net
Address 2: 2607:f8b0:4006:819::200e lga34s19-in-x0e.1e100.net

But requests do not:

$ minikube ssh "curl -v google.com"
* Failed to connect to google.com port 80: Connection timed out
curl: (7) Failed to connect to google.com port 80: Connection timed out
ssh: Process exited with status 7

docker run configured for local docker:

$ docker run hello-world

Hello from Docker!

docker run configured for minikube:

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.

The documentation linked to at #2956 (comment) did not help, as there's no proxy to configure.

What can I do to help debug?

@toothrot I'm having a very similar issue; what command did you use to reboot?