Minikube: hyperkit: VM is unable to access k8s.gcr.io (when VPN is in use)

Created on 13 Jan 2020  ·  21Comments  ·  Source: kubernetes/minikube

Starting minikube with the default VM driver (Hyperkit) makes the external network unreachable:

minikube -p test start
😄  [test] minikube v1.6.2 on Darwin 10.15.2
✨  Automatically selected the 'hyperkit' driver (alternates: [virtualbox vmwarefusion])
🔥  Creating hyperkit VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...
⚠️  VM is unable to access k8s.gcr.io, you may need to configure a proxy or set --image-repository
🐳  Preparing Kubernetes v1.17.0 on Docker '19.03.5' ...
🚜  Pulling images ...
❌  Unable to pull images, which may be OK: running cmd: "/bin/bash -c \"sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm config images pull --config /var/tmp/minikube/kubeadm.yaml\"": /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm config images pull --config /var/tmp/minikube/kubeadm.yaml": Process exited with 
status 1


stdout:

stderr:
W0113 20:47:22.189603 2751 common.go:77] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta1". Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
W0113 20:47:22.190387 2751 common.go:77] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta1". Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
W0113 20:47:22.192219 2751 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0113 20:47:22.192254 2751 validation.go:28] Cannot validate kubelet config - no validator is available
failed to pull image "k8s.gcr.io/kube-apiserver:v1.17.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher

And then trying to deploy any pod fails.

Starting with --vmdriver virtualbox works just fine.

The exact command to reproduce the issue:

kube apply -f utils.yaml

utils.yaml pulls in massenz/dnsutils:1.1 image (but this is reproducible with any Docker image).

The full output of the command that failed:

Normal BackOff 30s kubelet, minikube Back-off pulling image "massenz/dnsutils:1.1"
Warning Failed 30s kubelet, minikube Error: ImagePullBackOff
Normal Pulling 16s (x2 over 46s) kubelet, minikube Pulling image "massenz/dnsutils:1.1"
Warning Failed 1s (x2 over 31s) kubelet, minikube Failed to pull image "massenz/dnsutils:1.1": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 1s (x2 over 31s) kubelet, minikube Error: ErrImagePull

The output of the minikube logs command:


Jan 11 14:19:23 minikube kubelet[4757]: W0111 14:19:23.514145 4757 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for default/utils through plugin: invalid network status for
Jan 11 14:19:30 minikube kubelet[4757]: W0111 14:19:30.612595 4757 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for default/utils through plugin: invalid network status for
Jan 11 14:19:31 minikube kubelet[4757]: W0111 14:19:31.754128 4757 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for default/utils through plugin: invalid network status for

The operating system version:
MacOS 10.15.2

$ minikube version      
minikube version: v1.6.2
commit: 54f28ac5d3a815d1196cd5d57d707439ee4bb392
chyperkit help wanted kinbug lifecyclfrozen needs-solution-message prioritimportant-longterm
Source
picture massenz
👍13 ❤1 🎉1

Most helpful comment

At last it seems this issue is with the hyperkitVM, because It works fine with the virtual box.
Those who want their pod to be up and running, without worrying about the type of VM, Can follow these steps

  1. Delete the existing minikube by running _minikube delete_
  2. Install the virtual box _brew cask install virtualbox_ , make sure you do the installation proper way, if you are using macos it will ask for permission to be enabled from firevault, once enabled re-run the command until you see virtual box is installed successfully.
  3. once all is set up finally start the minikube using the virtual box _minikube start --vm-driver=virtualbox_
  4. At last you can run the pod using your usual yaml file _kubectl create -f "yourfilename.yaml"
  5. run kubectl get pods
  6. in case of error you can check _kubectl describe pods_ to check the detailed errors.
    If your yml is correct and path for the Image is correct, you should be good and will just require till step 5.
picture raghvendra1218 on 29 Jan 2020
👍7

All 21 comments

@massenz sorry that you face this issue, I am curious does
docker pull massenz/dnsutils:1.1 work on your own machine without problem ?

and also do you happen to use VPN or corp proxy?
and last question, does it make a difference if you do docker login ?

medyagh picture medyagh on 22 Jan 2020

Yes, Docker works just fine (with/without login)
Yes, I use a VPN (hopefully everyone does, these days...) but the problem persists with/without VPN.

Yep, the docker login was the first thing I tried as I assumed that would have been the case; but really, the problem must be around the egress network for the hyperkit VM: as mentioned, when I use --vm-driver virtualbox it works just fine; VBox, by default, creates a bridge and allows egress to the Internet for the VM, I'm guessing Hyperkit doesn't, and as the VM is on a separate subnet than the host network, then it doesn't know how to reach the external network.

BTW - this seems to be a good place to remark how awesome Minikube is - I absolutely love it, it has helped me immensely in exploring K8s, developing a whole stack of services, and generally making my developer's life so much easier: thank you deeply to the contributors!

massenz picture massenz on 23 Jan 2020

At last it seems this issue is with the hyperkitVM, because It works fine with the virtual box.
Those who want their pod to be up and running, without worrying about the type of VM, Can follow these steps

  1. Delete the existing minikube by running _minikube delete_
  2. Install the virtual box _brew cask install virtualbox_ , make sure you do the installation proper way, if you are using macos it will ask for permission to be enabled from firevault, once enabled re-run the command until you see virtual box is installed successfully.
  3. once all is set up finally start the minikube using the virtual box _minikube start --vm-driver=virtualbox_
  4. At last you can run the pod using your usual yaml file _kubectl create -f "yourfilename.yaml"
  5. run kubectl get pods
  6. in case of error you can check _kubectl describe pods_ to check the detailed errors.
    If your yml is correct and path for the Image is correct, you should be good and will just require till step 5.
raghvendra1218 picture raghvendra1218 on 29 Jan 2020
👍7

With minikube on virtualbox, all good till the deployment. My Problem is, pods running NodeJS service can't connect to the external services (e.g. MongoDB Atlas). Note: I have deployed on DigitalOcen, the same docker build work perfectly fine.

2020-02-24 17:05:02 [ info ] : server started on port 3333 (local)
Error: Error: querySrv ENOTFOUND _mongodb._tcp.twrdev-hokqm.mongodb.net
at NativeConnection. (/usr/src/app/services/mongoose.js:57:19)
at NativeConnection.emit (events.js:196:13)
at NativeConnection.EventEmitter.emit (domain.js:471:20)
at /usr/src/app/node_modules/mongoose/lib/connection.js:817:37
at processTicksAndRejections (internal/process/task_queues.js:82:9)
2020-02-24 17:05:02 [ error ] : Error connecting to DB : querySrv ENOTFOUND _mongodb._tcp.twrdev-hokqm.mongodb.net
Error: querySrv ENOTFOUND _mongodb._tcp.twrdev-hokqm.mongodb.net
at QueryReqWrap.onresolve [as oncomplete] (dns.js:203:19) {
errno: 'ENOTFOUND',
code: 'ENOTFOUND',
syscall: 'querySrv',
hostname: '_mongodb._tcp.twrdev-hokqm.mongodb.net'
}
[Unhandled Rejection] Node NOT Exiting...
Error: Could not connect to database
at NativeConnection. (/usr/src/app/services/mongoose.js:70:19)
at NativeConnection.emit (events.js:196:13)
at NativeConnection.EventEmitter.emit (domain.js:471:20)
at NativeConnection.set (/usr/src/app/node_modules/mongoose/lib/connection.js:125:12)
at /usr/src/app/node_modules/mongoose/lib/connection.js:711:26
at executeCallback (/usr/src/app/node_modules/mongodb/lib/operations/execute_operation.js:74:5)
at /usr/src/app/node_modules/mongodb/lib/operations/connect.js:209:23
at /usr/src/app/node_modules/mongodb/lib/operations/connect.js:299:21
at QueryReqWrap.callback (/usr/src/app/node_modules/mongodb/lib/core/uri_parser.js:56:21)
at QueryReqWrap.onresolve [as oncomplete] (dns.js:203:10)
2020-02-24 17:05:02 [ error ] : Disconnected from Database!
Any suggestion??
Thanks in advance

arvtiwar picture arvtiwar on 24 Feb 2020

For hyperkit, VM is unable to access k8s.gcr.io says to me that this is likely a DNS issue, and likely #3036

Do you mind sharing the output of the following two commands for me?

  • sudo lsof -i4UDP:53 -P -n
  • ps -afe | grep dns

Thank you!

tstromberg picture tstromberg on 4 Mar 2020
👍1

dnsmasq was causing issues for me:

brew uninstall dnsmasq
sudo lsof -i4UDP:53 -P -n

COMMAND PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
dnsmasq 123 nobody    4u  IPv4 0xb150fdd17fe0cd81      0t0  UDP 127.0.0.1:53

sudo kill -9 123

Minikube then runs without issues.

If you want to keep dnsmasq installed then I would try these instructions: https://github.com/kubernetes/minikube/issues/3036#issuecomment-423150899

ashleyconnor picture ashleyconnor on 7 Mar 2020

Same problem here, no dnsmasq installed, when using HyperKit.

The question is, which application does interfere on my machine?

MacOs Mojave: 10.14.6 (18G2022)

minikube version
minikube version: v1.7.3
commit: 436667c819c324e35d7e839f8116b968a2d0a3ff

kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-13T18:06:54Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

chrber picture chrber on 1 Apr 2020

I'm also running into this issue. Note: I also have AnyConnect running -- seems like this could be a common denominator. 

😄  minikube v1.9.2 on Darwin 10.15.4
✨  Automatically selected the hyperkit driver
👍  Starting control plane node m01 in cluster minikube
🔥  Creating hyperkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
🌟  Enabling addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

minikube version: v1.9.2
commit: 93af9c1e43cab9618e301bc9fa720c63d5efa393

I've tried re-installing with, 

minikube stop 
minikube delete
brew reinstall minikube

I'm running Docker Desktop 2.2.0.5
No dnsmasq

sudo lsof -i4UDP:53 -P -n
<yields nothing>

Following DNS Debugging, I see the following. 

$ for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done
E0415 18:58:18.027806       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
E0415 18:58:18.028150       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
E0415 18:58:18.028245       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7
linux/amd64, go1.13.6, da7f65b
E0415 18:58:17.954733       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
E0415 18:58:17.954834       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
E0415 18:58:17.954901       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7
linux/amd64, go1.13.6, da7f65b
riklopfer picture riklopfer on 15 Apr 2020

Hi team
I met the same error.
Mac Mojave , version 10.14.6. run the command minikube start --vm-driver=hyperkit to start the minikube, below is the logs

😄  minikube v1.9.2 on Darwin 10.14.6
✨  Using the hyperkit driver based on existing profile
👍  Starting control plane node m01 in cluster minikube
🔄  Restarting existing hyperkit VM for "minikube" ...
🐳  Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🌟  Enabling addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

I am using Ciso VPN, no matter i set the proxy or not, the result is the same.

kubectl -n kube-system get pods
NAME                               READY   STATUS    RESTARTS   AGE
coredns-66bff467f8-fnxht           1/1     Running   5          56m
coredns-66bff467f8-vxr5s           1/1     Running   5          56m
etcd-minikube                      1/1     Running   5          56m
kube-apiserver-minikube            1/1     Running   5          56m
kube-controller-manager-minikube   1/1     Running   5          56m
kube-proxy-pt5js                   1/1     Running   5          56m
kube-scheduler-minikube            1/1     Running   5          56m
storage-provisioner                1/1     Running   8          56m

Can get the dns log

kubectl -n kube-system logs coredns-66bff467f8-fnxht 
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
I0423 14:01:35.230402       1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.226674406 +0000 UTC m=+0.085300319) (total time: 30.002647734s):
Trace[2019727887]: [30.002647734s] [30.002647734s] END
E0423 14:01:35.230463       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0423 14:01:35.230738       1 trace.go:116] Trace[1427131847]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.22650071 +0000 UTC m=+0.085126637) (total time: 30.004178291s):
Trace[1427131847]: [30.004178291s] [30.004178291s] END
E0423 14:01:35.230753       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0423 14:01:35.235236       1 trace.go:116] Trace[939984059]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.232689948 +0000 UTC m=+0.091315902) (total time: 30.002522331s):
Trace[939984059]: [30.002522331s] [30.002522331s] END
E0423 14:01:35.235287       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

I can get the some docker images inside the VM

minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                     v1.18.0             43940c34f24f        4 weeks ago         117MB
k8s.gcr.io/kube-apiserver                 v1.18.0             74060cea7f70        4 weeks ago         173MB
k8s.gcr.io/kube-controller-manager        v1.18.0             d3e55153f52f        4 weeks ago         162MB
k8s.gcr.io/kube-scheduler                 v1.18.0             a31f78c7c8ce        4 weeks ago         95.3MB
kubernetesui/dashboard                    v2.0.0-rc6          cdc71b5a8a0e        5 weeks ago         221MB
k8s.gcr.io/pause                          3.2                 80d28bedfe5d        2 months ago        683kB
k8s.gcr.io/coredns                        1.6.7               67da37a9a360        2 months ago        43.8MB
kindest/kindnetd                          0.5.3               aa67fec7d7ef        5 months ago        78.5MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90        6 months ago        288MB
kubernetesui/metrics-scraper              v1.0.2              3b08661dc379        6 months ago        40.1MB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        2 years ago         80.8MB

But when i try to login the dockerhub, failed

$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: dllbh
Password:
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Some article said it has relationship with the DNS

sudo lsof -i4UDP:53 -P -n
COMMAND     PID           USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
mDNSRespo 93271 _mdnsresponder   33u  IPv4 0x1154dc77691251b5      0t0  UDP *:53

After kill the process, error still occurs. It seems a bug for minikube with hyperkit driver. How to fix this?

By the way,

minikube version
minikube version: v1.9.2
commit: 93af9c1e43cab9618e301bc9fa720c63d5efa393
majinghe picture majinghe on 23 Apr 2020
👍1

I'm also seeing this, trying to run on a Mac that has a "Cisco Anconnect" VPN software and hyperkit .. the easiest workaround is to use the --vm-driver=virtualbox option. I'm happy to provide config information if anyone really wants the details however my gut feeling is that the corporate installed Cisco VPN software is the culprit, it futzes with DNS even when its not "turned on" to ensure I'm not accessing "inappropriate" websites like say .. urban dictionary (I know, it seems unreasonable, but thats just collateral damage for a decent security posture so I put up with it).

crankbird picture crankbird on 24 Apr 2020

I'm also seeing this, trying to run on a Mac that has a "Cisco Anconnect" VPN software and hyperkit .. the easiest workaround is to use the --vm-driver=virtualbox option. I'm happy to provide config information if anyone really wants the details however my gut feeling is that the corporate installed Cisco VPN software is the culprit, it futzes with DNS even when its not "turned on" to ensure I'm not accessing "inappropriate" websites like say .. urban dictionary (I know, it seems unreasonable, but thats just collateral damage for a decent security posture so I put up with it).

I am using win10 pro and I have default hyper-v and cisco anyconnect
while running minikube it shows the exactly same error
however, if I turn-off the cisco anyconnect, it still shows the same error "
"! This VM is having trouble accessing https://k8s.gcr.io"

how can we turn off hyper-v on windows and start using --vm-driver=virtualbox

mjm19091979 picture mjm19091979 on 1 May 2020

Hi team
I met the same error.
Mac Mojave , version 10.14.6. run the command minikube start --vm-driver=hyperkit to start the minikube, below is the logs

😄  minikube v1.9.2 on Darwin 10.14.6
✨  Using the hyperkit driver based on existing profile
👍  Starting control plane node m01 in cluster minikube
🔄  Restarting existing hyperkit VM for "minikube" ...
🐳  Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🌟  Enabling addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

I am using Ciso VPN, no matter i set the proxy or not, the result is the same.

kubectl -n kube-system get pods
NAME                               READY   STATUS    RESTARTS   AGE
coredns-66bff467f8-fnxht           1/1     Running   5          56m
coredns-66bff467f8-vxr5s           1/1     Running   5          56m
etcd-minikube                      1/1     Running   5          56m
kube-apiserver-minikube            1/1     Running   5          56m
kube-controller-manager-minikube   1/1     Running   5          56m
kube-proxy-pt5js                   1/1     Running   5          56m
kube-scheduler-minikube            1/1     Running   5          56m
storage-provisioner                1/1     Running   8          56m

Can get the dns log

kubectl -n kube-system logs coredns-66bff467f8-fnxht 
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
I0423 14:01:35.230402       1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.226674406 +0000 UTC m=+0.085300319) (total time: 30.002647734s):
Trace[2019727887]: [30.002647734s] [30.002647734s] END
E0423 14:01:35.230463       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0423 14:01:35.230738       1 trace.go:116] Trace[1427131847]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.22650071 +0000 UTC m=+0.085126637) (total time: 30.004178291s):
Trace[1427131847]: [30.004178291s] [30.004178291s] END
E0423 14:01:35.230753       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0423 14:01:35.235236       1 trace.go:116] Trace[939984059]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.232689948 +0000 UTC m=+0.091315902) (total time: 30.002522331s):
Trace[939984059]: [30.002522331s] [30.002522331s] END
E0423 14:01:35.235287       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

I can get the some docker images inside the VM

minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                     v1.18.0             43940c34f24f        4 weeks ago         117MB
k8s.gcr.io/kube-apiserver                 v1.18.0             74060cea7f70        4 weeks ago         173MB
k8s.gcr.io/kube-controller-manager        v1.18.0             d3e55153f52f        4 weeks ago         162MB
k8s.gcr.io/kube-scheduler                 v1.18.0             a31f78c7c8ce        4 weeks ago         95.3MB
kubernetesui/dashboard                    v2.0.0-rc6          cdc71b5a8a0e        5 weeks ago         221MB
k8s.gcr.io/pause                          3.2                 80d28bedfe5d        2 months ago        683kB
k8s.gcr.io/coredns                        1.6.7               67da37a9a360        2 months ago        43.8MB
kindest/kindnetd                          0.5.3               aa67fec7d7ef        5 months ago        78.5MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90        6 months ago        288MB
kubernetesui/metrics-scraper              v1.0.2              3b08661dc379        6 months ago        40.1MB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        2 years ago         80.8MB

But when i try to login the dockerhub, failed

$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: dllbh
Password:
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Some article said it has relationship with the DNS

sudo lsof -i4UDP:53 -P -n
COMMAND     PID           USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
mDNSRespo 93271 _mdnsresponder   33u  IPv4 0x1154dc77691251b5      0t0  UDP *:53

After kill the process, error still occurs. It seems a bug for minikube with hyperkit driver. How to fix this?

By the way,

minikube version
minikube version: v1.9.2
commit: 93af9c1e43cab9618e301bc9fa720c63d5efa393

I have the probelm, can you please tell me how to get inside minikube vm?

mjm19091979 picture mjm19091979 on 1 May 2020

You can use minikube ssh

On Thu, Apr 30, 2020, 7:36 PM mjm19091979 notifications@github.com wrote:

Hi team
I met the same error.
Mac Mojave , version 10.14.6. run the command minikube start
--vm-driver=hyperkit to start the minikube, below is the logs

😄 minikube v1.9.2 on Darwin 10.14.6

✨ Using the hyperkit driver based on existing profile

👍 Starting control plane node m01 in cluster minikube

🔄 Restarting existing hyperkit VM for "minikube" ...

🐳 Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...

❗ This VM is having trouble accessing https://k8s.gcr.io

💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/

🌟 Enabling addons: default-storageclass, storage-provisioner

🏄 Done! kubectl is now configured to use "minikube"

I am using Ciso VPN, no matter i set the proxy or not, the result is the
same.

kubectl -n kube-system get pods

NAME READY STATUS RESTARTS AGE

coredns-66bff467f8-fnxht 1/1 Running 5 56m

coredns-66bff467f8-vxr5s 1/1 Running 5 56m

etcd-minikube 1/1 Running 5 56m

kube-apiserver-minikube 1/1 Running 5 56m

kube-controller-manager-minikube 1/1 Running 5 56m

kube-proxy-pt5js 1/1 Running 5 56m

kube-scheduler-minikube 1/1 Running 5 56m

storage-provisioner 1/1 Running 8 56m

Can get the dns log

kubectl -n kube-system logs coredns-66bff467f8-fnxht

[INFO] plugin/ready: Still waiting on: "kubernetes"

[INFO] plugin/ready: Still waiting on: "kubernetes"

I0423 14:01:35.230402 1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.226674406 +0000 UTC m=+0.085300319) (total time: 30.002647734s):

Trace[2019727887]: [30.002647734s] [30.002647734s] END

E0423 14:01:35.230463 1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

I0423 14:01:35.230738 1 trace.go:116] Trace[1427131847]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.22650071 +0000 UTC m=+0.085126637) (total time: 30.004178291s):

Trace[1427131847]: [30.004178291s] [30.004178291s] END

E0423 14:01:35.230753 1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

I0423 14:01:35.235236 1 trace.go:116] Trace[939984059]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.232689948 +0000 UTC m=+0.091315902) (total time: 30.002522331s):

Trace[939984059]: [30.002522331s] [30.002522331s] END

E0423 14:01:35.235287 1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

I can get the some docker images inside the VM

minikube ssh

                     _             _

        _         _ ( )           ( )

___ ___ (_) ___ (_)| |/') _ _ | |_ __

/' _ _\| |/' _ \| || , < ( ) ( )| '_\ /'__`\

| ( ) ( ) || || ( ) || || |`\ | (_) || |_) )( ___/

(_) (_) (_)(_)(_) (_)(_)(_) (_)\___/'(_,__/'____)

$ docker images

REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy v1.18.0 43940c34f24f 4 weeks ago 117MB
k8s.gcr.io/kube-apiserver v1.18.0 74060cea7f70 4 weeks ago 173MB
k8s.gcr.io/kube-controller-manager v1.18.0 d3e55153f52f 4 weeks ago 162MB
k8s.gcr.io/kube-scheduler v1.18.0 a31f78c7c8ce 4 weeks ago 95.3MB

kubernetesui/dashboard v2.0.0-rc6 cdc71b5a8a0e 5 weeks ago 221MB
k8s.gcr.io/pause 3.2 80d28bedfe5d 2 months ago 683kB
k8s.gcr.io/coredns 1.6.7 67da37a9a360 2 months ago 43.8MB

kindest/kindnetd 0.5.3 aa67fec7d7ef 5 months ago 78.5MB
k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90 6 months ago 288MB

kubernetesui/metrics-scraper v1.0.2 3b08661dc379 6 months ago 40.1MB
gcr.io/k8s-minikube/storage-provisioner v1.8.1 4689081edb10 2 years ago 80.8MB

But when i try to login the dockerhub, failed

$ docker login

Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.

Username: dllbh

Password:

Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Some article said it has relationship with the DNS

sudo lsof -i4UDP:53 -P -n

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

mDNSRespo 93271 _mdnsresponder 33u IPv4 0x1154dc77691251b5 0t0 UDP *:53

After kill the process, error still occurs. It seems a bug for minikube
with hyperkit driver. How to fix this?

By the way,

minikube version

minikube version: v1.9.2

commit: 93af9c1e43cab9618e301bc9fa720c63d5efa393

I have the probelm, can you please tell me how to get inside minikube vm?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/minikube/issues/6296#issuecomment-622220331,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABC2KAZUETHTFA3DC4KGUPLRPIYRZANCNFSM4KGIXIMQ
.

medyagh picture medyagh on 1 May 2020

We should update the error string to mention trying --driver=docker when a VPN is in use.

tstromberg picture tstromberg on 28 May 2020
👍2 👀1

we need a solution message, if can't pull images due to network issues, only on windows and macos, if they are not already using docker driver they should use docker driver.

medyagh picture medyagh on 3 Jun 2020

Still an issue.

sharifelgamal picture sharifelgamal on 17 Jun 2020 
| =>minikube delete


| =>minikube start --driver=docker
😄  minikube v1.11.0 on Darwin 10.15.4
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=1989MB) ...
🐳  Preparing Kubernetes v1.18.3 on Docker 19.03.2 ...
    ▪ kubeadm.pod-network-cidr=10.244.0.0/16
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

worked for me

OS: MacOs Catalina 10.15.4 (19E287)

vignesh-subburaj picture vignesh-subburaj on 20 Jun 2020

@vignesh-subburaj great. I'm not sure what the cause of this issue is, so for now it seems our recommended advice is to try to use the docker driver instead of the hyperkit driver.

If anyone would be interested in updating our warning message from:

This VM is having trouble accessing https://k8s.gcr.io

to something like

This VM is having trouble accessing https://k8s.gcr.io -- consider using the `docker` driver by running `minikube start --driver docker`
priyawadhwa picture priyawadhwa on 24 Jun 2020
👍2

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot picture fejta-bot on 22 Sep 2020

Still an issue for me:

😄  minikube v1.15.1 on Darwin 10.15.7
✨  Using the hyperkit driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating hyperkit VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/

minikube version: v1.15.1
commit: 23f40a012abb52eff365ff99a709501a61ac5876

EDIT: I should add that no VPN is in use

kylemclaren picture kylemclaren on 5 Dec 2020

same issue for me, similarly to @kylemclaren I have no VPN at all,

bmariesan picture bmariesan on 7 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

mrtkp9993 picture mrtkp9993  ·  3Comments
tnine picture tnine  ·  3Comments
RaeesBhatti picture RaeesBhatti  ·  3Comments
djschny picture djschny  ·  3Comments
xmnlab picture xmnlab  ·  3Comments