Minikube: China: failed to pull image... Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument

[hect1c]

Hey,

I'm trying to do a quick-start local setup on my MacOS Machine:

Run the following command

minikube start --vm-driver=hyperkit -v=9

And I get the following output / debug

`minikube v0.35.0 on darwin (amd64)
🔥 Creating hyperkit VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
...
...
About to run SSH command:
sudo mkdir -p /etc/sysconfig && printf %s "
CRIO_MINIKUBE_OPTIONS='--insecure-registry 10.96.0.0/12 '
" | sudo tee /etc/sysconfig/crio.minikube
SSH cmd err, output: :
CRIO_MINIKUBE_OPTIONS='--insecure-registry 10.96.0.0/12 '

...
📶 "minikube" IP address is 192.168.64.6
🐳 Configuring Docker as the container runtime ...
Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:62493
() Calling .GetVersion
Using API Version 1
() Calling .SetConfigRaw
() Calling .GetMachineName
✨ Preparing Kubernetes environment ...
(minikube) Calling .GetURL
🚜 Pulling images required by Kubernetes v1.13.4 ...
❌ Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.13.4": output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
: Process exited with status 1
🚀 Launching Kubernetes v1.13.4 using kubeadm ...
💣 Error starting cluster: kubeadm init:
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI

[init] Using Kubernetes version: v1.13.4
[preflight] Running pre-flight checks
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...

: Process exited with status 1

😿 Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉 https://github.com/kubernetes/minikube/issues/new`

Again my OS is MacOS High Sierra Version 10.13.6

I am able to pull the images using docker pull so don't think it's a network issue, however it seems unable to download or access the kubeadm, which I think is something I may need to install separately but having found any guides or documentation on this. I may be missing something but greatly appreciate your assistance on this ahead of time.

[huguesalary]

Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443

This is quite a weird IP. What does nslookup k8s.gcr.io return? It seems to me you have an internal DNS issue.

[hect1c]

This is my response

Server: 10.128.34.91
Address: 10.128.34.91#53

Non-authoritative answer:
Name: k8s.gcr.io
Address: 0.0.44.248

Please note: I am in China where google is blocked, but I do have my VPN on.

[huguesalary]

That’s what I thought, the issue is with the DNS returning an invalid IP. I’d have thought that through a VPN you’d get the correct ip.

Not sure what your options are since you are in China. For me the k8s.gcr.io resolves to 74.125.20.82.

[tstromberg]

Your DNS is responding with an address that apparently is not available. You can try this to possibly work-around the issue by fetching the images outside of the VM using:

minikube start --cache-images

Or by passing information to Docker so that it can use an HTTP proxy: https://github.com/kubernetes/minikube/blob/master/docs/http_proxy.md

Let me know what you find out! Once #3714 is merged, you'll be able to specify --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

[daniel-farina]

Deleting the local config fixed this issue for me:

rm -rf ~/.minikube

I run it again and all errors were gone.

[Kingonion]

In china, because of the GFW, you can't pull images from k8s.gcr.io without a VPN. But you can pull the needed images from registry.cn-hangzhou.aliyuncs.com/google_containers and tag them. If the needed images exist on the local system, the starting process will go on even with pulling error. I wrote a shell script that can help you install minikube on Linux OS easily. And you can also make a offline installation package by using the script. The project home is https://github.com/Kingonion/minikube-on-linux

[gbraad]

For China use a registry mirror when possible.

[maxnilz]

This is my response

Server: 10.128.34.91
Address: 10.128.34.91#53

Non-authoritative answer:
Name: k8s.gcr.io
Address: 0.0.44.248

Please note: I am in China where google is blocked, but I do have my VPN on.

Hi, I have the same problem with you, after some digging, I found the docker pull request will first go to docker daemon via REST endpoint /images/create, and the docker daemon will take over the pulling work. In order to make docker daemon to use the proxy, we need to set http_proxy & https_proxy to the place where dockerd running, i.e, root user.

Here is my scenario:
OS: I'm using ubuntu18.04 desktop
Where the docker daemon is running: the docker daemon run in root user
What supervisor host the docker daemon: systemd
Do I set the https_proxy & http_proxy: yes, I set them for the normal user, but didn't set them for root user.

How I solve this:
Because I just want to set http_proxy & https_proxy for docker daemon(dockerd) only for root, I set the proxy info into systemd service unit, like this(for my computer, the config file is located here: /lib/systemd/system/docker.service:

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
Environment="http_proxy=http://127.0.0.1:7777"
Environment="https_proxy=http://127.0.0.1:7777"
ExecStart=/usr/bin/dockerd -D -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

Hope this will help somebody who got the same error like me.

[singleye]

I had the same issue on 'MacOS 10.14.4 Majave'

Pull and tag images doesn't work for me

$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.0
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.1
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.0
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.0
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1

$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.0 k8s.gcr.io/kube-apiserver:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.0 k8s.gcr.io/kube-controller-manager:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.0 k8s.gcr.io/kube-scheduler:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.0 k8s.gcr.io/kube-proxy:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1

$ docker images
REPOSITORY                                                                    TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                                                         v1.14.0             5cd54e388aba        4 weeks ago         82.1MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.14.0             5cd54e388aba        4 weeks ago         82.1MB
k8s.gcr.io/kube-controller-manager                                            v1.14.0             b95b1efa0436        4 weeks ago         158MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.14.0             b95b1efa0436        4 weeks ago         158MB
k8s.gcr.io/kube-scheduler                                                     v1.14.0             00638a24688b        4 weeks ago         81.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.14.0             00638a24688b        4 weeks ago         81.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.14.0             ecf910f40d6e        4 weeks ago         210MB
k8s.gcr.io/kube-apiserver                                                     v1.14.0             ecf910f40d6e        4 weeks ago         210MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   1.3.1               eb516548c180        3 months ago        40.3MB
k8s.gcr.io/coredns                                                            1.3.1               eb516548c180        3 months ago        40.3MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.3.10              2c4adeb21b4f        4 months ago        258MB
k8s.gcr.io/etcd                                                               3.3.10              2c4adeb21b4f        4 months ago        258MB
k8s.gcr.io/pause                                                              3.1                 da86e6ba6ca1        16 months ago       742kB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.1                 da86e6ba6ca1        16 months ago       742kB

$ minikube start
😄  minikube v1.0.0 on darwin (amd64)
🤹  Downloading Kubernetes v1.14.0 images in the background ...
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
🔄  Restarting existing hyperkit VM for "minikube" ...
⌛  Waiting for SSH access ...r     y    d         r          r   s           7                   1                   4 m      a          2
📶  "minikube" IP address is 192.168.64.2
🐳  Configuring Docker as the container runtime ...
🐳  Version of container runtime is 18.06.2-ce
⌛  Waiting for image downloads to compe_te ...                               3                   d                   1  m      a         7
✨  Preparing Kubernetes environment ...                                      9                   1                   1  m      a         2
🚜  Pulling images required by Kubernetes v1.14.0 ...
❌  Unable to pull images, which may be OK: runnigz cmd: sudo kubeadm cgnfig images pull --config /var/lib/kubeadm.yaml: command faierdv sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.14.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting heade
rs)
, error: exit status 1
: Process exited with status 1
🔄  Relaunching Kubernetes v1.14.0 using kubeadm ...
⌛  Waiting for pods: apiserver proxy etcd scheduler controller dns           v                   5                   4 w     a           8
📯  Updating kube-proxy configuration ...
🤔  Verifying component health ......
💗  kubectl is now configured to use "minikube"
🏄  Done! Thank you for using minikube!

But specify '--image-repository' works for me

$ minikube start --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

$ minikube start --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
😄  minikube v1.0.0 on darwin (amd64)
🤹  Downloading Kubernetes v1.14.0 images in the background ...
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
🔄  Restarting existing hyperkit VM for "minikube" ...
⌛  Waiting for SSH access ...          "                        6 w     a           E      (     6 w     a                                         f
📶  "minikube" IP address is 192.168.64.2
🐳  Configuring Docker as the container runtime ...
🐳  Version of container runtime is 18.06.2-ce
⌛  Waiting for image downloads to complete ...                  6 w     a           E      (   6 w     a                                           v
✨  Preparing Kubernetes 4nvironment ..."                        6 w     a           E      (   6 w     a                                           h
🚜  Pulling images required by Kubernetes v1.14.0 ...
🔄  Relaunching Kubernetes v1.14.0 using kubeadm ...
⌛  Waiting for pods: apiserver proxy etcd skhrduler controller dnsw     a           E      (   6 w     a                                           g
📯  Updating kube-proxy configuration ...
🤔  Verifying component health .....
💗  kubectl is now configured to use "minikube"
🏄  Done! Thank you for using minikube!

BTW, I'm using minikube v1.0.0

$ minikube version
minikube version: v1.0.0

[tstromberg]

v1.0.1 now includes a user-friendly message to resolve the original requesters issue. Thank you for opening the bug report!

[BingerCoder]

rm -rf ~/.minikube
and with --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
works for me ~

[jiran2]

rm -rf ~/.minikube
and with --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
works for me ~

worked thanks

[hantsy]

Used the mirror repo(registry.cn-hangzhou.aliyuncs.com/google_containers), still failed, there is checksum still using Google service. saving checksum file: getting storage object: Get https://www.googleapis.com/storage/v1/b/minikube-preloaded-volume-tarballs/o/preloaded-images-k8s-v1-v1.17.3-docker-overlay2.tar.lz4?alt=json&prettyPrint=false&projection=full:

 minikube start --driver=hyperv --memory=4096m --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
* minikube v1.8.1 on Microsoft Windows 10 Pro 10.0.19041 Build 19041
* Using the hyperv driver based on user configuration
* Using image repository registry.cn-hangzhou.aliyuncs.com/google_containers
* Downloading VM boot image ...
* Creating hyperv VM (CPUs=2, Memory=4096MB, Disk=20000MB) ...
* Downloading preloaded images tarball for k8s v1.17.3 ...
E0310 01:54:28.001249   12292 cache.go:116] Error caching images:  **saving checksum file: getting storage object: Get https://www.googleapis.com/storage/v1/b/minikube-preloaded-volume-tarballs/o/preloaded-images-k8s-v1-v1.17.3-docker-overlay2.tar.lz4?alt=json&prettyPrint=false&projection=full:** dial tcp 216.58.200.42:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
E0310 01:54:28.020668   12292 config.go:71] Failed to preload container runtime Docker: copying file: sudo test -d \ && sudo scp -t \ && sudo touch -d "2020-03-10 01:10:26.0875112 +0800" \/preloaded.tar.lz4: Process exited with status 1
output: , falling back to caching images