Minikube: pulling kic base image does not respect --image-repository or --image-mirror-country flags for CN registry mirrors

[carvendy]

重现问题所需的命令：

失败的命令的完整输出：

minikube start --registry-mirror=https://registry.docker-cn.com --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --vm-driver=docker --alsologtostderr -v=8

minikube logs命令的输出：

W0406 10:08:19.436876 13086 exit.go:101] Failed to start docker container. "minikube start" may fix it.: recreate: creating host: create: creating: create kic node: create container: failed args: [run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=2200mb --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 gcr.io/k8s-minikube/kicbase:v0.0.8@sha256:2f3380ebf1bb0c75b0b47160fd4e61b7b8fef0f1f32f9def108d3eada50a7a81] output: Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.8@sha256:2f3380ebf1bb0c75b0b47160fd4e61b7b8fef0f1f32f9def108d3eada50a7a81' locally
docker: Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.
: exit status 125

使用的操作系统版本：
ubuntu 18.04

[listenerri]

可能跟我的问题一样，我的解决方案是：
https://listenerri.com/2020/04/07/minikube-%E5%90%AF%E5%8A%A8%E5%A4%B1%E8%B4%A5/

[sharifelgamal]

The above workaround of building from source will work for now, we'll work on making kicbase available somewhere.

[tstromberg]

We don't maintain the CN registry mirror - and do not know how to contact those who do.

[chenzhikang]

我也遇到了，正在想办法解决

[chenzhikang]

可能跟我的问题一样，我的解决方案是：
https://listenerri.com/2020/04/07/minikube-%E5%90%AF%E5%8A%A8%E5%A4%B1%E8%B4%A5/

你提供的解决方案很有价值。
但是我遇到的情况和你不大一样，我这里是确实连不上外网，本地docker images看不到kicbase镜像；

同时推断，当driver=docker 时，岂不是所有墙内的用户都无法完成minikube start了？

[chenzhikang]

可能跟我的问题一样，我的解决方案是：
https://listenerri.com/2020/04/07/minikube-%E5%90%AF%E5%8A%A8%E5%A4%B1%E8%B4%A5/

你好，你分享的经验很有价值！

结合我这边操作发现：
1：我这直接pull下来的镜像Digest是没问题的【这一点看到你那边pull下来就丢了Digest，感觉比较奇怪】
2：我这边是在docker tag时发现，new image的Digest确实丢了；【也就是说直接pull下载才会加上Digest，tag或其它操作就将Digest丢弃了】

供参考，握手！

[chenzhikang]

终于算是搞定这个问题了，成功启动minikube的docker驱动的集群，如图：

这个问题预计kicbase没同步到国内镜像源前，大陆用户都会遇到。下面说一下我的思路和方案：

方案1：根据源码重新编译minikube（连带kicbase），如sharifelgamal所述的方案。【根据源码编译带来更多不确定的问题，且minikube刚接触的都是入门用户，此方案不可靠，不推荐】

方案2：让minikube所在的机器走全局代理翻墙，假装身在国外，直接从gcr.io中pull kicbase镜像。【可行，但是kicbase镜像接近1G，翻墙下载估计会比较慢。】

方案3：想各种办法将kicbase镜像搞到本机（通过墙外机器pull下来然后save&load到本机、在国内的某些源中找到kicbase后pull到本地等），然后按照listenerri所提供的帖子中的方案，将镜像重新tag，然后恢复Digest，让minikube能识别，最后搞定。

这里我采用的是方案3，镜像是从阿里云的镜像仓库找到的（感谢上传镜像的同学！），如图：

其中1个的镜像地址：registry.cn-hangzhou.aliyuncs.com/smartliby/kicbase:v0.0.8

[lubocheng]

官方已推镜像,大家可以试试~~~
https://github.com/AliyunContainerService/minikube/issues/17

[chenzhikang]

官方已推镜像,大家可以试试~~~
AliyunContainerService#17

https://yq.aliyun.com/articles/221687
链接404呢？
@lubocheng

[lubocheng]

官方已推镜像,大家可以试试~~~
AliyunContainerService#17

https://yq.aliyun.com/articles/221687
链接404呢？
@lubocheng

链接官方自己删除了...可能过些日子就好了

[ukihsoroy]

使用新的1.10.0版本指定--base-image registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.10
可以正常启动了。

[spartucus]

使用新的1.10.0版本指定--base-image registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.10
可以正常启动了。

好像还是不行：

oc@ocserver:~$ minikube start --vm-driver docker --base-image registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.10 
😄  minikube v1.10.1 on Ubuntu 18.04
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
E0514 11:17:54.137973    3798 cache.go:129] Error downloading kic artifacts:  getting remote image: Get https://docker.pkg.github.com/v2/: dial tcp: lookup docker.pkg.github.com on 127.0.0.53:53: read udp 127.0.0.1:50986->127.0.0.53:53: i/o timeout
🤷  docker "minikube" container is missing, will recreate.
E0514 11:17:54.383291    3798 oci.go:79] docker daemon seems to be stuck. Please try restarting your docker. Will try to delete anyways: unknown state "minikube": docker inspect minikube --format={{.State.Status}}: exit status 1
stdout:


stderr:
Template parsing error: template: :1:8: executing "" at <.State.Status>: map has no entry for key "State"
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
🤦  StartHost failed, but will try again: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=3900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10: exit status 125
stdout:

stderr:
Unable to find image 'docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10' locally
docker: Error response from daemon: Get https://docker.pkg.github.com/v2/: dial tcp: lookup docker.pkg.github.com on 127.0.0.53:53: read udp 127.0.0.1:36426->127.0.0.53:53: i/o timeout.
See 'docker run --help'.

🤷  docker "minikube" container is missing, will recreate.
E0514 11:19:45.275536    3798 oci.go:79] docker daemon seems to be stuck. Please try restarting your docker. Will try to delete anyways: unknown state "minikube": docker inspect minikube --format={{.State.Status}}: exit status 1
stdout:


stderr:
Template parsing error: template: :1:8: executing "" at <.State.Status>: map has no entry for key "State"
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
😿  Failed to start docker container. "minikube start" may fix it: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=3900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10: exit status 125
stdout:

stderr:
Unable to find image 'docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10' locally
docker: Error response from daemon: Get https://docker.pkg.github.com/v2/: dial tcp: lookup docker.pkg.github.com on 127.0.0.53:53: read udp 127.0.0.1:60680->127.0.0.53:53: i/o timeout.
See 'docker run --help'.


💣  error provisioning host: Failed to start host: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=3900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10: exit status 125
stdout:

stderr:
Unable to find image 'docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10' locally
docker: Error response from daemon: Get https://docker.pkg.github.com/v2/: dial tcp: lookup docker.pkg.github.com on 127.0.0.53:53: read udp 127.0.0.1:60680->127.0.0.53:53: i/o timeout.
See 'docker run --help'.


😿  minikube is exiting due to an error. If the above message is not useful, open an issue:
👉  https://github.com/kubernetes/minikube/issues/new/choose

[ukihsoroy]

使用新的1.10.0版本指定--base-image registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.10
可以正常启动了。

好像还是不行：

oc@ocserver:~$ minikube start --vm-driver docker --base-image registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.10 
😄  minikube v1.10.1 on Ubuntu 18.04
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
E0514 11:17:54.137973    3798 cache.go:129] Error downloading kic artifacts:  getting remote image: Get https://docker.pkg.github.com/v2/: dial tcp: lookup docker.pkg.github.com on 127.0.0.53:53: read udp 127.0.0.1:50986->127.0.0.53:53: i/o timeout
🤷  docker "minikube" container is missing, will recreate.
E0514 11:17:54.383291    3798 oci.go:79] docker daemon seems to be stuck. Please try restarting your docker. Will try to delete anyways: unknown state "minikube": docker inspect minikube --format={{.State.Status}}: exit status 1
stdout:


stderr:
Template parsing error: template: :1:8: executing "" at <.State.Status>: map has no entry for key "State"
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
🤦  StartHost failed, but will try again: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=3900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10: exit status 125
stdout:

stderr:
Unable to find image 'docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10' locally
docker: Error response from daemon: Get https://docker.pkg.github.com/v2/: dial tcp: lookup docker.pkg.github.com on 127.0.0.53:53: read udp 127.0.0.1:36426->127.0.0.53:53: i/o timeout.
See 'docker run --help'.

🤷  docker "minikube" container is missing, will recreate.
E0514 11:19:45.275536    3798 oci.go:79] docker daemon seems to be stuck. Please try restarting your docker. Will try to delete anyways: unknown state "minikube": docker inspect minikube --format={{.State.Status}}: exit status 1
stdout:


stderr:
Template parsing error: template: :1:8: executing "" at <.State.Status>: map has no entry for key "State"
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
😿  Failed to start docker container. "minikube start" may fix it: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=3900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10: exit status 125
stdout:

stderr:
Unable to find image 'docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10' locally
docker: Error response from daemon: Get https://docker.pkg.github.com/v2/: dial tcp: lookup docker.pkg.github.com on 127.0.0.53:53: read udp 127.0.0.1:60680->127.0.0.53:53: i/o timeout.
See 'docker run --help'.


💣  error provisioning host: Failed to start host: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=3900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10: exit status 125
stdout:

stderr:
Unable to find image 'docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10' locally
docker: Error response from daemon: Get https://docker.pkg.github.com/v2/: dial tcp: lookup docker.pkg.github.com on 127.0.0.53:53: read udp 127.0.0.1:60680->127.0.0.53:53: i/o timeout.
See 'docker run --help'.


😿  minikube is exiting due to an error. If the above message is not useful, open an issue:
👉  https://github.com/kubernetes/minikube/issues/new/choose

试试这个，我正常启动了
minikube start --registry-mirror=https://registry.docker-cn.com --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --vm-driver=docker --alsologtostderr -v=8 --base-image registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.10

[spartucus]

@sigmaol 这样可以了，多谢~

[lubocheng]

@sigmaol 这样可以了，多谢~

不行...😪

[spartucus]

@lubocheng 不清楚跟网络有没有关系。我在移动网络下，就会失败。换成联通就好了。

[medyagh]

@carvendy @spartucus @lubocheng @sigmaol @lubocheng @chenzhikang

in latest version of minikube (v1.11.0) we added the kicbase image(https://github.com/kubernetes/minikube/pull/7943) to docker hub, do you mind trying the latest version and see if that works for you without adding any tricks?

this should just work
download latest minikube verison: https://minikube.sigs.k8s.io/docs/start/

minikube delete
minikube start --driver=docker

here is the image on docker hub:
https://hub.docker.com/r/kicbase/stable

--- google translate:

在最新版本的minikube（v1.11.0）中，我们将kicbase映像添加到了docker hub，您是否介意尝试最新版本并查看它是否对您有效而未添加任何技巧？

这应该工作
下载最新的minikube版本：https://minikube.sigs.k8s.io/docs/start/

minikube删除
minikube start-驱动程序= docker

这是Docker Hub上的图像：
https://hub.docker.com/r/kicbase/stable

[medyagh]

this issue is Partially fixed by providing docker hub image but it is still a problem that it should respect the user-provided mirror as said in this comment https://github.com/kubernetes/minikube/issues/7915#issuecomment-622025458

and minikube should try to download the kicbase image from the user provided mirror for better speed

[spartucus]

I tried latest version(v1.11.0) on ubuntu18.04, minikube started and downloaded images successfully! Everything looks good to me, well done!

[medyagh]

@spartucus glad to see that was fixed, in v1.11.0 we pushed a fallback image to dockerhub that probably fixed it.
we still need to respect the registrry mirrors for the base image,

so if user specificies a mirror, we should download the base image from that mirror.

[danieltanfh95]

使用新的1.10.0版本指定--base-image registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.10
可以正常启动了。

好像还是不行：

oc@ocserver:~$ minikube start --vm-driver docker --base-image registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.10 
😄  minikube v1.10.1 on Ubuntu 18.04
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
E0514 11:17:54.137973    3798 cache.go:129] Error downloading kic artifacts:  getting remote image: Get https://docker.pkg.github.com/v2/: dial tcp: lookup docker.pkg.github.com on 127.0.0.53:53: read udp 127.0.0.1:50986->127.0.0.53:53: i/o timeout
🤷  docker "minikube" container is missing, will recreate.
E0514 11:17:54.383291    3798 oci.go:79] docker daemon seems to be stuck. Please try restarting your docker. Will try to delete anyways: unknown state "minikube": docker inspect minikube --format={{.State.Status}}: exit status 1
stdout:


stderr:
Template parsing error: template: :1:8: executing "" at <.State.Status>: map has no entry for key "State"
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
🤦  StartHost failed, but will try again: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=3900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10: exit status 125
stdout:

stderr:
Unable to find image 'docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10' locally
docker: Error response from daemon: Get https://docker.pkg.github.com/v2/: dial tcp: lookup docker.pkg.github.com on 127.0.0.53:53: read udp 127.0.0.1:36426->127.0.0.53:53: i/o timeout.
See 'docker run --help'.

🤷  docker "minikube" container is missing, will recreate.
E0514 11:19:45.275536    3798 oci.go:79] docker daemon seems to be stuck. Please try restarting your docker. Will try to delete anyways: unknown state "minikube": docker inspect minikube --format={{.State.Status}}: exit status 1
stdout:


stderr:
Template parsing error: template: :1:8: executing "" at <.State.Status>: map has no entry for key "State"
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
😿  Failed to start docker container. "minikube start" may fix it: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=3900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10: exit status 125
stdout:

stderr:
Unable to find image 'docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10' locally
docker: Error response from daemon: Get https://docker.pkg.github.com/v2/: dial tcp: lookup docker.pkg.github.com on 127.0.0.53:53: read udp 127.0.0.1:60680->127.0.0.53:53: i/o timeout.
See 'docker run --help'.


💣  error provisioning host: Failed to start host: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=3900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10: exit status 125
stdout:

stderr:
Unable to find image 'docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.10' locally
docker: Error response from daemon: Get https://docker.pkg.github.com/v2/: dial tcp: lookup docker.pkg.github.com on 127.0.0.53:53: read udp 127.0.0.1:60680->127.0.0.53:53: i/o timeout.
See 'docker run --help'.


😿  minikube is exiting due to an error. If the above message is not useful, open an issue:
👉  https://github.com/kubernetes/minikube/issues/new/choose

试试这个，我正常启动了
minikube start --registry-mirror=https://registry.docker-cn.com --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --vm-driver=docker --alsologtostderr -v=8 --base-image registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.10

This works!

[priyawadhwa]

It looks like this is working with registry mirrors as well! I'm going to go ahead and close this issue; but if anyone is still seeing errors please reopen by including /reopen in your comment.