Kubeadm: [Kubeadm] Failing to pull images

Created on 31 Oct 2018  路  16Comments  路  Source: kubernetes/kubeadm

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version: v1.11.* (reproduced on all v1.11 and v1.12 releases)

Environment:

  • Kubernetes version : v1.11.* (reproduced on all v1.11 and v1.12 releases)
  • OS: RHEL 7.5
  • Kernel: 3.10.0-862.11.6.el7.x86_64

What happened?

When running kubeadm init the images fail to be pulled:

[root@pink01 ~]# kubeadm init --pod-network-cidr=192.168.0.0/16 --kubernetes-version=v1.11.1
[init] using Kubernetes version: v1.11.1
[preflight] running pre-flight checks
I1031 06:34:55.007573    7218 kernel_validator.go:81] Validating kernel version
I1031 06:34:55.007833    7218 kernel_validator.go:96] Validating kernel config
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[preflight] Some fatal errors occurred:
        [ERROR ImagePull]: failed to pull image [k8s.gcr.io/kube-apiserver-amd64:v1.11.1]: exit status 1
        [ERROR ImagePull]: failed to pull image [k8s.gcr.io/kube-controller-manager-amd64:v1.11.1]: exit status 1
        [ERROR ImagePull]: failed to pull image [k8s.gcr.io/kube-scheduler-amd64:v1.11.1]: exit status 1
        [ERROR ImagePull]: failed to pull image [k8s.gcr.io/kube-proxy-amd64:v1.11.1]: exit status 1
        [ERROR ImagePull]: failed to pull image [k8s.gcr.io/pause:3.1]: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

What you expected to happen?

Images should be pulled

How to reproduce it (as minimally and precisely as possible)?

kubeadm init --pod-network-cidr=192.168.0.0/16 --kubernetes-version=v1.11.1

Anything else we need to know?

Also reproduced on latest (v1.12.2). Tried manually too:

[root@pink01 ~]# kubeadm config images pull
failed to pull image "k8s.gcr.io/kube-apiserver:v1.12.2": output: Error response from daemon: Get https://k8s.gcr.io/v2/kube-apiserver/manifests/v1.12.2: Get https://k8s.gcr.io/v2/token?scope=repository%3Akube-apiserver%3Apull&service=k8s.gcr.io: net/http: TLS handshake timeout
, error: exit status 1
picture ocofaigh

Most helpful comment

I had the same problem.
I have a proxy but docker refuse to take it so have to configure the proxy to the docker client https://www.thegeekdiary.com/how-to-configure-docker-to-use-proxy/
And it's work.

picture Norskel on 22 May 2019
👍4 🎉1

All 16 comments

seems that you don't have internet connectivity to this server.
if that's an issue for you can pull the images from a PC that has connectivity, place them in a local container registry and then use the "imageRepository" field of the v1alpha3 config in version 1.12.x
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha3

for more questions please try the support channels:
https://github.com/kubernetes/community/blob/master/contributors/guide/issue-triage.md#user-support-response-example

neolit123 picture neolit123 on 31 Oct 2018

@neolit123 I have been using kubeadm on this machine for months without issues. It definitely has internet access. I have also reproduced on other machines too. Has anyone else reported such an issue? was working fine on Friday.

ocofaigh picture ocofaigh on 31 Oct 2018

works fine for me at the moment with 1.12.x.

  • we haven't seen any reports about this recently.
neolit123 picture neolit123 on 31 Oct 2018

@neolit123 Any suggestions on how to debug? Here is what I get when I try to deploy the latest 1.12.x:

[init] using Kubernetes version: v1.12.2
[preflight] running pre-flight checks
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'

[preflight] Some fatal errors occurred:
    [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.12.2: output: Error response from daemon: Get https://k8s.gcr.io/v1/_ping: net/http: TLS handshake timeout
, error: exit status 1
    [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.12.2: output: Error response from daemon: Get https://k8s.gcr.io/v1/_ping: net/http: TLS handshake timeout
, error: exit status 1
    [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.12.2: output: Error response from daemon: Get https://k8s.gcr.io/v1/_ping: net/http: TLS handshake timeout
, error: exit status 1
    [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.12.2: output: Pulling repository k8s.gcr.io/kube-proxy
Network timed out while trying to connect to https://k8s.gcr.io/v1/repositories/kube-proxy/images. You may want to check your internet connection or if you are behind a proxy.
, error: exit status 1
    [ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v1/_ping: net/http: TLS handshake timeout
, error: exit status 1
    [ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Error response from daemon: Get https://k8s.gcr.io/v2/etcd/manifests/3.2.24: Get https://k8s.gcr.io/v2/token?scope=repository%3Aetcd%3Apull&service=k8s.gcr.io: net/http: TLS handshake timeout
, error: exit status 1
    [ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.2: output: Error response from daemon: Get https://k8s.gcr.io/v1/_ping: net/http: TLS handshake timeout
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

On same machine, curl works:

[root@pink01 ~]# curl https://k8s.gcr.io/v2/etcd/manifests/3.2.24
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 950,
         "digest": "sha256:7b073bdab8c52dc23dfb3e2101597d30304437869ad8c0b425301e96a066c408",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 950,
         "digest": "sha256:4f93210747a2150605fdbc764a86b549824f322f2168b7c762bd4e9a9a264860",
         "platform": {
            "architecture": "arm",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 950,
         "digest": "sha256:f1dcb048bb5a5c6d74e07db03f0a006a8bdcea4dc034722d1d9e24da15901390",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 951,
         "digest": "sha256:bf275fe898f9efda3db428e3287970b348a7e60d1086af36770aa78c9caec886",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 951,
         "digest": "sha256:857376ef4d252c5ce1817902216dc92fedcc133a225144b3a427e5f89c88a17f",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}
ocofaigh picture ocofaigh on 31 Oct 2018

can you pull the images using the docker CLI:
docker pull ....
?

neolit123 picture neolit123 on 31 Oct 2018

@neolit123 Nope:

[root@pink01 ~]# docker pull k8s.gcr.io/kube-apiserver-amd64:v1.12.2
Error response from daemon: Get https://k8s.gcr.io/v1/_ping: net/http: TLS handshake timeout

[root@pink01 ~]# docker pull k8s.gcr.io/kube-apiserver-amd64:v1.11.1
Error response from daemon: Get https://k8s.gcr.io/v1/_ping: net/http: TLS handshake timeout
ocofaigh picture ocofaigh on 31 Oct 2018

@ocofaigh
you seem to have a connectivity issue unrelated to kubeadm.

$ docker pull k8s.gcr.io/kube-apiserver-amd64:v1.12.2
v1.12.2: Pulling from kube-apiserver-amd64
90e01955edcd: Already exists 
70b78fdead23: Pull complete 
Digest: sha256:07337a4302eab1c868f4c4b3273b01d6abaaf869131aa374cb0685f7d642647e
Status: Downloaded newer image for k8s.gcr.io/kube-apiserver-amd64:v1.12.2

please try resolving it and try again.

neolit123 picture neolit123 on 31 Oct 2018

A third person has just reported the same issue to me. He said:

whether on IBM鈥檚 network or at home (with the VPN shut down), doesn鈥檛 work for me

% docker pull k8s.gcr.io:5000/kube-apiserver-amd64:v1.11.4
Error response from daemon: Get https://k8s.gcr.io:5000/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

ocofaigh picture ocofaigh on 31 Oct 2018

I'm with the same problem, but it's not related with kubeadm.

yagonobre picture yagonobre on 31 Oct 2018

@ocofaigh i suggested this earlier, but try pulling the kubeadm required images on a machine that has connectivity and upload them to a private container registry somewhere.
then using "imageRepository" in the 1.12 kubeadm config you can pull images from the private repo:
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha3

neolit123 picture neolit123 on 31 Oct 2018

UPDATE: its seems intermittent

yagonobre picture yagonobre on 31 Oct 2018

@neolit123 The problem is I cannot pull the images from any machine. Tried 5 times and got different errors each time:

[root@pink01 ~]# kubeadm config images pull
failed to pull image "k8s.gcr.io/kube-apiserver:v1.12.2": output: Pulling repository k8s.gcr.io/kube-apiserver
Network timed out while trying to connect to https://k8s.gcr.io/v1/repositories/kube-apiserver/images. You may want to check your internet connection or if you are behind a proxy.
, error: exit status 1

[root@pink01 ~]# kubeadm config images pull
failed to pull image "k8s.gcr.io/kube-apiserver:v1.12.2": output: Error response from daemon: Get https://k8s.gcr.io/v2/kube-apiserver/manifests/v1.12.2: net/http: TLS handshake timeout
, error: exit status 1

[root@pink01 ~]# kubeadm config images pull
failed to pull image "k8s.gcr.io/kube-apiserver:v1.12.2": output: Error response from daemon: Get https://k8s.gcr.io/v2/kube-apiserver/manifests/v1.12.2: net/http: TLS handshake timeout
, error: exit status 1

[root@pink01 ~]# kubeadm config images pull
[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.12.2
failed to pull image "k8s.gcr.io/kube-controller-manager:v1.12.2": output: Pulling repository k8s.gcr.io/kube-controller-manager
unauthorized: authentication required
, error: exit status 1

[root@pink01 ~]# kubeadm config images pull
failed to pull image "k8s.gcr.io/kube-apiserver:v1.12.2": output: Error response from daemon: Get https://k8s.gcr.io/v2/kube-apiserver/manifests/v1.12.2: net/http: TLS handshake timeout
, error: exit status 1
ocofaigh picture ocofaigh on 31 Oct 2018

there could be connectivity issues from your network to k8s.gcr.io.
GCR has separate regions so it might be a case where some of the regional servers are currently down.

but again, not an issue for the kubeadm issue tracker.

neolit123 picture neolit123 on 31 Oct 2018

@neolit123 Thanks for the info. Wonder if there is any way to confirm this. I'll try again tomorrow anyway

ocofaigh picture ocofaigh on 31 Oct 2018

try posting in the #kubeadm channel of the k8s slack.
this might provide some reports if other users are experiecing the same problem.

neolit123 picture neolit123 on 31 Oct 2018
👍1

I had the same problem.
I have a proxy but docker refuse to take it so have to configure the proxy to the docker client https://www.thegeekdiary.com/how-to-configure-docker-to-use-proxy/
And it's work.

Norskel picture Norskel on 22 May 2019
👍4 🎉1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

luxas picture luxas  路  3Comments
tmjd picture tmjd  路  4Comments
jessfraz picture jessfraz  路  3Comments
jbrandes picture jbrandes  路  4Comments
shenshouer picture shenshouer  路  3Comments