kubeadm init fails to pull some k8s images

[sectorsize512]

Is this a BUG REPORT

BUG REPORT

Versions

kubeadm version (use kubeadm version):

# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

[root@faas-A01 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

• Cloud provider or hardware configuration:

VM provisioned in our Datacenter.

• OS (e.g. from /etc/os-release):

# cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core) 

• Kernel (e.g. uname -a):

[root@faas-A01 ~]# uname -a
Linux faas-A01.sl.cloud9.ibm.com 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

What happened?

I'm trying to deploy new cluster using kubeadm, which fails with the following output:

[root@faas-A01 ~]# kubeadm -v 100 init
I0717 18:51:19.929233  125277 interface.go:360] Looking for default routes with IPv4 addresses
I0717 18:51:19.929336  125277 interface.go:365] Default route transits interface "eth0"
I0717 18:51:19.929851  125277 interface.go:174] Interface eth0 is up
I0717 18:51:19.929948  125277 interface.go:222] Interface "eth0" has 3 addresses :[10.91.0.3/24 9.12.246.3/24 fe80::4d4:1fff:fe45:3858/64].
I0717 18:51:19.929977  125277 interface.go:189] Checking addr  10.91.0.3/24.
I0717 18:51:19.929992  125277 interface.go:196] IP found 10.91.0.3
I0717 18:51:19.930010  125277 interface.go:228] Found valid IPv4 address 10.91.0.3 for interface "eth0".
I0717 18:51:19.930023  125277 interface.go:371] Found active IP 10.91.0.3 
I0717 18:51:20.757702  125277 feature_gate.go:230] feature gates: &{map[]}
I0717 18:51:20.757753  125277 init.go:250] [init] validating feature gates
[init] using Kubernetes version: v1.11.1
[preflight] running pre-flight checks
I0717 18:51:20.757916  125277 checks.go:581] validating kubernetes and kubeadm version
I0717 18:51:20.758105  125277 checks.go:179] validating if the firewall is enabled and active
I0717 18:51:20.774437  125277 checks.go:216] validating availability of port 6443
I0717 18:51:20.774607  125277 checks.go:216] validating availability of port 10251
I0717 18:51:20.774665  125277 checks.go:216] validating availability of port 10252
I0717 18:51:20.774722  125277 checks.go:291] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml
I0717 18:51:20.774745  125277 checks.go:291] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml
I0717 18:51:20.774762  125277 checks.go:291] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml
I0717 18:51:20.774777  125277 checks.go:291] validating the existence of file /etc/kubernetes/manifests/etcd.yaml
I0717 18:51:20.774796  125277 checks.go:438] validating if the connectivity type is via proxy or direct
I0717 18:51:20.774859  125277 checks.go:474] validating http connectivity to first IP address in the CIDR
I0717 18:51:20.774888  125277 checks.go:474] validating http connectivity to first IP address in the CIDR
I0717 18:51:20.774904  125277 checks.go:138] validating if the service is enabled and active
I0717 18:51:20.788974  125277 checks.go:340] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0717 18:51:20.789111  125277 checks.go:340] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0717 18:51:20.789180  125277 checks.go:653] validating whether swap is enabled or not
I0717 18:51:20.789237  125277 checks.go:381] validating the presence of executable crictl
I0717 18:51:20.789279  125277 checks.go:381] validating the presence of executable ip
I0717 18:51:20.789305  125277 checks.go:381] validating the presence of executable iptables
I0717 18:51:20.789340  125277 checks.go:381] validating the presence of executable mount
I0717 18:51:20.789367  125277 checks.go:381] validating the presence of executable nsenter
I0717 18:51:20.789393  125277 checks.go:381] validating the presence of executable ebtables
I0717 18:51:20.789422  125277 checks.go:381] validating the presence of executable ethtool
I0717 18:51:20.789445  125277 checks.go:381] validating the presence of executable socat
I0717 18:51:20.789471  125277 checks.go:381] validating the presence of executable tc
I0717 18:51:20.789503  125277 checks.go:381] validating the presence of executable touch
I0717 18:51:20.789528  125277 checks.go:523] running all checks
I0717 18:51:20.790846  125277 kernel_validator.go:81] Validating kernel version
I0717 18:51:20.791017  125277 kernel_validator.go:96] Validating kernel config
I0717 18:51:20.815017  125277 checks.go:411] checking whether the given node name is reachable using net.LookupHost
I0717 18:51:20.815055  125277 checks.go:622] validating kubelet version
I0717 18:51:20.986776  125277 checks.go:138] validating if the service is enabled and active
I0717 18:51:20.997542  125277 checks.go:216] validating availability of port 10250
I0717 18:51:20.997664  125277 checks.go:216] validating availability of port 2379
I0717 18:51:20.997729  125277 checks.go:253] validating the existence and emptiness of directory /var/lib/etcd
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
I0717 18:51:20.999020  125277 checks.go:840] pulling  k8s.gcr.io/kube-apiserver-amd64:v1.11.1
I0717 18:51:30.727269  125277 checks.go:840] pulling  k8s.gcr.io/kube-controller-manager-amd64:v1.11.1
I0717 18:51:35.491180  125277 checks.go:840] pulling  k8s.gcr.io/kube-scheduler-amd64:v1.11.1
I0717 18:51:40.280577  125277 checks.go:840] pulling  k8s.gcr.io/kube-proxy-amd64:v1.11.1
I0717 18:51:45.174670  125277 checks.go:840] pulling  k8s.gcr.io/pause-amd64:3.1
I0717 18:51:45.205674  125277 checks.go:840] pulling  k8s.gcr.io/etcd-amd64:3.2.18
I0717 18:51:45.234767  125277 checks.go:840] pulling  k8s.gcr.io/coredns:1.1.3
[preflight] Some fatal errors occurred:
    [ERROR ImagePull]: failed to pull image [k8s.gcr.io/kube-apiserver-amd64:v1.11.1]: exit status 1
    [ERROR ImagePull]: failed to pull image [k8s.gcr.io/kube-controller-manager-amd64:v1.11.1]: exit status 1
    [ERROR ImagePull]: failed to pull image [k8s.gcr.io/kube-scheduler-amd64:v1.11.1]: exit status 1
    [ERROR ImagePull]: failed to pull image [k8s.gcr.io/kube-proxy-amd64:v1.11.1]: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

I.e., the pulling of some images succeeds but others fail:

[root@faas-A01 ~]# docker images
REPOSITORY               TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/coredns       1.1.3               b3b94275d97c        7 weeks ago         45.6 MB
k8s.gcr.io/etcd-amd64    3.2.18              b8df3b177be2        3 months ago        219 MB
k8s.gcr.io/pause-amd64   3.1                 da86e6ba6ca1        6 months ago        742 kB

I also tried kubeadm -v 100 config images pull:

[root@faas-A01 ~]# kubeadm -v 100 config images pull
I0717 18:53:08.170155  125765 interface.go:360] Looking for default routes with IPv4 addresses
I0717 18:53:08.170264  125765 interface.go:365] Default route transits interface "eth0"
I0717 18:53:08.170796  125765 interface.go:174] Interface eth0 is up
I0717 18:53:08.171003  125765 interface.go:222] Interface "eth0" has 3 addresses :[10.91.0.3/24 9.12.246.3/24 fe80::4d4:1fff:fe45:3858/64].
I0717 18:53:08.171035  125765 interface.go:189] Checking addr  10.91.0.3/24.
I0717 18:53:08.171050  125765 interface.go:196] IP found 10.91.0.3
I0717 18:53:08.171069  125765 interface.go:228] Found valid IPv4 address 10.91.0.3 for interface "eth0".
I0717 18:53:08.171082  125765 interface.go:371] Found active IP 10.91.0.3 
I0717 18:53:09.004241  125765 feature_gate.go:230] feature gates: &{map[]}
failed to pull image "k8s.gcr.io/kube-apiserver-amd64:v1.11.1": exit status 1

What you expected to happen?

I was expecting the kubeadm init to succesfully pull all k8s images and proceed with cluster setup.

How to reproduce it (as minimally and precisely as possible)?

For me fresh installation of Centos 7.4.1708 node with Docker and Kubeadm/Kubectl/... installed via yum install manifested the issue.

[Multiply]

Same for us, just plain arm variant instead.

[Multiply]

Related: https://github.com/kubernetes/kubernetes/issues/65674#issuecomment-405770560

[neolit123]

failed to pull image "k8s.gcr.io/kube-apiserver-amd64:v1.11.1": exit status 1

the 1.11.1 part should not happen, given your kubeadm version is 1.11.0.
if you add --kubernetes-version=1.11.0 it should work.

but i can reproduce this with the latest master, which suggests that the 1.11.1 images could be missing from the recent cut?

/cc @chuckha
/cc @kubernetes/sig-cluster-lifecycle-bugs

[neolit123]

Related: kubernetes/kubernetes#65674 (comment)

Yes. Image and package pushes lag the tag/binary cuts slightly

that explains it.
except that's not exactly a good release practice. :

[foxish]

Looks like it could be a bug in the release script which pushed them to staging and not k8s.gcr.io.
Tracking in https://github.com/kubernetes/release/issues/587

[chuckha]

We use https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/v1alpha3/defaults.go#L36 as the default k8s version which then gets normalized to the most recent release on that branch...which happens to be 1.11.1 in this case. As noted above, the images are lagging the release and the workaround is also noted above.

Retries with various versions feels like it will be painful...

[luxas]

Closing this as an issue with the release machinery https://github.com/kubernetes/release/issues/587

[pawanbahuguna]

So whats the solution?