Kubeadm: Installing crictl requires use of --ignore-preflight-errors=cri

[bruce2409]

Versions

kubeadm version: v1.9.1
kubectl version: v1.9.1
Centos 7.4 w/ kernel: 3.10.0-693.11.6.el7.x86_64

What happened?

If running kubeadm init without crictl being in the system path you get the expected warning of:
[WARNING FileExisting-crictl]: crictl not found in system path but the process continues as normal.

However if you have got crictl in your system path, kubeadm init will fail on this output:
[ERROR CRI]: unable to check if the container runtime at "/var/run/dockershim.sock" is running: exit status 1 which means you need to provide --ignore-preflight-errors=cri and then the init will work as expected.

The same behaviour occurs when running kubeadm join on a node.

What you expected to happen?

To have the join/init process to work the same way regardless of the presence of crictl.

How to reproduce it (as minimally and precisely as possible)?

• Try to run kubeadm init on a server without crictl in $PATH
• Try to run kubeadm init on a server with crictl in $PATH

Anything else we need to know?

The issue seems to be related to the fact that the dockershim.sock file is created by the kubelet service but, unless I'm missing something, this can't be started without having run kubeadm init/join in the first place.

[dixudx]

/kind bug
/assign

Have sent out PR to fix crictl preflight check.

[fredmj]

Hello @dixudx ,
I still have this issue.
Is it really corrected in the merge pull request #58802 (https://github.com/kubernetes/kubernetes/pull/58802)?

Environment is CentOS7.4 and VMs hosted through virtualbox 5.2.8

[fredmj@Lagrange ~]$ cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core) 
[fredmj@Lagrange ~]$ vboxmanage --version
5.2.8r121009

kubeadm v 1.9.6 ; kubectl v 1.9.6; kubelet v 1.9.6

[root@master ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
[root@master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
[root@master ~]# kubelet --version
Kubernetes v1.9.6

go and crictl in my $PATH

[root@master ~]# cat ~/.bash_profile 
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:/usr/local/go/bin
export PATH
export GOPATH=$HOME/go
[root@master ~]# go version
go version go1.10 linux/amd64
[root@master ~]# crictl --version
crictl version 1.0.0-alpha.0

kubeadm init still complain about my missing --ignore-preflight-errors=cri

[root@master ~]# kubeadm init
[init] Using Kubernetes version: v1.9.6
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
    [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[preflight] Some fatal errors occurred:
    [ERROR CRI]: unable to check if the container runtime at "/var/run/dockershim.sock" is running: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

kubeadmin will succeed with the --ignore-preflight-errors flag

[root@master ~]# kubeadm init --ignore-preflight-errors=cri
[init] Using Kubernetes version: v1.9.6
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
    [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
    [WARNING CRI]: unable to check if the container runtime at "/var/run/dockershim.sock" is running: exit status 1
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [master.kubernetes kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.2.111]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 99.003851 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node master.kubernetes as master by adding a label and a taint
[markmaster] Master master.kubernetes tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: 450518.9af12e3fd8302eb6
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token 450518.9af12e3fd8302eb6 192.168.2.111:6443 --discovery-token-ca-cert-hash sha256:a2b0e2e11b0442bd7be31cbfdc7efd529b94ef56c71a3b0bca0daa7b641c0033

[fredmj]

/kind bug
/assign

[dixudx]

@fredmj That bug-fix is only applicable after release v1.10. Currently you can try it in all v1.10.0 beta versions, v1.10.0 rc version and v1.11.0 alpha version.

kubeadm v 1.9.6 ; kubectl v 1.9.6; kubelet v 1.9.6

We did not back-port the fix to v1.9 yet.

@timothysc @luxas Shall we back-port this to v1.9?

[fredmj]

@dixudx, @timothysc , @luxas I'll try that with v10.x.x and eventually open a PR

[MikeSpreitzer]

It looks to me like https://github.com/kubernetes/kubernetes/pull/58802 does not really fix the problem. Looking at the changes in that PR, they tweak the code that generates the crictl -r /var/run/dockershim.sock info command, but they do not change the fact that a failure from that command counts as a preflight failure.

Let's be clear about this: preflight checks should not require that the dockershim already be in place.

[dixudx]

do not change the fact that a failure from that command counts as a preflight failure
Let's be clear about this: preflight checks should not require that the dockershim already be in place.

@MikeSpreitzer CRICheck is removed from the []Checker` list. That is how cri preflight check is skipped.

[MikeSpreitzer]

@dixudx: There are _two_ lists of checkers. CRICheck was removed from the list in RunInitMasterChecks but not the list in RunJoinNodeChecks.

[yunghoy]

Circular dependency alert.

kubeadm requires dockershim.sock(created by kubelet) to be run.
kubelet requires pki and manifests(created by kubeadm) to be run.

[dixudx]

@yunghoy Please try to use kubeadm reset to clean up your environments. Just be aware when you really want to do this.

Then run kubeadm init with your customized flags.

[yunghoy]

@dixudx Sadly, I did that more than 10 times.

[yunghoy]

The only solution I found is to run "kubeadm init" with "--ignore-preflight-errors=cri" option. Thanks

[bart0sh]

Another possible workaround is remove crictl from your system.

[luxas]

I think https://github.com/kubernetes/kubernetes/pull/62481 at HEAD/the upcoming v1.11 will fix the circular dependency @yunghoy mentioned at least. Does that make sense to you?
BTW; we're packaging crictl in the debs/rpms in v1.11, and will make the tool being locally kind of mandatory (you don't _have to have it_, but recommended and used inside of kubeadm on more ways)