kubeadm init/join CRI preflight check fails unnecessarily

[MikeSpreitzer]

What keywords did you search in kubeadm issues before filing this one?

I searched for: unable to check if the container runtime at "/var/run/dockershim.sock" is running

I found #657 but it was already closed.

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

root@mjs-worker-dal10-a:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

root@mjs-worker-dal10-a:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

• Cloud provider or hardware configuration:
  No kube CloudProvider. SoftLayer VM with 16 CPUs and 32 GB of main memory.

• OS (e.g. from /etc/os-release): 16.04.4 LTS

• Kernel (e.g. uname -a): Linux mjs-worker-dal10-a 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

root@mjs-worker-dal10-a:~# kubeadm join --token t3c20c.s0s7q3md5lkgn5d6 --discovery-token-ca-cert-hash 1044b1e3ef183d51c50030533d8f33764a2928f8b76193a020eab2d855619ab4 10.171.126.235:6443
[preflight] Running pre-flight checks.
[preflight] Some fatal errors occurred:
    [ERROR CRI]: unable to check if the container runtime at "/var/run/dockershim.sock" is running: fork/exec /usr/local/bin/crictl -r /var/run/dockershim.sock info: no such file or directory
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

I have crictl installed and there is no dockershim running yet.

root@mjs-worker-dal10-a:~# ls -ld /var/run/dock*
drwx------ 6 root root   120 May  8 00:03 /var/run/docker
-rw-r--r-- 1 root root     5 May 12 15:41 /var/run/docker.pid
srw-rw---- 1 root docker   0 May 12 15:41 /var/run/docker.sock

What you expected to happen?

I expected kubeadm to not complain about the fact that it has not made a dockershim yet. I expected #657 to be fixed, since I am using version 1.10.2 of kubeadm.

How to reproduce it (as minimally and precisely as possible)?

Just like #657

Anything else we need to know?

[MikeSpreitzer]

/kind bug
@kubernetes/sig-cluster-lifecycle-bugs

[luxas]

Thanks @MikeSpreitzer. I believe https://github.com/kubernetes/kubernetes/pull/63297 and https://github.com/kubernetes/kubernetes/pull/63849 will do

[timothysc]

closing as PRs have been merged.

[MikeSpreitzer]

@luxas: No, this issue is not about kubeadm reset, so those PRs you cited are not relevant.

[MikeSpreitzer]

@timothysc: I am not sure which PR you are referring to, but https://github.com/kubernetes/kubernetes/blob/v1.10.2/cmd/kubeadm/app/preflight/checks.go#L981 still includes CRICheck and in master I see https://github.com/kubernetes/kubernetes/blob/41a531317ade9656e7daee0d82beabe41c71942c/cmd/kubeadm/app/preflight/checks.go#L963

[chuckha]

@MikeSpreitzer is the kubelet running?

[chuckha]

Oh nevermind, I understand now. My mistake.

[chuckha]

This is the PR that fixes the issue you're seeing. If you define a CRI socket that isn't the docker shim, kubeadm will use crictl. If you have the default (dockershim) defined then kubeadm will check the docker service.

Feel free to reopen if you have more questions or would like to discuss some other resolution

[MikeSpreitzer]

The problem here is not about CRI vs docker. It is about requiring one of them for kubeadm join. #657 complained about this, and was closed due to https://github.com/kubernetes/kubernetes/pull/58802 --- but that was only a partial fix. In cmd/kubeadm/app/preflight/checks.go, CRICheck was listed in _both_ RunInitMasterChecks and RunJoinNodeChecks (when useCRI), but that PR removed only the former, not also the latter. The later PR https://github.com/kubernetes/kubernetes/pull/62481 removed even the partial fix, by collecting the common checks into one procedure, and deciding that CRICheck is one of the common checks (when CRI is chosen). When doing kubeadm join, with CRI chosen, it should not be required that the dockershim already be in place.

[MikeSpreitzer]

/reopen

[MikeSpreitzer]

@chuckha please re-open this, I do not know how.

[dixudx]

@MikeSpreitzer kubernetes/kubernetes#62481 solved this. Currently this fix is only available for v1.11.x releases.

https://github.com/kubernetes/kubernetes/blob/2fcac6abf27a3912a71f590300b72c2e48266120/cmd/kubeadm/app/preflight/checks.go#L966-L971

You can see, for default dockershim.sock, it will check docker status. For other cri sockets, cricheck is enabled.

I think this issue can be closed now.

[MikeSpreitzer]

There is a bug in the release-1.10 branch. Looking at cmd/kubeadm/app/preflight/checks.go in that branch, compare RunInitMasterChecks with RunJoinNodeChecks. You will see that in no circumstances does RunInitMasterChecks insist on finding a CRI socket, but RunJoinNodeChecks will --- if it finds crictl --- insist that the CRI socket already exists. I think the behavior of RunJoinNodeChecks is wrong, and RunInitMasterChecks may be wrong too. It may be that release 1.11 has better behavior, and it may be that some back-porting would be the right way to fix release-1.10.

[dixudx]

@MikeSpreitzer Filed PR #63907 to back port release-1.10. PTAL. Thanks.

[dixudx]

/cc @luxas

[SpComb]

Per https://github.com/kubernetes/kubernetes/pull/63907#issuecomment-390002464 this does not seem to be getting a 1.10 backport... if I'm understanding this issue correctly, then with the latest 1.10.x release, installing crictl will break kubeadm join for the default Docker runtime:

$ sudo kubeadm join ...
[preflight] Running pre-flight checks.
[preflight] Some fatal errors occurred:
    [ERROR CRI]: unable to check if the container runtime at "/var/run/dockershim.sock" is running: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

Mitigation is to not install crictl, which is a bit odd because kubeadm join then suggests that you do exactly that:

        [WARNING FileExisting-crictl]: crictl not found in system path
    Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl

---

To be clear, this is a different issue from the confusingly similar issue affecting 1.10.x when using kubeadm join --cri-socket /var/run/crio/crio.sock with a non-Docker runtime and having crictl installed, which is broken in 1.10.2 and fixed in 1.10.3 by https://github.com/kubernetes/kubernetes/pull/63849:

[preflight] Running pre-flight checks.
    [WARNING DirAvailable--etc-kubernetes-manifests]: /etc/kubernetes/manifests is not empty
[preflight] Some fatal errors occurred:
    [ERROR CRI]: unable to check if the container runtime at "/var/run/crio/crio.sock" is running: fork/exec /usr/local/bin/crictl -r /var/run/crio/crio.sock info: no such file or directory
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

[luxas]

I'm closing this as fixed in v1.11 as this issue is targeting the v1.11 milestone which we are doing housekeeping on. If you feel strongly about getting a v1.10 backport, please open a new issue for specifically that and we might consider it.

[aivanzipper]

$ sudo apt-get remove cri-tools

[kulong0105]

i also met the issue in v1.10.12

cannot workaround it by remove cri-tools, since kubeadm depends on cri-tools

[root@skyaxe-app-0 SkyDiscovery-2019-01-10]# yum remove cri-tools
Loaded plugins: fastestmirror
Resolving Dependencies
--> Running transaction check
---> Package cri-tools.x86_64 0:1.12.0-0 will be erased
--> Processing Dependency: cri-tools >= 1.11.0 for package: kubeadm-1.10.12-0.x86_64
--> Running transaction check
---> Package kubeadm.x86_64 0:1.10.12-0 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================
 Package                                   Arch                                   Version                                      Repository                                                Size
==============================================================================================================================================================================================
Removing:
 cri-tools                                 x86_64                                 1.12.0-0                                     @SkyDiscovery-kubernetes                                  18 M
Removing for dependencies:
 kubeadm                                   x86_64                                 1.10.12-0                                    @SkyDiscovery-kubernetes                                 105 M

Transaction Summary
==============================================================================================================================================================================================
Remove  1 Package (+1 Dependent package)

Installed size: 123 M
Is this ok [y/N]: n

any idea ?

[rvesse]

You can workaround this by adding --ignore-preflight-errors=CRI to the kubeadm join commands