Minikube: sudo crictl: not found if installed to a directory outside of sudoers secure_path

[afbjorklund]

We have some things breaking on CentOS, which does not have /usr/local/bin in the PATH.
That is: users do have it in _their_ PATH, but root does not. And sudo runs with the root PATH.

So if you are trying to run something that is not from a package, you will get a "command not found"
This goes both for "crio version" (when run as root) and for "sudo crictl", only work with /usr/bin.

One workaround is to move everything to /usr/bin, which is violating the FHS but whatever.

It would be nice if minikube was able to find crio and crictl - also in their _default_ location ?

This would have worked better if crio and cri-tools would be available in the default distribution.

But it doesn't (at the moment), and when you try to install it locally you run into these issues...

[afbjorklund]

Actually the same thing happens with minikube itself:

$ sudo install minikube-linux-amd64 /usr/local/bin/minikube
$ sudo minikube version
sudo: minikube: command not found

$ sudo rpm -ivh minikube-1.4.0.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:minikube-1.4.0-0                 ################################# [100%]
$ sudo minikube version
minikube version: v1.4.0
commit: 7969c25a98a018b94ea87d949350f3271e9d64b6

That is, anything installed in /usr/local is not found.

[afbjorklund]

Here is how things work in CentOS:

[admin@7d58aaaae462 ~]$ echo $PATH
/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/admin/.local/bin:/home/admin/bin
[admin@7d58aaaae462 ~]$ sudo -i
[root@7d58aaaae462 ~]# echo $PATH
/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
[root@7d58aaaae462 ~]# logout
[admin@7d58aaaae462 ~]$ sudo su -
[root@7d58aaaae462 ~]# echo $PATH
/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
[root@7d58aaaae462 ~]# logout

i.e. when using sudo, /usr/local/bin is missing from the PATH.

(Normally "sudo" itself wasn't installed either, but seems to be now...)

Tested with:

• CentOS Linux release 7.6.1810 (Core)
• CentOS Linux release 8.0.1905 (Core)

RH Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1166185
Fixed in: sudo-1.8.22-0.1.b1.fc28 and up (but not in "c8" branch)

Fedora: https://src.fedoraproject.org/rpms/sudo/blob/f28/f/sudoers
Defaults secure_path = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
CentOS: https://git.centos.org/rpms/sudo/blob/c8/f/SOURCES/sudoers
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin

Note that /usr/local/sbin is still in the PATH, added by /etc/profile.
(probably by accident, since it is using a different "pathmunge" function)

[tstromberg]

I wouldn't want to make it the default everywhere, but alternatively, we could use sudo -E for calls to crictl and other necessary places?

This is definitely not a CentOS only issue, but it's admittedly less common elsewhere.

[RA489]

/assign

[RA489]

@afbjorklund I tried replicating this on CentOS and following two approaches works for me.

1. Changing the secure_path to /usr/local/bin

[rohit@localhost root]$ sudo visudo
Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

1. copying to usr/bin solves this issue.

Let me know which approach can be good for this issue.

[RA489]

IIUC The secure_pathvalue, if set, will be used as PATH environment variable for the commands you run using sudo

[afbjorklund]

I think the plan was to change the sudo invocation in minikube, rather than have the user change...

i.e. either do sudo -E crictl or perhaps sudo /usr/local/bin/crictl (using which)

[sharifelgamal]

We should probably fix this at some point, having a centos machine to test on would help.

[afbjorklund]

Arguably, the default config is stupid (and indeed, they did fix it for later versions)

Most likely we will go with the /usr/local/bin/crictl workaround, when needed.
Installing /usr/bin/crictl with the cri-tools package would be the _preferred_ approach.

Problem is that upstream sometimes suggests to use the tarball and /usr/local instead:

https://github.com/kubernetes-sigs/cri-tools#install-crictl

sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin

The correct way would be to install the rpm package from the kubernetes repositories:

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

yum install cri-tools

But this gets confusing with minikube, since we use our own binaries for the other three...
And since docker doesn't use crictl ("yet"), we don't install it but leave it to the poor user.

[afbjorklund]

We currently use it from 4 different places (before refactoring, one was hiding in "logs"):

pkg/minikube/cruntime/cri.go:       c := exec.Command("sudo", "crictl", "ps", "-a", fmt.Sprintf("--name=%s", filter), fmt.Sprintf("--state=%s", state), "--quiet")
pkg/minikube/cruntime/cri.go:       rr, err = cr.RunCmd(exec.Command("sudo", "crictl", "ps", "-a", fmt.Sprintf("--state=%s", state), "--quiet"))
pkg/minikube/cruntime/cri.go:   cmd.WriteString("sudo crictl logs ")
pkg/minikube/logs/logs.go:  cmds["container status"] = "sudo crictl ps -a || sudo docker ps -a"

So maybe do a which call, before those sudo calls.

Only difference would be running /usr/bin/crictl.