Cri-o: calico not working on fresh k8s installation with cri-o as CRI

[kubealex]

Description

Following the guide on kubernetes installation, I wanted to try the combination:

cri-o as container runtime
calico as cni plugin

The scenario is an HA install with:

2 Master nodes
2+ worker nodes

I am actually able to install cri-o and deploy the first master, set the --pod-network-cidr during kubeadm init phase then the problem rises.

By default, cri-o uses the crio-bridge cni in /etc/cni/net.d, and when calico gets installed, it should switch on the newly created config for calico on the same dir BUT!

Jun  3 21:35:01 k8s-master-0 crio: time="2020-06-03 21:35:01.707957151Z" level=warning msg="Error validating CNI config file /etc/cni/net.d/10-calico.conflist: [failed to find plugin \"calico\" in path [/usr/libexec/cni]]"
Because calico plugin is in:
[kube@k8s-master-0 ~]$ ls /opt/cni/bin/ |grep calico
calico
calico-ipam

Why just dont add to default plugin dirs in /etc/crio/crio.conf the /opt/cni/bin/ directory?
Default config of cri-o doesn't take into account that plugin dir, making it incompatible with suggested cni plugin for k8s.

Steps to reproduce the issue:

1. Install cri-o
2. Bootstrap first k8s master
3. Install calico CNI plugin4. Enjoy your pods being scheduled with the same IP on all nodes. (lol)

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):
Happens every time, at the moment combination k8s-cri-o-calico is  not working.
Output of crio --version:

[kube@k8s-master-0 ~]$ crio --version
crio version 
Version:       1.18.0
GitCommit:     
GitTreeState:  
BuildDate:     
GoVersion:     go1.13.6
Compiler:      gc
Platform:      linux/amd64
Linkmode:      unknown: `ldd /home/kube/crio` failed: ldd: /home/kube/crio: No such file or directory

Additional environment details (AWS, VirtualBox, physical, etc.): libvirt

[haircommander]

interesting, from what I can tell, the default plugin_dirs is [/opt/cni/bin]

how did you install cri-o? and (for sanity check) what's the output of crio config | grep -A 3 plugin_dirs

[kubealex]

crio config | grep -A 3 plugin_dirs

I followed the instructions on kubernetes website:

On centos/rhel7:

# Install prerequisites
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_7/devel:kubic:libcontainers:stable.repo
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:1.18.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:1.18/CentOS_7/devel:kubic:libcontainers:stable:cri-o:1.18.repo
# Install CRI-O
yum install -y cri-o

Below you can find what outputs from my config:

[kube@k8s-master-0 ~]$ crio config | grep -A 3 plugin_dirs
INFO Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_KILL
plugin_dirs = [
"/usr/libexec/cni",
]

[haircommander]

also, while we're at it, what's
crio config | grep network_dir

[haircommander]

I will look at this tomorrow and see what's going on

[kubealex]

[kube@k8s-master-0 ~]$ crio config | grep network_dir
INFO Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_KILL

CRI-O will pick-up the first one found in network_dir.

network_dir = "/etc/cni/net.d/"

Please don't hesitate to assing me tasks if you need further analysis, if I can help, it would be a pleasure.

[haircommander]

@dougsland should be putting up a fix soon :smiley:

[kubealex]

I can confirm that changing crio.conf to handle /opt/cni/bin instead of /usr/libexec/cni/ makes everything work like a charm.

[haircommander]

@kubealex I believe this is fixed. can this ticket be closed?

[kubealex]

Uhm, actually it isn't.

[kube@k8s-master-0 ~]$ crio config | grep -A 3 plugin_dirs
INFO Using default capabilities: CAP_NET_RAW, CAP_MKNOD, CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_KILL
plugin_dirs = [
"/usr/libexec/cni",
]