Cri-o: Delete network failure when destroy a pod

[shaoyuta]

Description
Run k8s+crio+runc in clearlinux, deploy a pod then delete it, found that the network interface created by kubelet can't be deleted.
I'm not sure this is an issue related about cri-o, but all are ok when I do same operations if k8s+containerd+runc.

Steps to reproduce the issue:

1. https://github.com/clearlinux/cloud-native-setup/blob/master/clr-k8s-examples/README.md
2. ./create_stack.sh minimal
3. # kubectl create -f https://raw.githubusercontent.com/clearlinux/cloud-native-setup/master/clr-k8s-examples/tests/deploy-svc-ing/test-deploy-runc.yaml   // wait until pod is created 
4. # kubectl delete -f https://raw.githubusercontent.com/clearlinux/cloud-native-setup/master/clr-k8s-examples/tests/deploy-svc-ing/test-deploy-runc.yaml

Describe the results you received:

Form service crio's log:

# journalctl -u crio -f
Jul 10 04:34:46 v-30190-n1 crio[413]: time="2019-07-10 04:34:46.893330511Z" level=error msg="Error deleting network: failed to convert major version part "": strconv.Atoi: parsing "": invalid syntax"
Jul 10 04:34:46 v-30190-n1 crio[413]: time="2019-07-10 04:34:46.893413425Z" level=error msg="Error while removing pod from CNI network "cbr0": failed to convert major version part "": strconv.Atoi: parsing "": invalid syntax"

Observe directory: /var/lib/cni/networks/cbr0/

# ls /var/lib/cni/networks/cbr0/ -l
-rw-r--r-- 1 root root 64 Jul 10 07:07 10.244.1.4
-rw-r--r-- 1 root root 64 Jul 10 07:26 10.244.1.5    // this address is create when "kubectl create ...", but is not released after "kubectl delete ..."
-rw-r--r-- 1 root root 10 Jul 10 07:26 last_reserved_ip.0
-rwxr-x--- 1 root root  0 Jul 10 04:26 lock

Output of crio --version:

# crio --version
crio version 1.14.4
commit: unknown

# crictl version
Version:  0.1.0
RuntimeName:  cri-o
RuntimeVersion:  1.14.4
RuntimeApiVersion:  v1alpha1

Additional environment details (AWS, VirtualBox, physical, etc.):

# kubelet --version
Kubernetes v1.15.0

# runc --version
runc version 1.0.0-rc5
spec: 1.0.0

[sysrich]

We are also seeing this issue on openSUSE Kubic

@saschagrunert do you have any immediate thoughts/hints/theoretical workarounds for me try out?

[saschagrunert]

Hey @shaoyuta and @sysrich, thanks for reporting the issue. It looks like that the cniVersion is not set correctly either by the plugin or by the local config in the network_dir folder (default /etc/cni/net.d).

@shaoyuta can you try to update the CNI plugins to the latest available version (0.8.1) and run CRI-O 1.15.x with Kubernetes 1.15.x ?

@sysrich To reproduce the issue (which I can't on Tumbleweed), how does the CNI configuration look like on the test machine?

Thanks for the additional information :pray:

[thkukuk]

On Fri, Jul 19, Sascha Grunert wrote:

@sysrich To reproduce the issue (which I can't on Tumbleweed), how does the CNI
configuration look like on the test machine?

It's openSUSE Kubic running the k8s e2e test suite triggering the
problem.

The content of /etc/cni/net.d/10-flannel.conflist (no idea who
creates that):

{
"name": "cbr0",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}

--
Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS
SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany
GF: Felix Imendoerffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nuernberg)

[sysrich]

The content of /etc/cni/net.d/10-flannel.conflist (no idea who creates that):

In typical CNI practice, the flannel.conflist is made by the flannel container..and my personal experience shows that it's not uncommon for different CNI providers to do quirky things with their config

In our case though, we're seeing this with both the upstream flannel container and our openSUSE built one.

[thkukuk]

On Fri, Jul 19, Richard Brown wrote:

In typical CNI practice, the flannel.conflist is made by the flannel
container..and my personal experience shows that it's not uncommon for
different CNI providers to do quirky things with their config

I think we need this fix:
https://github.com/coreos/flannel/pull/1135

--
Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS
SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany
GF: Felix Imendoerffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nuernberg)

[saschagrunert]

I think we need this fix: coreos/flannel#1135

Yes, for flannel this fix is needed.

[thkukuk]

Fixed for openSUSE Kubic, passes test suite now.

[shaoyuta]

@thkukuk , Yes, coreos/flannel#1135 can resolve this issue.
Thanks a lot.

On Fri, Jul 19, Richard Brown wrote: In typical CNI practice, the flannel.conflist is made by the flannel container..and my personal experience shows that it's not uncommon for different CNI providers to do quirky things with their config
I think we need this fix: coreos/flannel#1135
…
-- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: Felix Imendoerffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nuernberg)