Runtime: Internet Connectivity issues in kata-qemu on Kubernetes CRIO Runtime

[shreyasbapat]

Description of problem

Followed exact steps in this for Kubernetes 1.14+
https://github.com/kata-containers/packaging/tree/master/kata-deploy

When I run the sample deployment, the pod does not have internet access. It can not even ping host machine.

Tried out the solution in https://github.com/kata-containers/runtime/issues/1364 without success (As 8.8.8.8 or 1.1.1.1 is not reachable).

ip a does not list proper network interfaces which causes this.

Expected result

Internet connection working

Actual result

Output of ip a :

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

If I just disable the runtimeclass, and run the container in CRIO,

ip a gives:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 26:37:17:61:9c:e8 brd ff:ff:ff:ff:ff:ff
    inet 10.233.106.67/32 brd 10.233.106.67 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::2437:17ff:fe61:9ce8/64 scope link 
       valid_lft forever preferred_lft forever

The eth0 interface here makes internet access possible.

Further information

I would appreciate getting any pointers on this, I have followed the manual as-is. Thanks in advance.

[fidencio]

@shreyasbapat, I'm not the most appropriate person to help you here, but I'd like to ask some preliminary information. Would you mind to run kata-collect-data.sh and paste the output in the issue? This script will collect all the possible information about your configuration and will help us to, at least, be able to reproduce the issue.

[fidencio]

Also, basic information about flannel version and host OS would be very very welcome.

[shreyasbapat]

Hey @fidencio , where will I find this script? I installed kata using this: https://github.com/kata-containers/packaging/tree/master/kata-deploy#kubernetes-quick-start

I am not sure where the script is located in. I would be glad if you could help me out there.

[caoruidong]

@shreyasbapat https://github.com/kata-containers/runtime/blob/master/data/kata-collect-data.sh.in

[jodh-intel]

@shreyasbapat - the script should be located at /opt/kata/bin/kata-collect-data.sh.

[shreyasbapat]

It errors for me:
I assume this is because of the runtimeclasses in the Kubernetes Cluster?

ubuntu@ip-10-1-32-46:~$ sudo bash /opt/kata/bin/kata-collect-data.sh
ERROR: kata-collect-data.sh: cannot find runtime 'kata-runtime'

EDIT: I recently discovered that the kata-deploy pod was down from the cluster. I am investigating and then will post the output again.

[shreyasbapat]

Also, basic information about flannel version and host OS would be very very welcome.

We don't use flannel, we use calico. Host OS is Ubuntu 18.04

[jodh-intel]

Does the following work for you?

$ sudo -E PATH="$PATH:/opt/kata/bin" bash -c 'kata-collect-data.sh'

[shreyasbapat]

Thanks @jodh-intel that helped. This is the output of the script: https://gist.github.com/shreyasbapat/e9cc6b20f32f3af1d2847fad1679c712

[jodh-intel]

Thanks @shreyasbapat. Could you try enabling debug output and re-running the script?

Note that the script output is designed to be pasted directly into this issue as a comment: that way, it gets nicely formatted :smile:

[shreyasbapat]

Thanks @shreyasbapat. Could you try enabling debug output and re-running the script?

Note that the script output is designed to be pasted directly into this issue as a comment: that way, it gets nicely formatted 😄

Sure, let me do that

[shreyasbapat]

Thanks @shreyasbapat. Could you try enabling debug output and re-running the script?

Note that the script output is designed to be pasted directly into this issue as a comment: that way, it gets nicely formatted 😄

Hey @jodh-intel , I am not able to do that:

ubuntu@worker:~$ sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers
install: cannot stat '/usr/share/defaults/kata-containers/configuration.toml': No such file or directory

[fidencio]

/cc @ariel-adam, can you point this one to Stefano Brivio? I couldn't find him on GitHub.