Kubevirt: MTU inside VM doesn't reflect tap device MTU

[booxter]

MTU mismatch is a widely known root cause of TCP connection timeouts and other issues.

In the pod:

[vagrant@node02 ~]$ sudo docker exec -it  $(sudo docker ps | grep virt-launcher-ovm-cirros | grep entrypoint.sh | awk -e '{print $1}') ip a
...
3: eth0@if55: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br1 state UP group default 
    link/ether 72:c3:ee:4e:61:a7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::70c3:eeff:fe4e:61a7/64 scope link 
       valid_lft forever preferred_lft forever
4: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 72:c3:ee:4e:61:a7 brd ff:ff:ff:ff:ff:ff
    inet 169.254.75.86/32 brd 169.254.75.86 scope global br1
       valid_lft forever preferred_lft forever
    inet6 fe80::70c3:eeff:fe4e:61a7/64 scope link 
       valid_lft forever preferred_lft forever
5: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master br1 state UNKNOWN group default qlen 1000
    link/ether fe:58:0a:f4:01:33 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc58:aff:fef4:133/64 scope link 
       valid_lft forever preferred_lft forever

In the VM:

$ ip a                                                     
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000                                         
    link/ether 0a:58:0a:f4:01:33 brd ff:ff:ff:ff:ff:ff     
    inet 10.244.1.51/24 brd 10.244.1.255 scope global eth0 
       valid_lft forever preferred_lft forever             
    inet6 fe80::858:aff:fef4:133/64 scope link             
       valid_lft forever preferred_lft forever             

Note that it's 1450 in the pod but 1500 inside the VM.

Connectivity issue can be confirmed with the following commands executed from the VM:

$ ping -s 1422 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1422 data bytes
1430 bytes from 8.8.8.8: seq=0 ttl=56 time=14.883 ms
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 14.883/14.883/14.883 ms
$ ping -s 1423 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1423 data bytes
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

As you can see, anything larger than 1422 (+28 bytes for IP header overhead) doesn't get through the network.

The fix here is to advertise desired MTU to the VM via DHCP option: http://www.networksorcery.com/enp/protocol/bootp/option026.htm (Note it assumes DHCPv4 enabled; IPv6-only images, or images with static IP configuration, won't work; time will tell if these are legit scenarios to handle for kubevirt, so this issue is for the base case of DHCPv4 enabled images only.)

[rmohr]

@booxter would you also like to work on a PR for that? Just let us know.

[booxter]

Yes @rmohr I am working on it, I just don't see an option for me to self-assign the issue (btw the fix will probably also require some preparatory steps in https://github.com/vishvananda/netlink to expose interface MTU).

[vladikr]

btw the fix will probably also require some preparatory steps in https://github.com/vishvananda/netlink to expose interface MTU

I think, maybe I'm wrong, if you'd get a link to the device, you could read/write the MTU via the attrs
https://godoc.org/github.com/vishvananda/netlink#LinkAttrs

[booxter]

You are right @vladikr I already figured it out and have the code, will post later.

[booxter]

I have a fix (see above) though I need to add a functional test case before proposing it for merge.

The test I envision will: 1) extract eth0 MTU from result of 'ip a' called on the VM; 2) ping the VM with the interface MTU sized payload, and vice versa, and check that ping succeeds. I will probably need to extract remote command execution logic from an Expecter usage in tests/vm_userdata_test.go and reuse it in the new test case.

[rmohr]

The scenario sounds ok in general. Another one would be to start a vm, open serial console connection, check MTU there, then open a kubectl exec connection to the pod of the vm and check mtu on the bridge too ...

We have helper functions for executing commands inside pods and inside vms, they may help you: ExecuteCommandOnPod and LoggedInCirrosExpecter.