Libnetwork: Overlay network issues with setting MTU and performance implications

[bauerm97]

Just to start with my set up: I am running stock Ubuntu 16.04 LTS release on two nodes in a test cluster environment. We are testing the feasibility of scaling docker to our HPC environment, and thus these two specific test nodes are connected via IPoIB and are running as a swarm (not in swarm mode). The docker version output can be seen below.

mibauer@node5:/$ docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64

This is a seemingly very complex issue and has several parts to it, so I will break it down into sections

Overlay bandwidth issues

First we need to know the performance of IPoIB in ideal conditions, from host to host. Running iperf3 between the two nodes we see performance like this:

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  13.0 GBytes  11.1 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  13.0 GBytes  11.1 Gbits/sec                  receiver

iperf Done.

Next we check the bandwidth from a container on node 4 to a container on node 5 using iperf. To start, fire up a container on node4 and run iperf3 in server mode:

mibauer@node4:~$ docker network create --driver overlay --subnet=192.168.20.0/24 overlay1
mibauer@node4:~$ docker run -it --name=o1t1 --network=overlay1 192.168.2.1:5000/fairsoft.nosrc
root@c8de2553174e:/# iperf3 -s

Then fire one up on node5 and run iperf3 as the client,

mibauer@node5:/$ docker run -it --name=o1t2 --network=overlay1 192.168.2.1:5000/fairsoft.nosrc
root@bd44c7f8ad60:/# iperf3 -c o1t1 -l 1M -Z

Now check the performance of iperf3 after it finishes:

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  2.56 MBytes  2.15 Mbits/sec  531             sender
[  4]   0.00-10.00  sec  2.44 MBytes  2.04 Mbits/sec                  receiver

iperf Done.

One thing I had seen while tuning our IPoIB for optimal performance, is that our IPoIB interface is running in connected mode with an MTU of 65520 (on both nodes), therefore increasing the MTU should increase the performance significantly.

I tested this theory using the calico libnetwork plugin. When running with the default MTU we see similar performance (~2Mb/s), but when upping the container and host side of the virtual interface's MTUs, we see performance more in line with the host (~13 Gb/s). This brings me to the second issue

Unable to set the MTU as a driver option

According to #1349 it should now be possible to set the MTU by specifying a driver option on creation of the network. According to @aboch in #1021 I should first need to ensure that the docker_gwbridge network has a properly set MTU. I have tried this two ways: 1) by passing the daemon --mtu=65520 as an arg and 2) by manually creating the docker_gwbridge network. This works fine, as I can see in my container and on my host that the docker_gwbridge has an MTU of 65520. Now that the gwbridge is properly configured, I should be able to create an overlay network with the driver option of mtu, and it should be honored.

Now we try the network creation again:

mibauer@node4:~$ docker network create --opt com.docker.network.mtu=65520 --driver overlay --subnet=192.168.22.0/24 overlay2
mibauer@node4:~$ docker run -it --name=o2t1 --network=overlay2 192.168.2.1:5000/fairsoft.nosrc
root@830662f22ca1:/# ip addr
284: eth0@if285: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 02:42:c0:a8:16:02 brd ff:ff:ff:ff:ff:ff
    inet 192.168.22.2/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:c0ff:fea8:1602/64 scope link 
       valid_lft forever preferred_lft forever

Inside the container I can immediately see the MTU of this interface is not properly set. Performance testing again shows the same speed (~2Mb/s). At this point in time, I just want to know if the MTU is actually the issue, so I set about manually setting the MTU of the container side interface. The MTU of all interfaces on the host correctly obey the MTU settings. This brings me to the third and most perplexing problem that I'm having:

It seems to be impossible to manually set the MTU

I wanted to go in and manually change the MTU now, just to see if this is the issue. I use nsenter to execute in the network namespace of my container, and use ifconfig to set the interfaces MTU. This sets the interface MTU, and now I test with iperf3 again:

root@c7f436c4d489:/# iperf3 -c o2t1 -l 1M -Z
Connecting to host o2t1, port 5201
[  4] local 192.168.22.3 port 38858 connected to 192.168.22.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  2.38 MBytes  19.9 Mbits/sec    3   21.4 KBytes       
[  4]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   21.4 KBytes       
[  4]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   21.4 KBytes       
[  4]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   21.4 KBytes  

This result really puzzles me. When iperf3 hangs like this it is because of an MTU conflict somewhere on the path, somewhere the MTU is too low for the outgoing traffic from my container. However, I have already set the MTU on the host as well as in the container (even accounting for 50 byte overhead required by vxLan), which should be all of the interfaces I need to set. Closer inspection of the container side interface leads me to a different story:

mibauer@node5:/$ sudo nsenter -n -t $(sudo docker inspect --format '{{ .State.Pid }}' o2t2) /bin/bash
root@node5:/# ip addr 
249: eth0@if250: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue state UP group default 
    link/ether 02:42:c0:a8:16:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.22.3/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:c0ff:fea8:1603/64 scope link 
       valid_lft forever preferred_lft forever

The eth0 interface on the container (if index 249) claims to be linked to the interface with an index of 250. On the host side, we see that there is no interface 250 to speak of. The only new interface I see is this veth interface:

252: veth84d801c@if251: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc noqueue master docker_gwbridge state UP group default 
    link/ether 92:cc:ce:d2:5c:85 brd ff:ff:ff:ff:ff:ff link-netnsid 11
    inet6 fe80::90cc:ceff:fed2:5c85/64 scope link 
       valid_lft forever preferred_lft forever

This is interface index 252 linked with interface index 251 (which appears to be in namespace nsid 11). It appears to me that the interface on the container is connected to interface index 250 which is in network namespace nsid 11 according to the host. The data is then moved to interface index 251 inside nsid 11 and then connects up again to the veth interface with index 252 on the host.

I have attempted to put myself into nsid 11 using nsenter however that appears to be impossible. The nsenter package requires the process ID which controls the namespace (which I don't know for nsid 11) or the file where the namespace is located (which I again don't know). I ran a python script that scrapes through the /proc/ folder and inspects the /proc/*/ns/net file against the inode number of the hosts network namespace, however again it only returned the PID of the container itself, not this mysterious "middle man". Also, the ip netns package appears to provide no functionality to find the location of a namespace from the nsid:

mibauer@node5:/$ ip netns list-id
nsid 0 
nsid 1 
nsid 2 
nsid 3 
nsid 4 
nsid 5 (iproute2 netns name: test)
nsid 6 
nsid 7 
nsid 8 
nsid 10 
nsid 11 

Notice how nsid 5 has a name attached: this is because there is a symlink existing that is named test, that exists inside a specified directory that ip netns looks in. All ip netns commands (inclduing exec) require the name to execute a command inside that namespace, therefore the nsid appears to be useless. Upon examining the source code of ip netns, I can't find any helpful information on a way to actually backtrack to the location of these namespaces.

Why am I seeing this behavior with the mysterious pass through namespace? Why does this namespace not belong to any process running on my system? Where does this namespace exist on my file system and why does it exist? How can I manually set the MTU of these interfaces if I can't access this namespace?

Thank you for reading, if you have any information at all please let me know. If there is code that needs to be fixed, I would be willing to try and help.

[aboch]

@bauerm97

--opt com.docker.network.mtu=65520

It should be com.docker.network.driver.mtu

We just recently fixed the documentation, which was suggesting the wrong option (sorry for that): https://github.com/docker/docker/blob/master/docs/reference/commandline/network_create.md

[aboch]

I have attempted to put myself into nsid 11 using nsenter however that appears to be impossible.

Get into /var/run/docker/netns, identify the file which name start with d-<first digits of overlay network id> (where d is an integer). Run nsenter --net=<that file name> and you will get into the overlay network sandbox (network namespace).

I can't answer your other more detailed questions about the network namespace mgmt. For that we neeed @mrjana 's help.

[bauerm97]

@aboch

Get into /var/run/docker/netns, identify the file which name start with d- (where d is an integer). Run nsenter --net= and you will get into the overlay network sandbox (network namespace).

This works, I can now see inside this network namespace. Thank you very much for letting me know this location. Is there a reason why there isn't a symlink created to the /var/run/netns folder to have this namespace accessible via ip netns? I would be willing to add support for this if wanted

overlay network sandbox (network namespace).

Specifically to touch on this point: this was one of the ideas I had floating around for what this mysterious network namespace was. Would it be useful to potentially document this fact somewhere? I spent a few hours rummaging through the code and couldn't find any indication of if the sandbox is an actual network namespace or just an abstract idea, however I may not have looked in the right locations.

--opt com.docker.network.mtu=65520
It should be com.docker.network.driver.mtu

We just recently fixed the documentation, which was suggesting the wrong option (sorry for that): https://github.com/docker/docker/blob/master/docs/reference/commandline/network_create.md

Awesome, glad to hear the documentation got fixed. If there are any necessary implementations for better support of MTU options, please let me know and I would be happy to help.

Also, would you be able to tell me exactly how the traffic gets routed through the system. We monitored the network traffic on the host namespace with nload and only saw outgoing traffic through the ib0 interface, no incoming traffic was seen on any other interfaces. How exactly does the data get to the ib0 interface?

[bauerm97]

It should be com.docker.network.driver.mtu

We just recently fixed the documentation, which was suggesting the wrong option (sorry for that): https://github.com/docker/docker/blob/master/docs/reference/commandline/network_create.md

Just to post a quick update on this part, I'm back at work this morning and I tested out the bandwidth with the overlay network using the proper MTU driver option. Our throughput is now approximately equal to the host to host benchmarks.

[aboch]

Closing as submitter confirmed mtu setting on overlay driver networks works fine.