Compose: The order in which networks are assigned to interfaces does not match the order they are listed in compose file

[larsks]

The following report was produced using Docker version 17.03.0-ce and compose version docker-compose version 1.11.2, build dfed245.

Given a docker-compose.yml that looks like this:

version: "2"

services:
  server:
    image: alpine
    command: sleep 999
    networks:
      - nw0
      - nw1
      - nw2
      - nw3

networks:
  nw0:
  nw1:
  nw2:
  nw3:

I would expect the networks to be assigned to interface in order, such that eth0 is attached to nw0, eth1 is attached to nw1, etc. This is exactly the behavior I see if I start a container using docker run and then attach additional networks with docker network connect.

However, when using docker-compose and the above compose file, the ordering of networks and interfaces appears to be inconsistent. Assuming that the above compose file is in a directory named nwtest, this script will demonstrate the problem:

#!/bin/sh

docker-compose up -d

for nw in 0 1 2 3; do
    nw_cidr=$(docker network inspect -f '{{ (index .IPAM.Config 0).Subnet }}' \
        nwtest_nw${nw})
    if_cidr=$(docker exec -it nwtest_server_1 ip addr show eth${nw} |
        awk '$1 == "inet" {print $2}')

    nw_net=$(ipcalc -n $nw_cidr | cut -f2 -d=)
    if_net=$(ipcalc -n $if_cidr | cut -f2 -d=)

    echo "nw${nw} $nw_net eth${nw} ${if_net}"

    if [ "$if_net" != "$nw_net" ]; then
        echo "MISMATCH: nw${nw} = $nw_net, eth${nw} = $if_net" >&2
    fi
done

docker-compose stop

On my system, that produces as output:

Starting nwtest_server_1
nw0 192.168.32.0 eth0 192.168.32.0
nw1 192.168.48.0 eth1 192.168.48.0
nw2 192.168.64.0 eth2 192.168.80.0
MISMATCH: nw2 = 192.168.64.0, eth2 = 192.168.80.0
nw3 192.168.80.0 eth3 192.168.64.0
MISMATCH: nw3 = 192.168.80.0, eth3 = 192.168.64.0
Stopping nwtest_server_1 ... done

For comparison, here is a script that performs the same test using docker run and manual attachment:

#!/bin/sh

docker rm -f nwtest_server_1
docker run -d --name nwtest_server_1 --network nwtest_nw0 \
    alpine sleep 999

for nw in 1 2 3; do
    docker network connect nwtest_nw${nw} nwtest_server_1
done

for nw in 0 1 2 3; do
    nw_cidr=$(docker network inspect -f '{{ (index .IPAM.Config 0).Subnet }}' \
        nwtest_nw${nw})
    if_cidr=$(docker exec -it nwtest_server_1 ip addr show eth${nw} |
        awk '$1 == "inet" {print $2}')

    nw_net=$(ipcalc -n $nw_cidr | cut -f2 -d=)
    if_net=$(ipcalc -n $if_cidr | cut -f2 -d=)

    echo "nw${nw} $nw_net eth${nw} ${if_net}"

    if [ "$if_net" != "$nw_net" ]; then
        echo "MISMATCH: nw${nw} = $nw_net, eth${nw} = $if_net" >&2
    fi
done

docker rm -f nwtest_server_1

This always runs without error.

[johnharris85]

Initial guess would be networks is implemented as a dict. Will take a look this evening.

[tgogos]

Thanks @johnharris85
There is also an open bounty at SO (link): , where you can also add your answer.

[an-tex]

Same issue. Gets tricky when you have to rely on the order e.g. for iptables rules like "iptables -t nat -A POSTROUTING -o eth3 -j MASQUERADE"

[akshshar]

+1. Trying to spin up some router topologies using docker-compose. This completely screws up the connections for the protocols.

[narayan-iyengar]

+1 form me as well.
Editing my response to add some details. I thought about what @johnharris85 mentioned, so I numbered my network names in the order I want them to be connected and I can get around the issue for now.

For example part of my compose looks like this:
networks:
-- 1firstnw
-- 2nw
-- 3nw

Hope this helps someone.

[LuisPiedra]

+1

Workaround of setting alphabetically order is not working for me in docker-compose version 1.13.0, build 1719ceb, it seems to be adding ifaces randomly.

In any case, even if alphabetically works, I still need to have the same network assigned to different interfaces in different containers, so still need to force the ordering.

As @johnharris85 mentioned, network configuration is passed to docker engine as an unordered dict if networks are created with the container. One possible implementation would be to first create the container and then connect the container to each network one by one. This behaviour could be also configurable.

[LuisPiedra]

I have been playing with the scripts provided by @larsks , and docker engine keeps the order of the interfaces only when networks are connected to a running container. Otherwise, it will also create the interfaces randomly.

If the container is started after connecting the networks, it will also have mismatches

docker rm -f nwtest_server_1
docker create --name nwtest_server_1 --network nwtest_nw0 \
    alpine sleep 999

for nw in 1 2 3 4 5 6 7 8 9; do
    docker network connect nwtest_nw${nw} nwtest_server_1
done

docker start nwtest_server_1

Same happens if container is restarted

docker rm -f nwtest_server_1
docker run -d --name nwtest_server_1 --network nwtest_nw0 \
    alpine sleep 999

for nw in 1 2 3 4 5 6 7 8 9; do
    docker network connect nwtest_nw${nw} nwtest_server_1
done

docker stop nwtest_server_1
docker start nwtest_server_1

Anyway, even if docker engine offers support for this behaviour (this is already adressed: https://github.com/moby/moby/issues/25181), docker-compose won't respect the definition order since original network list is converted to an unordered list.

In addition, if connections include parameters like a fixed ip address, networks is not a list at all, it is implemented as a dictionary. Maybe some additional parameter like order or priority would be needed if docker engine implements this behaviour at some point.

[mwiget]

I thought I had an easy fix for this, but it turns out docker network connect plays its own game, as @LuisPiedra pointed out correctly. It is not enough to sort the calls to connect_container_to_network() ...

[Ageraluon]

+1, any progress on this yet?

[shin-]

Issue should be fixed by #5566 - at least the Compose aspect of it.

[eugenepark1]

@shin can you elaborate what exactly is the fix? there werent much details on the link

[gentunian]

extending @larsks example I've created this gist that demonstrate that ordering is dict-based and not yaml-based.

The gist include everything you need to run the test. Just chmod+x it.

@shin is this working on docker swarm?

[shin-]

https://docs.docker.com/compose/compose-file/compose-file-v2/#priority

Swarm features should be requested on the Swarm repo or Moby repo.

[gentunian]

@shin it seems that priority is missing in version 3, so my guess is that is deprecated.

[jfellus]

@gentunian after looking at the code, I don't see any difference in handling network priorities between v2 and v3.
See https://github.com/docker/compose/blob/master/compose/network.py#L325, and PR docker/compose#5566 clearly show that v3 is ok since at least jan 2018...

however, a rapid test shows that priorities are currently ignored by docker, would you choose v2 or v3... lexical order of network names seems to be overriding the priorities you set. see https://gist.github.com/jfellus/cfee9efc1e8e1baf9d15314f16a46eca

(the answer to this may also answer https://github.com/docker/cli/issues/1372)