Libnetwork: cant start docker 1.10 with ipv6 disabled via ipv6.disable=1 and an existing bridge with an IP

[ppickfor]

If OS is booted with ipv6.disable=1, to disable ipv6, and docker is started on an existing bridge with an IP address then docker fails to start with the following error:

"Error starting daemon: Error initializing network controller: Error creating default \"bridge\" network: Failed to setup IP tables, cannot acquire Interface address: Interface docker0 has no IPv4 addresses"

Removing ipv6.disable=1 is a work around but I would prefer to run without ipv6 enabled.

I veified that there was an IP address before attempting to start docker. After attempting to start docker there is no IP address on the bridge. Docker can start if is creates the bridge but not a second time as the bridge already exists.

to duplicate:
boot os with ipv6.disable=1
create a bridge
assign and ip address to the bridge
attempt to start docker with -b bridgename
(or start docker twice without specifying a bridge when booted with ipv6.disable=1)

[ricardobranco777]

This bug happens on Docker 1.10.1 too.

[mavenugo]

ping @aboch

[fudanchii]

I tried to debug this and found that netlink.AddrList(i.Link, netlink.FAMILY_V6) leaks ipv4 in environment with ipv6.disable=1, which in result will be removed at setupVerifyAndReconcile since it's assumed as IPv6.
We can work around this by filtering AddrList result for IPv6, I patch bridgeInterface.addresses like this:

diff --git a/vendor/src/github.com/docker/libnetwork/drivers/bridge/interface.go b/vendor/src/github.com/docker/libnetwork/drivers/bridge/interface.go
index cdf6883..a913cbc 100644
--- a/vendor/src/github.com/docker/libnetwork/drivers/bridge/interface.go
+++ b/vendor/src/github.com/docker/libnetwork/drivers/bridge/interface.go
@@ -4,6 +4,7 @@ import (
        "fmt"
        "net"

+       "github.com/docker/libnetwork/types"
        "github.com/vishvananda/netlink"
 )

@@ -52,14 +53,23 @@ func (i *bridgeInterface) addresses() (netlink.Addr, []netlink.Addr, error) {
                return netlink.Addr{}, nil, err
        }

-       v6addr, err := netlink.AddrList(i.Link, netlink.FAMILY_V6)
+       v6addrUnFiltered, err := netlink.AddrList(i.Link, netlink.FAMILY_V6)
        if err != nil {
                return netlink.Addr{}, nil, err
        }

        if len(v4addr) == 0 {
-               return netlink.Addr{}, v6addr, nil
+               return netlink.Addr{}, v6addrUnFiltered, nil
        }
+
+       // Ensure netlink.AddrList doesn't leak IPv4
+       var v6addr []netlink.Addr
+       for _, ipv6 := range v6addrUnFiltered {
+               if !types.CompareIPNet(ipv6.IPNet, v4addr[0].IPNet) {
+                       v6addr = append(v6addr, ipv6)
+               }
+       }
+
        return v4addr[0], v6addr, nil
 }

Will put a pull request if this looks ok

[fudanchii]

Apparently a fix has been proposed here
Please disregard my comment above, thanks.

[chenchun]

@ppickfor @fudanchii Can you share your kernel version?
I can't reproduce this with the following step

root@docker-1:/home/vagrant# ip ad show docker0
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:28:3f:59:29 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
root@docker-1:/home/vagrant# cat netlink.go 
package main

import (
    "fmt"

    "github.com/vishvananda/netlink"
)

func main() {
    link, _ := netlink.LinkByName("docker0")
    addrs, _ := netlink.AddrList(link, netlink.FAMILY_V4)
    for _, addr := range addrs {
        fmt.Printf("addrv4 %s\n", addr.String())
    }

    addrs, _ = netlink.AddrList(link, netlink.FAMILY_V6)
    for _, addr := range addrs {
        fmt.Printf("addrv6 %s\n", addr.String())
    }
}

root@docker-1:/home/vagrant# go run netlink.go 
addrv4 172.17.0.1/16 docker0

My environment

root@docker-1:/home/vagrant# uname -a
Linux docker-1 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt9-2 (2015-04-13) x86_64 GNU/Linux
root@docker-1:/home/vagrant# sysctl -p
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

root@docker-1:/vagrant/docker# echo $GOPATH
/vagrant/docker/vendor/
root@docker-1:/vagrant/docker# git branch
* (detached from v1.10.0)

[fudanchii]

Ubuntu 14.04.3 and arch Linux with the latest kernel. IPv6 disabled completely at boot, not via sysctl. Try passing ipv6.disable=1 to kernel parameter at grub menu. sysctl -a won't show any ipv6 parameter if it's disabled completely.

edit:
my test env:

adie@archr:~ $ sysctl -a | grep ipv6
sysctl: permission denied on key 'fs.protected_hardlinks'
sysctl: permission denied on key 'fs.protected_symlinks'
sysctl: permission denied on key 'kernel.cad_pid'
sysctl: permission denied on key 'kernel.usermodehelper.bset'
sysctl: permission denied on key 'kernel.usermodehelper.inheritable'
sysctl: permission denied on key 'net.ipv4.tcp_fastopen_key'
adie@archr:~ $ go/sandbox/nltest    
IPv4> "192.168.122.201/24"
IPv6> "192.168.122.201/24"
adie@archr:~ $ cat go/sandbox/nltest.go 
package main

import (
    "fmt"
    "log"
    "os"

    "github.com/vishvananda/netlink"
)

func main() {
    ifname := "eth0"
    if len(os.Args) > 1 {
        ifname = os.Args[1]
    }

    iflink, err := netlink.LinkByName(ifname)
    if err != nil {
        log.Fatalf("can not get link for %s: %s", ifname, err.Error())
    }

    v4addr, err := netlink.AddrList(iflink, netlink.FAMILY_V4)
    if err != nil {
        log.Fatalf("can not get IPv4 addresses: %s", err.Error())
    }

    v6addr, err := netlink.AddrList(iflink, netlink.FAMILY_V6)
    if err != nil {
        log.Fatalf("can not get IPv6 addresses: %s", err.Error())
    }

    for _, v4 := range v4addr {
        fmt.Printf("IPv4> %q\n", v4.IPNet)
    }

    for _, v6 := range v6addr {
        fmt.Printf("IPv6> %q\n", v6.IPNet)
    }
}
adie@archr:~ $ uname -a
Linux archr.vm.kuon 4.4.1-2-ARCH #1 SMP PREEMPT Wed Feb 3 13:12:33 UTC 2016 x86_64 GNU/Linux

With IPv6 enabled:

adie@archr:~ $ go/sandbox/nltest 
IPv4> "192.168.122.201/24"
IPv6> "fe80::5e2:5d54:9aa2:aa46/64"

[chenchun]

I can reproduce now following "start docker twice without specifying a bridge". I tried to patch https://github.com/vishvananda/netlink/pull/91 to docker and it really solves the issue. After https://github.com/vishvananda/netlink/pull/91 getting merged, maybe @fudanchii you can push a PR to libnetwork ?

[fudanchii]

Sure, will do :+1:

[ppickfor]

applied
force AddList to get rid of unexpected information
from vishvananda/netlink#91
to docker master
commit b16536aeb519e8d449a8231606b49520a160e44
( I suspect there's a better way )

I can start up with ipv6.disable=1 and docker does not wipe the bridge
ip address

looks OK to me

looking forward to a binary I can try with this

Thanks

Peter

[liusdu]

Sorry for seeing this issue so late. And the PR vishvananda/netlink#91 is for this issue. Thanks @fudanchii !

[aboch]

Thanks @fudanchii.

Now that https://github.com/vishvananda/netlink/pull/91 is merged, can you please push a PR to libnetwork to import the latest netlink.

Thanks.

[dkiser]

Was this introduced in 1.10.0?

[aboch]

@dkiser Yes

[dkiser]

@aboch any workarounds for us that can't upgrade to 1.11.0 yet?

[aboch]

@dkiser Sorry, not many options other than

• booting your os with ipv6.disable=0 or
• start the daemon with --iptables=false

[ricardobranco777]

The other option is the net.ipv6.conf.all.disable_ipv6=1 sysctl. Consider upgrading to 1.11 because it seems mostly a bugfix release based on the changelog.

[jmfee-usgs]

We had a similar issue on a centos like system Scientific Linux release 7.2, with disable_ipv6=1 and using iptables instead of firewalld. The docker0 interface was left behind, and the service failed to start with a docker0 has no ipv4 addresses error.

Docker version 1.10.3

I added a systemd override file to manually clean up the "orphaned" docker0 interface whenever the service is stopped or fails to start.

Add an override file (or configure the ExecStopPost command elsewhere) /etc/systemd/system/docker.service.d/cleanup_docker0_bridge.conf:

[Service]
ExecStopPost=/bin/bash -c "/sbin/ip link show docker0 && /sbin/ip link delete docker0"

Then update systemd with:

/usr/bin/systemctl daemon-reload