Libnetwork: Unable to create docker0 if VPN is active

[Manouchehri]

~ > ip route
0.0.0.0/1 via 10.8.0.5 dev tun0 
default via 192.168.1.1 dev wlp3s0  proto dhcp  src 192.168.1.191  metric 10 
default via 192.168.1.1 dev enp2s0  proto dhcp  src 192.168.1.181  metric 20 
10.8.0.1 via 10.8.0.5 dev tun0 
10.8.0.5 dev tun0  proto kernel  scope link  src 10.8.0.6 
23.196.166.175 via 192.168.1.1 dev wlp3s0 
128.0.0.0/1 via 10.8.0.5 dev tun0 
192.168.1.0/24 dev enp2s0  proto kernel  scope link  src 192.168.1.181 
192.168.1.0/24 dev wlp3s0  proto kernel  scope link  src 192.168.1.191 
192.168.1.1 dev wlp3s0  proto dhcp  scope link  src 192.168.1.191  metric 10 
192.168.1.1 dev enp2s0  proto dhcp  scope link  src 192.168.1.181  metric 20 
~ > sudo docker -d
Warning: '-d' is deprecated, it will be removed soon. See usage.
WARN[0000] please use 'docker daemon' instead.          
INFO[0000] API listen on /var/run/docker.sock           
INFO[0000] [graphdriver] using prior storage driver "btrfs" 
INFO[0000] Firewalld running: false                     
FATA[0000] Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network 

If I stop my VPN, then I'm able to create the interface.

~ > sudo systemctl stop [email protected]
~ > ip route
default via 192.168.1.1 dev wlp3s0  proto dhcp  src 192.168.1.191  metric 10 
default via 192.168.1.1 dev enp2s0  proto dhcp  src 192.168.1.181  metric 20 
23.196.166.175 via 192.168.1.1 dev wlp3s0 
192.168.1.0/24 dev enp2s0  proto kernel  scope link  src 192.168.1.181 
192.168.1.0/24 dev wlp3s0  proto kernel  scope link  src 192.168.1.191 
192.168.1.1 dev wlp3s0  proto dhcp  scope link  src 192.168.1.191  metric 10 
192.168.1.1 dev enp2s0  proto dhcp  scope link  src 192.168.1.181  metric 20 
~ > sudo docker -d
Warning: '-d' is deprecated, it will be removed soon. See usage.
WARN[0000] please use 'docker daemon' instead.          
INFO[0000] API listen on /var/run/docker.sock           
INFO[0000] [graphdriver] using prior storage driver "btrfs" 
INFO[0000] Firewalld running: false                     
INFO[0000] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
INFO[0000] Loading containers: start.                   

INFO[0000] Loading containers: done.                    
INFO[0000] Daemon has completed initialization          
INFO[0000] Docker daemon                                 commit=a34a1d5-dirty execdriver=native-0.2 graphdriver=btrfs version=1.9.1
^CINFO[0003] Processing signal 'interrupt'  

After the interface is created, then there's no issues with starting the daemon while my VPN is active.

~ > sudo systemctl start [email protected]
~ > ip route
0.0.0.0/1 via 10.8.0.5 dev tun0 
default via 192.168.1.1 dev wlp3s0  proto dhcp  src 192.168.1.191  metric 10 
default via 192.168.1.1 dev enp2s0  proto dhcp  src 192.168.1.181  metric 20 
10.8.0.1 via 10.8.0.5 dev tun0 
10.8.0.5 dev tun0  proto kernel  scope link  src 10.8.0.6 
23.196.166.175 via 192.168.1.1 dev wlp3s0 
128.0.0.0/1 via 10.8.0.5 dev tun0 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 
192.168.1.0/24 dev enp2s0  proto kernel  scope link  src 192.168.1.181 
192.168.1.0/24 dev wlp3s0  proto kernel  scope link  src 192.168.1.191 
192.168.1.1 dev wlp3s0  proto dhcp  scope link  src 192.168.1.191  metric 10 
192.168.1.1 dev enp2s0  proto dhcp  scope link  src 192.168.1.181  metric 20 
~ > sudo docker -d
Warning: '-d' is deprecated, it will be removed soon. See usage.
WARN[0000] please use 'docker daemon' instead.          
INFO[0000] API listen on /var/run/docker.sock           
INFO[0000] [graphdriver] using prior storage driver "btrfs" 
INFO[0000] Firewalld running: false                     
INFO[0000] Default bridge (docker0) is assigned with an IP address 172.17.0.1/16. Daemon option --bip can be used to set a preferred IP address 
INFO[0000] Loading containers: start.                   

INFO[0000] Loading containers: done.                    
INFO[0000] Daemon has completed initialization          
INFO[0000] Docker daemon                                 commit=a34a1d5-dirty execdriver=native-0.2 graphdriver=btrfs

[hkjn]

A colleague of mine reported the same behavior on Ubuntu, also while connected to VPN (tun0 device created with openvpn).

Their /var/log/upstart/docker.log output:

redefined network 
/var/run/docker.sock is up
INFO [graphdriver] using prior storage driver "aufs" 
INFO API listen on /var/run/docker.sock           
INFO Firewalld running: false                     
FATA Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network 

Their /etc/os-release:

NAME="Ubuntu"
VERSION="14.04.2 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.2 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

Their docker version:

Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

[FrontSide]

Experiencing the same problem.
Docker 1.9.0 and 1.9.1 on Centos7 (AWS instance).

docker-client

Client:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   76d6bc9
 Built:        Tue Nov  3 18:00:05 UTC 2015
 OS/Arch:      linux/amd64

uname -r

3.10.0-123.8.1.el7.x86_64

service docker status

Redirecting to /bin/systemctl status  docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2016-01-07 11:52:22 UTC; 52s ago
     Docs: https://docs.docker.com
  Process: 25925 ExecStart=/usr/bin/docker daemon --insecure-registry docker-registry.company.net:5000 (code=exited, status=2)
 Main PID: 25925 (code=exited, status=2)
   CGroup: /system.slice/docker.service
           └─25960 mkfs.ext4 -E nodiscard,lazy_itable_init=0,lazy_journal_init=0 /dev/mapper/docker-202:1-8798356-base

Jan 07 11:50:52 hostname systemd[1]: Starting Docker Application Container Engine...
Jan 07 11:50:52 hostname docker[25925]: time="2016-01-07T11:50:52.232391369Z" level=info msg="API listen on /var/run/docker.sock"
Jan 07 11:50:52 hostname docker[25925]: time="2016-01-07T11:50:52.300489554Z" level=warning msg="Usage of loopback devices is strongly discouraged for production use. Ple...ev section."
Jan 07 11:52:22 hostname systemd[1]: docker.service start operation timed out. Terminating.
Jan 07 11:52:22 hostname systemd[1]: docker.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 07 11:52:22 hostname systemd[1]: Failed to start Docker Application Container Engine.
Jan 07 11:52:22 hostname systemd[1]: Unit docker.service entered failed state.
Jan 07 11:52:22 hostname systemd[1]: docker.service failed.

What's confusing me the most is, that it works if I wait a bit, stop the service and restart it. It still takes longer than it should, but it doesn't time out.

Also, it works instantly and without any problems on the openstack instance (same linux distro, same docker version), just not on aws.

[ramilexe]

the same problem

[aboch]

I think it is because of the default static route installed by the VPN:
0.0.0.0/1 via 10.8.0.5 dev tun0

When docker daemon comes up with _no_ docker0 bridge on the host, it will try to pick a subnet to assign to the default "bridge" network from a predefined list of IP pools .
While choosing, It checks whether the pool overlaps with any interface on the system, if so it discards it and move to the next one. The overlap check is run against the route list in here.

Given 0.0.0.0 will overlap with every subnet, none of the predefined pool is chosen, therefore the failure.

[FrontSide]

Ok I could solve the problem for me. It wasn't caused by VPN after all.
Apparently I installed it the wrong way or something.

I'll just leave the lines I changed here:

Did not work with:

wget https://yum.dockerproject.org/repo/main/centos/7/Packages/docker-engine-selinux-1.9.1-1.el7.centos.noarch.rpm -O docker-selinux.rpm
rpm --install --hash --verbose docker-selinux.rpm

wget https://yum.dockerproject.org/repo/main/centos/7/Packages/docker-engine-1.9.1-1.el7.centos.x86_64.rpm -O docker.rpm
rpm --install --hash --verbose docker.rpm

Worked fine with:

docker_repo_file=/etc/yum.repos.d/docker.repo

echo "[dockerrepo]" > $docker_repo_file
echo "name=Docker Repository" >> $docker_repo_file
echo "baseurl=https://yum.dockerproject.org/repo/main/centos/7/" >> $docker_repo_file
echo "enabled=1" >> $docker_repo_file
echo "gpgcheck=1" >> $docker_repo_file
echo "gpgkey=https://yum.dockerproject.org/gpg" >> $docker_repo_file

yum install docker-engine-1.9.1 -y

Note that in latter version the docker-engine-selinux dependency is handled automatically whereas, previously I had to install it manually otherwise docker-engine wouldn't install.

[KostyaEsmukov]

Same here.
Debian jessie, Docker 1.10.1, OpenVPN 2.3.4.

root@host:~# docker daemon -D
DEBU[0000] docker group found. gid: 999
DEBU[0000] Server created for HTTP on unix (/var/run/docker.sock)
DEBU[0000] Using default logging driver json-file
INFO[0000] [graphdriver] using prior storage driver "aufs"
DEBU[0000] Using graph driver aufs
INFO[0000] Graph migration to content-addressability took 0.00 seconds
DEBU[0000] Option DefaultDriver: bridge
DEBU[0000] Option DefaultNetwork: bridge
INFO[0000] Firewalld running: false
DEBU[0000] /sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -D PREROUTING]
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT]
DEBU[0000] /sbin/iptables, [--wait -t nat -F DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -X DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -F DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -X DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION]
DEBU[0000] /sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION]
DEBU[0000] /sbin/iptables, [--wait -t nat -n -L DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -N DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -n -L DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -N DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION]
DEBU[0000] /sbin/iptables, [--wait -t filter -N DOCKER-ISOLATION]
DEBU[0000] /sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -j RETURN]
DEBU[0000] /sbin/iptables, [--wait -I DOCKER-ISOLATION -j RETURN]
DEBU[0000] Registering ipam driver: "default"
DEBU[0000] Allocating IPv4 pools for network bridge (54597afee3fc122b6fc34bdf51024492ad50539118f67ce45ecf4776ef445dda)
DEBU[0000] RequestPool(LocalDefault, , , map[], false)
DEBU[0000] Cleaning up old shm/mqueue mounts: start.
FATA[0000] Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network

But Docker starts flawlessly when openvpn is stopped. So this is a workaround:

service docker stop
service openvpn stop

service docker start
service openvpn start

[KostyaEsmukov]

Same workaround using systemd drop-ins:

# ensure that openvpn connections are started after openvpn.service
mkdir -p /lib/systemd/system/[email protected]
cat > /lib/systemd/system/[email protected]/10-openvpn.conf << EOF
[Unit]
After=openvpn.service
EOF

# ensure that docker is started before openvpn.service
mkdir -p /lib/systemd/system/docker.service.d
cat > /lib/systemd/system/docker.service.d/10-openvpn.conf << EOF
[Unit]
Before=openvpn.service
EOF

systemctl daemon-reload
systemctl show docker | grep Before  # should have openvpn.service listed

[m3adow]

I encountered a similar error. When running docker on a Host with an active OpenVPN connection, I could not use docker network create without explicitly stating the --subnet range. I assume the reason for that is the default OpenVPN 0.0.0.0/1 route as @aboch mentioned. Error message is Error response from daemon: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network.
Is it possible to define a "forced" network pool which is utilised despite possible address conflicts?

[TwistTheNeil]

Same issue here
Debian sid
Docker version 1.10.3, build 20f81dd

[kamihicouki]

hi, everyone.
i had same problem.
i resolved this problem as below.
docker network rm $(docker network ls -q)

[codekoala]

I was able to resolve this problem on my systems by setting --bip in /etc/default/docker, as described by @DominicBoettger in https://github.com/docker/docker/issues/18113#issuecomment-208436349

[ufoscout]

Same issue on Ubuntu 16.04 LTS and Docker 1.11.1

[kog13]

Same issue here with Docker 1.11.1 + OpenVPN.

Cropped up when upgrading to docker-compose 1.7.1 where 'up' now creates a new custom defined network. Not sure what the long-term solution is but for the moment I am using docker-compose overrides for a CI environment and manually specifying the service network_mode: "bridge". This just avoids the creation of a network (not really ideal...) but at least gets around the: "could not find an available predefined network"

[ghost]

Same issue if I'm using openconnect.

[ghost]

Same problem here on gentoo with docker 1.11.0 - i always have to stop openvpn when a network needs to be created :( :angry:

[netsuso]

I had the same problem with docker and openvpn. It seems the networks 0.0.0.0/1 and 128.0.0.0/1 routes are created by openvpn if the "redirect-gateway def1" flag is provided from the server. As explained above, docker cannot find any free network range as those two /1 ranges cover all possible addresses.

What I did to solve it (apart from bugging the guy responsible for the openvpn service to remove the def1 flag :D) was giving openvpn a command to drop those two routes and set a true default route:

$ openvpn --config vpn_config_file --route-up fix-routes.sh

And the fix-routes.sh script:

#!/bin/sh

echo "Adding default route to $route_vpn_gateway with /0 mask..."
ip route add default via $route_vpn_gateway

echo "Removing /1 routes..."
ip route del 0.0.0.0/1 via $route_vpn_gateway
ip route del 128.0.0.0/1 via $route_vpn_gateway

With a default /0 route docker would no longer have problems finding a valid subnet

[ghost]

I can confirm the comment from @netsuso - when disabling the /1 partitioning of the networks docker networking stuff works well even with enabled openvpn.

[ivarec]

@netsuso what would be $gw and $route_vpn_gateway in op's case?

[netsuso]

Sorry about the $gw, I pasted a simplified version of my script and forgot to change this one (I've just edited my comment)

The only variable that is used is $route_vpn_gateway, and it's set by openvpn when executing the --route-up script (in the op's case it would be 10.8.0.5)

[equivalent]

had similar issue because I'm using TorGuard VPN client (which is using openvpn), when I disconnected from it suddenly docker compose start working

[danielmotaleite]

People without vpn, using multiple network cards, check this post: https://github.com/docker/docker/issues/18113#issuecomment-208436349

This one fixed for me. the magic option is --bip=(machine main ip)/(ip netmask)

I only had to execute /usr/bin/docker daemon --bip=192.168.111.3/24 one, docker then creates some cache and after that, starting via systemd works fine

[clvx]

Same issue here. The worst part is that I can't disable the vpn to create the network interface, because I'm working on servers behind NAT in different locations.

[aboch]

@clvx
As others have already suggested in this thread, this problem can be overcome by specifying the docker0 address/mask via the --bip daemon flag.

Same thing if you need to create other networks, specify --subnet during the network create command.

If you cannot change the init config file, then manually create a docker0 bridge

ip link add docker0 type bridge
ip addr add dev docker0 A.B.C.D/MM
ip link set docker0 up

then restart the docker service.

The above should all work because libnetwork does not run an overlap check when the address pool is explicitly passed.

[purplesrl]

+1 / ubuntu 16.04

Client:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 22:01:48 2016
OS/Arch: linux/amd64

[armand1m]

Same issue on ubuntu 16.04 xenial when trying to sudo apt install docker-engine with an active openvpn connection. The docker cli is installed normally, but it can't execute the post installation script that configures the docker0 network. The same happens if I do try to start the docker daemon with an active vpn connection.

[aboch]

I locked this issue because it become a bucket where to drop "I have this issue too" comments, while the root cause of the issue has been explained long ago and workarounds have been provided.

[GordonTheTurtle]

@Manouchehri It has been detected that this issue has not received any activity in over 6 months. Can you please let us know if it is still relevant:

• For a bug: do you still experience the issue with the latest version?
• For a feature request: was your request appropriately answered in a later version?

Thank you!
This issue will be automatically closed in 1 week unless it is commented on.
For more information please refer to https://github.com/docker/libnetwork/issues/1926