Libnetwork: --internal networks can contact IP addresses on the host

[bboreham]

Example:

$ docker network create --internal aaa
59dfd142b4efe70cea7f62ae13e798df55e50c69d875663e50f890cf0c5e03d7
$ docker network inspect 59df
[
    {
        "Name": "aaa",
        "Id": "59dfd142b4efe70cea7f62ae13e798df55e50c69d875663e50f890cf0c5e03d7",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": {
            "Driver": "default",
            "Config": [
                {
                    "Subnet": "172.25.0.0/16",
                    "Gateway": "172.25.0.1/16"
                }
            ]
        },
        "Containers": {},
        "Options": {}
    }
]

Am I missing something? Isn't part of the point of --internal that there is no gateway?

[aboch]

Thanks, we thought of that, but then the container would not even be reachable from the host.
Which may be fine, we need to see if this would interfere with the common use of internal networks. Then we can make the call.

[mrjana]

@bboreham The reason internal networks have a gateway is because contrary to popular belief docker networks are not a single l2 segment. It can consist of multiple l2 segments and a driver supporting multiple l2 segments should be able to route between the segments. That is why each segment has a gateway IP. What internal networks don't have is default gateway as expected because it is not expected to communicate with the external world.

[bboreham]

OK, I understand the answer, and indeed the gateway address does not forward off the local machine, but I do not follow "What internal networks don't have is default gateway".

$ docker run -ti --net=aaa alpine /bin/sh
# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.25.0.1      0.0.0.0         UG    0      0        0 eth0
172.25.0.0      *               255.255.0.0     U     0      0        0 eth0

This container has a default route which has a gateway. @mrjana what did you mean by "default gateway"?

[bboreham]

Getting curiouser; networks created using the overlay driver and --internal do not get a default route or gateway, even if you specify one with --gateway:

$ docker network create --driver overlay --subnet=10.0.11.0/24 --gateway=10.0.11.1 --internal m
y-net3
$ docker run -ti --net=my-net3 alpine /bin/sh
/ # ip route
10.0.11.0/24 dev eth0  src 10.0.11.2

[mrjana]

I meant a default route to a gateway. But if you are seeing a default route in a network aaa is a bug if it is meant to be internal. Which driver was used to create aaa?

For the overlay network that you created even though you added a gateway there won't be any route using that as a nexthop unless you have a network with multiple subnets. If you create an overlay network with multiple subnets then you will see that behavior in the route dump.

[bboreham]

Which driver was used to create aaa?

This was in the context of the issue description:

$ docker network create --internal aaa

i.e. the bridge driver.

If I'm reading correctly, @mrjana says this network having a gateway is a bug and @aboch says it's by design. The docs at https://docs.docker.com/engine/reference/commandline/network_create/ say:

--internal               Restricts external access to the network

which is not 100% clear whether it means "external to the container network" or "external to the host".

[mrjana]

@bboreham In docker a network by default is a private network. The only way to access containers outside of the network is using the ports that the container exposes. So if you discard @aboch's comment(he probably meant something else) what the doc says is pretty unambiguous. --internal means the containers cannot be accessed from outside, whether be it from the host or outside the host. The host is part of the external world.

[bboreham]

When I try it, I find connectivity between the host and the container but not the external world.

$ docker run -ti --net=aaa alpine /bin/sh
/ # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.25.0.1      0.0.0.0         UG    0      0        0 eth0
172.25.0.0      *               255.255.0.0     U     0      0        0 eth0
/ # ping 172.17.0.1 # <--- this is the docker bridge
PING 172.17.0.1 (172.17.0.1): 56 data bytes
64 bytes from 172.17.0.1: seq=0 ttl=64 time=0.070 ms
64 bytes from 172.17.0.1: seq=1 ttl=64 time=0.061 ms
^C
--- 172.17.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.061/0.065/0.070 ms
/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
669: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:19:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.25.0.2/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe19:2/64 scope link 
       valid_lft forever preferred_lft forever

Then from outside the container:

$ ping 172.25.0.2
PING 172.25.0.2 (172.25.0.2) 56(84) bytes of data.
64 bytes from 172.25.0.2: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from 172.25.0.2: icmp_seq=2 ttl=64 time=0.041 ms
64 bytes from 172.25.0.2: icmp_seq=3 ttl=64 time=0.041 ms
^C
--- 172.25.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.041/0.042/0.045/0.005 ms

[mrjana]

Yeah that's the bug.

[dan-richards]

When attempting to connect to the host from the container, it seems that the packets are ending up in the INPUT chain in the filter table of the firewall because the packets are destined for local delivery.

Here are some workaround options which you can use until this is fixed.

This will drop all packets originating from the user-defined network with the short ID of e6f77d1d552e:

iptables -t filter -A INPUT -i br-e6f77d1d552e -j DROP

This will drop all packets originating from any libnetwork bridge (and also any non-libnetwork interfaces whose names start with "br-"):

iptables -t filter -A INPUT -i br-+ -j DROP

You will need to insert these rules further up the chain if there are some `ACCEPT`-jumping rules that you need to precede. You could instead use a `DROP` default policy and only permit packets originating from the interfaces you want. This will set the default policy and add the rule to only accept packets from ethernet interfaces:

iptables -t filter -A INPUT -i eth+ -j ACCEPT
iptables -t filter -P INPUT DROP

--- **Example:**

root@default:/home/docker# ifconfig docker0 | grep 'inet '
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
root@default:/home/docker# docker run -itd --network e6f77d1d552e test-image
085066273b9422774f097ca26aec289794b2df323391a596b7d55be2ce131dc3
root@default:/home/docker# docker exec -it 085066273b94 bash
root@085066273b94:/# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.21.0.1      0.0.0.0         UG    0      0        0 eth0
172.21.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0
root@085066273b94:/# ifconfig eth0 | grep 'inet '
          inet addr:172.21.0.2  Bcast:0.0.0.0  Mask:255.255.0.0

Before:

root@default:/home/docker# ping -c1 -W1 172.21.0.2 | grep loss
1 packets transmitted, 1 packets received, 0% packet loss
root@default:/home/docker# docker exec -it 085066273b94 bash
root@085066273b94:/# (
    ping -c1 -W1 172.21.0.1
    ping -c1 -W1 172.17.0.1
) | grep loss
1 packets transmitted, 1 received, 0% packet loss, time 0ms
1 packets transmitted, 1 received, 0% packet loss, time 0ms
root@085066273b94:/# nmap -sS -p 22,2376 172.21.0.1
# ...
Host is up (0.000030s latency).
PORT     STATE SERVICE
22/tcp   open  ssh
2376/tcp open  docker
# ...
Nmap done: 1 IP address (1 host up) scanned in 13.43 seconds

After:

root@default:/home/docker# ping -c1 -W1 172.21.0.2 | grep loss
1 packets transmitted, 0 packets received, 100% packet loss
root@default:/home/docker# docker exec -it 085066273b94 bash
root@085066273b94:/# (
    ping -c1 -W1 172.21.0.1
    ping -c1 -W1 172.17.0.1
) | grep loss
1 packets transmitted, 0 received, 100% packet loss, time 0ms
1 packets transmitted, 0 received, 100% packet loss, time 0ms
root@085066273b94:/# nmap -sS -p 22,2376 172.21.0.1
# ...
Host is up (0.000026s latency).
PORT     STATE    SERVICE
22/tcp   filtered ssh
2376/tcp filtered docker
# ...
Nmap done: 1 IP address (1 host up) scanned in 14.44 seconds

For some reason, the INPUT chain rule I used for dropping the packets "from br-e6f77d1d552e" also prevent the host connecting to the container. Although that's what we want, I'm not sure _why_ this happened. Perhaps it's because they technically originated from the br-e6f77d1d552e (at 172.21.0.1)?

---

My best guess is that libnetwork should create a new chain and a rule to unconditionally jump to it. This rule should be at the top of the filter table's INPUT chain, much like the rule to jump to the DOCKER-ISOLATION chain at the top of that table's FORWARD chain. I have experience with neither Go nor the libnetwork code base, so I'm probably not the best person to move this issue forward.

@mrjana, what do you think of this approach?

[dan-richards]

The only way to access containers outside of the network is using the ports that the container exposes

Exposing ports also does not appear to be working correctly, even without my addition to the INPUT chain:

root@default:~# docker network inspect e6f77d1d552e --format '{{ .Internal }}'
true
root@default:~# docker run -itd --name nginx --network e6f77d1d552e -p 1337:80 nginx
799e2d2d41e088191b82a68de5c5f25e30e752373fccd621460b62674eaccb38
root@default:~# docker port nginx # should list the mappings
root@default:~# docker port nginx 80  # should list the mapping
Error: No public port '80/tcp' published for nginx
root@default:~# curl -sSI 0.0.0.0:1337 | grep '^HTTP' # should yield "HTTP/1.1 200 OK"
curl: (7) Failed to connect to 0.0.0.0 port 1337: Connection refused
root@default:~# curl -sSI 127.0.0.1:1337 | grep '^HTTP' # should yield "HTTP/1.1 200 OK"
curl: (7) Failed to connect to 127.0.0.1 port 1337: Connection refused
root@default:~# curl -sSI 172.21.0.3:80 | grep '^HTTP' # connection should be refused
HTTP/1.1 200 OK

Am I missing something? :disappointed:
On the plus side, with my firewall rule added, the request to 172.21.0.3:80 times out. :v:

[RRAlex]

The following issue might be related:

The docker daemon is where firewall rules should be dynamically managed, as they already are, otherwise things will leak or do things in the wrong order and/or be a management hell as bridge name will change, etc...

https://github.com/docker/docker/issues/14041

[GordonTheTurtle]

@bboreham It has been detected that this issue has not received any activity in over 6 months. Can you please let us know if it is still relevant:

• For a bug: do you still experience the issue with the latest version?
• For a feature request: was your request appropriately answered in a later version?

Thank you!
This issue will be automatically closed in 1 week unless it is commented on.
For more information please refer to https://github.com/docker/libnetwork/issues/1926

[jwidauer]

This bug seems to still not be fixed and is still replicatable with the same instructions @bboreham gave. Proof:

$ docker version
Client:
 Version:           18.09.6
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        481bc77
 Built:             Sat May  4 02:35:57 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.6
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       481bc77
  Built:            Sat May  4 01:59:36 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Create the driver:

$ docker network create --internal aaa

Run in image:

$ docker run -ti --net=aaa alpine /bin/sh
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
bdf0201b3a05: Pull complete 
Digest: sha256:28ef97b8686a0b5399129e9b763d5b7e5ff03576aa5580d6f4182a49c5fe1913
Status: Downloaded newer image for alpine:latest
/ # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.22.0.1      0.0.0.0         UG    0      0        0 eth0
172.22.0.0      *               255.255.0.0     U     0      0        0 eth0
/ # ping 172.17.0.1  <-- docker bridge
PING 172.17.0.1 (172.17.0.1): 56 data bytes
64 bytes from 172.17.0.1: seq=0 ttl=64 time=0.192 ms
64 bytes from 172.17.0.1: seq=1 ttl=64 time=0.174 ms
64 bytes from 172.17.0.1: seq=2 ttl=64 time=0.195 ms
^C
--- 172.17.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.174/0.187/0.195 ms
/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
60: eth0@if61: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:16:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.22.0.2/16 brd 172.22.255.255 scope global eth0
       valid_lft forever preferred_lft forever

From outside the container:

$ ping 172.18.0.2
PING 172.18.0.2 (172.18.0.2) 56(84) bytes of data.
64 bytes from 172.18.0.2: icmp_seq=1 ttl=64 time=0.081 ms
64 bytes from 172.18.0.2: icmp_seq=2 ttl=64 time=0.097 ms
^C
--- 172.18.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.081/0.089/0.097/0.008 ms

All of this was run on an up to date Ubuntu 18.04 machine.

[RRAlex]

Is that last ping from an external machine on which you've added a route or just locally?

[jwidauer]

Locally in the same machine where I ran the container.