Libnetwork: Ubuntu 18.04 + Docker 17.12.1-ce break DNS resolution

[mprobst]

This might be a dupe of #1654 (which is closed - but then this is still happening). /CC @sanimej who's been looking at that last year.

The default installation of Docker on Ubuntu 18.04 is broken if Google DNS servers 8.8.8.8 and 8.8.4.4 are not reachable for some reason.

Out of the box Ubuntu runs systemd, which creates a nameserver running on localhost:

$ docker --version
Docker version 17.12.1-ce, build 7390fc6
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"

$ cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
# [...]

nameserver 127.0.0.53

Docker ignores the localhost nameserver:

$ docker run  -ti busybox cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
# [...]
nameserver 8.8.8.8
nameserver 8.8.4.4

Because on my network, 8.8.8.8 is not reachable for some reason (different bug), name lookups time out:

$ docker run  -ti busybox nslookup www.google.com
Server:    8.8.8.8
Address 1: 8.8.8.8

nslookup: can't resolve 'www.google.com'

For completeness sake, the symptom that lead me to find this is that when building a Go based image and running go get in a build step, I get a Could not resolve host: github.com (took me a while to piece this one together...):

Step 6/9 : RUN go get ./
 ---> Running in f64c6b120d88
# cd .; git clone https://github.com/julienschmidt/httprouter /go/src/github.com/julienschmidt/httprouter
Cloning into '/go/src/github.com/julienschmidt/httprouter'...
fatal: unable to access 'https://github.com/julienschmidt/httprouter/': Could not resolve host: github.com
package github.com/julienschmidt/httprouter: exit status 128

[fcrisciani]

@mprobst in case of standalone container, you are in control on which dns to use, if you don't specify anything you will get by default the google ones. if 8.8.8.8 does not work for you, you can specify others with --dns as per https://docs.docker.com/engine/reference/commandline/run

[mprobst]

@fcrisciani my host system has a /etc/resolv.conf that has a working entry for name resolution. Docker ignores this, and then fails at runtime. That does seem like a bug to me. Can you clarify? It does seem very surprising.

If this is how Docker does and will behave, I guess this is an upstream bug in Ubuntu's package setup.

[fcrisciani]

@mprobst
a container spawned with docker run and not networking options will also create a network namespace, isolating it from the host namespace. this will not allow the container to communicate with any of the host resources, in this case the 127.0.0.53 dns.

[comdw]

I agree there is an issue here somewhere...

I have docker 18.03.0-ce running on Ubuntu 16.04 with custom DNS entries in /etc/resolv.conf, e.g.: nameserver 172.99.1.7, and in this environment if I look at /etc/resolv.conf inside a container I see the same thing: nameserver 172.99.1.7

Now on a new machine running docker 18.05.0-ce on Ubuntu 18.04 with the same DNS setup but configured with netplan (which is part of 18.04 changes). The /etc/resolv.conf file should not be edited by hand and uses systemd-resolved - it has an internal nameserver 127.0.0.53 entry in /etc/resolv.conf. Now inside the container I don't see this entry copied but instead get the default google ones (8.8.8.8, 8.8.4.4).

Clearly Ubuntu is different w.r.t. DNS in 18.04, but I haven't confirmed whether Docker is also behaving differently, or just defaults to Google DNS because it somehow knows that 127.0.0.53 is not going to work inside the container.

My workaround for now is to use the --dns option when starting containers to manually specify the hosts, but I would rather the DNS inheritance from the host continued to work!

Documentation ref:
https://docs.docker.com/config/containers/container-networking/#dns-services
"By default, a container inherits the DNS settings of the Docker daemon, including the /etc/hosts and /etc/resolv.conf. You can override these settings on a per-container basis"

Related issue?
https://github.com/docker/libnetwork/issues/2068

[diegoquintanav]

Probably related https://github.com/moby/moby/issues/36153
At that moment it was with ubuntu 16.04, now with 18.04

If it's worth something, I posted in SO with updated information https://stackoverflow.com/questions/51105875/internet-connection-not-working-networkmanager-not-working-after-installing-dock

[ptink]

I really don't think this issue should have been closed- docker is broken out of the box on any corporate network, and in a non-obvious way that will usually require the hunting down of this issue in order to find a work around.

Would it not be possible for docker to detect the presence of nameserver 127.0.0.53 in resolv.conf in a similar way to how systemd-resolve does and use /run/systemd/resolv/resolv.conf (which contains the actual DNS nameserver rather than the local) instead?

[kaneg]

I also encountered the same issue after upgrading to Ubuntu 18.04.
The root cause of the issue is that in Ubuntu 18.04, the /etc/resolve.conf is controlled by systemd-resolve which is using local address 127.0.0.53 as DNS server. Meanwhile, the docker is also dependent on the DSN in /etc/resolve.conf. In this case since the 127.0.0.53 is only meaningful for the host instead of each Docker container. Thus, Docker choose a fallback way, which set DNS for each container to Google's public DNS server which is 8.8.8.8.

Although Docker has a --dns to explicitly set external DNS server, it is not a ideal way.
I'm not sure if there is any elegant way to resolve the issue. For now, I use below workaround by disabling systemd as default DSN resolver:
ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

[ctelfer]

I looked into this a bit deeper to add some color as to what is going on here. Docker spawns a resolver for all user-defined networks. From moby/moby:

        if !containertypes.NetworkMode(n.Name()).IsUserDefined() {
                createOptions = append(createOptions, libnetwork.CreateOptionDisableResolution())
        }    

and from libnetwork:

func CreateOptionDisableResolution() EndpointOption {
        return func(ep *endpoint) {
                ep.disableResolution = true
        }
}
...
func (ep *endpoint) needResolver() bool {
        ep.Lock()
        defer ep.Unlock()
        return !ep.disableResolution
}
...
func (sb *sandbox) populateNetworkResources(ep *endpoint) error {
...
        if ep.needResolver() {
                sb.startResolver(false)
        }

So for non-user-defined networks (e.g. bridge) docker/libnetwork specifically does not start any sort of proxying resolver. I verified this experimentally by creating a fresh bridge network and running docker run -it --network=mynet --privileged nicolaka/netshoot iptables -L -t nat -n and seeing the IPtables redirects to the proxy resolver.

...
Chain DOCKER_OUTPUT (1 references)
target     prot opt source               destination         
DNAT       tcp  --  0.0.0.0/0            127.0.0.11           tcp dpt:53 to:127.0.0.11:40937
DNAT       udp  --  0.0.0.0/0            127.0.0.11           udp dpt:53 to:127.0.0.11:57704
...

Running the same command without the --network=mynet (putting the container in the default bridge network) will show an empty set of iptables rules.

Containers without resolver proxying have no way to reach the host's 127.0.0.* network. As @fcrisciani said above, any network traffic directed to such an address will go to the container's network namespace's loopback address which is not plumbed to answer such requests. This is why libnetwork specifically filters all 127.* nameservers from the generated /etc/resolv.conf that goes in each container. Docker engine reads the (configurably-located if I read the code correctly) resolv.conf file for the system, but then filters out those localhost nameservers in the resolv.conf that it generates for the container because it knows the container can't reach them.

This is why @comdw saw the name server's being copied in, but others have not in ubuntu 18.04. A simple workaround if your system is unable to reach 8.8.8.8 / 8.8.4.4 (the defaults provided when no other host resolution seems possible) may well be to place your containers in a user-defined bridge network rather than the default bridge. (Haven't tested this yet on 18.04, though. If someone would that would be helpful.)

Given how annoying this would be in general, the right solution would seem to be a change of policy (or an override) in moby to re-enable proxy resolution for the bridge network. But it doesn't look like something we can fix from libnetwork ... at least not cleanly.

[mushkevych]

As this thread is referenced by Google, let me provide "how-to" on resolving the issue with the docker builds on Ubuntu 18.04.

• find out the DNS servers with command (your IP addressed will be different):

$ nmcli dev show | grep DNS
IP4.DNS[1]:                             10.11.12.13
IP4.DNS[2]:                             10.11.12.14

• register these IPs with the /etc/resolv.conf:

sudo -s
echo "nameserver 10.11.12.13" >> /etc/resolvconf/resolv.conf.d/tail
echo "nameserver 10.11.12.14" >> /etc/resolvconf/resolv.conf.d/tail

• reboot the OS and check whether the IPs have been registered:

cat /etc/resolv.conf   

[bmurphy1976]

We solved this by adding the following to every docker run command. Using this approach we at least didn't have to modify any system wide files on our file system.

--dns `cat /run/systemd/resolve/resolv.conf | grep nameserver | head -n 1 | awk '{print $2}'`

The fact that we had to do this on all of our Ubuntu 18.04 servers is obnoxious.

[diegoquintanav]

@mushkevych What OS are you using? in 18.04 there's no resolv.conf.d/tail out of the box.

[mushkevych]

@diegoquintanav I am using Ubuntu 18.04
You are correct that resolv.conf.d/tail is not present out of the box, so the commands above are creating it.

[diegoquintanav]

@mushkevych It didn't solve it for me, do you have perhaps resolvconf installed? It does not come shipped with 18.04 neither.

[yaleman]

Has anyone found a solution for docker-compose yet? I can do it for individual containers, but so far I'm looking at having to dynamically create the docker-compose.yml to set DNS on my dev machine.

[carletes]

Has anyone found a solution for docker-compose yet? I can do it for individual containers, but so far I'm looking at having to dynamically create the docker-compose.yml to set DNS on my dev machine.

I followed @kaneg's approach:

$ ls -l /etc/resolv.conf 
lrwxrwxrwx 1 root root 32 Aug  6 12:58 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf
$

According to the man page systemd-resolved(8) on my Ubuntu 18.04 system, this is a clean and supported approach (see section "/ETC/RESOLV.CONF" there).

With this setup in place, name resolution inside containers works just fine for me.

[yaleman]

Thanks @carletes and @kaneg that works perfectly in my testing.

[glaux]

I definitely agree that this should be treated like a bug. The solution @kaneg suggests unfortunately doesn't work on my system (Ubuntu 18.04, Docker 18.06.0-ce). This is my output:

$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 32 Aug 28 10:06 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf

however the container still have the incorrect configuration when started with docker-compose.

The only 'workaround' I've found is to manually append the contents of /etc/resolv.conf from the host to the same file in the container. As others have pointed out, ADD or COPY on build doesn't work. Perhaps the file could be populated with the correct dns info with an entrypoint script, but honestly I've already wasted a few hours trying to get this to work and I don't restart my containers too often.

[caparzo]

@mushkevych
sudo -s echo "nameserver 10.11.12.13" >> /etc/resolvconf/resolv.conf.d/tail echo "nameserver 10.11.12.14" >> /etc/resolvconf/resolv.conf.d/tail
worked for me, you just have to be sure that the file "/etc/resolv.conf is generated by "resolvconf".
You don't need to restart the OS, just restart the resolvconf service.
Thanks

[Joseluismantilla]

Guys

I tried all the solutions but unfortunatelly, none workd for me!.

The best way for me is to force the container to use the same network as the host by using the parameter --network host

Example:
$ docker run --name test --network host --rm -id personal/centos6-chef
854d161cd8a16040830f7c58bc541b0d992dc9567d8804edc745ecebb6738cb8

$ docker exec -it test bash
[root@CO-IT01776 /]# ping google.com
PING google.com (216.58.217.206) 56(84) bytes of data.
64 bytes from lax17s05-in-f206.1e100.net (216.58.217.206): icmp_seq=1 ttl=49 time=120 ms
64 bytes from lax17s05-in-f206.1e100.net (216.58.217.206): icmp_seq=2 ttl=49 time=120 ms

With docker-compose.yml

version: '3.2'
services:
  remy:
    image: personal/centos6-chef:latest
    environment:
    network_mode: host
    privileged: true
    volumes:

Greetings from Colombia
Thanks

[kenden]

I believe it should be fixed in Docker CE 18.09, by PR https://github.com/moby/moby/pull/37485

[wilfried-ecolutis]

I still have the issue on 18.09.0, build 4d60db4.
I bypass it by adding a dns list.

# using OpenDNS
dns:
  - "208.67.222.222"
  - "208.67.220.220"

[Jeansen]

I have the same problem. Starting a plain image like docker run --rm -it debian up to recently had no problem when doing a ping to any domain on the interweb. Since the latest upgrade to version 18.09.1 I can no longer resolve any host. Not even locally! Going with the solution in https://github.com/docker/libnetwork/issues/2187#issuecomment-435193013 helped. That is when running docker run --rm --network host -it debian I can again do a simple ping to any domain without problems.

[MohammedEssehemy]

@kaneg solution worked perfect I just had to restart docker daemon to take effect.

[ImTheDeveloper]

Tried a few of the options above with no luck.

Docker version 18.09.1, build 4c52b90

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"

I'm using docker-compose with a user-defined bridge network setup in the following manner:

version: '3.7'

networks:
  web:
    driver: bridge
    internal: false
  app:
    driver: bridge
    internal: true
  db:
    driver: bridge
    internal: true

My particular container with the issue is added to all 3 networks.

I added my tests here: https://superuser.com/questions/1411111/docker-dns-resolution-slow-for-http-calls-in-node-application-ipv6-bridge-netw

From the above, I'm getting a 4 second timeout before being able to then get a response from a HTTP request. The resolution of the domain is causing issues and it specifically seems to be waiting for the ipv6 failure to happen before retuning a response. I do not have ipv6 enabled on my dev machine nor production server.

Listening on port 53 is systemd-resolve still so I went the route of symlink fix as per comment https://github.com/docker/libnetwork/issues/2187#issuecomment-403372190 but this did not change anything.

tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      989/systemd-resolve 

Since systemd-resolve is in use I could normally switch to dnsmasq I guess however I've seen that this can break custom bridge networks as documented in the first post here https://github.com/docker/libnetwork/issues/2068#issue-294280940

05:35:12.465921 IP (tos 0x0, ttl 64, id 47820, offset 0, flags [DF], proto UDP (17), length 60)
    192.168.1.141.53 > 192.168.1.116.59581: [udp sum ok] 24220 q: AAAA? localtunnel.me. 0/0/0 (32)
05:35:12.522916 IP (tos 0x0, ttl 63, id 62222, offset 0, flags [DF], proto UDP (17), length 62)
    192.168.1.116.39734 > 10.0.0.2.53: [bad udp cksum 0xcc59 -> 0x9375!] 1710+ A? api.telegram.org. (34)
05:35:16.523303 IP (tos 0x0, ttl 63, id 16594, offset 0, flags [DF], proto UDP (17), length 62)
    192.168.1.116.53416 > 8.8.8.8.53: [bad udp cksum 0xd267 -> 0x57f5!] 1710+ A? api.telegram.org. (34)
05:35:16.543163 IP (tos 0x0, ttl 119, id 49810, offset 0, flags [none], proto UDP (17), length 78)
    8.8.8.8.53 > 192.168.1.116.53416: [udp sum ok] 1710 q: A? api.telegram.org. 1/0/0 api.telegram.org. [2m29s] A 149.154.167.220 (50)

Is there anything else I can try or any more info I can provide?

[diegoquintanav]

On a sidenote, In my case my problem was related to conflicting addresses with the DNS server in my local network. The solution is explained in https://superuser.com/questions/1336567/installing-docker-ce-in-ubuntu-18-04-breaks-internet-connectivity-of-host

[BenKettlewell]

I'm not sure if there is any elegant way to resolve the issue. For now, I use below workaround by disabling systemd as default DSN resolver:
ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Just a warning when I ran this command I ended up losing internet access for the whole machine.

[soga-denken]

Try this. My system is ubuntu 18.04.

Set DNS

sudo apt install resolvconf
echo "nameserver 10.11.12.13" >> /etc/resolvconf/resolv.conf.d/tail
echo "nameserver 10.11.12.14" >> /etc/resolvconf/resolv.conf.d/tail

Then,

cat /etc/systemd/resolved.conf

[Resolve]
#DNS=
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#Cache=yes
#DNSStubListener=yes

Uncoment DNSStubListener=yes and set DNSStubListener=no

after reboot, I get

$ lsof -i4:53

Hope this works to you.

[ImTheDeveloper]

I've done some more investigation into my issue with debug mode and checking the journal.

Mar 12 12:26:30 dell-desktop dockerd[1926]: time="2019-03-12T12:26:30.151162466Z" level=debug msg="Name To resolve: api.telegram.org."
Mar 12 12:26:30 dell-desktop dockerd[1926]: time="2019-03-12T12:26:30.151364644Z" level=debug msg="[resolver] query api.telegram.org. (A) from 192.168.0.4:58560, forwarding to udp:10.0.0.2"
Mar 12 12:26:34 dell-desktop dockerd[1926]: time="2019-03-12T12:26:34.151603785Z" level=debug msg="[resolver] read from DNS server failed, read udp 192.168.0.4:58560->10.0.0.2:53: i/o timeout"
Mar 12 12:26:34 dell-desktop dockerd[1926]: time="2019-03-12T12:26:34.151881138Z" level=debug msg="[resolver] query api.telegram.org. (A) from 192.168.0.4:43810, forwarding to udp:8.8.8.8"
Mar 12 12:26:34 dell-desktop dockerd[1926]: time="2019-03-12T12:26:34.175549311Z" level=debug msg="[resolver] received A record \"149.154.167.220\" for \"api.telegram.org.\" from udp:8.8.8.8"

Above I see a failure and then a retry which is accounting for 4 second delay.

Current etc/resolv.conf on host:

cat /etc/resolv.conf 
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 192.168.93.1
nameserver 8.8.4.4

Inside container:

root@af2080737f7f:/etc# cat resolv.conf 
nameserver 127.0.0.11
options ndots:0

Docker daemon:

{
    "dns": ["10.0.0.2", "8.8.8.8"],
    "debug": true
}

I have since removed the 10.0.0.2 dns entry and I have rebuilt the containers after killing the daemon and restarting. However I still see the dns resolution going to the same address and failing before then trying google.

Mar 12 13:01:26 dell-desktop dockerd[1926]: time="2019-03-12T13:01:26.772409502Z" level=debug msg="[resolver] read from DNS server failed, read udp 192.168.48.3:55436->10.0.0.2:53: i/o timeout"
Mar 12 13:01:26 dell-desktop dockerd[1926]: time="2019-03-12T13:01:26.772812209Z" level=debug msg="[resolver] query api.telegram.org. (A) from 192.168.48.3:48482, forwarding to udp:8.8.8.8"
Mar 12 13:01:26 dell-desktop dockerd[1926]: time="2019-03-12T13:01:26.791847645Z" level=debug msg="[resolver] received A record \"149.154.167.220\" for \"api.telegram.org.\" from udp:8.8.8.8"

[cpuguy83]

@ImTheDeveloper This sounds a lot like https://www.weave.works/blog/racy-conntrack-and-dns-lookup-timeouts

[m00dy]

@kaneg thanks buddy. It works like a charm.

[almas-istybaev]

I also encountered the same issue after upgrading to Ubuntu 18.04.
The root cause of the issue is that in Ubuntu 18.04, the /etc/resolve.conf is controlled by systemd-resolve which is using local address 127.0.0.53 as DNS server. Meanwhile, the docker is also dependent on the DSN in /etc/resolve.conf. In this case since the 127.0.0.53 is only meaningful for the host instead of each Docker container. Thus, Docker choose a fallback way, which set DNS for each container to Google's public DNS server which is 8.8.8.8.

Although Docker has a --dns to explicitly set external DNS server, it is not a ideal way.
I'm not sure if there is any elegant way to resolve the issue. For now, I use below workaround by disabling systemd as default DSN resolver:
ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Thanks. It's work for me