Libnetwork: DNS broken out-of-the-box on Ubuntu Artful

[akdor1154]

To reproduce -

• Install Ubuntu Artful or Bionic desktop with default settings, connect it to a network by plugging in a cable and letting NetworkManager do its magic. Block access to 8.8.8.8 and 8.8.4.4 because you are on a corporate network.
• Install docker-ce from docker's stable deb repo. (17.12.0-ce)

• Create a container:
  docker run -it ubuntu bash
• Observe that DNS is broken in the container:
  apt update
Err:1 http://security.ubuntu.com/ubuntu xenial-security InRelease Temporary failure resolving 'security.ubuntu.com' Err:2 http://archive.ubuntu.com/ubuntu xenial InRelease Temporary failure resolving 'archive.ubuntu.com'

I believe this is because Ubuntu uses systemd-networkd on the host by default, and /etc/resolv.conf lists only

nameserver 127.0.0.53

.

Expected behaviour:
docker should work out of the box on ubuntu artful.

Not-quite-good-enough-workarounds:

• sudo apt install dnsmasq; echo '{"dns": ["172.17.0.1"]}' > /etc/docker/daemon.json
  This fixes default containers, but breaks default custom networks - docker network create --driver bridge brokenDNS; docker run --network brokenDNS -it ubuntu bash.

[ddebroy]

I suspect the name server is not correctly reflected in the host's /etc/resolv.conf. When I try a Ubuntu 17.04 based VM, I see the following in /etc/resolv.conf with 172.31.0.2 correctly pointing to the desired external DNS server.

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 172.31.0.2
nameserver 127.0.0.53

Can you check why the Ubuntu host's /etc/resolv.conf does not have any entry pointing to external DNS beyond 127.0.0.53?

[maxdwit]

This also applies to BIONIC BEAVER

[diegoquintanav]

Probably related https://github.com/moby/moby/issues/36153
At that moment it was with ubuntu 16.04, now with 18.04

If it's worth something, I posted in SO with updated information https://stackoverflow.com/questions/51105875/internet-connection-not-working-networkmanager-not-working-after-installing-dock

[awilkins]

TLDR: Please, devs, implement a flag for the daemon that replicates the user-defined network DNS behaviour in the default network. Put all the scary warnings about backward compatibility in the docs you like. :-)

why the Ubuntu host's /etc/resolv.conf does not have any entry pointing to external DNS beyond 127.0.0.53

Because in 17.10 Ubuntu moved from a NetworkManager spawned instance of dnsmasq to using systemd-resolved for local DNS caching. systemd-resolved expects applications to use glib API calls or dbus to make DNS requests, but provides a stub DNS listener on 127.0.0.53.

In user-defined docker networks this is fine because the docker DNS proxy on 127.0.0.11 is configured and routes queries through whatever is configured on the host system. I've confirmed this goes through systemd-resolved in some form by disabling it and watching lookups timeout in the container.

In the default network this breaks.

It was broke before when it was using dnsmasq because docker assumed the 127.0.1.1 listen address it used was not a real DNS server and copied the Google DNS servers into your container. In the main this works (and wouldn't work if you didn't do it, because dnsmasq doesn't listen to the docker0 bridge by default), but fails in the scenarios above where access to Google DNS is limited for good reasons, or you need to resolve names inside a VPN.

But you could work around it with the --dns setting, and by reconfiguring the dnsmasq instance to listen to the docker0 bridge in addition to the loopback adapter, as per this answer.

systemd-resolved's stub listener can't be reconfigured to listen to additional adapters (and this is an explicit design decision), so in order to effect this workaround you now have to disable it's stub listener, and reconfigure NetworkManager to start it's spawned instance of dnsmasq again, but to listen on 127.0.0.53 as that's now what /etc/resolv.conf says. Doing this is not trivial. I did it once on a colleague's VM, it took me about an hour of reading docs to work it all out, I am thankful I'm still using the LTS from 2016 which hasn't been inflicted with systemd-resolved yet.

A flag for dockerd that replicates the user-defined network DNS behaviour in the default network would be a neat, tidy way to fix this and would not cause issues for I guess 99% of Ubuntu users in a corporate environment.

If you put a daemon.json with it switched on in the packages for Ubuntu, everyone will... well, not notice, because things will work nicely. But I'll be happy.

[j3p0uk]

Suggested on related issue #2187

https://github.com/docker/libnetwork/issues/2187#issuecomment-402982366

In my case, the /run/systemd/resolve/resolv.conf contains what I want docker to use in containers I spin up. If I could simply tell the daemon to treat that resolv.conf in the same way as it is currently treating /etc/resolv.conf I think it would solve the issue (and the need to do so is likely discoverable).

This would provide a much more user-friendly experience.

[fcrisciani]

@j3p0uk this commit https://github.com/moby/moby/pull/37485 addresses the systemd-resolved case