Libnetwork: Ability to use localhost DNS required
Currently if /etc/resolv.conf contains only localhost nameservers, the following error occurs when starting containers (from the docker daemon logs):
time="2017-02-19T16:10:50.505005194Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
The problem with this behaviour is that in some company networks access to all non-managed DNS servers. Compounded with the changes to systemd-resolved which as of version 232 of systemd no longer uses a symlinked resolv.conf but proxies through a server running on 127.0.0.53, I can't create containers without specifying the DNS for every container I start (which is not tolerant to my development laptop moving between the company and my personal network and is pretty inconvenient).
In my opinion, the behaviour that should be adopted is that a DNS proxy (like the one that is used on the default network) runs on the network's default gateway and that all embedded DNS server instances forward to this server which can then use the host machine's DNS configuration. In general, I don't really feel like setting the nameservers to a fixed value by default is a good practice, even if it is Google DNS which you set it to.
Note that containers on the default network, do not have this problem as they presumably do not use the embedded DNS server and route their DNS requests to the gateway IP.
Is there a reason (a security one maybe) that you don't allow localhost DNS nameservers other than the fact they would not be available to the container using the localhost IP?
I am not sure if it's related to #1548 but no one there is reporting the log error I have mentioned and are in fact saying that adding 8.8.8.8 to their /etc/resolv.conf is helping, so probably not.
This issue would be helped with the resolution of #1046 which would allow alternate but is still not useful for a mobile device where the private DNS server is only available on the network that is forcing the use of it.
flungo
All 20 comments
@flungo This is fixed in the docker master and will be available in the next release. This works for containers on user defined networks because Docker embedded DNS is required to differentiate if the resolver is a host loopback IP or the container's loopback (when you have a local resolver in the container itself).
sanimej
on 19 Feb 2017
Still seem to be experiencing a problem using Docker 17.05.0-ce. Should this have been resolved or is the fix not yet included in the Docker release? If the change should be in 17.05.0-ce then I will try to provide more detail on my scenario and what's not working.
My current docker version output:
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.8.1
Git commit: 89658bed64
Built: Fri May 5 22:40:58 2017
OS/Arch: linux/amd64
Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.8.1
Git commit: 89658bed64
Built: Fri May 5 22:40:58 2017
OS/Arch: linux/amd64
Experimental: false
@flungo It should work in 17.05. Can you share your env details about how you are creating the containers with the host loopback IP as resolver ?
sanimej
on 13 Jun 2017
This does not work in 17.05. AFAIK the internal DNS resolver runs in the container namespace, which can't see a resolver on the host.
What is desperately needed is for the internal resolver to setup datagram and stream sockets so it can actually communicate with a process on the host that can then forward queries in host-space to the host's resolvers.
wrouesnel
on 27 Jun 2017
I am assuming you are using user defined networks because the docker DNS server is not enabled for the default bridge (docker0 bridge) network. Here is a capture showing how the hostmode resolver works with docker DNS. I am running a 17.06 RC binary in this node, but 17.05 should work as well.
ubuntu@ip-172-31-10-183:~$ sudo dnsmasq -d --user=root --resolv-file=/home/ubuntu/dnsmasq_resolv.conf --log-queries
sudo: unable to resolve host ip-172-31-10-183: Connection refused
dnsmasq: started, version 2.75 cachesize 150
dnsmasq: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify
dnsmasq: reading /home/ubuntu/dnsmasq_resolv.conf
dnsmasq: using nameserver 172.31.0.2#53
dnsmasq: read /etc/hosts - 7 addresses
ubuntu@ip-172-31-10-183:~$ docker network create -d bridge docker
02679f9e944717ed03bdd33f768bdc7b16d0962b1331d5516accce31b20929cb
ubuntu@ip-172-31-10-183:~$ docker run -it --name dnstest --net docker sanimej/ubuntu bash
root@f2c6ac1599c2:/# dig +short google.com
74.125.28.102
74.125.28.113
74.125.28.138
74.125.28.139
74.125.28.100
74.125.28.101
root@f2c6ac1599c2:/# dig +short AAAA google.com
2607:f8b0:400a:807::200e
root@f2c6ac1599c2:/#
dnsmasq logs
dnsmasq: query[A] google.com from 127.0.0.1
dnsmasq: forwarded google.com to 172.31.0.2
dnsmasq: reply google.com is 74.125.28.102
dnsmasq: reply google.com is 74.125.28.113
dnsmasq: reply google.com is 74.125.28.138
dnsmasq: reply google.com is 74.125.28.139
dnsmasq: reply google.com is 74.125.28.100
dnsmasq: reply google.com is 74.125.28.101
dnsmasq: query[AAAA] google.com from 127.0.0.1
dnsmasq: forwarded google.com to 172.31.0.2
dnsmasq: reply google.com is 2607:f8b0:400a:807::200e
I will go ahead and close this issue. If you have issues getting it to work can you show a reproducible procedure with the relevant details and also the deamon logs in debug mode when an external query is sent from a container.
sanimej
on 3 Jul 2017
Sorry for the delay. I'll document the current behavior on 17.05.0 CE then upgrade and see if anything changes. It is possible that the problem I am experiencing is different from what I first reported. My pre-upgrade environment is:
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.8.1
Git commit: 89658bed64
Built: Fri May 5 22:40:58 2017
OS/Arch: linux/amd64
Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.8.1
Git commit: 89658bed64
Built: Fri May 5 22:40:58 2017
OS/Arch: linux/amd64
Experimental: false
In order to make it easy to identify when there is a problem and replicate the problem I was having on my Univerity campus where DNS requests to external servers is restricted I added rules equilient to these to my firewall:
iptables -I FORWARD -o WAN -m tcp --dport 53 -j DROP
iptables -I FORWARD -o WAN -m udp --dport 53 -j DROP
The local recursor which I am using is systemd-resolved and the /etc/resolv.conf on the host is a symlink to ../usr/lib/systemd/resolv.conf with the contents:
# This is a static resolv.conf file for connecting local clients to
# systemd-resolved via its DNS stub listener on 127.0.0.53.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
systemd-resolved is using the upstream DNS server given by DHCP which is the firewall which I added the rules to and on the host I am able to resolve addresses.
If I now run docker run --rm flungo/netutils dig google.com I get:
; <<>> DiG 9.10.4-P6 <<>> google.com
;; global options: +cmd
;; connection timed out; no servers could be reached
Looking at the docker logs, the following has been logged:
Jul 03 20:11:09 my-hostname dockerd[18063]: time="2017-07-03T20:11:09.016166930+01:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Jul 03 20:11:09 my-hostname dockerd[18063]: time="2017-07-03T20:11:09.016191091+01:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Running docker run --rm flungo/netutils cat /etc/resolv.conf confirms that this is how the DNS has been configured inside of the container:
# This is a static resolv.conf file for connecting local clients to
# systemd-resolved via its DNS stub listener on 127.0.0.53.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 8.8.8.8
nameserver 8.8.4.4
This becomes even more frustrating when trying to build an image. If I try to build my netutils image with docker build -t flungo/latex src/ you get:
Sending build context to Docker daemon 2.048kB
Step 1/4 : FROM alpine
---> 4a415e366388
Step 2/4 : MAINTAINER Fabrizio Lungo <[email protected]>
---> Running in faa10fca375a
---> facc0229a08b
Removing intermediate container faa10fca375a
Step 3/4 : RUN apk add --update iproute2 bind-tools curl openssl nmap nmap-scripts tcpdump
---> Running in ffee7793cbb4
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.5/main: temporary error (try again later)
WARNING: Ignoring APKINDEX.c51f8f92.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.5/community: temporary error (try again later)
WARNING: Ignoring APKINDEX.d09172fd.tar.gz: No such file or directory
ERROR: unsatisfiable constraints:
bind-tools (missing):
required by: world[bind-tools]
curl (missing):
required by: world[curl]
iproute2 (missing):
required by: world[iproute2]
nmap (missing):
required by: world[nmap]
nmap-scripts (missing):
required by: world[nmap-scripts]
openssl (missing):
required by: world[openssl]
tcpdump (missing):
required by: world[tcpdump]
The command '/bin/sh -c apk add --update iproute2 bind-tools curl openssl nmap nmap-scripts tcpdump' returned a non-zero code: 7
The solutions would be for me to change the DNS setup on the machine and instead write the DNS servers provided over DHCP into the host's /etc/resolv.conf or (what I have been doing) adding a dropin when on the University network to the docker service which sets DNS flag to manually specify the university DNS (annoyingly different depending on where on campus you are).
Given in my first post I said it worked on default networking, either the way default networking works has been changed or this may be a different issue. Runing in a custom network appears to work.
Logs from debug mode after trying to run docker run --rm flungo/netutils dig google.com:
Jul 03 20:48:03 my-hostname systemd[1]: Started Docker Application Container Engine.
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.520751675+01:00" level=debug msg="Calling GET /_ping"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.521973303+01:00" level=debug msg="Calling POST /v1.29/containers/create"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.522316094+01:00" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":[\"dig\",\"google.com\"],\"Domainname\":\"\",\"Entrypoint\":null,\"Env\":[],\"HostConfig\":{\"AutoRemove\":true,\"Binds\":null,\"BlkioDeviceReadBps\":null,\"BlkioDeviceReadIOps\":null,\"BlkioDeviceWriteBps\":null,\"BlkioDeviceWriteIOps\":null,\"BlkioWeight\":0,\"BlkioWeightDevice\":null,\"CapAdd\":null,\"CapDrop\":null,\"Cgroup\":\"\",\"CgroupParent\":\"\",\"ConsoleSize\":[0,0],\"ContainerIDFile\":\"\",\"CpuCount\":0,\"CpuPercent\":0,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpuShares\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"DeviceCgroupRules\":null,\"Devices\":[],\"DiskQuota\":0,\"Dns\":[],\"DnsOptions\":[],\"DnsSearch\":[],\"ExtraHosts\":null,\"GroupAdd\":null,\"IOMaximumBandwidth\":0,\"IOMaximumIOps\":0,\"IpcMode\":\"\",\"Isolation\":\"\",\"KernelMemory\":0,\"Links\":null,\"LogConfig\":{\"Config\":{},\"Type\":\"\"},\"Memory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":-1,\"NanoCpus\":0,\"NetworkMode\":\"default\",\"OomKillDisable\":false,\"OomScoreAdj\":0,\"PidMode\":\"\",\"PidsLimit\":0,\"PortBindings\":{},\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyRootfs\":false,\"RestartPolicy\":{\"MaximumRetryCount\":0,\"Name\":\"no\"},\"SecurityOpt\":null,\"ShmSize\":0,\"UTSMode\":\"\",\"Ulimits\":null,\"UsernsMode\":\"\",\"VolumeDriver\":\"\",\"VolumesFrom\":null},\"Hostname\":\"\",\"Image\":\"flungo/netutils\",\"Labels\":{},\"NetworkingConfig\":{\"EndpointsConfig\":{}},\"OnBuild\":null,\"OpenStdin\":false,\"StdinOnce\":false,\"Tty\":false,\"User\":\"\",\"Volumes\":{},\"WorkingDir\":\"\"}"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.558197414+01:00" level=debug msg="container mounted via layerStore: /var/lib/docker/btrfs/subvolumes/99f8ebfca16a750cfbb7fb979395faccc8ecd9b0aebfa3883d300617eba2bf66"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.572647411+01:00" level=debug msg="Calling POST /v1.29/containers/a6c2415084f3016811b4c5a8048ac15777bcd51aee05c73d1276e998ee4ea107/attach?stderr=1&stdout=1&stream=1"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.573067638+01:00" level=debug msg="attach: stderr: begin"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.573102810+01:00" level=debug msg="attach: stdout: begin"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.575429049+01:00" level=debug msg="Calling GET /v1.29/events?filters=%7B%22container%22%3A%7B%22a6c2415084f3016811b4c5a8048ac15777bcd51aee05c73d1276e998ee4ea107%22%3Atrue%7D%2C%22type%22%3A%7B%22container%22%3Atrue%7D%7D"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.576710378+01:00" level=debug msg="Calling POST /v1.29/containers/a6c2415084f3016811b4c5a8048ac15777bcd51aee05c73d1276e998ee4ea107/start"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.577854476+01:00" level=debug msg="container mounted via layerStore: /var/lib/docker/btrfs/subvolumes/99f8ebfca16a750cfbb7fb979395faccc8ecd9b0aebfa3883d300617eba2bf66"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.578456224+01:00" level=debug msg="Assigning addresses for endpoint relaxed_kilby's interface on network bridge"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.578501145+01:00" level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[])"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.595836537+01:00" level=debug msg="Assigning addresses for endpoint relaxed_kilby's interface on network bridge"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.612216625+01:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.612254794+01:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.623693648+01:00" level=debug msg="Programming external connectivity on endpoint relaxed_kilby (dd51276f3e21ee6a078e3f0ae1c991f7af5766c442efc0c5ca77b0b87ccbca7f)"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.755644958+01:00" level=debug msg="sandbox set key processing took 75.94953ms for container a6c2415084f3016811b4c5a8048ac15777bcd51aee05c73d1276e998ee4ea107"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.864562046+01:00" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"start-container\", Id:\"a6c2415084f3016811b4c5a8048ac15777bcd51aee05c73d1276e998ee4ea107\", Status:0x0, Pid:\"\", Timestamp:(*timestamp.Timestamp)(0xc420862a60)}"
Jul 03 20:48:44 my-hostname dockerd[23972]: time="2017-07-03T20:48:44.873545700+01:00" level=debug msg="libcontainerd: event unhandled: type:\"start-container\" id:\"a6c2415084f3016811b4c5a8048ac15777bcd51aee05c73d1276e998ee4ea107\" timestamp:<seconds:1499111324 nanos:864378399 > "
Jul 03 20:49:02 my-hostname dockerd[23972]: time="2017-07-03T20:49:02.970777011+01:00" level=debug msg="containerd: process exited" id=a6c2415084f3016811b4c5a8048ac15777bcd51aee05c73d1276e998ee4ea107 pid=init status=9 systemPid=24194
Jul 03 20:49:02 my-hostname dockerd[23972]: time="2017-07-03T20:49:02.977664192+01:00" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"exit\", Id:\"a6c2415084f3016811b4c5a8048ac15777bcd51aee05c73d1276e998ee4ea107\", Status:0x9, Pid:\"init\", Timestamp:(*timestamp.Timestamp)(0xc42080f0a0)}"
Jul 03 20:49:02 my-hostname dockerd[23972]: time="2017-07-03T20:49:02.977843639+01:00" level=debug msg="attach: stdout: end"
Jul 03 20:49:02 my-hostname dockerd[23972]: time="2017-07-03T20:49:02.977861370+01:00" level=debug msg="attach: stderr: end"
Jul 03 20:49:02 my-hostname dockerd[23972]: time="2017-07-03T20:49:02.978206042+01:00" level=debug msg="Revoking external connectivity on endpoint relaxed_kilby (dd51276f3e21ee6a078e3f0ae1c991f7af5766c442efc0c5ca77b0b87ccbca7f)"
Jul 03 20:49:02 my-hostname dockerd[23972]: time="2017-07-03T20:49:02.981256036+01:00" level=debug msg="DeleteConntrackEntries purged ipv4:0, ipv6:0"
Jul 03 20:49:03 my-hostname dockerd[23972]: time="2017-07-03T20:49:03.105721336+01:00" level=debug msg="Releasing addresses for endpoint relaxed_kilby's interface on network bridge"
Jul 03 20:49:03 my-hostname dockerd[23972]: time="2017-07-03T20:49:03.105756719+01:00" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)"
Jul 03 20:49:03 my-hostname dockerd[23972]: time="2017-07-03T20:49:03.248830525+01:00" level=debug msg="Client context cancelled, stop sending events"
If this is a valid bug, then most users wont be noticing as most users aren't using a localhost resolver and sitting in a network where DNS requests to google DNS would be blocked after falling back.
Going to upgrade now and post back if this is still the same.
flungo
on 3 Jul 2017
Upgraded to the latest version (built from source using the AUR package) and can still reproduce as described in my last post.
Client:
Version: unknown-version
API version: 1.30
Go version: go1.8.3
Git commit: 3dfb8343
Built: Mon Jul 3 20:27:00 2017
OS/Arch: linux/amd64
Server:
Version: 17.06.0-dev
API version: 1.31 (minimum version 1.12)
Go version: go1.8.3
Git commit: c2ad549533
Built: Mon Jul 3 21:27:15 2017
OS/Arch: linux/amd64
Experimental: false
@flungo Using the host DNS resolver is possible only when Docker embedded DNS server is enabled. Like I mentioned earlier Docker DNS server is activated only on user created networks and not on the default bridge network.
Instead of doing..
docker run --rm flungo/netutils dig google.com
try
docker network create -d bridge testnet
docker run --rm --net testnet flungo/netutils dig google.com
@sanimej Using a custom network creates a resolver which works as expected now. The issue is currently is with the default network and so I think this may be different to the one I originally posted. The default network should also be made to work when using a localhost resolver.
Is the default network in the scope of libnetwork or do I need to report on a different repo?
Thanks for your assistance.
flungo
on 3 Jul 2017
# docker version
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:20:42 2017
OS/Arch: linux/amd64
Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:20:42 2017
OS/Arch: linux/amd64
Experimental: false
Docker daemon launch command line: --dns 127.0.0.53 --storage-driver=aufs --log-driver=journald
/etc/resolv.conf:
$ cat /etc/resolv.conf
# This is a static resolv.conf file for connecting local clients to
# systemd-resolved via its DNS stub listener on 127.0.0.53.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
Commands:
$ docker network inspect testnet
[
{
"Name": "testnet",
"Id": "a55432658701c3dc9b40e9777516d39929ecef5f5d40abfc0c0f34918b097d11",
"Created": "2017-07-04T06:27:29.498205627+10:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.21.0.0/16",
"Gateway": "172.21.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"Containers": {
"3b723de150d4590150d528858de93f2e0bd84ba177402f042646d83ecfd51a7b": {
"Name": "inspiring_lalande",
"EndpointID": "958680a5678b8f103474526f7f31fd814e0c0c1370e6156fb51cde66139ce43b",
"MacAddress": "02:42:ac:15:00:02",
"IPv4Address": "172.21.0.2/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
Now the container:
$ docker run -it --rm --net testnet ubuntu:zesty bash
root@3b723de150d4:/# apt-get update
Err:1 http://security.ubuntu.com/ubuntu zesty-security InRelease
Temporary failure resolving 'security.ubuntu.com'
Err:2 http://archive.ubuntu.com/ubuntu zesty InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu zesty-updates InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu zesty-backports InRelease
Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/zesty/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/zesty-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/zesty-backports/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/zesty-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
root@3b723de150d4:/# cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0
Wireshark shows no requests from the container successfully making it to the local host resolver.
AFAIK this is completely expected because the internal resolver still runs in the network namespace of the container, not the host - so localhost requests wouldn't make it out.
The only reliable way I've fixed this is by forcing --dns 172.17.0.1 and running a dnsmasq instance on my host listening on that interface, and forwarding to systemd.
Probing around in a privileged container on a user network, I note the following iptables rules being inserted for docker:
# iptables -L -n -v -t nat
Chain PREROUTING (policy ACCEPT 39 packets, 3431 bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 3 packets, 201 bytes)
pkts bytes target prot opt in out source destination
1 67 DOCKER_OUTPUT all -- * * 0.0.0.0/0 127.0.0.11
Chain POSTROUTING (policy ACCEPT 4 packets, 268 bytes)
pkts bytes target prot opt in out source destination
1 67 DOCKER_POSTROUTING all -- * * 0.0.0.0/0 127.0.0.11
Chain DOCKER_OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 DNAT tcp -- * * 0.0.0.0/0 127.0.0.11 tcp dpt:53 to:127.0.0.11:44649
1 67 DNAT udp -- * * 0.0.0.0/0 127.0.0.11 udp dpt:53 to:127.0.0.11:37340
Chain DOCKER_POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
0 0 SNAT tcp -- * * 127.0.0.11 0.0.0.0/0 tcp spt:44649 to::53
0 0 SNAT udp -- * * 127.0.0.11 0.0.0.0/0 udp spt:37340 to::53
And for completeness this is the ip routing table:
# ip route
default via 172.21.0.1 dev eth0
172.21.0.0/16 dev eth0 proto kernel scope link src 172.21.0.2
The hosts iptables while the test container is running:
# iptables -L -n -v -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
8 591 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 33 packets, 2231 bytes)
pkts bytes target prot opt in out source destination
3 219 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 33 packets, 2231 bytes)
pkts bytes target prot opt in out source destination
6 373 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
9 592 MASQUERADE all -- * !br-a55432658701 172.21.0.0/16 0.0.0.0/0
28 2183 RETURN all -- * * 192.168.122.0/24 224.0.0.0/24
0 0 RETURN all -- * * 192.168.122.0/24 255.255.255.255
0 0 MASQUERADE tcp -- * * 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
0 0 MASQUERADE udp -- * * 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
0 0 MASQUERADE all -- * * 192.168.122.0/24 !192.168.122.0/24
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
5 321 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- br-a55432658701 * 0.0.0.0/0 0.0.0.0/0
# iptables -L -n -v
Chain INPUT (policy ACCEPT 632 packets, 38849 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53
0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67
0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
47446 41M DOCKER-ISOLATION all -- * * 0.0.0.0/0 0.0.0.0/0
30107 40M ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
17339 987K ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
27990 39M ACCEPT all -- * br-a55432658701 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * br-a55432658701 0.0.0.0/0 0.0.0.0/0
15662 882K ACCEPT all -- br-a55432658701 !br-a55432658701 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- br-a55432658701 br-a55432658701 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0
0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0 0.0.0.0/0
0 0 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain OUTPUT (policy ACCEPT 646 packets, 37339 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT udp -- * virbr0 0.0.0.0/0 0.0.0.0/0 udp dpt:68
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- br-a55432658701 docker0 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- docker0 br-a55432658701 0.0.0.0/0 0.0.0.0/0
47446 41M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Partial process tree of docker while the container was running:
root 2850 0.5 0.7 468796 59664 ? Ssl 06:43 0:05 dockerd -H fd:// --dns 127.0.0.53 --storage-driver=aufs --log-driver=journald
root 2862 0.1 0.1 441512 11464 ? Ssl 06:43 0:01 \_ docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
root 7440 0.0 0.0 274536 3516 ? Sl 06:55 0:00 \_ docker-containerd-shim 965d370c54f8b513645a52a5d1f9c331c57f1134b27457037755075ab0ed8f6a /var/run/docker/libcontainerd/965d370c54f8b513645a52a5d1f9c331c57f1134b27457037755075ab0ed8f6a docker-runc
root 7459 0.0 0.0 18380 2468 pts/4 Ss+ 06:55 0:00 \_ bash
This is consistent with every other time I've dealt with this - the embedded DNS resolver still cannot use a localhost resolver on the host if it's not listening on a public or bridge interface.
wrouesnel
on 3 Jul 2017
@flungo Container can be configured with either its own loopback IP as the resolver (cases where the DNS server is running in the container namespace) or host loopback IP as the resolver (when it comes from host's /etc/resolv.conf). Docker DNS server runs in the host namespace but listens to the DNS queries from the container. So it can handle these two cases differently and route the queries to the host resolver in the latter case.
Its not possible to do this for containers on the default bridge network because the queries are directly sent to the configured resolver IP; ie: it can't' differentiate between local loopback vs host loopback. If this use case is a requirement for you consider using user defined networks.
sanimej
on 3 Jul 2017
@wrouesnel If you have --dns 127.0.0.53 configured for the docker daemon this is equivalent to configuring --dns 127.0.0.53 on every docker run. This will take precedence over the resolver IPs from host's /etc/resolv.conf. So the queries will be sent to 127.0.0.53 in the container's network namespace which obviously won't work if your resolver is running in the host 127.0.0.0.53.
Can you try without the --dns 127.0.0.53 in the docker daemon config ?
sanimej
on 3 Jul 2017
@sanimej Ok that seems to be work. Still leaves me with nothing for bridge0 and builds, so I still have to do the workaround but so long as DNS gets to the host properly that's progress.
wrouesnel
on 3 Jul 2017
@wrouesnel docker build does take a --network config. See if that works for you.
sanimej
on 3 Jul 2017
I get that the default network doesn't use the embedded DNS server for backward compatibility reasons - but I feel like it would be useful to have the option to enable it in there, which would be a neat, easy, one-line-in-a-config-file solution to these problems (which I think many people are still running into).
awilkins
on 25 Jul 2018
I am running containers that do not just depend on a host-localhost dns proxy but a similar http proxy. Creating a new network for that is a joke. The proxies on the host will have to change config and restart to start listening on the new bridge. And after a reboot they will fail to come up, because the bridges are not there anymore. Again change config and restart ... Or run an instance per bridge ...?
Just give us the reverse of "--publish" to make host-localhost services available in containers by port. I bet there are a thousand more valid use-cases for that (i.e. printing localhost:631)
henning-schild
on 29 Jun 2019
Containers on that default bridge can also not deal with network-roaming hosts. The copy of resolv.conf will get outdated. You will have to manually update it or restart your container. A problem that a host-localhost resolver proxy could solve ... but there is no way to get to it.
henning-schild
on 29 Jun 2019
Such a port mapping could be done with iptables or something like socat. Fun fact, when starting dockerd from an init system it will report running before the default bridge is up. So your services starting after dockerd might still fail to listen on the bridge.
henning-schild
on 29 Jun 2019
For clarification, my containers in user-defined networks need to access DNS running in a default-bridge container (on same host). Docker host has DHCP reservation (Host IP = DNS IP, so /etc/resolv.conf nameserver entry points to itās own IP address - not ālocalhostā). This does not appear to work. I cannot resolve DNS from the user-defined network containers.
HunterDG
on 28 Dec 2019
Related issues
BrianAdams
Ā·
9Comments
fcrisciani
Ā·
4Comments
blop
Ā·
4Comments
BSWANG
Ā·
10Comments
naanuswaroop
Ā·
9Comments
Most helpful comment
I get that the default network doesn't use the embedded DNS server for backward compatibility reasons - but I feel like it would be useful to have the option to enable it in there, which would be a neat, easy, one-line-in-a-config-file solution to these problems (which I think many people are still running into).