Compose: Host DNS servers not included on Centos 7

Created on 29 May 2018  路  11Comments  路  Source: docker/compose

Description of the issue

I found out that docker-compose seems to not include the nameservers of the host by default as it should when using a simple bridge network. This was reported a long time ago here: https://github.com/docker/compose/issues/2847 but it seems like it is still not fixed. I tried to disable firewalld without any luck. It worked though if I specify the dns directly for the containers who need them.

Context information (for bug reports)

docker-compose version 1.21.2, build a133471
docker-py version: 3.3.0
CPython version: 3.6.5
OpenSSL version: OpenSSL 1.0.1t  3 May 2016

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:20:16 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:23:58 2018
  OS/Arch:      linux/amd64
  Experimental: false

services:
  database:
    image: postgres:10.3
    ports:
    - 5432:5432/tcp
    restart: always
    volumes:
    - /home/degel/data:/var/lib/postgresql/data:rw
  letsencrypt:
    image: jrcs/letsencrypt-nginx-proxy-companion
    volumes:
    - /home/degel/nginx/certs:/etc/nginx/certs:rw
    - /home/degel/nginx/vhost.d:/etc/nginx/vhost.d:rw
    - /home/degel/nginx/html:/usr/share/nginx/html:rw
    - /var/run/docker.sock:/var/run/docker.sock:ro
  nginx-proxy:
    image: jwilder/nginx-proxy
    labels:
      com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: ''
    ports:
    - 80:80/tcp
    - 443:443/tcp
    restart: always
    volumes:
    - /home/degel/nginx/certs:/etc/nginx/certs:ro
    - /home/degel/nginx/vhost.d:/etc/nginx/vhost.d:rw
    - /home/degel/nginx/html:/usr/share/nginx/html:rw
    - /var/run/docker.sock:/tmp/docker.sock:ro
  service:
    depends_on:
    - database
    - letsencrypt
    - nginx-proxy
    entrypoint:
    - java
    - -Djava.security.egd=file:/dev/./urandom
    - -jar
    - -Dspring.profiles.active=deploy
    - app.jar
    environment:
      LETSENCRYPT_EMAIL: ****************
      LETSENCRYPT_HOST: ****************
      VIRTUAL_HOST: localhost,*****************
      VIRTUAL_PORT: '8080'
    image: ************************
    ports:
    - 8080:8080/tcp
    restart: always
version: '3.0'

Steps to reproduce the issue

  1. Setup a standard centos 7 installation
  2. docker-compose up -d with a valid docker-compose.yml
  3. docker exec -it nginx-proxy sh
  4. wget http://www.perdu.com

Observed result

It will say:
wget: unable to resolve host address 'www.perdu.com'

Expected result

It should access the website and download the index.html

Additional information

Centos 7

picture Sytten

Most helpful comment

Seems that this still and ongoing issue,

TEST IT
First test your connection, the IP should work:

$ sudo docker run --rm busybox ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=52 time=15.678 ms 64 bytes from 8.8.8.8: seq=1 ttl=52 time=15.884 ms 64 bytes from 8.8.8.8: seq=2 ttl=52 time=13.808 ms 64 bytes from 8.8.8.8: seq=3 ttl=52 time=14.874 ms

But the DNS resolution should fail:

$ sudo docker run --rm busybox ping www.google.com ping: bad address 'www.google.com'

HOW TO FIX IT
Follow the regular recommendation of adding the DNS server globally into docker host:

$ sudo vim /etc/docker/daemon.json

{ "dns": ["172.17.0.1", "8.8.8.8"], "ipv6": false }
And then restart the docker service

$ sudo service docker restart

This probably resolve the most majority of the problems, but for a few more, be sure that your docker interface is whitelist on your firewall, try the following:

$ sudo firewall-cmd --permanent --zone=trusted --change-interface=docker0 $ sudo firewall-cmd --reload
That should do the trick for most people, now just try it more time:

$ sudo docker run --rm busybox ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=52 time=15.678 ms 64 bytes from 8.8.8.8: seq=1 ttl=52 time=15.884 ms 64 bytes from 8.8.8.8: seq=2 ttl=52 time=13.808 ms 64 bytes from 8.8.8.8: seq=3 ttl=52 time=14.874 ms

Easy and simple,

picture AlexandeRiverou on 16 May 2020
👍2

All 11 comments

I'm not able to reproduce the issue;

When running on the default bridge network, the embedded DNS is not used, so you'll see the same DNS servers configured as on the host;

docker run --rm busybox cat /etc/resolv.conf
; Created by cloud-init on instance boot automatically, do not edit.
;
nameserver 67.207.67.3
nameserver 67.207.67.2
search localdomain

When running on a custom network the embedded DNS server (127.0.0.11) is used in the container;

docker network create mynetwork
docker run --rm --network=mynetwork busybox cat /etc/resolv.conf
search localdomain
nameserver 127.0.0.11
options ndots:0

However, the embedded DNS will forward requests it cannot resolve to the DNS servers that are configured on the host;

First, enable debug mode (assuming there's no existing daemon.json configuration file);

mkdir /etc/docker
echo '{"debug":true}' > /etc/docker/daemon.json
systemctl reload docker.service

Run a container that attempts to resolve www.perdu.com:

docker run --rm --network=mynetwork busybox ping -c3 www.perdu.com

PING www.perdu.com (208.97.177.124): 56 data bytes
64 bytes from 208.97.177.124: seq=0 ttl=52 time=96.856 ms
64 bytes from 208.97.177.124: seq=1 ttl=52 time=96.050 ms
64 bytes from 208.97.177.124: seq=2 ttl=52 time=96.165 ms

--- www.perdu.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 96.050/96.357/96.856 ms

Check the daemon logs will show that the embedded DNS forwarded requests to 67.207.67.3;

journalctl -n50 -u docker.service
...
May 29 11:16:59 centos-s-2vcpu-4gb-ams3-01 dockerd[9739]: time="2018-05-29T11:16:59.477816344Z" level=debug msg="Name To resolve: www.perdu.com."
May 29 11:16:59 centos-s-2vcpu-4gb-ams3-01 dockerd[9739]: time="2018-05-29T11:16:59.478017802Z" level=debug msg="[resolver] query www.perdu.com. (AAAA) from 172.18.0.2:55743, forwarding to udp:67.207.67.3"
May 29 11:16:59 centos-s-2vcpu-4gb-ams3-01 dockerd[9739]: time="2018-05-29T11:16:59.567582043Z" level=debug msg="[resolver] external DNS udp:67.207.67.3 did not return any AAAA records for \"www.perdu.com.\""
May 29 11:16:59 centos-s-2vcpu-4gb-ams3-01 dockerd[9739]: time="2018-05-29T11:16:59.568139089Z" level=debug msg="Name To resolve: www.perdu.com."
May 29 11:16:59 centos-s-2vcpu-4gb-ams3-01 dockerd[9739]: time="2018-05-29T11:16:59.568309985Z" level=debug msg="[resolver] query www.perdu.com. (A) from 172.18.0.2:55001, forwarding to udp:67.207.67.3"
May 29 11:16:59 centos-s-2vcpu-4gb-ams3-01 dockerd[9739]: time="2018-05-29T11:16:59.708061598Z" level=debug msg="[resolver] received A record \"208.97.177.124\" for \"www.perdu.com.\" from udp:67.207.67.3"

Trying to do the same using wget in the nginx-proxy image;

docker run --rm --network=mynetwork --entrypoint="" jwilder/nginx-proxy wget http://www.perdu.com

--2018-05-29 11:30:42--  http://www.perdu.com/
Resolving www.perdu.com (www.perdu.com)... 208.97.177.124
Connecting to www.perdu.com (www.perdu.com)|208.97.177.124|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 204 [text/html]
Saving to: 'index.html'

     0K                                                       100% 14.1M=0s

2018-05-29 11:30:42 (14.1 MB/s) - 'index.html' saved [204/204]
  • What DNS servers do you have configured on the host? Are they internal DNS servers, or perhaps a DNS server running on the local machine?
  • If you enable debug mode, do the logs show any information what's happening? (e.g., the embedded DNS forwarding the request but getting no response?)
  • Does it work if you manually set a DNS on the container (--dns=8.8.8.8)?
thaJeztah picture thaJeztah on 29 May 2018

Thanks for the fast reply, I will test that today.
The DNS servers are internal, but not running on the localhost.
I will enable debug right now to check.
Yes it works if I set the DNS directly, that's my temporary fix right now

Sytten picture Sytten on 29 May 2018

docker run --rm --network=mynetwork busybox ping -c3 www.perdu.com
ping: bad address 'www.perdu.com'

Logs:

May 29 07:50:32 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:32.971496190-04:00" level=debug msg="Name To resolve: www.perdu.com."
May 29 07:50:32 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:32.971867382-04:00" level=debug msg="[resolver] query www.perdu.com. (AAAA) from 172.19.0.2:36074, forwarding to udp:10.
May 29 07:50:32 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:32.980850800-04:00" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/start
May 29 07:50:32 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:32.980976997-04:00" level=debug msg="[resolver] external DNS udp:10.32.127.22 did not return any AAAA records for \"www.
May 29 07:50:32 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:32.986033773-04:00" level=debug msg="Name To resolve: www.perdu.com."
May 29 07:50:32 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:32.986238441-04:00" level=debug msg="[resolver] query www.perdu.com. (AAAA) from 172.19.0.2:33674, forwarding to udp:10.
May 29 07:50:32 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:32.992188749-04:00" level=debug msg="[resolver] external DNS udp:10.32.127.22 did not return any AAAA records for \"www.
May 29 07:50:33 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:32.996040570-04:00" level=debug msg="Name To resolve: www.perdu.com."
May 29 07:50:33 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:32.996230146-04:00" level=debug msg="[resolver] query www.perdu.com. (AAAA) from 172.19.0.2:59413, forwarding to udp:10.
May 29 07:50:33 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:33.001565748-04:00" level=debug msg="[resolver] external DNS udp:10.32.127.22 did not return any AAAA records for \"www.
May 29 07:50:33 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:33.005004252-04:00" level=debug msg="Name To resolve: www.perdu.com."
May 29 07:50:33 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:33.005194780-04:00" level=debug msg="[resolver] query www.perdu.com. (A) from 172.19.0.2:52784, forwarding to udp:10.32.
May 29 07:50:33 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:33.011920333-04:00" level=debug msg="[resolver] external DNS udp:10.32.127.22 did not return any A records for \"www.per
May 29 07:50:33 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:33.015930644-04:00" level=debug msg="Name To resolve: www.perdu.com."
May 29 07:50:33 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:33.016126097-04:00" level=debug msg="[resolver] query www.perdu.com. (A) from 172.19.0.2:37974, forwarding to udp:10.32.
May 29 07:50:33 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:33.018696538-04:00" level=debug msg="[resolver] external DNS udp:10.32.127.22 did not return any A records for \"www.per
May 29 07:50:33 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:33.023338192-04:00" level=debug msg="Name To resolve: www.perdu.com."
May 29 07:50:33 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:33.023510940-04:00" level=debug msg="[resolver] query www.perdu.com. (A) from 172.19.0.2:56259, forwarding to udp:10.32.
May 29 07:50:33 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:33.026758142-04:00" level=debug msg="[resolver] external DNS udp:10.32.127.22 did not return any A records for \"www.per
May 29 07:50:33 s6iprojet02.gel.usherbrooke.ca dockerd[846]: time="2018-05-29T07:50:33.174746479-04:00" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/exit

I think I know the issue, since the first external resolver returns false, it fails without trying the other ones. My resolv.conf is the following:

search gel.usherbrooke.ca
nameserver 10.32.127.22
nameserver 132.210.*.*
nameserver 10.34.104.32

EDIT:
I found out that this first DNS server (that I received from DHCP) is not able to resolve anything external to the subnet, but is still returning a NXDOMAIN. So I guess docker just doesn't care and thinks it might now the IP still.

nslookup perdu.com 10.32.127.22
Server:     10.32.127.22
Address:    10.32.127.22#53

** server can't find perdu.com: NXDOMAIN

On my host if I try without specifying a server

nslookup perdu.com
;; Got recursion not available from 10.32.127.22, trying next server
Server:     132.210.7.13
Address:    132.210.7.13#53

Non-authoritative answer:
Name:   perdu.com
Address: 208.97.177.124
Sytten picture Sytten on 29 May 2018

Ok it is a known issue. I am closing the issue here since it's not a docker-compose issue nor centos 7.
For reference:
https://github.com/moby/moby/issues/20494
https://github.com/docker/libnetwork/issues/1664

The problem is that docker does not fallback on the second DNS server if the first fails to resolve the name, but responds.
Thanks @thaJeztah to point me in the right direction ;)

Sytten picture Sytten on 29 May 2018

Looks like those logs were truncated; can you check if it's actually attempting to continue with the next DNS server? 

nameserver 132.210.*.*

That entry looks a bit odd; is that actually valid? Looking at RESOLV.CONF(5);

       nameserver Name server IP address
              Internet address of a name server that the resolver should
              query, either an IPv4 address (in dot notation), or an IPv6
              address in colon (and possibly dot) notation as per RFC 2373

Hm, actually, looking at the code; https://github.com/docker/docker-ce/blob/v18.03.1-ce/components/engine/vendor/github.com/docker/libnetwork/resolver.go#L493-L522, I see that both in a situation where the DNS returned _something_ and in a situation where the DNS returned _no response_ (resp == nil), a break is done; https://github.com/docker/docker-ce/blob/v18.03.1-ce/components/engine/vendor/github.com/docker/libnetwork/resolver.go#L522 (i.e. libnetwork leaves the loop, and won't try the next DNS.

In your situation it looks like we're ending up in https://github.com/docker/docker-ce/blob/v18.03.1-ce/components/engine/vendor/github.com/docker/libnetwork/resolver.go#L515-L517, which means the DNS returned a result for www.perdu.com, but the answers section was empty or didn't contain an IPv4 (A) or IPv6 (AAAA) record.

thaJeztah picture thaJeztah on 29 May 2018

Ah, haha, our comments just crossed :smile;

thaJeztah picture thaJeztah on 29 May 2018
👍1

The fix for https://github.com/docker/libnetwork/issues/1664

is actually included in Docker 18.03.1; https://github.com/docker/docker-ce/blob/v18.03.1-ce/components/engine/vendor/github.com/docker/libnetwork/resolver.go#L494-L498

Looks like this situation is slightly different (DNS didn't return a FAIL, but an empty response / answers section.

thaJeztah picture thaJeztah on 29 May 2018

I truncated the DNS server to avoid exposing to much info about my network (its a routable IP).
Thats true it might be a different issue though we should open a new issue in the libnetwork maybe?

Sytten picture Sytten on 29 May 2018

Yes, planning to open a quick PR for discussion

thaJeztah picture thaJeztah on 29 May 2018

Perfect let me know, I can provide more logs and do more tests if needed (gtg for now).
Again thanks for the fast replies!

Sytten picture Sytten on 29 May 2018

Seems that this still and ongoing issue,

TEST IT
First test your connection, the IP should work:

$ sudo docker run --rm busybox ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=52 time=15.678 ms 64 bytes from 8.8.8.8: seq=1 ttl=52 time=15.884 ms 64 bytes from 8.8.8.8: seq=2 ttl=52 time=13.808 ms 64 bytes from 8.8.8.8: seq=3 ttl=52 time=14.874 ms

But the DNS resolution should fail:

$ sudo docker run --rm busybox ping www.google.com ping: bad address 'www.google.com'

HOW TO FIX IT
Follow the regular recommendation of adding the DNS server globally into docker host:

$ sudo vim /etc/docker/daemon.json

{ "dns": ["172.17.0.1", "8.8.8.8"], "ipv6": false }
And then restart the docker service

$ sudo service docker restart

This probably resolve the most majority of the problems, but for a few more, be sure that your docker interface is whitelist on your firewall, try the following:

$ sudo firewall-cmd --permanent --zone=trusted --change-interface=docker0 $ sudo firewall-cmd --reload
That should do the trick for most people, now just try it more time:

$ sudo docker run --rm busybox ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=52 time=15.678 ms 64 bytes from 8.8.8.8: seq=1 ttl=52 time=15.884 ms 64 bytes from 8.8.8.8: seq=2 ttl=52 time=13.808 ms 64 bytes from 8.8.8.8: seq=3 ttl=52 time=14.874 ms

Easy and simple,

AlexandeRiverou picture AlexandeRiverou on 16 May 2020
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

CrimsonGlory picture CrimsonGlory  路  3Comments
AvdN picture AvdN  路  3Comments
bitver picture bitver  路  3Comments
davidbarratt picture davidbarratt  路  3Comments
HackerWilson picture HackerWilson  路  3Comments