Compose: docker-compose up did not create /etc/hosts entry for linked container

[anentropic]

I had a yaml like:

redis:
  image: redis
  expose:
    - 6379

myapp:
  links:
    - postgres
    # forgot to link redis here

I did a docker-compose up -d and saw the problem with my app, it couldn't connect to hostname redis

no problem, I edited the yaml:

redis:
  image: redis
  expose:
    - 6379

myapp:
  links:
    - postgres
    - redis

I did a docker-compose up -d again expecting it to be fixed

nope ConnectionError: Error -2 connecting to redis:6379. Name or service not known

tried a docker-compose restart myapp... not fixed

restarted all services

docker-compose stop
docker-compose up -d

not fixed

exec'd into myapp and had a look in /etc/hosts ...I only have these entries:

172.17.0.219    postgres adddb7bd4709 oceania_postgres_1
172.17.0.219    postgres_1 adddb7bd4709 oceania_postgres_1
172.17.0.219    oceania_postgres_1.bridge
172.17.0.220    oceania_redis_1
172.17.0.220    oceania_redis_1.bridge

so, obviously problem is due to the missing link entry for redis container

it's a bit weird because normally this 'just works'

[anentropic]

docker-compose stop myapp
docker-compose rm -v myapp
docker-compose up -d

...fixed it

pretty sure I shouldn't have to go that far though?

[dnephin]

Are you using docker-compose 1.4.0 ? We fixed a bug like this in 1.4.2.

[anentropic]

oh yep 1.4.0

[dnephin]

Cool, upgrading should fix it!

[xaka]

I'm having the same issue with the following environment under OSX and latest docker toolbox:

$ docker -v
Docker version 1.8.3, build f4bf5c7

$ docker-compose -v
docker-compose version: 1.4.2

cat /etc/hosts
172.17.0.113    8fff64540333
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.113    name_service_2
172.17.0.113    name_service_2.bridge

I'd expect to see an entry for service, but it's not there.

[dnephin]

I think this is caused by a race condition in docker when writing out this file. I don't think there is anything we can do from the compose side. I believe it's being worked on for the docker 1.9 release.

If you stop, rm, and up again, does it still fail?

[xaka]

Unfortunately yes, it does. The sad part is that it is a permanent problem now with latest docker/docker-compose releases.

[dnephin]

We haven't had many reports of this, could you provide your docker-compose.yml ? Is the cat /etc/hosts from a run or exec from a container created by up ?

[igr]

We also experience the very same issue. On my machine with docker 1.8.1 this works, but on other machine with docker 1.8.3, this is not working - /etc/hosts on container does not get updated.

For me, this update happens only after first docker-compose start, so I need to start/stop/start in order to have it applied.

[josephearl]

Also having this issue. Noticed it because CI was failing but worked on developers machines - CI was running 1.5.0 and developers 1.4.0. Upgraded to 1.5.1 on dev machines and now seeing this problem every time.

All the link entries in /etc/hosts have the numbers and are prefixed with the project name.

For me the hosts file is similar to @xaka - from run.

docker-compose version: 1.5.1
docker-py version: 1.5.0
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.2a 19 Mar 2015

Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 18:01:15 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 18:01:15 UTC 2015
 OS/Arch:      linux/amd64

Edit: solved my issue - I had to specify both -f docker-compose.yml and -f docker-compose.prod.yml -- I had assumed it would automatically load the base compose file (which contained the link).

[dnephin]

@JosephEarl that is expected if you're running with --x-networking, is that the case?

[josephearl]

@dnephin just fixed my issue, see edited comment - was a problem with docker-compose overlays

Also I was not using --x-networking.

[pydevops]

I have ran into the same issue with the following Docker Toolbox versions, as of latest today.

• Docker version 1.10.3, build 20f81dd
• docker-compose version 1.6.2, build 4d72027

Tried stop,rm all stopped container and up again, still the same issue.

[ElijahGlover]

I can confirm the same issue with docker 1.10.3 and docker-compose version 1.6.2, build 4d72027

[pydevops]

I figured out a solution for docker-compose 1.10.x
Upgrade docker-compose.yml to Version '2'.
Please read https://docs.docker.com/compose/networking/ and try it out with docker exec bash inside the docker container by "ping my_service".This is part of service discovery of docker-compose.

docker-compose.yml

version: '2'
services:

[ElijahGlover]

@pydevops had version 2 all along still getting nothing in /etc/hosts or env vars

[pydevops]

@Elemecca it uses /etc/resolv.conf instead for DNS. no need to worry about /etc/hosts. "ping your_service" would work inside the docker container. "your_service" is the service name for "your_service" in docker-compose.yml. Link provides extra alias to work.
I think it uses the new software defined network instead of the legacy bridge network.

[ElijahGlover]

@pydevops thanks, not broken just misunderstood

[kkorada]

facing same issue,
when i am linking containers i am not able to see its entry in /etc/hosts but ping is still working
i ama using docker-compose version 1.7.1, build 0a9ab35
and docker version is 1.11.1

[LightningK0ala]

^ same here

[xwang2713]

me too. Is there a way to find how many containers per service in the master container which has "links" configured if no entry inserted in /etc/hosts? After scale there will be multiple containers for a service. Use "getent hosts _" to iterate probably is not a good way.

[LightningK0ala]

@dnephin should we reopen this issue?

[dnephin]

No, this is the expected behaviour. Any V2 compose files uses user defined networks, which does not modify /etc/hosts.

[xwang2713]

How does the service/container defined "links" find other linked service/containers? Particularly after "docker-compose scale".

[dnephin]

DNS , https://docs.docker.com/compose/networking/

[xwang2713]

look up service name only return first container ip or the service name as hostname only reference the first container in that service. If "docker-compose scale" to add more containers how do get their hostnames or its? In Kubernetes user can use restful API with URL to get container ips.

[aroneous]

I'm also having issues with a v1 config file, under Docker 1.11.2, docker-compose 1.7.1.

I find that links are not registered if the target container is not running at the time the linked container is created. Here, test-two has a link to test-one. You can see that if test-one is created but stopped when test-two is created, there are no link definitions. If test-one is running when test-two is created, correct links are created.

Aron@aron-xps MINGW64 ~/Development/docker/compose-test
$ docker-compose create test-one
Creating test-one-container

Aron@aron-xps MINGW64 ~/Development/docker/compose-test
$ docker-compose create test-two
Creating test-two-container

Aron@aron-xps MINGW64 ~/Development/docker/compose-test
$ docker inspect --format="{{.HostConfig.Links}}" test-two-container
[]

Aron@aron-xps MINGW64 ~/Development/docker/compose-test
$ docker-compose rm -f test-two
Going to remove test-two-container
Removing test-two-container ... done

Aron@aron-xps MINGW64 ~/Development/docker/compose-test
$ docker-compose start test-one
Starting test-one ... done

Aron@aron-xps MINGW64 ~/Development/docker/compose-test
$ docker-compose create test-two
test-one-container is up-to-date
Creating test-two-container

Aron@aron-xps MINGW64 ~/Development/docker/compose-test
$ docker inspect --format="{{.HostConfig.Links}}" test-two-container
[/test-one-container:/test-two-container/one /test-one-container:/test-two-container/test-one-container]

Using: docker-compose.yml.txt

[Kaijun]

same here while using docker-compose version 1.6.2

Even though the host of linked container is not added into /etc/hosts,

but the ip of the linked container could be resolved actually.

Try running:

docker exec xxx getent hosts container-name

then you will see

XXX.XXX.XXX.XXX      linked-container-name linked-container-alias

[aanand]

@aroneous This is a known issue with old-style links and will not be changed. You can fix it by upgrading to the v2 Compose file format.

@Kaijun This is correct behaviour. Docker networking does not make use of /etc/hosts.

[dhrp]

When using the following sequence:

1. docker-compose up -d mongodb
2. docker-compose _run_ api bash

mongodb also does not get added to the /etc/hosts file of the api container. This is unfortunate, because golang's mgo driver then fails to find the mongodb host (even though ping and telnet do allow to connect to it). When using the local network IP for the mgo driver also allows it to connect successfully.

[edit] I read more above..

No, this is the expected behavior. Any V2 compose files uses user defined networks, which does not modify /etc/hosts.

OK, I see. But it exposes another _bug?_ -- Docker's DNS server does something funky such that the mgo driver can't use it, while it can use the same database when reached through the 'public' network. e.g. myhost.dyndns.org:27017.

[edit2]
No, the MGO driver is to blame. @jbielick (below) points to the root cause.

my docker-compose file:

version: '2'
services:
  api:
    image: golang:1.6
    ports:
      - 8080:8080
    volumes:
      - /Users/tpeskens/Develop/gostart/:/go/
    links:
      - mongodb:mongodb
    command: bash

mongodb:
    image: mongo
    volumes:
      - mongodb_volume:/data/db

volumes:
  mongodb_volume:
     driver: local

Server Version: 1.12.0-rc3
docker-compose version 1.8.0-rc1, build 9bf6bc6

[aanand]

That's strange, but if the mongodb hostname works from other programs (e.g. if you can run docker-compose run api ping mongodb) then it sounds like the problem is at the application level, i.e. in the mgo driver.

[jbielick]

@dhrp I think it's indeed in the mgo library itself. I had this very same problem with an almost identical docker-compose.yml. The discussion that made me suspicious was here, where this is metioned

but mgo has a really nutty address
resolver that first connects using UDP (where the closed port on ::1
is not detectable), then casts the UDPAddr into a TCPAddr (?!), and
connects to the IP address:

and the corresponding go-mgo/mgo code is here. Since it's parsing the Dial string for an IP and only finds the link alias, tries to resolve the IP, but fails.

[dhrp]

hmm. Thanks @jbielick. I found a related issue on Mgo, referenced above. We should probably move this discussion there. (or open a new issue on Mgo).

[dimakievua]

Hi,
same issue with me. docker-compose.yml:

  web:
    image: nginx
    ports:
      - "80:80"
    links:
      - php:php
    depends_on:
      - php

  php:
    image: php56
    depends_on:
      - db

$ docker-compose -f docker-compose.yml up -d
$ docker-compose exec web cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.18.0.4 6310ca711c0a
$ docker-compose exec web ping php
root@9a88778bbfc1:/# ping php
ping: unknown host php

$ docker version
Client:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built: Thu Jul 28 22:00:36 2016
OS/Arch: linux/amd64

Server:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built: Thu Jul 28 22:00:36 2016
OS/Arch: linux/amd64

$ docker-compose version
docker-compose version 1.8.0, build f3628c7
docker-py version: 1.9.0
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013

[aanand]

@dimakievua This will happen if php isn't running by the time web starts (e.g. if it exits immediately after starting).

[joedborg]

I'm getting the same on Mac, except I can communicate to the linked container even though it's not in /etc/hosts, nor do the env vars exist.

$ docker-compose rm
Going to remove docssitedocker_http_1, docssitedocker_application_1
Are you sure? [yN] y 
Removing docssitedocker_http_1 ... done
Removing docssitedocker_application_1 ... done

$ docker-compose up -d
Creating docssitedocker_application_1
Creating docssitedocker_http_1

$ docker-compose ps
            Name               Command   State         Ports        
-------------------------------------------------------------------
docssitedocker_application_1   php-fpm   Up      9000/tcp           
docssitedocker_http_1          nginx     Up      0.0.0.0:80->80/tcp 

$ docker-compose exec http cat /etc/hosts
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.19.0.3  b4162fbf29c6

$ docker-compose exec http ping application
PING application (172.19.0.2): 56 data bytes
64 bytes from 172.19.0.2: seq=0 ttl=64 time=0.101 ms
64 bytes from 172.19.0.2: seq=1 ttl=64 time=0.098 ms
64 bytes from 172.19.0.2: seq=2 ttl=64 time=0.090 ms

$ docker-compose exec http env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=b4162fbf29c6
no_proxy=*.local, 169.254/16
HOME=/root

$ docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.7.1
 Git commit:   6f9534c
 Built:        Thu Sep  8 10:31:18 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 17:52:38 2016
 OS/Arch:      linux/amd64

$ docker-compose --version
docker-compose version 1.8.1, build 878cff1

I need access to these, so I can use scale and then inject the hosts into a load balancer config.

[dimakievua]

Went deeper into documentation. docker_compose use internal DNS server and even there is no line in /etc/hosts it's possible to reach another container. The issue that there is more visibility if you can see hostnames and address directly in hosts file.

[ykshatroff]

dnephin commented on 20 Oct 2015

We fixed a bug like this in 1.4.2.

Reproduced exactly the same thing on docker 1.12.3 + docker-compose 1.9.0_rc4: restarting containers did not ensure new extra_hosts availability, until the containers were removed.

The yaml snippet:

  external_links:
      - dockers_redis_1:redis
      - dockers_nginx_1:nginx

The hosts were available in the mean time by their internal names a la dockers_nginx_1 but not by nginx.

OS: Gentoo Linux. Maybe the problem is in Gentoo portage build system, though.

[dkniffin]

I spent way too much time with this issue. Is it documented _anywhere_ that the links are no longer in /etc/hosts? I found a bunch of stackoverflows referencing /etc/hosts, and so I expected it the links to be defined there, and the documentation didn't say it _wasn't_ there, so I assumed that was correct.

[hadsed]

I'm facing this problem as well [*]. I used to use /etc/hosts to verify my links are working properly. What can I do now? I'm having redirect problems with nginx proxying and I'd like to know if it's misconfiguration in docker or otherwise.

[*] Versions:

docker-compose version 1.9.0, build 2585387
docker-py version: 1.10.6
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1t  3 May 2016

Client:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:42:17 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:42:17 2016
 OS/Arch:      linux/amd64

[AnthonyWC]

@dkniffin It doesn't have much details but it's documented here: https://docs.docker.com/engine/userguide/networking/configure-dns/

Container will use the embedded DNS server @ 127.0.0.11 to resolve for DNS name (/etc/resolv.conf)

Also see https://github.com/docker/docker/issues/19474

As of Docker 1.10, the docker daemon implements an embedded DNS server which provides built-in service discovery for any container created with a valid name or net-alias or aliased by link. The exact details of how Docker manages the DNS configurations inside the container can change from one Docker version to the next. So you should not assume the way the files such as /etc/hosts, /etc/resolv.conf are managed inside the containers and leave the files alone and use the following Docker options instead...

[mohammedSlimani]

if you use docker-compose run make sure you are using the --use-aliases param