Compose: Container linking doesn't work reliably under docker-compose 1.6

[matevarga]

Environment: docker 1.10, docker-compose 1.6, Ubuntu 15.10

Background:

I've just updated docker to 1.10 and docker-compose to 1.7 on my Ubuntu 15.10 box. I am using around 7-8 containers linked with docker-compose, and it was working perfectly under docker-compose 1.6, but with 1.7, some containers don't see other (linked) containers. This seems to be related to the changed name resolution (changed from /etc/hosts to the /etc/resolv.conf-based approach).

Let's say I have containers named A and B. B is linked to A, and the Java code running inside the container wants to connect to 'A'. B starts up, I am getting exceptions saying "cannot connect to ". Note that is the random (?) ID of the container, not its name.

1) How does B know the ID of A? Somehow the DNS must return this ID when trying to look up 'A'.
2) If the DNS returns this ID, then why can't B connect to A through its ID?

So somehow, something must have regressed between 1.6 and 1.7.

Note that linking works for some containers (always), and doesn't for some others (always).

[dnephin]

It sounds like DNS is working, but the service might not actually be ready and listening to connection when another service tries to make the connection.

See https://github.com/docker/compose/issues/374#issuecomment-157822991

[matevarga]

Ok, found the issue -- container A sends back its hostname to B, and pinging A using its ID worked before 1.10/1.6, it doesn't work now. So it's not a real issue, but maybe useful to document.

[dnephin]

The container short id should be available as a hostname alias.

[matevarga]

but the service might not actually be ready and listening to connection

No, this is not the case.

The container short id should be available as a hostname alias.

I can confirm this is not the case. pkb-tomcat is linked to pkb-tolven, and while it can reach it by hostname, it can't connect by container ID.

➜  local git:(develop) docker ps | grep pkb-tolven
9c7062960a7c        pkbdev/pkb-tolven:base              "/bin/sh -c ./startTo"   5 minutes ago       Up 5 minutes        1099/tcp, 8081/tcp, 0.0.0.0:20021->20021/tcp               local_pkb-tolven_1

➜  local git:(develop) docker-compose run pkb-tomcat bash 
root@ea9dd4cec55b:/usr/local/tomcat# ping pkb-tolven
PING pkb-tolven (172.18.0.6): 56 data bytes
64 bytes from 172.18.0.6: icmp_seq=0 ttl=64 time=0.125 ms
64 bytes from 172.18.0.6: icmp_seq=1 ttl=64 time=0.101 ms
^C--- pkb-tolven ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.101/0.113/0.125/0.000 ms
root@ea9dd4cec55b:/usr/local/tomcat# ping 9c7062960a7c
ping: unknown host

[dnephin]

Thanks for the bug report!

I just tried it out. Looking at docker inspect <container name> I noticed the short id we're adding is 10 characters long, where as the docker cli uses a 12 character long short id. The ping works with the 10 character id.

I'm not sure why this is, or if it changed recently. We'll have to look to see if 12 is the standard representation for short id and if it is, I guess we should use that instead.

For now, using the first 10 characters should work.

[matevarga]

For now, using the first 10 characters should work.

Yep, that's a regression though -- in my case, container B connects to container A, and then container A sends back a hostname that B should use for some other service. In this specific scenario, that hostname is A, meaning that 'A' finds out its own hostname, and then sends it to B... and you see the problem.

[dnephin]

Yes, it is a bug, it should be fixed in the next bug fix release.

[chanux]

I thought I had the same issue and waited for the latests (v1.7.1) to land in pypi. I just upgraded but the problem is still there.

Docker version 1.11.1, build 5604cbe

docker-compose version 1.7.1, build 6c29830

Linux nim 3.16.0-70-generic #90~14.04.1-Ubuntu SMP Wed Apr 6 22:56:34 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I am just trying to link a mysql container to a web container via linking.

How can I help you help me?

[LaughingCoffin]

same issue with me ,

my docker-compose looks like this :

version: '2'
services:
db:
image: db:latest
ports:
- "3306:3306"
environment:
MYSQL_PASS: dpamdin
ON_CREATE_DB: dis
MYSQL_USER: admin
MYSQL_ROOT_PASSWORD: dpadmin
expose:
- "3306"
webapp:
build: ./tomcat
links:
- db
ports:
- "8080:8080"
- "8443:8080"
depends_on:
- db
expose:
- "8080"

When i run mysql docker separately , i can connect to mysql from my host machine, using the command : mysql -uadmin -pdpadmin -h0.0.0.0 -P3306

But when i run them through my docker-compose , my tomcat can't connect to mysql , even i can't connect to mysql from my host-machine , is this something to do with , volumes ? that store the configurations that u ran the first time ? or is it somthing else , plz help

[timkettering]

Ive been struggling with this same issue all morning and while I don't know exactly what is going on, here's what I'm able to confirm...

1. this problem occurs when my java program tries to look up the host of another container that's been stood up by docker-compose.
2. if exec bash in the container and try a networking command such as ping the hostname is resolved okay, so I know its specific to java.
3. if i stand up the individual docker instances using docker run with --name instead then my java program is able to successfully resolve the host.

I believe the issue revolves around the names that docker-compose creates for the containers, it seems to be a composite key such as foocompose_containername_number. whatever DNS resolution is occuring under the hood is allowing the service name to be resolved to the correct docker compose name, but that same operation is failing under java.

[abelsromero]

Same here as @timkettering says, I have a Java process that calls another. From a shell in the calling container it works, but the process is not able to reach the other :/

[capfel]

@timkettering @abelsromero
I found out here #2172 that the current docker version is using an embedded DNS server for resolving the host name.
Reading this article, http://www.myhowto.org/java/42-understanding-host-name-resolution-and-dns-behavior-in-java/ , I found out that Java caches the host names of the embedded DNS server and never updates it. That is why the host name is not available if you start all services at once via docker-compose up.
To verify this, I started my services one at a time with docker-compose start <service> with some pause in between, and then it actually worked.
Hope this helps.