Libnetwork: internal DNS has wrong ip address mappings

[BrianAdams]

Started two containers via docker-compose. 78b39860a0de (ip:172.23.0.3) and 8da0903d79a6 (ip:172.23.0.2). When I attempt to resolve the 8da0903d79a6 container from either the host or from container 78b39860a0de it gives me container 78b39860a0de ip.

docker exec -it 78b39860a0de ping -c1 8da0903d79a6
PING 8da0903d79a6 (172.23.0.3): 56 data bytes
64 bytes from 172.23.0.3: seq=0 ttl=64 time=0.048 ms

That should have been 172.23.0.2 returned from the internal DNS server but it was not.

docker --version
Docker version 1.12.0-rc2, build 906eacd, experimental

docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                              NAMES
78b39860a0de        badevguru/dives_api   "/bin/sh -c '/bin/sh "   4 minutes ago       Up 4 minutes        3000/tcp, 0.0.0.0:5858->5858/tcp                   riak_dives-api_1
8da0903d79a6        badevguru/riakts      "/usr/lib/riak/riak-c"   11 hours ago        Up 6 minutes        0.0.0.0:32840->8087/tcp, 0.0.0.0:32839->8098/tcp   riak_riak-ts_1

docker exec -it 78b39860a0de cat /etc/hosts
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.23.0.3  78b39860a0de

docker exec -it 8da0903d79a6 cat /etc/hosts
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.23.0.2  8da0903d79a6

And the compose file:

version: "2"
services:
  riak-ts:
    image: badevguru/riakts
    ports:
      - "8087"
      - "8098"
    labels:
      - "com.basho.riakts.cluster.name=test"
    environment:
      - PRIMARY_NODE=riak-ts
  dives-api:
    image: badevguru/dives_api
    links:
      - riak-ts
    ports:
      - "5858:5858"
    environment:
      - RIAK_SERVER=riak-ts
      - DEBUG_FLAG=--debug-brk

[barodeur]

I'm having having exactly the same problem.

I'm using docker for macOS

$ docker --version
Docker version 1.12.0-rc4, build e4a0dbc, experimental

$ docker-compose --version
docker-compose version 1.8.0-rc2, build c72c966

I noticed that in my case, sometimes the DNS resolves correctly and sometimes not.
As you can see bellow, redis is resolved from my app container as 172.20.0.2 the first time and then 172.20.0.3 the second time.

$ dc exec app bash
root@b1aa03402c57:/app# ping redis
PING redis (172.20.0.2): 56 data bytes
64 bytes from 172.20.0.2: icmp_seq=0 ttl=64 time=0.228 ms
^C--- redis ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.228/0.228/0.228/0.000 ms
root@b1aa03402c57:/app# ping redis
PING redis (172.20.0.3): 56 data bytes
64 bytes from 172.20.0.3: icmp_seq=0 ttl=64 time=0.078 ms
^C--- redis ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.078/0.078/0.078/0.000 ms

I tried to recreate the docker network, but the problem was still there.

[simdevmon]

I am having the same problem too.

I run docker at Ubuntu 16.04.1 LTS

$ docker --version
Docker version 1.12.0, build 8eab29e

$ docker-compose--version
docker-compose version 1.8.0, build f3628c7

$ docker network inspect ...
            "b98613018f8ba02aa96b1fce47753ac6e9d61a277c8a7fb1df99203eea7e21be": {
                "Name": "dockerfiles_db-system-test_1",
                "EndpointID": "2df87cd6f9784f738ad31b36a5306991f52a357e549389c56ab350a08fcb930c",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            },
            "eac47e83736ca066bcec5308897926341f4011fffb66435538be2af881297d10": {
                "Name": "dockerfiles_db-integration-test_1",
                "EndpointID": "9ed4c3dbdc224b1d93fdf218b1b21dd8964f04c87e9271163482b5ab0ce3dbab",
                "MacAddress": "02:42:ac:11:00:03",
                "IPv4Address": "172.17.0.3/16",
                "IPv6Address": ""
            }

With docker compose I link both db-integration-test and db-system-test. When I ping it directs to the wrong container:

$ ping db-system-test
PING db-system-test (172.17.0.3): 56 data bytes
64 bytes from 172.17.0.3: icmp_seq=0 ttl=64 time=0.106 ms
64 bytes from 172.17.0.3: icmp_seq=1 ttl=64 time=0.075 ms

$ ping db-integration-test
PING db-integration-test (172.17.0.2): 56 data bytes
64 bytes from 172.17.0.2: icmp_seq=0 ttl=64 time=0.131 ms
64 bytes from 172.17.0.2: icmp_seq=1 ttl=64 time=0.067 ms

After restart sometime it works fine and after another restart it points to the wrong container again.

[sanimej]

@simdevmon Can you share your compose file ? If you are images are private I can try with one of the standard disto images from the hub.

[simdevmon]

@sanimej Sure, the Jenkins Dockerfile is a custom one, but extended from the offical Jenkins Dockerfile (latest).

# --- jenkins configuration
jenkins:
  build: "./ci-jenkins"
  ports:
   - "8080:8080"
   - "50000:50000"
  tty: true
  volumes:
   -  /opt/nmc-ci/jenkins_home/:/var/jenkins_home      
  links:
  - db-integration-test
  - db-system-test

# --- database for integration tests
db-integration-test:
   image: postgres
   ports:
    - "5432:5432"
   environment:
   - POSTGRES_USER=dbit
   - POSTGRES_PASSWORD=dbit

# --- database for system tests
db-system-test:
   image: postgres
   environment:
   - POSTGRES_USER=dbst
   - POSTGRES_PASSWORD=dbst

[simdevmon]

The problem still persists in Docker 1.12.3

[janten]

I am seeing the same issue in Docker version 1.12.6, build d5236f0.

[Spechal]

Also appears with Docker version 17.03.1-ce, build c6d412e && docker-compose version 1.11.2, build dfed245

[jzako]

+1 the same issue
Docker version 17.03.1-ce, build c6d412e
docker-compose version 1.11.2, build dfed245
osx

[fcrisciani]

fixed by: #1796