Compose: Links between containers doesn't work with version 2

[braska]

I use docker-compose v1.6.0 and docker engine 1.10.0.

I have the following docker-compose.yml:

version: '2'

services:
  db:
    image: postgres
    environment:
      POSTGRES_DB: test
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test1234
    volumes:
      - db-data:/var/lib/postgresql/data
  web:
    build: .
    command: /bin/bash -c "bundle exec rake db:migrate && bundle exec rails s -p 3000 -b 0.0.0.0"
    environment:
      RAILS_ENV: production
      DB_HOST: db
      DB_PORT: 5432
      DB_USER: test
      DB_DB: test
      DB_PASSWORD: test1234
      SECRET_KEY_BASE: 5ffbf99dcae1dee8500abd759114f473a942cb23a23db2ef8ce61972d2eaeedbb656ddb184ff0175ab4f8c5bcb29fa860505a140a6442608192dee1e3b168928
    ports:
      - "3000:3000"
    volumes:
      - public:/myapp/public
volumes:
  db-data:
    driver: local
  public:
    driver: local

But while starting containers with docker-compose up command, I see following error: web_1 | PG::ConnectionBad: could not translate host name "db" to address: Name or service not known.

I tried to find the reasons for this error:

[user@localhost test_project]$ docker-compose run web bash
Starting test_project_db_1
root@9bd26d6496bd:/myapp# cat /etc/hosts
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.18.0.3  9bd26d6496bd
root@9bd26d6496bd:/myapp# ping db
ping: unknown host

I added the following to web section in the docker-compose, but this has no effect:

    links:
      - db:db
    depends_on:
      - db

[dmaumenee]

Since version 1.10 Docker embeded a DNS server. The /etc/hosts file is not modified.

According to docker-compose network documentation, it should work without link declaration.

_By default Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name._

With my own containers, it's OK.
You can list the networks via docker network ls and inspect your default network with docker network inspect XXX_default

[braska]

Network test_project_default was successfully created with following address: 172.18.0.0/16. Both containers was connected to this network. In containers /etc/resolv.conf I see 127.0.0.11. But link between containers doesn't work (I can't ping).

According to docker-compose network documentation, it should work without link declaration.

I expect this behavior, but can't get it.

[braska]

I can ping another container by ip (172.18.0.2 can ping 172.18.0.3), but can't ping by hostname (web can't ping db)

[mavenugo]

@braska can you please share the output for docker info ? If you using centos/fedora/RHEL OS, we have a known issue in 1.10.0, where the workaround is to disable firewalld. We are working to fix that in the upcoming 1.10.1.

[braska]

@mavenugo, yes, I use Fedora 23.

Output for docker info:

Containers: 22
 Running: 2
 Paused: 0
 Stopped: 20
Images: 105
Server Version: 1.10.0
Storage Driver: devicemapper
 Pool Name: docker-253:0-27134065-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 4.546 GB
 Data Space Total: 107.4 GB
 Data Space Available: 102.8 GB
 Metadata Space Used: 9.851 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.138 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.109 (2015-09-22)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.3.4-300.fc23.x86_64
Operating System: Fedora 23 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 11.63 GiB
Name: localhost.localdomain
ID: 7ZRC:4NVN:TMPA:FEQ2:BLCL:J5GF:KT6E:T7IY:N5QN:7YC3:APBC:ZIDD
Username: braska
Registry: https://index.docker.io/v1/

[mavenugo]

@braska can you please disable firewalld and try out the tests ?

[braska]

@mavenugo, thank you so much for workaround! Now I can continue my work!

I got the result after sudo systemctl stop firewalld && sudo systemctl restart docker.
Without sudo systemctl restart docker I got following error: ERROR: Failed to Setup IP tables: Unable to enable SKIP DNAT rule: (iptables failed: iptables --wait -t nat -I DOCKER -i br-4d440d4c14f8 -j RETURN: iptables: No chain/target/match by that name. (exit status 1))

[mavenugo]

FYI, this is being addressed in https://github.com/docker/docker/issues/20026. You can close this issue.

[mariochavez]

Hi,

I'm having the same issues running a Docker Swarm on EC2.

Running docker info I got:

 Running: 9
 Paused: 0
 Stopped: 0
Images: 15
Server Version: swarm/1.2.3
Role: primary
Strategy: spread
Filters: health, port, containerslots, dependency, affinity, constraint
Nodes: 3
 node1: 50.112.2.111:2376
  └ ID: 3ZSM:QJGV:PJHX:WR2R:UCSI:WNUP:7N2J:B7YA:VZVZ:3J5M:IVTW:LBNV
  └ Status: Healthy
  └ Containers: 2
  └ Reserved CPUs: 0 / 2
  └ Reserved Memory: 0 B / 4.051 GiB
WARNING: No kernel memory limit support
  └ Labels: executiondriver=, kernelversion=4.2.0-18-generic, operatingsystem=Ubuntu 15.10, provider=amazonec2, storagedriver=aufs
  └ UpdatedAt: 2016-06-28T02:58:58Z
  └ ServerVersion: 1.11.2
 node2: 52.37.69.144:2376
  └ ID: M3X3:265Y:PI2H:EY67:ZD43:XCW5:GG6U:KQDJ:GO4A:PZ4M:TA5G:4QTP
  └ Status: Healthy
  └ Containers: 2
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.016 GiB
  └ Labels: executiondriver=, kernelversion=4.2.0-18-generic, operatingsystem=Ubuntu 15.10, provider=amazonec2, storagedriver=aufs
  └ UpdatedAt: 2016-06-28T02:58:46Z
  └ ServerVersion: 1.11.2
 node3: 50.112.70.206:2376
  └ ID: BGW3:M52K:SLC2:62BS:ZM2O:A2TT:3BZR:LNNA:YYR3:3EZW:VTGG:WLA6
  └ Status: Healthy
  └ Containers: 5
  └ Reserved CPUs: 0 / 2
  └ Reserved Memory: 0 B / 4.051 GiB
  └ Labels: executiondriver=, kernelversion=4.2.0-18-generic, operatingsystem=Ubuntu 15.10, provider=amazonec2, storagedriver=aufs
  └ UpdatedAt: 2016-06-28T02:59:10Z
  └ ServerVersion: 1.11.2
Plugins:
 Volume:
 Network:
Swarm:
 NodeID:
 IsManager: No
Security Options:
Kernel Version: 4.2.0-18-generic
Operating System: linux
Architecture: amd64
CPUs: 5
Total Memory: 9.118 GiB
Name: f221e2a87f30
Docker Root Dir:
Debug Mode (client): false
Debug Mode (server): false

My docker compose file is as follows:

version: "2"

services:
  web:
    image: custom-image
    env_file:
      - ./config/staging.env
    depends_on:
      - "redis"
    ports:
      - "7020:7020"
    restart: always
    environment:
      - REDIS_PORT_6379_TCP_ADDR=redis
      - REDIS_PORT_6379_TCP_PORT=6379
  redis:
    image: redis
    ports:
      - "6379:6379"
  proxy:
    image: jwilder/nginx-proxy:latest
    ports:
      - "80:80"
    volumes:
      - "/var/run/docker.sock:/tmp/docker.sock"

networks:
  default:
    driver: overlay

It starts and overlay network where once I start the compose file I can see all the containers in there:
docker network inspect app_default

[
    {
        "Name": "app_default",
        "Id": "1e9204c55f0a09b4fdce588789b2926f080a0ad65e22a061b209b32a9aefdd77",
        "Scope": "global",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1/24"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "9a0eb4a229e5765941ffe875e158285fa74ac3fd5c880b1c2d23ad47c73d243c": {
                "Name": "app_web_run_1",
                "EndpointID": "5fbecf0024e48b26a74a137fa4d63f9b78ff8f249f681706eacd491db6cb0c20",
                "MacAddress": "02:42:0a:00:01:05",
                "IPv4Address": "10.0.1.5/24",
           },
            "a1b3b3df8008817f859657ca74b4c23c3d15972c7d77fc3c41c2980661247514": {
                "Name": "app_proxy_1",
                "EndpointID": "7f8f0324a5ffb6d9c794c6e3ca92e6722689fd037b8df1fb876c58c0ccd2ce8c",
                "MacAddress": "02:42:0a:00:01:02",
                "IPv4Address": "10.0.1.2/24",
                "IPv6Address": ""
            },
            "b6a3ec7f880a7d5cab196a43b7306736a4261da677b6f472ae49abf6e9440c7f": {
                "Name": "app_redis_1",
                "EndpointID": "09b829d9192e407ea0a2ae69188ac3cc72a931bcbe5f5cae210b5a8d853fb94d",
                "MacAddress": "02:42:0a:00:01:03",
                "IPv4Address": "10.0.1.3/24",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {},
        "Engine": {
            "ID": "M3X3:265Y:PI2H:EY67:ZD43:XCW5:GG6U:KQDJ:GO4A:PZ4M:TA5G:4QTP",
            "IP": "52.37.xx.xxx",
            "Addr": "52.37.xx.xxx:2376",
            "Name": "master",
            "Cpus": 1,
            "Memory": 1039294464,
            "Labels": {
                "executiondriver": "",
                "kernelversion": "4.2.0-18-generic",
                "operatingsystem": "Ubuntu 15.10",
                "provider": "amazonec2",
                "storagedriver": "aufs"
            },
            "Version": "1.11.2"
        }
    }
]

My web app run in node1 while redis starts in node2, if I attach to bash session on node1 and I try to ping redis by name or IP, ping just hang at PING redis (10.0.1.3): 56 data bytes.

If I force both containers to run in the same node, then I can ping redis.

My nodes do not have firewalld running. I'm not sure what I'm missing here.

[benzid-wael]

I have the same issue, and I didn't find any way to fix it.

My environment:

OS:

Ubuntu 16.04

Docker:

Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:33:38 2016
OS/Arch: linux/amd64

Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:33:38 2016
OS/Arch: linux/amd64

Compose:

docker-compose version 1.8.1, build 878cff1
docker-py version: 1.10.3
CPython version: 2.7.12
OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016

Note that even for some containers, I need to run them with --net==host to be able to access them form host

[baykier]

@benzid-wael it's same to me ,same ubuntu1604,and same docker version ,only the docker-compose is 1.9.0 ship by a docker image . i had run --link manuly and is ok! looks like the docker-compose bugs

[nicooga]

I had this issue with a dockerized DigitalOcean VM running Ubuntu 16.04.
Upgrading docker engine on the remote host to latest solved the problem, though I also upgraded docker-compose locally.
It's strange because it worked fine before with the old versions.
Currently I'm running this versions:

~bash
$ ssh remote-host docker -v
Docker version 1.13.1, build 092cba3
$ docker -v
Docker version 1.13.1, build 092cba3
$ docker-compose -v
docker-compose version 1.11.1, build 7c5d5e4
~

[MikeC711]

FWIW: I had trouble with a pip installed docker-compose 1.12.x on Xenial. I downgraded to 1.11.2 and it resolved it ... then, for some apparent reason it started again. If I put a delay in the command and append to /etc/hosts ... then it works OK, but one of the containers leaves me as non-root w/no sudo ... so I can't fix that one. Not sure why it stopped working. I removed all images and started again ... seems to be a bit of an issue for many folks

[kopax]

is this still open? I have it...

[codegyaan]

This issue exists for me as well.
Docker version: 1.12.6
Docker compose version: 1.9.0

Docker running Ubuntu.

docker info:
Containers: 16
Running: 0
Paused: 0
Stopped: 16
Images: 322
Server Version: 1.12.6
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 399
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null overlay bridge host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.41-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.951 GiB
Name: moby
ID: 5VOX:G6E5:DTL2:AQW7:P7M7:64VG:LXHS:KZ3V:4T3Q:EUP2:6Z7I:ULIS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 22
Goroutines: 39
System Time: 2017-12-14T09:33:43.099410038Z
EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
WARNING: No kernel memory limit support
Insecure Registries:
127.0.0.0/8