Compose: Port conflict with multiple "host:<port range>:port" services

[andrei-m]

Description of the issue

If a compose file defines multiple services that share an overlapping port range, a port conflict occurs when docker-compose up is executed. This behavior started to happen for me when upgrading to Docker Desktop 2.2.0 (Stable) for Mac. The version I was previously running (I believe 2.1.5) was able to select distinct ports for the two services without conflicting. Here is a POC Docker Compose file:

version: '2.1'
services:
  postgres-foo:
    image: postgres
    ports:
      - "127.0.0.1:32768-61000:5432"

  postgres-bar:
    image: postgres
    ports:
      - "127.0.0.1:32768-61000:5432"

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.25.2, build 698e2846
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

services:
  postgres-bar:
    image: postgres
    ports:
    - 127.0.0.1:32768-61000:5432/tcp
  postgres-foo:
    image: postgres
    ports:
    - 127.0.0.1:32768-61000:5432/tcp
version: '2.1'

Steps to reproduce the issue

1. docker-compose up

Observed result

Services fail to start and a port conflict error occurs

Expected result

Services start up. An available port within the overlapping range is selected for each service.

Stacktrace / full error message

docker-compose up
Starting docker-bug-poc_postgres-foo_1 ...
Starting docker-bug-poc_postgres-bar_1 ... error
Starting docker-bug-poc_postgres-foo_1 ... error
ERROR: for docker-bug-poc_postgres-bar_1  Cannot start service postgres-bar: Ports are not available: listen tcp 127.0.0.1:32768: bind: address already in use

ERROR: for docker-bug-poc_postgres-foo_1  Cannot start service postgres-foo: Ports are not available: listen tcp 127.0.0.1:37587: socket: too many open files

ERROR: for postgres-bar  Cannot start service postgres-bar: Ports are not available: listen tcp 127.0.0.1:32768: bind: address already in use

ERROR: for postgres-foo  Cannot start service postgres-foo: Ports are not available: listen tcp 127.0.0.1:37587: socket: too many open files
ERROR: Encountered errors while bringing up the project.

Additional information

I uploaded diagnostic information with the ID 37C41A27-71E7-4431-AFBE-5DA0EEC74A3C/20200126225028

[cosmincrihan]

I am experiencing a similar issue that may come from the same cause.
The context is the same: upgrading to Docker Desktop 2.2.0 on MacOS breaks the port publishing when trying to run an NFS server container.
To reproduce:

docker run -d --privileged --restart=always -v /exported_folder:/exported_folder -e NFS_EXPORT_DIR_1=/exported_folder -e NFS_EXPORT_DOMAIN_1=\* -e NFS_EXPORT_OPTIONS_1=rw,no_root_squash -p 111:111 -p 111:111/udp -p 2049:2049 -p 2049:2049/udp -p 32765:32765 -p 32765:32765/udp -p 32766:32766 -p 32766:32766/udp -p 32767:32767 -p 32767:32767/udp fuzzle/docker-nfs-server:latest

The error shown:

docker: Error response from daemon: driver failed programming external connectivity on endpoint <CONTAINER_NAME> (ad6b1dccbfe77e3708e687b5eed9311cf1a6f828e36c739f18f5daa5803461b7): Error starting userland proxy: listen tcp 0.0.0.0:111: bind: address already in use

I have already taken down every component related to NFS on the Mac itself, and nothing is occupying that port.
The output of sudo lsof -iTCP -sTCP:LISTEN -n -P:

COMMAND    PID    USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
launchd      1    root    8u  IPv4 0xd9679bbeeddece79      0t0  TCP *:22 (LISTEN)
launchd      1    root    9u  IPv6 0xd9679bbeedde7269      0t0  TCP *:22 (LISTEN)
launchd      1    root   11u  IPv4 0xd9679bbeeddece79      0t0  TCP *:22 (LISTEN)
launchd      1    root   14u  IPv6 0xd9679bbeedde7269      0t0  TCP *:22 (LISTEN)
dnscrypt-  955  nobody    8u  IPv4 0xd9679bbef0fb0e79      0t0  TCP 127.0.0.1:53 (LISTEN)
com.docke 1190 <USERNAME>    8u  IPv4 0xd9679bbef5ad2801      0t0  TCP 127.0.0.1:49273 (LISTEN)

Just downgrading to Docker Desktop 2.1.x solves everything.
Does anybody have any idea for a workaround?

[thaJeztah]

Wondering how this would've worked before, as it wouldn't be possible to have two processes listening on the same port 🤔

[cosmincrihan]

@thaJeztah should I file a separate issue, as it is not directly related to Docker compose?

Wondering how this would've worked before, as it wouldn't be possible to have two processes listening on the same port 🤔

[ravishaheshan]

Also when you add scale and run the command multiple times, it will create more containers than the "scale" value. For example when I ran it with --scale=3 for the first time it creates one container successfully and two containers fail with port conflicts. Second time 2 containers success 1 fails, 3rd time 2 success 1 fails. when I execute docker ps it shows 5 containers.
Note: I also have port range in my compose file. Before the 2.2.0 update It worked perfectly and created 3 containers and assigned ports within the given range.

[MartinLyne]

Any news on this? After upgrading docker on mac to > 2.0 my compose script using port ranges on scalable service also fails. It works perfectly on docker < 2.0.

[weimingwill]

@MartinLyne Did you find any solutions for this?