Watchtower: Recreated container has different external port

[saitho]

Describe the bug
I'm running a basic image to test if Watchtower works for my use case. The exposed port is decided by the system, in my case it was "32768->8080" for the manual startup (i.e. internal port 8080 is exposed to 32768 which is configured in Nginx reverse proxy).

After pushing an update to Docker Hub, Watchtower pulls the new image and recreates the container.
However the external port is different (now: 32769). I'd expect the container to be recreated with the external port it had before.

To Reproduce
Steps to reproduce the behavior:

1. Create a container with random external port (docker run -d -p :80 saitho/docker-test:latest)
2. Push a new version of the Docker image to Docker Hub
3. Wait for Watchtower to pull the latest image
4. Check the external port of the recreated container. It does not match the initial external port.

Expected behavior
I expect my container to be restarted with the same external port, even if it was determined by the system.

Environment

• Ubuntu 18.10
  
  
  
  • Docker version 19.03.13, build 4484c46d9d
  
  

Logs from running watchtower with the --debug option

time="2020-11-28T16:10:57Z" level=info msg="Stopping /app (b3dcf79ca1630af3130704835f2f36b5d63163a3d3adb73986cec9ebe345a937) with SIGTERM"
time="2020-11-28T16:11:07Z" level=debug msg="Removing container b3dcf79ca1630af3130704835f2f36b5d63163a3d3adb73986cec9ebe345a937"
time="2020-11-28T16:11:07Z" level=debug msg="This is the watchtower container /watchtower"
time="2020-11-28T16:11:07Z" level=info msg="Creating /app"
time="2020-11-28T16:11:07Z" level=debug msg="Starting container /app (28540459504a057ae5798451ce93aa2c49b219cf8dcd7d3772d9c1e69dbfa66d)"

Additional context

Docker Inspect of the container

{
  "Id": "d7df03cb8198f29ad513a533687c723df7692762653dc722f880977b46744d9a",
  "Created": "2020-11-28T14:42:38.659724815Z",
  "Path": "/docker-entrypoint.sh",
  "Args": [
    "nginx",
    "-g",
    "daemon off;"
  ],
  "State": {
    "Status": "running",
    "Running": true,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 19252,
    "ExitCode": 0,
    "Error": "",
    "StartedAt": "2020-11-28T14:42:39.144688812Z",
    "FinishedAt": "0001-01-01T00:00:00Z"
  },
  "Image": "sha256:60c6a8fa6d515e7019eeb1b375bf82c3133db883b16df47711c195ed3785c431",
  "ResolvConfPath": "/var/lib/docker/containers/d7df03cb8198f29ad513a533687c723df7692762653dc722f880977b46744d9a/resolv.conf",
  "HostnamePath": "/var/lib/docker/containers/d7df03cb8198f29ad513a533687c723df7692762653dc722f880977b46744d9a/hostname",
  "HostsPath": "/var/lib/docker/containers/d7df03cb8198f29ad513a533687c723df7692762653dc722f880977b46744d9a/hosts",
  "LogPath": "/var/lib/docker/containers/d7df03cb8198f29ad513a533687c723df7692762653dc722f880977b46744d9a/d7df03cb8198f29ad513a533687c723df7692762653dc722f880977b46744d9a-json.log",
  "Name": "/stackhead-watch_saitho_me-app",
  "RestartCount": 0,
  "Driver": "overlay2",
  "Platform": "linux",
  "MountLabel": "",
  "ProcessLabel": "",
  "AppArmorProfile": "docker-default",
  "ExecIDs": null,
  "HostConfig": {
    "Binds": null,
    "ContainerIDFile": "",
    "LogConfig": {
      "Type": "json-file",
      "Config": {}
    },
    "NetworkMode": "default",
    "PortBindings": {
      "80/tcp": [
        {
          "HostIp": "0.0.0.0",
          "HostPort": "0"
        }
      ]
    },
    "RestartPolicy": {
      "Name": "always",
      "MaximumRetryCount": 0
    },
    "AutoRemove": false,
    "VolumeDriver": "",
    "VolumesFrom": null,
    "CapAdd": null,
    "CapDrop": null,
    "Capabilities": null,
    "Dns": null,
    "DnsOptions": null,
    "DnsSearch": null,
    "ExtraHosts": null,
    "GroupAdd": null,
    "IpcMode": "private",
    "Cgroup": "",
    "Links": null,
    "OomScoreAdj": 0,
    "PidMode": "",
    "Privileged": false,
    "PublishAllPorts": false,
    "ReadonlyRootfs": false,
    "SecurityOpt": null,
    "UTSMode": "",
    "UsernsMode": "",
    "ShmSize": 67108864,
    "Runtime": "runc",
    "ConsoleSize": [
      0,
      0
    ],
    "Isolation": "",
    "CpuShares": 0,
    "Memory": 0,
    "NanoCpus": 0,
    "CgroupParent": "",
    "BlkioWeight": 0,
    "BlkioWeightDevice": null,
    "BlkioDeviceReadBps": null,
    "BlkioDeviceWriteBps": null,
    "BlkioDeviceReadIOps": null,
    "BlkioDeviceWriteIOps": null,
    "CpuPeriod": 0,
    "CpuQuota": 0,
    "CpuRealtimePeriod": 0,
    "CpuRealtimeRuntime": 0,
    "CpusetCpus": "",
    "CpusetMems": "",
    "Devices": null,
    "DeviceCgroupRules": null,
    "DeviceRequests": null,
    "KernelMemory": 0,
    "KernelMemoryTCP": 0,
    "MemoryReservation": 0,
    "MemorySwap": 0,
    "MemorySwappiness": null,
    "OomKillDisable": false,
    "PidsLimit": null,
    "Ulimits": null,
    "CpuCount": 0,
    "CpuPercent": 0,
    "IOMaximumIOps": 0,
    "IOMaximumBandwidth": 0,
    "MaskedPaths": [
      "/proc/asound",
      "/proc/acpi",
      "/proc/kcore",
      "/proc/keys",
      "/proc/latency_stats",
      "/proc/timer_list",
      "/proc/timer_stats",
      "/proc/sched_debug",
      "/proc/scsi",
      "/sys/firmware"
    ],
    "ReadonlyPaths": [
      "/proc/bus",
      "/proc/fs",
      "/proc/irq",
      "/proc/sys",
      "/proc/sysrq-trigger"
    ]
  },
  "GraphDriver": {
    "Data": {
      "LowerDir": "/var/lib/docker/overlay2/1537164a89892d21c953102a0e50c467598dc31fe8f25bc4f4f6204e3807333f-init/diff:/var/lib/docker/overlay2/930f239dd451bbee33099d37471f5d5dbce0c13b80956b50c5e68eb78802bc00/diff:/var/lib/docker/overlay2/50f1dee2b3fcf04783a897302e8ff7022af29c104300a169b82c8954fd41bcf0/diff:/var/lib/docker/overlay2/4d8095646af03ed7f3958c03e5232f85f09ebec1f52040b43dadbc341457f368/diff:/var/lib/docker/overlay2/a5963377b32b254375b615fbc1279b589cbdc5ea7aeec8e666caf686655583d5/diff:/var/lib/docker/overlay2/4728c21ded685d444f7cdfb5112c102c5b724262dcddf2e13c7e41ed1e8b3b68/diff:/var/lib/docker/overlay2/00b9943438414979975f2c770de672b3155e52f8f65dcb2acbc794f69f64e74e/diff",
      "MergedDir": "/var/lib/docker/overlay2/1537164a89892d21c953102a0e50c467598dc31fe8f25bc4f4f6204e3807333f/merged",
      "UpperDir": "/var/lib/docker/overlay2/1537164a89892d21c953102a0e50c467598dc31fe8f25bc4f4f6204e3807333f/diff",
      "WorkDir": "/var/lib/docker/overlay2/1537164a89892d21c953102a0e50c467598dc31fe8f25bc4f4f6204e3807333f/work"
    },
    "Name": "overlay2"
  },
  "Mounts": [],
  "Config": {
    "Hostname": "28fa402de186",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "ExposedPorts": {
      "80/tcp": {}
    },
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "NGINX_VERSION=1.19.5",
      "NJS_VERSION=0.4.4",
      "PKG_RELEASE=1~buster"
    ],
    "Cmd": [
      "nginx",
      "-g",
      "daemon off;"
    ],
    "Image": "saitho/docker-test:latest",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": [
      "/docker-entrypoint.sh"
    ],
    "OnBuild": null,
    "Labels": {
      "maintainer": "NGINX Docker Maintainers <[email protected]>"
    },
    "StopSignal": "SIGQUIT"
  },
  "NetworkSettings": {
    "Bridge": "",
    "SandboxID": "0fab225a57abf1e1e476030a24121a41cbcf569d6bfac2f038f95c734b0ee22f",
    "HairpinMode": false,
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "Ports": {
      "80/tcp": [
        {
          "HostIp": "0.0.0.0",
          "HostPort": "32771"
        }
      ]
    },
    "SandboxKey": "/var/run/docker/netns/0fab225a57ab",
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null,
    "EndpointID": "",
    "Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "IPAddress": "",
    "IPPrefixLen": 0,
    "IPv6Gateway": "",
    "MacAddress": "",
    "Networks": {
      "stackhead-network-watch_saitho_me": {
        "IPAMConfig": {},
        "Links": null,
        "Aliases": [
          "app",
          "28fa402de186",
          "d7df03cb8198"
        ],
        "NetworkID": "9e0143923862b250283e2bb1dafe65bd47c0d4123314342e01d0562338d7df6a",
        "EndpointID": "65834269c64126ee06d58c212c03d7c6311719fe4ab9864ef17e1d97dbed49ae",
        "Gateway": "172.18.0.1",
        "IPAddress": "172.18.0.2",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "MacAddress": "02:42:ac:12:00:02",
        "DriverOpts": null
      }
    }
  }
}

(Link to uploaded file: docker-inspect.txt)

Checking via docker inspect it looks like the external port is stored in NetworkSettings.Ports["80/tcp"].HostPort.
I haven't really looked into what to adjust to make this work. But I'd be willing to contribute that fix.

[github-actions[bot]]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[simskij]

Hey @saitho and welcome to the community!

That would be grand! I'd be more than happy to assist you with reviews and what not to get this through.

Best,
Simme

[saitho]

I was to be using Watchtower with Terraform where I noticed that won't work as it breaks Terraform management.
I'll rely on the Terraform mechanisms for updating containers, so I won't be contributing the feature/bugfix to Watchtower for now.