Watchtower: Linked container will no longer be stopped and restarted

[samuel-ye]

Now docker seems to treat --link (as well as docker-compose links) within NetworkSettings Links instead of inside HostConfig. When new dependent docker image is fetched by watchtower, only such container is auto closed and restarted, other linked containers (which links to updated container and should be restarted together) are not handled correctly by watchtower.

[stffabi]

Thanks for the repport. Which version of docker are you using?

[samuel-ye]

my docker version:
Client:
Version: 1.12.6
API version: 1.24
Package version: docker-1.12.6-61.git85d7426.el7.centos.x86_64
Go version: go1.8.3
Git commit: 85d7426/1.12.6
Built: Tue Oct 24 15:40:21 2017
OS/Arch: linux/amd64

Server:
Version: 1.12.6
API version: 1.24
Package version: docker-1.12.6-61.git85d7426.el7.centos.x86_64
Go version: go1.8.3
Git commit: 85d7426/1.12.6
Built: Tue Oct 24 15:40:21 2017
OS/Arch: linux/amd64

[flipace]

I think we might have the same issue. None of our (multiple) linked containers are restarted when the container is updated.

Our Version info:

Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:24:16 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:22:56 2017
 OS/Arch:      linux/amd64
 Experimental: false

Is there any way to work around this? We now have to manually restart the containers to get it to work :(

EDIT: Oh, and we're using watchtower:latest in this case

[stffabi]

@flipace thanks for your report. Would it be possible for you to give me some example docker run commands with linked containers where watchtower is failing? That would greatly help me, to track it down.

[qqilihq]

I'm quite new to the Watchtower party, so I'm not sure whether I'm mis-interpreting the documentation or I'm also hitting this issue. A maximally reduced minimal example as follows:

version: '3'
services:

  nginx:
    image: nginx
    links:
      - service

  service:
    image: my-image
    labels:
      - com.centurylinklabs.watchtower.enable=true

  watchtower:
    image: v2tec/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: --interval 10 --label-enable --debug --no-pull

Note, I'm using most of the watchtower parameters just for local testing.

Dockerfile for my-image:

FROM nginx
RUN echo "Hello world!" > /usr/share/nginx/html/index.html

Test it:

$ docker build -t my-image .
$ docker-compose up

Then, modify the Dockerfile for my-image and rebuild it:

$ docker build -t my-image .

After ~ 10 seconds, Watchtower detects the changes and the following happens:

watchtower_1  | time="2018-04-15T18:53:39Z" level=debug msg="Checking containers for updated images" 
watchtower_1  | time="2018-04-15T18:53:39Z" level=debug msg="Retrieving running containers" 
watchtower_1  | time="2018-04-15T18:53:39Z" level=info msg="Found new my-image:latest image (sha256:db72a62491ddd7ece16905d8557bfdd7d6a72628191210ef614f47c07672ac40)" 
watchtower_1  | time="2018-04-15T18:53:39Z" level=info msg="Stopping /watchtowertest_service_1 (7f029015bce8df1fd7002a7ca76ff12929dc5e1c73ed025b71ccbd6095619e76) with SIGTERM" 
watchtowertest_service_1 exited with code 0
watchtower_1  | time="2018-04-15T18:53:40Z" level=debug msg="Removing container 7f029015bce8df1fd7002a7ca76ff12929dc5e1c73ed025b71ccbd6095619e76" 
watchtower_1  | time="2018-04-15T18:53:40Z" level=info msg="Creating /watchtowertest_service_1" 
watchtower_1  | time="2018-04-15T18:53:40Z" level=debug msg="Starting container /watchtowertest_service_1 (fa697efe3ecd9d396752d83516cef92aa73136a0ce833d1ff8e67cef450e1854)" 
watchtower_1  | time="2018-04-15T18:53:41Z" level=debug msg="Scheduled next run: 2018-04-15 18:53:49 +0000 UTC" 

But the nginx service is not being restarted. According to the docs, I would have expected, that nginx would be restarted as well?

[flipace]

@qqilihq this looks like it's the same issue i have - i didn't have time to make the repro though, but yours looks good! we had to disable wt because of this :(

[alallier]

Are there any alternative projects that provide the same functionality?

[workingmachine]

I have detect the same issue, see watchtower log:

nextcloud and ttrss are started with a link to mariadb.

Unfortunately mariadb was stopped before nextcloud and ttrss were stopped.
At the starting, nexcloud and ttrss are started before maraidb was started.

root@workingmachine ~ # docker logs watchtower
time="2018-10-26T14:41:59+02:00" level=info msg="First run: 2018-10-26 14:46:59 +0200 CEST"
time="2018-10-26T14:47:06+02:00" level=info msg="Found new linuxserver/tt-rss:latest image (sha256:577e619a928c23c026cca6f789a9ea7df47ff425f47493db6e8eb9b0bfb834c3)"
time="2018-10-26T14:47:18+02:00" level=info msg="Found new nextcloud:14 image (sha256:dbcf87f7f2897ca0763ece1276172605bd18d00565f0b8a86ecfc2341e62a3f4)"
time="2018-10-26T14:47:28+02:00" level=info msg="Found new mariadb:10.2 image (sha256:16f756926078fa66ed64c0940ea4944e91d60b46483b28b241a28974dc3704b5)"
time="2018-10-26T14:47:28+02:00" level=info msg="Stopping /mariadb (a30ba4ff0115251b70c93f5c6ca008f83f6d569102fcf0b76c287ae69178bdc2) with SIGTERM"
time="2018-10-26T14:47:30+02:00" level=info msg="Stopping /nextcloud (bae15b97043a056386513dde4807014d185ff46f74d0727ff40c4ece0f771e59) with SIGTERM"
time="2018-10-26T14:47:31+02:00" level=info msg="Stopping /ttrss (60a5a066b7480f5a2e763151daba15b0099c02309abcc1271c4f1cebf3c02b05) with SIGTERM"
time="2018-10-26T14:47:35+02:00" level=info msg="Creating /ttrss"
time="2018-10-26T14:47:36+02:00" level=info msg="Removing image sha256:9724f106ac94d6ea7c542f138ac60069095d88bbb3550ad1a7b5d85e46b4c55b"
time="2018-10-26T14:47:36+02:00" level=info msg="Creating /nextcloud"
time="2018-10-26T14:47:37+02:00" level=info msg="Removing image sha256:a36a7b14ffa47969ed0fc002e54a1dd55e6ca7a8369cfb7cf43dc18f592b3a3f"
time="2018-10-26T14:47:40+02:00" level=info msg="Creating /mariadb"
time="2018-10-26T14:47:40+02:00" level=info msg="Removing image sha256:26ea4885da20491a5051470f1bc13ad691bfb8e7dda95eaf71eda0037a58d48b"

[simskij]

As docker links are considered deprecated, this isnt something I will prioritize. If someone would like to have a stab at providing a PR, I'd be happy to review it however.

Docker official documentation deprecation notice added as reference.