Watchtower: Dependent containers are not shutdown or restarted

[kje1]

Describe the bug
I recently commented on https://github.com/containrrr/watchtower/issues/504#issuecomment-653531801 which has now been closed as the suggested fix was to use the com.centurylinklabs.watchtower.depends-on label for dependent containers. I have done this but unfortuantely this is still not working for me. Today my vpn container updated and a service which uses the vpn container for it's network, was initially not shut down but also not restarted again after the vpn image was updated. To solve this I have to destroy the dependent container and recreate it as the container ID of the vpn changes after an update.

As you can see below, I have both depends_on and the com.centurylinklabs.watchtower.depends-on label setup on the dependent container but there is nothing in the logs indicating that watchtower did anything with this containers.

Am I using the label wrong by any chance?

To Reproduce
Steps to reproduce the behavior:

1. Container A with dependency link to container B
2. Watchtower updates container B

Expected behavior

1. Container A should be stoppped
2. Container B should be updated and restarted
3. Container A should be restarted and connect to new container B

Docker Compose
Extract from docker compose file...

#WatchTower
  watchtower:
   image: containrrr/watchtower
   hostname: watchtower
   container_name: watchtower
   restart: always
   volumes:
    - /var/run/docker.sock:/var/run/docker.sock
   command: --interval "3600" --cleanup --debug
   environment:
    - TZ=${TZ}

#vpn        
  vpn:
    image: dperson/openvpn-client
    container_name: vpn
    # cap_add, security_opt, and volume required for the image to function
    cap_add:
      - net_admin
    environment:
      - TZ=Europe/London
      - FIREWALL=""
    networks:
      - default
    dns:
      - 8.8.8.8
      - 8.8.4.4
    read_only: true
    tmpfs:
      - /tmp
      - /run
    restart: always
    security_opt:
      - label:disable
    stdin_open: true
    tty: true
    volumes:
      - /dev/net:/dev/net:z
      - ${USERDIR}/docker/vpn:/vpn
    ports:
      - "XXXX:XXXX"

#Ombi
  ombi:
   image: linuxserver/ombi
   container_name: ombi
   restart: always
   volumes:
    - ${USERDIR}/docker/ombi:/config
    - ${USERDIR}/docker/shared:/shared
   environment:
    - PUID=${PUID}
    - PGID=${PGID}
    - TZ=${TZ}
   network_mode: service:vpn
   depends_on:
    - vpn
   labels:
    - "com.centurylinklabs.watchtower.depends-on=vpn"

Logs from running watchtower with the --debug option
time="2020-07-07T15:00:25+01:00" level=debug msg="This is the watchtower container /watchtower" time="2020-07-07T15:00:25+01:00" level=info msg="Stopping /vpn (bef6f210ff074c07efcce051ca1980f12bf1b4eddc5e6a16800b9d78b45de6c9) with SIGTERM" time="2020-07-07T15:00:31+01:00" level=debug msg="Removing container bef6f210ff074c07efcce051ca1980f12bf1b4eddc5e6a16800b9d78b45de6c9" time="2020-07-07T15:00:31+01:00" level=info msg="Creating /vpn" time="2020-07-07T15:00:31+01:00" level=debug msg="Starting container /vpn (1acd36abab38d472c93b26c8d914a42a94f5ad25fde70478866721286814a368)" time="2020-07-07T15:00:32+01:00" level=info msg="Removing image sha256:2403135f134b336da433a70a127d88f24a831463ad7c17a2ad1a3bbe1c376226" time="2020-07-07T15:00:32+01:00" level=debug msg="Scheduled next run: 2020-07-07 16:00:01 +0100 BST"

[github-actions[bot]]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[piksel]

I think you need to specify the label as:

   labels:
     com.centurylinklabs.watchtower.depends-on: vpn

[kje1]

Is this really the case? I've got labels specified like that for containers that I proxy via traefik that work as expected using my syntax. The label also appears in portainer as I would expect as well.

[piksel]

Maybe not, but it looked like it might be unintentional, as it would be parsed as an array of strings, rather than a key/value map, but I guess it's smart enough to detect that. Also, I made mistake in the syntax above. There shouldn't be any dashes before map items.

[kje1]

Hopefully it is just a syntax issue rather than the label not actually working correctly.

I just double checked all of my traefik labels and what I originally said was slightly wrong. It seems some of them have the syntax that you wrote and some have the syntax I wrote, as below. I must admit, I am fairly new to all of this and followed a pretty comprehensive guide when setting up my proxy with labels so I don't know much about how the syntax should be for strings or key / value map. What I can see though is that all of my traefik labels have the dash before them but your suggestion is to not include that?

Example traefik labels - 1 with equals, one with colon;

labels
  - "traefik.frontend.headers.STSPreload=true"
  - "traefik.frontend.headers.customFrameOptionsValue: allow-from https:${DOMAINNAME}"

The documentation here doesn't give any indication on how to use the label in other containers so for a novice like myself it has been a case of trial and error, but unforutantely with no success.

So based on all of that, how should the label be written? With or without a dash? With a colon instead of an equal sign? Does it need to be encompassed in quotation marks? I'll give your original suggestion a try too and hopefully that will fix the issue I'm having.

[piksel]

In this case, labels is technically an array of strings, equivalent of the following json:

"labels": [
  "traefik.frontend.headers.STSPreload=true",
  "traefik.frontend.headers.customFrameOptionsValue: allow-from https:${DOMAINNAME}"
]

compared to the key/value object:

"labels": {
  "traefik.frontend.headers.STSPreload": true,
  "traefik.frontend.headers.customFrameOptionsValue": "allow-from https:${DOMAINNAME}"
}

But I think docker-compose parses them and splits them to key-values, but I couldn't find it in the documentation. I have seen other instance of this in docker-compose files, so both probably work.
But you could try changing the label to:

    com.centurylinklabs.watchtower.depends-on: vpn

and see if it makes a difference, it's the only thing I found that stood out. It looks like it should work when looking through the code,

I'm thinking that Watchtower should probably output in the log why it's restarting containers, maybe a note about how many dependencies it found for a updated container? It really is quite hard to tell what is happening in these cases right now.

[simskij]

@Saicheg might be able to verify this, but from a) looking at the code and b) knowing that what it tries to emulate is links, I think this feature works in the opposite direction of what you've set up.

Depends on, from what I can tell, does not include a list of containers that you depend on, but rather a list of containers that depend on you. In your case, if ombi is updated, It would also restart vpn. A more obvious naming of the label would likely have been com.centurylinklabs.watchtower.children, which must have blown past me while reviewing it.

Could you please try moving your label to the VPN container instead and specify ombi as the value?

Thanks.

[kje1]

I came to a very similar conclusion yesterday actually. I'll reverse the dependencies and let you know if that solves the problem. Thanks!

[kje1]

Hi,

I've added the label like this as suggested. Could you confirm that this is the correct syntax please? Unfortunately it's now just a case of waiting for the the vpn to update to see if this works correctly. I guess I could try adding a depends on to another container as a trial...

I think @piksel makes a very good point above. Something in the logs to show what dependencies exist for a particular container would be really useful. That would give confidence during the inital setup process that everything should work as intended when containers do update.

#vpn        
  vpn:
    image: dperson/openvpn-client
    container_name: vpn
    # cap_add, security_opt, and volume required for the image to function
    cap_add:
      - net_admin
    environment:
      - TZ=Europe/London
      - FIREWALL=""
    networks:
      - default
    dns:
      - 8.8.8.8
      - 8.8.4.4
    read_only: true
    tmpfs:
      - /tmp
      - /run
    restart: always
    security_opt:
      - label:disable
    stdin_open: true
    tty: true
    volumes:
      - /dev/net:/dev/net:z
      - /home/kje1/docker/vpn:/vpn
    ports:
      - "XXXX:XXXX"
      - "XXXX:XXXX"
      - "XXXX:XXXX"
      - "XXXX:XXXX"
      - "XXXX:XXXX"
      - "XXXX:XXXX"
    labels:
      - "com.centurylinklabs.watchtower.depends-on: jackett, radarr, sonarr, qbittorrent, tautulli, ombi"

[simskij]

    labels:
      - "com.centurylinklabs.watchtower.depends-on: jackett, radarr, sonarr, qbittorrent, tautulli, ombi"

I would write it as:

labels:
  com.centurylinklabs.watchtower.depends-on: jackett, radarr, sonarr, qbittorrent, tautulli, ombi

just to remove any ambiguity regarding how it will be parsed. Other than that, yeah - this should work.

[zethis]

Same problem for me, depends_on: works perfectly with docker-compose.
Why not with watchtower ?

Edit:
It appears depends_on: it's only available with docker-compose, not with docker

Edit2:
Issue talking about that here: https://github.com/docker/compose/issues/5608
Not merged, Closed.

So if Watchtower use docker run after update, it's logical 👍

[xAlstrat]

I can't make it work. I think I tried all the possible options:

    labels:
      com.centurylinklabs.watchtower.depends-on: "python" # Also tried without the quotes (")

And the string version:

    labels:
       - "com.centurylinklabs.watchtower.depends-on=python"

I also added the label at build time in Dockerfile for labeling the image:

LABEL com.centurylinklabs.watchtower.depends-on="python"

The command docker inspect shows that the label is set correctly (for both image and container), so I think there is no problem with the label format. Here is my docker inspect raspbian (that depends on python container/service):

            ...
            "Labels": {
                "com.centurylinklabs.watchtower.depends-on": "python",
                "com.docker.compose.config-hash": "83fd007a7de0d61d8576236c62e335195d0f978fff8e85228a70c5d9f0b515e1",
                ...

This is my docker-compose.yml:

version: '3.7'

services:
  python:
    image: my-repo/python:latest
    restart: always
    container_name: python
    labels:
      com.centurylinklabs.watchtower.depends-on: raspbian
  raspbian:
    image: my-repo/raspbian:latest
    restart: always
    depends_on:
      - python # Obviously this doesn't work
    container_name: raspbian
    labels:
      com.centurylinklabs.watchtower.depends-on: python
  watchtower:
    restart: always
    image: containrrr/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./config.json:/config.json
    command: --interval 30 --cleanup --include-stopped --revive-stopped

When Watchtower detects a new python image, raspbian isn't stopped and logs shows no error about that container.

I even tried using depends-on label in python instead of raspbian (as you can see in the docker-compose file) like a kind of 'those-depends-on-me' label, but didn't work.

I'm running Watchtower and other services with docker-compose up.

UPDATE:

For some reason, I think my Watchtower is handling container names with a slash '/' as preffix (as shown in the logs in a previous post: /vpn), docker inspect is also showing my container name as /python. So I tried using "/python" instead of "python", but didn't work.

The interesting thing was when I configured both containers to depends-on the other:

  python:
    ...
    container_name: python
    labels:
      com.centurylinklabs.watchtower.depends-on: "/raspbian"
  raspbian:
    ...
    container_name: raspbian
    labels:
      com.centurylinklabs.watchtower.depends-on: "/python"

Using '/' preffix, Watchtower detected a circular dependency showing it in the logs:

watchtower_1_c63e785908fe | time="2020-07-31T17:18:03Z" level=info msg="circular reference to /raspbian"

When a new 'python' imagen is detected, the logs shows that a new image was found but 'python' container is not getting stopped/updated (neither 'raspbian' container). It seems ok for me because is there a circular reference.

So, using slashes, Watchtowers detects circular references but no unidirectional references (I tried referencing from python->raspbian and raspbian->python when pushing a new python image).

[xAlstrat]

Well, the code showed Watchtower was getting the container links but was not using them for shutting down/restarting the containers. I managed to make it work with some changes.

Linked containers were stopped using the watchtower image I built but I couldn't watch them restart because of this issue #584.

[Mattie112]

I also have a issue with the depends-on feature:

time="2020-09-07T12:14:07Z" level=info msg="Found new docker-registry.xxx.net/xxx/sites-generic-portal/nginx:master image (sha256:aaa)"
time="2020-09-07T12:16:30Z" level=info msg="Found new docker-registry.xxx.net/xxx/sites-generic-portal/php-fpm:master image (sha256:bbb)"
time="2020-09-07T12:17:38Z" level=info msg="Stopping /sites-generic-portal_phpfpm-master_1 (ccc) with SIGTERM"
time="2020-09-07T12:17:49Z" level=info msg="Stopping /sites-generic-portal_nginx-master_1 (ddd) with SIGTERM"
time="2020-09-07T12:17:50Z" level=info msg="Creating /sites-generic-portal_nginx-master_1"
time="2020-09-07T12:17:52Z" level=info msg="Creating /sites-generic-portal_phpfpm-master_1"

Yes it is correct that both a new nginx and a new php-fpm image are pushed. However I expected other containers to be restarted also. Here is my docker-compose.yml

version: '3.5'
services:
  nginx-router:
    image: nginx
    ports:
      - 80
    labels:
      com.centurylinklabs.watchtower.enable: true
      com.centurylinklabs.watchtower.depends-on: "nginx-master,phpfpm-master,frontend-master"
    restart: always
    depends_on:
      - nginx-master

  phpfpm-master:
    image: docker-registry.xxx.net/xx/sites-generic-portal/php-fpm:master
    labels:
      com.centurylinklabs.watchtower.enable: true
      com.centurylinklabs.watchtower.depends-on: "nginx-master,frontend-master,nginx-router"
    restart: always
  frontend-master:
    image: docker-registry.xxx.net/xx/sites-generic-portal-frontend:master
    labels:
      com.centurylinklabs.watchtower.enable: true
      com.centurylinklabs.watchtower.depends-on: "nginx-master,phpfpm-master,nginx-router"
    restart: always
  nginx-master:
    image: docker-registry.xxx.net/xx/sites-generic-portal/nginx:master
    labels:
      com.centurylinklabs.watchtower.enable: true
      com.centurylinklabs.watchtower.depends-on: "phpfpm-master,frontend-master,nginx-router"
    restart: always
    depends_on:
      - phpfpm-master
      - frontend-master

(simplified + not present in this file are the 'beta' and 'stable' containers that are exactly the same structure)

Basically I want to restart everything as soon as there is an update as I keep having connectivity issues between the nginx-router and for example the nginx-master.
Things I have tried:

• Only using the depends-on on the nginx-router
• Only using the depends-on on the nginx-master (and beta/stable)
• Using the "full" name as generated by docker-compose (sites-generic-portal_nginx-master_1 etc)
• Without the depends-on (hoping it would see the docker-compose depends_on)
• and probably some more/other combinations

I am 100% the label is set in the container (docker inspect) but I simply do not see that for example my nginx-router gets restarted. Any ideas?

[joshuapullin]

Is there any update on this or anybody got this working with docker compose?
I have got this setup based on the comments within this issue, with the vpn container having the label with three dependency containers in a comma-separated listed. I also tried with a space and without, so the split (looking at the code), will not have any trailing spaces.
Whenever the vpn container updates, there is no mention of any dependent containers in the log (with debug enabled).
I have added the label to one of the dependent containers to see if it works the other way round, and it still does not pick up the dependency.

I am trialing the label on various other containers in different formats to see if they work in different ways. These aren't really "dependent" on the others referenced, but just seeing if anything is missing.

Just to note, that I am using a socket-proxy container. I don't know if this needs anything explicit to be allowed? Before the depends-on label was referenced, containers were updating correctly (without any dependent containers being updated).

[simskij]

@xAlstrat

Using '/' preffix, Watchtower detected a circular dependency showing it in the logs

If your containers have a leading slash in their name, then you need to include it as well, as watchtower doesn't operate on the references or names in your docker-compose file, but rather the actual docker state.

Regarding the circular dependency, I first want to make clear that I'm not trying to be accusative or condescending here. I would however like to know what you expected to happen in this case?

It is hard for both containers to depend on each other, as that requirement creates a deadlock in the shutdown order: "Before shutting down python, shut down raspbian, but before you shutdown raspbian; shutdown python" 😅

@joshuapullin

Basically I want to restart everything as soon as there is an update

Looks like this circular dependency behavior is what you're going for as well. I'm afraid we don't support bidirectional dependencies at this point.

[joshuapullin]

@simskij
I’m not trying to get the circular dependency behaviour that @xAlstrat has referenced. The only reason I currently have it this way is test and see if any containers that have a dependency referenced also update, which they don’t.

I have three containers dependent on one vpn container. When any of the three dependent containers update I can still connect; when the vpn container updates the three dependent containers are not reachable.
I followed the comments on this issue relating to the original post, and put the label on the vpn container. None of the three containers are also stopped and restarted. There is also nothing in the logs. Looking at the code, the logs don’t get updated with anything relating to dependent containers either.
I have looked at the docs and apart from the one page relating to depends-on, which I have followed, I can’t see anything else needed (nor via the code) or any examples.

Just to reiterate, I have added the label to various other containers in different formats (based on the comments here regarding quoting) to see if a specific format is needed, but no luck yet.

I have Watchtower using a socket-proxy container for connecting to the docker host. I don’t know if Watchtower needs additional access via the socket-proxy for the dependencies but it updates okay currently?

Any help would be much appreciated?

@kje1 did you ever get this working in the end?

Thanks.

[xAlstrat]

Regarding the circular dependency, I first want to make clear that I'm not trying to be accusative or condescending here. I would however like to know what you expected to happen in this case?

I wasn't specting anything, I was just playing with dependencies to figure out how it worked. Unidirectional dependencies were not detected, but circle ones does.

Well, that's not the point.

I checked the source code (long ago) and... dependencies restarting logic wasn't completely implemented. So I build my own Watchtower docker image with the missing code to get it working.

My changes xAlstrat@1915a5dc04558fefd0df0bcb0537fb22fedc0d9f

[joshuapullin]

Thanks @xAlstrat, I will try your code this week if I get some time.
Will let you know.
Have skimmed over it and looks what it is needed.

[kje1]

Sorry for being quite on this one recently. No, I never got this working in the end despite trying numerous different formats for the labels. I too tried on different containers as @joshuapullin has done but with no luck. I left it as is because my VPN container never really updates and so it wasn't causing me too much of an issue but cleary there is something wrong.

I'll also give your code a try @xAlstrat and let you know if it works or not.