Watchtower: Watchtower working but running docker-compose up -d still rebuilds container

[kevindd992002]

So Watchtower is working just fine for me as it pulls the latest image for a container daily at 6AM and I get email notifications regarding that. Here's one example of a notification I got today:

Subject: Watchtower updates on 62aace002bd4
Body:

2019-06-16 06:01:20 (info): Found new mdhiggins/sonarr-sma:latest image (sha256:aed9328022607ff84cd4ce48ab3ebd62b83888eb4641edbbe75c65dd1e2162a7)
2019-06-16 06:01:38 (info): Stopping /sonarr (b426c35fe58a016210b0583347406d333dbf70bf637e3cbb3a616b807420aed8) with SIGTERM
2019-06-16 06:01:43 (info): Creating /sonarr
2019-06-16 06:01:44 (info): Removing image sha256:293fdeef3265d13ec61235bd1ebcf207e44cac5504b60d7610fb5d18084123eb

So based on those logs, the sonarr container got recreated/updated automatically. To confirm this, I checked the hash of the current image that is in my system and inspected the image used by the running container and they were the same, so all good. To furhter confirm, I did these (and their corresponding results were):

docker-compose pull -> as expected, no image was pulled because it was already the latest
docker-compose up -d -> surprisingly, the sonarr container still got recreated.

So why was the container still recreated if the hash of the running container is the same as the one in the database anyway? I'm confused. Sorry, I'm a docker/Linux noob here.

[kevindd992002]

Any ideas?

[simskij]

I'm not really sure why docker-compose choses to recreate it. If I understand you correctly you've verified that the container actually got updated?

Do you do anything out of the ordinary in your compose-file that might cause this behavior? If the config has changed in any way during docker-compose up it will recreate the container even if the image hasn't changed.

[simskij]

Two other things to try, just out of curiosity:

1. Do a docker inspect of the container, then implicitly recreate it using docker-compose up -d and do another docker inspect. Any difference?

2. If you do two consecutive docker-compose up -d, does the container get recreated both times?

[kevindd992002]

Correct, the container gets actually updated but docker-compose still chooses to recreate it. The updated image hash in the repository even has the same exact image has inside the container when I expect it. I believe Watchtower does not use docker-compose to actually recreate the container and docker-compose doesn't want that, not sure? When a container gets recreated by Watchtower, does it respect the compose-file settings and retain the configuration in there?

No, I'm not touching my compose-file at all when this happens. And no, nothing special there really. This is happening for all the containers.

As for your questions:

1. Here's one that got updated yesterday:

Before running the command: https://pastebin.com/CBHqJFM6

After running the command and the container gets recreated: https://pastebin.com/JyriG9P6

Difference between the two: https://www.diffchecker.com/RzD7qrWk (could it possibly be because of the difference highlighted in line 217?)

1. The container gets recreated only once during the first docker-compose up -d run.

[controlcde]

I could confirm - exactly the same happens on my system.

[kevindd992002]

Thank you for confirming. At least I know now that this isn't an isolated case.

[kevindd992002]

@simskij , do you have any other ideas?

[simskij]

Sorry, Im currently on vacation. I'll have a look as soon as I get home.

[NDuggan]

I’m experiencing the same thing (not a big problem). I’m wondering if it’s something to do with a docker-compose cache or config. Because Watchtower initiated the update, it’s like Docker Compose doesn’t know about it.

I don’t have much knowledge about Docker Compose, so not sure if this is relevant but if you run docker-compose bundle to generate a .dab file, the image ID doesn’t match the current version (updated by watchtower).

Using @kevindd992002’s example:

• Sonarr original image ID: foo
• Sonarr image ID after watchtower update: bar
• Sonarr container image ID (after update): bar
• Sonarr image ID inside the .dab (bundle) file: foo

[simskij]

Oh, interesting. I've had my beard full rebasing and getting pre-/post-update lifecycle hooks in the last couple of days. This is definitely on my agenda, not just super prioritized as it doesnt really make any difference from a production perspective.

[NDuggan]

@simskij agreed, it's not a high priority in my point of view and not sure if my findings are just red herrings.

I also only see it recreate the containers once when running docker-compose up -d. If I run it again it doesn't recreate, so maybe it is just that Docker Compose isn't aware of the latest image.

[kevindd992002]

I understand how it's not a priority, no worries. At least it'a now acknowledged that it's some sort of a bug but of course is something that needs to be fixed. Sometimes I run docker-compose up -d manually and it really shouldn't recreate containers that were already updated by Watchtowerr.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kevindd992002]

How can we keep this not stale?

[cutzenfriend]

Same issue here! This should be fixed...

[simskij]

You are more than welcome to fix it if you like. 🙏🏻 I fail to see how it makes any difference at this point, more than cosmetic. 😊

[controlcde]

I fail to see how it makes any difference at this point, more than cosmetic. 😊

@simskij It´s more than a cosmetic thing. Normally with docker-compose up only modified containers will restart. Every container that watchtower has already updated (and restarted) will restart after executing docker-compose up so I could not update my compose file in working hour because it could restart necessary containers.

[kevindd992002]

@controlcde , you make a very good point if use docker containers in production. I hope they fix this issue.

[volschin]

I also think this bug is not a cosmetic one. Having lots of containers running in one compose file it‘s a problem all watchtower updated containers are recreated again only with a small change in the compose file.

[volschin]

Probably difficult to solve as docker compose seem to calculate its own hash as Label com.docker.compose.config-has. If this isn‘t updated docker-compose thinks with the next run the Container is outdated.

[kevindd992002]

Well, I hope the devs start to look into this.

[volschin]

Perhaps this helps https://docs.docker.com/compose/reference/config/
The —hash option was added in August 2018.

[jnidzwetzki]

I was facing the same problem today. I have looked for the reason and would like to document my findings in this ticket.

As @volschin already wrote, the problem is, that docker-compose creates a hash over the configuration of the container. When docker-compose up is called and the hash of a running container is different, then the container is recreated.

The hash is stored in a label called com.docker.compose.config-hash:

$  docker inspect <container>
[...]
"Labels": {
   "com.docker.compose.config-hash": "b42031fd69f7b4a2bde8f978ce32bab88aac4feeae9b232a22a9e2b3e63d8c72",
   "com.docker.compose.container-number": "1",
   "com.docker.compose.oneoff": "False",
   "com.docker.compose.project": "docker-compose",
   "com.docker.compose.project.config_files": "docker-compose.yml",
   "com.docker.compose.project.working_dir": "/root/docker-compose",
   "com.docker.compose.service": "hello-world",
   "com.docker.compose.version": "1.25.0"
}

The reason for the current behavior is that the hash value also contains the used image of the container. Watchtower updates the container image without recalculating the hash value. Therefore, docker-compose up detects a configuration change and recreates the container.

To solve the problem, the hash needs to be recalculated by watchtower. Unfortunately, the hash contains many values. The calculation of the hash can be found here (see also the config_dict function in line 685):

https://github.com/docker/compose/blob/2887d82d1649825ab21ed1f455b3d5d33a8739d5/compose/service.py#L682

I think the biggest challenge is to get all necessary information from the container runtime (e.g., secrets) to calculate the hash correctly.

[simskij]

Thank you very much @jnidzwetzki for your thorough investigation! 🙏🏼 I'm hesitating to pursue this feature in watchtower at all. To be totally honest, it sounds like recalculating the hash would be very fragile and could break without warning.

Any suggestions on how to proceed? I seldom modify my compose files, or run docker-compose up so I kind of need your input here. And do note that "this should be fixed" doesn't count as input. 😜

[jnidzwetzki]

To solve this problem, watchtower would need a re-implementation of the complete JSON hash algorithm (https://github.com/docker/compose/blob/dd240787c2d0018feb215344207dfe8093a798fc/compose/utils.py#L103).

The algorithm and the needed values depend on the docker-compose version and this would be very fragile. Maybe, we could implement different versions depending on the com.docker.compose.version label. In addition, it could be hard to get all the needed information for the hash from the container runtime.

Another solution for the problem might be to ask the docker-compose project to include only the _image name_ and not the _image id_ into the hash. Then the hash would be stable and don't need to be changed during image updates. However, how image updates in docker-compose up are handled is currently discussed in https://github.com/docker/compose/issues/3574 . Maybe we should wait until the docker-compose project has found a solution how to deal with image updates.

[volschin]

This discussion is running since 2016 with latest comment from 2018. I don‘t think something happens there.

[volschin]

Perhaps it would be an idea to link the binary of Docker-compose to a place in the container. This allows to use the hash algorithm from the installed Version.

[oliverpool]

If you have the --no-restart flag set and restart:always in the docker-compose config, does docker-compose restarts it with the updated image?

[simskij]

To solve this problem, watchtower would need a re-implementation of the complete JSON hash algorithm

This discussion is running since 2016 with latest comment from 2018. I don‘t think something happens there.

In the light of these two comments, I'm gonna go ahead and close this issue as a won't fix. For now, docker-compose users will just have to live with this as there is nothing, within reason, we can do from this end.

[kevindd992002]

To solve this problem, watchtower would need a re-implementation of the complete JSON hash algorithm

This discussion is running since 2016 with latest comment from 2018. I don‘t think something happens there.

In the light of these two comments, I'm gonna go ahead and close this issue as a won't fix. For now, docker-compose users will just have to live with this as there is nothing, within reason, we can do from this end.

But why? Isn't this a valid issue?

[werty1st]

I found a workaround. Unfortunately without involving watchtower.
https://gist.github.com/werty1st/3d2d8690260cc87fd9fd9cf5f88b79c1

[kevindd992002]

No email notif for that, I guess?