Watchtower: Cleanup not does not work with rolling restarts

[maxnl]

Hi all,

I'm running watchtower with the argument --cleanup but it doesn't seem to be cleaning up any images after completing container update

I was running in debug mode and it doesn't seem to be even attempting to cleanup the old images. Attached is an abridged log (I removed the lines regarding checking for updates for images with no updates available)

time="2020-10-15T15:53:06-07:00" level=debug msg="Checking containers for updated images",
time="2020-10-15T15:53:06-07:00" level=debug msg="Retrieving running containers",
time="2020-10-15T15:53:14-07:00" level=debug msg="Pulling homeassistant/home-assistant:stable for /home-assistant",
time="2020-10-15T15:53:14-07:00" level=debug msg="No credentials for homeassistant in /config.json",
time="2020-10-15T15:53:14-07:00" level=debug msg="Got image name: homeassistant/home-assistant:stable",
time="2020-10-15T15:58:46-07:00" level=info msg="Found new homeassistant/home-assistant:stable image (sha256:3497f22ac2fc18728bafa6d8c7416061e842ab012bddfbf8c31b0215665ff91e)",
time="2020-10-15T15:58:59-07:00" level=info msg="Stopping /home-assistant (f74a5243b4f5d075dd238eeefbf7a65cd440ff1618690066fc4b50b316ae8267) with SIGTERM",
time="2020-10-15T15:59:26-07:00" level=debug msg="Removing container f74a5243b4f5d075dd238eeefbf7a65cd440ff1618690066fc4b50b316ae8267",
time="2020-10-15T16:00:08-07:00" level=info msg="Creating /home-assistant",
time="2020-10-15T16:04:43-07:00" level=debug msg="Starting container /home-assistant (af57d020e5ef6010b5fa6cd24b8e9297cbf321256834522dac1e9032870c6869)",
time="2020-10-15T16:04:47-07:00" level=debug msg="Scheduled next run: 2020-10-15 16:53:06 -0700 PDT"

I have also tried running with Environment Variable WATCHTOWER_CLEANUP=true though I didn't have debug on at the time. Logs show stopping and creation of new container but nothing regarding cleanup (though I'm not sure if the standard logging would show anything regarding cleanup)

time="2020-10-19T00:10:46-07:00" level=info msg="Found new linuxserver/qbittorrent:latest image (sha256:9e62a4578b009d07b66ba0c9f3fcbecfc1dc32e9eb4e3c8afc99371e53d703e2)",
time="2020-10-19T00:10:54-07:00" level=info msg="Stopping /Qbittorrent (d9da4a68c4384d4257f462f4adc220ece17d52d2681bd0cdcc5f7e69b2451a2b) with SIGTERM",
time="2020-10-19T00:11:06-07:00" level=info msg="Creating /Qbittorrent"

Other than old imagines remaining, watchtower does seem to be properly pulling updated imagines and redeploying containers as expected.

Thanks in advance!

[github-actions[bot]]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[piksel]

(This previously claimed no output would be emitted on update, but I just presumed (wrongly) client was the docker client)

Every image removed should emit an info level log entry:
https://github.com/containrrr/watchtower/blob/30b51c4b45f7f01850ca30f82a0833d7d77d2d6d/pkg/container/client.go#L289-L300

So it should be present in the log even without debug enabled.

It does seem like rollingRestart doesn't update the map though, maybe that's what is happening:
https://github.com/containrrr/watchtower/blob/30b51c4b45f7f01850ca30f82a0833d7d77d2d6d/internal/actions/update.go#L69-L82

What other ENV or arguments are you using to run watchtower?

[maxnl]

TZ=America/Los_Angeles
WATCHTOWER_POLL_INTERVAL=3600
WATCHTOWER_ROLLING_RESTART=true
WATCHTOWER_TIMEOUT=60s
WATCHTOWER_CLEANUP=true

Let me try removing WATCHTOWER_ROLLING_RESTART and see if that makes a difference

[maxnl]

Well, that appears to have done it. Running without WATCHTOWER_ROLLING_RESTART produced the expected result:

time="2020-10-21T05:46:41-07:00" level=info msg="Found new oznu/homebridge:latest image (sha256:56f615b454699553a72029d91267cf8f1a56f651522c4593101733a732955b2f)",time="2020-10-20T23:43:55-07:00" level=info msg="Starting Watchtower and scheduling first run: 2020-10-21 00:43:55 -0700 PDT m=+3601.270274164",
time="2020-10-21T05:46:55-07:00" level=info msg="Creating /Homebridge",
time="2020-10-21T05:46:41-07:00" level=info msg="Stopping /Homebridge (3421271931a7eb8befa157f0b31c51d174b54d6cb519ad57f6d1fd15fd7ec2af) with SIGTERM",
time="2020-10-21T05:48:08-07:00" level=info msg="Removing image sha256:de5f8434df6ec52ee17fa0529570c6d1d95c345c20c1c6f41d3d3f860d1cd2ac"

[piksel]

Yeah, that's what I figured. Renaming the issue!

Ref #619

[EVOTk]

hello, I have exactly the same problem!

[piksel]

@EVOTk I have created a PR with a fix, so it will probably be included in the next release. Until then, either disable the rolling update or manually do the cleanup. Sorry for the inconvenience!

[EVOTk]

@piksel
Thank you very much for your reply !
Indeed, I clean manually for the moment.
When do you think the next release will be available?

[piksel]

Almost all focus is still on #669, so I think it's probably when we think that has been sufficiently well solved.

[devbots-lock-issue[bot]]

To avoid important communication to get lost in a closed issues no one monitors, I'll go ahead and lock this issue. If you want to continue the discussion, please open a new issue. Thank you! 🙏🏼

[piksel]

This was included in v1.1.0, that was just released.