Watchtower: 0.3.3 appears to break email notifications

Created on 15 Apr 2019 · 14Comments · Source: containrrr/watchtower

With version 0.3.3, I no longer get email notifications.

All versions of containrrr/watchtower up to 0.3.2 send me email notifications, as desired.
v2tec/watchtower sends emails notifications, as desired

To Reproduce
This is my Dockerfile that works and sends emails (0.3.2):

docker run -d \
--name watchtower \
--hostname=azure.dockerhost001 \
-v /var/run/docker.sock:/var/run/docker.sock \
-e WATCHTOWER_NOTIFICATIONS=email \
-e [email protected] \
-e [email protected] \
-e WATCHTOWER_NOTIFICATION_EMAIL_SERVER=email-smtp.us-east-1.amazonaws.com \
-e WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER=REMOVED \
-e WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=REMOVED \
-- containrrr/watchtower:0.3.2 \
--cleanup

This is my Dockerfile that works but does not send emails (0.3.3)

Expected behavior
Email notifications should continue to be received in 0.3.3

Environment

Platform:
Azure VM

Architecture
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
Docker version:
18.06.2-ce, build 6d37f41

Logs from running watchtower with the --debug option
time="2019-04-15T10:25:46Z" level=debug msg="Retrieving running containers"
time="2019-04-15T10:25:47Z" level=debug msg="Checking if /my.minimaker.space is a watchtower instance."
time="2019-04-15T10:25:47Z" level=debug msg="Checking if /my.digitalsign.space is a watchtower instance."
time="2019-04-15T10:25:47Z" level=debug msg="Checking if /james.minimaker.space is a watchtower instance."
time="2019-04-15T10:25:47Z" level=debug msg="Checking if /mangorei.roboteam.space is a watchtower instance."
time="2019-04-15T10:25:47Z" level=debug msg="Checking if /mystics.mangoreischool.cricket is a watchtower instance."
time="2019-04-15T10:25:47Z" level=debug msg="Checking if /daniel.minimaker.space is a watchtower instance."
time="2019-04-15T10:25:47Z" level=debug msg="Checking if /www.annesmithart.com is a watchtower instance."
time="2019-04-15T10:25:47Z" level=debug msg="Checking if /www.vibehairdezine.co.nz is a watchtower instance."
time="2019-04-15T10:25:47Z" level=debug msg="Checking if /www.richardsplace.net is a watchtower instance."
time="2019-04-15T10:25:47Z" level=debug msg="Checking if /https-portal is a watchtower instance."
time="2019-04-15T10:25:47Z" level=debug msg="Checking if /mysql is a watchtower instance."
time="2019-04-15T10:25:47Z" level=debug msg="Checking if /portainer is a watchtower instance."
time="2019-04-15T10:25:47Z" level=debug msg="Starting Watchtower and scheduling first run: 2019-04-15 10:30:47 +0000 UTC m=+300.011851552"

High Awaiting user Completed Bug

Source

SmallFriendlyKiwi

All 14 comments

Thanks for your report! 🙏

I'll try to repro the error as soon as I get a chance. I'll also see if we may add some unit tests to make sure we wont get any regression in the future.

simskij on 15 Apr 2019

I'm looking into this a bit further by examining some of my other hosts- Please disregard for now

SmallFriendlyKiwi on 15 Apr 2019

👍1

Alright, let me know if the problem persists!

simskij on 15 Apr 2019

I'm happy, after further testing, that 0.3.3 appears to break email notifications for my configuration, at least. More accurate details, below:

__Tested__ 0.3.1, 0.3.2 and 0.3.3 of containrrr/watchtower
__Tested__ v2tec/watchtower

__0.3.1__: Sends emails
__0.3.2__: Sends emails
__0.3.3__: __Does not send__ emails
__v2tec__: Sends emails

__Expected behavior__
Email notifications should continue to be received in 0.3.3

__Environment__

__Platform__
Azure VM

__Architecture__
PRETTY_NAME=Debian GNU/Linux 8 (jessie)
NAME=Debian GNU/Linux
VERSION_ID=8
VERSION=8 (jessie)
ID=debian

__Docker version__
18.06.2-ce, build 6d37f41

__To Reproduce__
This is my Dockerfile that works and sends emails (0.3.1):

```docker run -d \
--name watchtower \
--hostname=azure.dockerhost031dbg \
-v /var/run/docker.sock:/var/run/docker.sock \
-e WATCHTOWER_NOTIFICATIONS=email \
-e [email protected] \
-e [email protected] \
-e WATCHTOWER_NOTIFICATION_EMAIL_SERVER=email-smtp.us-east-1.amazonaws.com \
-e WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER=REMOVED \
-e WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=REMOVED \
-- containrrr/watchtower:0.3.1 \
--cleanup \
--debug

Email Received:

Subject: Watchtower updates on azure.dockerhost031dbg
Body: 2019-04-16 07:14:34 (info): First run: 2019-04-16 07:19:34 +0000 UTC

Docker Logs:

root@dockerhost001:~/docker-watch-tower# docker logs -f watchtower
time="2019-04-16T07:14:34Z" level=debug msg="Retrieving running containers"
time="2019-04-16T07:14:34Z" level=info msg="First run: 2019-04-16 07:19:34 +0000 UTC"


This is my Dockerfile that works and sends emails (0.3.2):

```docker run -d \
  --name watchtower \
  --hostname=azure.dockerhost032dbg \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e WATCHTOWER_NOTIFICATIONS=email \
  -e [email protected] \
  -e [email protected] \
  -e WATCHTOWER_NOTIFICATION_EMAIL_SERVER=email-smtp.us-east-1.amazonaws.com \
  -e WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER=REMOVED \
  -e WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=REMOVED \
  -- containrrr/watchtower:0.3.2 \
  --cleanup \
  --debug


Email Received:

Subject: Watchtower updates on azure.dockerhost032dbg
Body: 2019-04-16 07:10:12 (info): First run: 2019-04-16 07:15:12 +0000 UTC m=+299.461395055


Docker Logs:

root@dockerhost001:~/docker-watch-tower# docker logs -f watchtower
time="2019-04-16T07:10:12Z" level=debug msg="Retrieving running containers"
time="2019-04-16T07:10:12Z" level=debug msg="Checking if /my.minimaker.space is a watchtower instance."
time="2019-04-16T07:10:12Z" level=debug msg="Checking if /my.digitalsign.space is a watchtower instance."
time="2019-04-16T07:10:12Z" level=debug msg="Checking if /james.minimaker.space is a watchtower instance."
time="2019-04-16T07:10:12Z" level=debug msg="Checking if /mangorei.roboteam.space is a watchtower instance."
time="2019-04-16T07:10:12Z" level=debug msg="Checking if /mystics.mangoreischool.cricket is a watchtower instance."
time="2019-04-16T07:10:12Z" level=debug msg="Checking if /daniel.minimaker.space is a watchtower instance."
time="2019-04-16T07:10:12Z" level=debug msg="Checking if /www.annesmithart.com is a watchtower instance."
time="2019-04-16T07:10:12Z" level=debug msg="Checking if /www.vibehairdezine.co.nz is a watchtower instance."
time="2019-04-16T07:10:12Z" level=debug msg="Checking if /www.richardsplace.net is a watchtower instance."
time="2019-04-16T07:10:12Z" level=debug msg="Checking if /https-portal is a watchtower instance."
time="2019-04-16T07:10:12Z" level=debug msg="Checking if /mysql is a watchtower instance."
time="2019-04-16T07:10:12Z" level=debug msg="Checking if /portainer is a watchtower instance."
time="2019-04-16T07:10:12Z" level=info msg="First run: 2019-04-16 07:15:12 +0000 UTC m=+299.461395055"

This is my Dockerfile that works but __appears not to send__ emails (0.3.3):

```docker run -d \
--name watchtower \
--hostname=azure.dockerhost033dbg \
-v /var/run/docker.sock:/var/run/docker.sock \
-e WATCHTOWER_NOTIFICATIONS=email \
-e [email protected] \
-e [email protected] \
-e WATCHTOWER_NOTIFICATION_EMAIL_SERVER=email-smtp.us-east-1.amazonaws.com \
-e WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER=REMOVED \
-e WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=REMOVED \
-- containrrr/watchtower:0.3.3 \
--cleanup \
--debug

No email Received:

Docker Logs:

root@dockerhost001:~/docker-watch-tower# docker logs -f watchtower
time="2019-04-16T07:12:21Z" level=debug msg="Retrieving running containers"
time="2019-04-16T07:12:21Z" level=debug msg="Checking if /my.minimaker.space is a watchtower instance."
time="2019-04-16T07:12:21Z" level=debug msg="Checking if /my.digitalsign.space is a watchtower instance."
time="2019-04-16T07:12:21Z" level=debug msg="Checking if /james.minimaker.space is a watchtower instance."
time="2019-04-16T07:12:21Z" level=debug msg="Checking if /mangorei.roboteam.space is a watchtower instance."
time="2019-04-16T07:12:21Z" level=debug msg="Checking if /mystics.mangoreischool.cricket is a watchtower instance."
time="2019-04-16T07:12:21Z" level=debug msg="Checking if /daniel.minimaker.space is a watchtower instance."
time="2019-04-16T07:12:21Z" level=debug msg="Checking if /www.annesmithart.com is a watchtower instance."
time="2019-04-16T07:12:21Z" level=debug msg="Checking if /www.vibehairdezine.co.nz is a watchtower instance."
time="2019-04-16T07:12:21Z" level=debug msg="Checking if /www.richardsplace.net is a watchtower instance."
time="2019-04-16T07:12:21Z" level=debug msg="Checking if /https-portal is a watchtower instance."
time="2019-04-16T07:12:21Z" level=debug msg="Checking if /mysql is a watchtower instance."
time="2019-04-16T07:12:21Z" level=debug msg="Checking if /portainer is a watchtower instance."
time="2019-04-16T07:12:21Z" level=debug msg="Starting Watchtower and scheduling first run: 2019-04-16 07:17:21 +0000 UTC m=+299.724866568"
```

SmallFriendlyKiwi on 16 Apr 2019

Extremely weird. I've dug through the diffs between 3.2 and 3.3 and cant really see any related changes. I'll dig a bit deeper and get back to you. Thanks again!

simskij on 16 Apr 2019

I think I can see the problem in the code - I'm just not suitably familiar with Go, git and github to do a pull request - Sorry :-(

It looks like there has been some refactoring and a line has been missed at the end.
notifier.SendNotification()

__Currently__

func runUpdatesWithNotifications(filter container.Filter) {
    notifier.StartNotification()
    updateParams := actions.UpdateParams{
        Filter:      filter,
        Cleanup:     cleanup,
        NoRestart:   noRestart,
        Timeout:     timeout,
        MonitorOnly: monitorOnly,
    }
    err := actions.Update(client, updateParams)
    if err != nil {
        log.Println(err)
    }
}

__Should be?__

func runUpdatesWithNotifications(filter container.Filter) {
    notifier.StartNotification()
    updateParams := actions.UpdateParams{
        Filter:      filter,
        Cleanup:     cleanup,
        NoRestart:   noRestart,
        Timeout:     timeout,
        MonitorOnly: monitorOnly,
    }
    err := actions.Update(client, updateParams)
    if err != nil {
        log.Println(err)
    }
        notifier.SendNotification()
}

SmallFriendlyKiwi on 16 Apr 2019

Sounds reasonable! Thanks for your great work!

simskij on 16 Apr 2019

👍1

Should be fixed with ccd746e9cc53f78f3c3ca5edb79df42fe4366855.
Would you mind building it from hash and see if it solves the problem for you?

I'm still working on getting OK test coverage for these things, and until then; things are a bit more
cumbersome. 😄

simskij on 16 Apr 2019

👍1

I'll give it a go :-) I'll do some reading and see how I go. It will be a voyage of discovery for me!

SmallFriendlyKiwi on 17 Apr 2019

I was able to make the Go executable, watchtower.

However, I'm struggling to build a docker image that works and I've been unable to resolve it - sorry :-(.

The container always exits with:

rich@ubuntu002:~/Development/watchtower/watchtower$ sudo docker logs -f watchtower
standard_init_linux.go:207: exec user process caused "no such file or directory"

More details, below, of how I approached it:

rich@ubuntu002:~/Development/watchtower/watchtower$ ll
total 15692
drwxr-xr-x 9 rich rich     4096 Apr 17 15:45 ./
drwxr-xr-x 4 rich rich     4096 Apr 17 15:39 ../
drwxr-xr-x 2 rich rich     4096 Apr 17 15:39 actions/
drwxr-xr-x 2 rich rich     4096 Apr 17 15:39 app/
drwxr-xr-x 2 rich rich     4096 Apr 17 15:39 .circleci/
-rw-r--r-- 1 rich rich     3328 Apr 17 15:39 CODE_OF_CONDUCT.md
drwxr-xr-x 3 rich rich     4096 Apr 17 15:39 container/
-rw-r--r-- 1 rich rich     1234 Apr 17 15:39 CONTRIBUTING.md
-rw-r--r-- 1 rich rich      366 Apr 17 15:39 Dockerfile
drwxr-xr-x 8 rich rich     4096 Apr 17 15:40 .git/
drwxr-xr-x 3 rich rich     4096 Apr 17 15:39 .github/
-rw-r--r-- 1 rich rich       45 Apr 17 15:39 .gitignore
-rw-r--r-- 1 rich rich     2311 Apr 17 15:39 go.mod
-rw-r--r-- 1 rich rich     1327 Apr 17 15:39 goreleaser.yml
-rw-r--r-- 1 rich rich    15143 Apr 17 15:39 go.sum
-rw-r--r-- 1 rich rich    11526 Apr 17 15:39 LICENSE.md
-rw-r--r-- 1 rich rich     4772 Apr 17 15:39 main.go
drwxr-xr-x 2 rich rich     4096 Apr 17 15:39 notifications/
-rw-r--r-- 1 rich rich    19033 Apr 17 15:39 README.md
-rwxr-xr-x 1 rich rich 15946334 Apr 17 15:41 watchtower*

sudo docker build -t sfk/watchtower:0.3.4 .

Sending build context to Docker daemon  16.61MB
Step 1/8 : FROM alpine:latest as alpine
 ---> cdf98d1859c1
Step 2/8 : RUN apk add --no-cache     ca-certificates     tzdata
 ---> Running in bcb0f0bb77ca
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/2) Installing ca-certificates (20190108-r0)
(2/2) Installing tzdata (2019a-r0)
Executing busybox-1.29.3-r10.trigger
Executing ca-certificates-20190108-r0.trigger
OK: 10 MiB in 16 packages
Removing intermediate container bcb0f0bb77ca
 ---> 807bdd1d4dec
Step 3/8 : FROM scratch
 ---> 
Step 4/8 : LABEL "com.centurylinklabs.watchtower"="true"
 ---> Running in 4afcbf54aeff
Removing intermediate container 4afcbf54aeff
 ---> 7a4fe3c22fac
Step 5/8 : COPY --from=alpine     /etc/ssl/certs/ca-certificates.crt     /etc/ssl/certs/ca-certificates.crt
 ---> 25939f135d90
Step 6/8 : COPY --from=alpine     /usr/share/zoneinfo     /usr/share/zoneinfo
 ---> 56ee46205734
Step 7/8 : COPY watchtower /
 ---> 20bab2c719d1
Step 8/8 : ENTRYPOINT ["/watchtower"]
 ---> Running in 456017a9fcd9
Removing intermediate container 456017a9fcd9
 ---> bd276bc51ff0
Successfully built bd276bc51ff0
Successfully tagged sfk/watchtower:0.3.4

rich@ubuntu002:~/Development/watchtower/watchtower$ sudo docker run -d --name watchtower -v /var/run/docker.sock:/var/run/docker.sock -- sfk/watchtower:0.3.4
0443b4551ab75cf191c7b3b45ef5cb046847ec86903e16e5f099e6bdb225ac3f
rich@ubuntu002:~/Development/watchtower/watchtower$ sudo docker ps -a
CONTAINER ID        IMAGE                  COMMAND             CREATED             STATUS                     PORTS               NAMES
0443b4551ab7        sfk/watchtower:0.3.4   "/watchtower"       6 seconds ago       Exited (1) 5 seconds ago                       watchtower

SmallFriendlyKiwi on 17 Apr 2019

I used Dockerfile to build watchtower. E-mails are working again :-)

svengo on 17 Apr 2019

Awesome @svengo!

@SmallFriendlyKiwi its crucial that you have the watchtower binary in the same folder as your building from, as well as having it built for the same arch and os as you're gonna run it on. Feel free to try out @svengo's dockerfile and let me know whether it works.

Otherwise, 0.3.4 will be released - containing the fix - sometime during this weekend.

Thanks again guys for helping out 🙏

simskij on 17 Apr 2019

👍1

Yes - works for me. Thanks for all of the help - I've learnt a lot :-)

Do you think it would be useful have a link to @svengo 's Dockerfile in the CONTRIBUTING.md file?
Also, perhaps, something more explicit in the CONTRIBUTING.md file than

go build                               # compiles and packages an executable binary, watchtower

From @svengo 's Dockerfile I was able to use the following command in my dev environment and then the Dockerfile in this repo worked fine for me.

GO111MODULE=on CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' .

SmallFriendlyKiwi on 18 Apr 2019

@SmallFriendlyKiwi Sure, feel free to open a pull request with the information you find useful!
I'll go ahead and close this issue now. Thanks again for all your collaboration guys! 🙏

simskij on 18 Apr 2019

Was this page helpful?

0 / 5 - 0 ratings