Watchtower: Synology error: Invalid Character 'p'

[xbmcnut]

In the last 10 days or so, without any Docker mods from me, I've started to get constant errors emailed through and I don't know where to start. Could be a Synology update that has killed things?

date,stream,content
2018-11-24 11:10:23,stderr,time="2018-11-24T11:10:23Z" level=info msg="Waiting for running update to be finished..." 

2018-11-24 10:46:49,stderr,"time=\"2018-11-24T10:46:49Z\" level=info msg=\"Unable to update container /watchtower, err='Error response from daemon: unauthorized: authentication required'. Proceeding to next.\" 
"
2018-11-24 10:21:44,stderr,"time=\"2018-11-24T10:21:44Z\" level=info msg=\"Unable to update container /unifi, err='Error response from daemon: error parsing HTTP 404 response body: invalid character 'p' after top-level value: \"404 page not found\n\"'. Proceeding to next.\" 
"
2018-11-24 10:06:45,stderr,"time=\"2018-11-24T10:06:45Z\" level=info msg=\"Unable to update container /tasmoadmin, err='Error response from daemon: unauthorized: authentication required'. Proceeding to next.\" 
"
2018-11-24 09:11:50,stderr,"time=\"2018-11-24T09:11:50Z\" level=info msg=\"Unable to update container /watchtower, err='Error response from daemon: error parsing HTTP 404 response body: invalid character 'p' after top-level value: \"404 page not found\n\"'. Proceeding to next.\" 
"
2018-11-24 09:11:41,stderr,"time=\"2018-11-24T09:11:41Z\" level=info msg=\"Unable to update container /mqtt, err='Error response from daemon: error parsing HTTP 404 response body: invalid character 'p' after top-level value: \"404 page not found\n\"'. Proceeding to next.\" 

2018-11-24 08:21:43,stderr,"time=\"2018-11-24T08:21:43Z\" level=info msg=\"Unable to update container /unifi, err='Error response from daemon: error parsing HTTP 404 response body: invalid character 'p' after top-level value: \"404 page not found\n\"'. Proceeding to next.\" 

Any clues?

[xbmcnut]

Here is my Docker file that's been running sweet for the last two months.

docker run -d \
  --name watchtower \
  --restart=always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e WATCHTOWER_NOTIFICATIONS=email \
  -e [email protected] \
  -e [email protected] \
  -e WATCHTOWER_NOTIFICATION_EMAIL_SERVER=smtp.gmail.com \
  -e [email protected] \
  -e WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=xxxxxxxxxx \
  v2tec/watchtower --cleanup

[zenofile]

Same here, not only on a Synology NAS but on Archlinux aswell:

Docker version 18.09.0-ce, build 4d60db472b

time="2018-11-23T23:00:04+01:00" level=info msg="Unable to update container /watchtower, err='Error response from daemon: error parsing HTTP 404 response body: invalid character 'p' after top-level value: "404 page not found\n"'. Proceeding to next." 
time="2018-11-24T02:00:03+01:00" level=info msg="Unable to update container /watchtower, err='Error response from daemon: unauthorized: authentication required'. Proceeding to next." 
time="2018-11-24T03:00:02+01:00" level=info msg="Unable to update container /adminer, err='Error response from daemon: unauthorized: authentication required'. Proceeding to next." 
time="2018-11-24T05:00:01+01:00" level=info msg="Unable to update container /resilio, err='Error response from daemon: error parsing HTTP 404 response body: invalid character 'p' after top-level value: "404 page not found\n"'. Proceeding to next." 
time="2018-11-24T06:00:01+01:00" level=info msg="Unable to update container /resilio, err='Error response from daemon: error parsing HTTP 404 response body: invalid character 'p' after top-level value: "404 page not found\n"'. Proceeding to next." 
time="2018-11-24T07:00:02+01:00" level=info msg="Unable to update container /adminer, err='Error response from daemon: error parsing HTTP 404 response body: invalid character 'p' after top-level value: "404 page not found\n"'. Proceeding to next." 
time="2018-11-24T10:00:01+01:00" level=info msg="Unable to update container /resilio, err='Error response from daemon: error parsing HTTP 404 response body: invalid character 'p' after top-level value: "404 page not found\n"'. Proceeding to next." 
time="2018-11-24T11:00:03+01:00" level=info msg="Unable to update container /watchtower, err='Error response from daemon: unauthorized: authentication required'. Proceeding to next." 
time="2018-11-24T15:00:02+01:00" level=info msg="Unable to update container /adminer, err='Error response from daemon: error parsing HTTP 404 response body: invalid character 'p' after top-level value: "404 page not found\n"'. Proceeding to next." 
time="2018-11-24T15:00:03+01:00" level=info msg="Unable to update container /watchtower, err='Error response from daemon: unauthorized: authentication required'. Proceeding to next." 
time="2018-11-24T23:00:02+01:00" level=info msg="Unable to update container /adminer, err='Error response from daemon: unauthorized: authentication required'. Proceeding to next."

The watchtower container is started through docker-compose.

docker-compose.yml:

version: '3'
services:
  watchtower:
    container_name: watchtower
    image: v2tec/watchtower
    userns_mode: "host"
    restart: unless-stopped
    environment:
      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER=xxx
      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=xxx
      - WATCHTOWER_NOTIFICATIONS=email
      - WATCHTOWER_NOTIFICATION_EMAIL_FROM=xxx
      - WATCHTOWER_NOTIFICATION_EMAIL_TO=xxx
      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER=smtp.eu.mailgun.org
      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT=587
    labels:
      - com.centurylinklabs.watchtower.enable=true
    command: --label-enable --cleanup --schedule @hourly
    network_mode: bridge
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

[iyerusad]

Seeing same from similar WatchTower docker set up.

Shutting off container in meantime - possibly retrying efforts may have contributed to blowing through a bandwidth cap.

[circa10a]

Did you add auth to your registry and not update the watchtower config?

[samara8609]

seeing the same issue. Nothing in the setup has changed.

section of my compose for watchtower:

#Watchtower
watchtower:
container_name: watchtower
image: v2tec/watchtower
restart: unless-stopped
volumes:
- /var/run/docker.sock:/var/run/docker.sock
command: --cleanup
networks:
- bridgeRestrictIP

[Ezwen]

Same problem here on Debian, possibly after updating docker itself. Maybe a change was made in the default configuration of docker? Or maybe does the default docker registry now requires authentication...?

Also I am not using any private registry.

[a10kiloham]

Same here on Ubuntu 18

[zenofile]

I tried a 10 second interval with debug mode enabled and the error shows only sporadically this way, about every five minutes, if at all.

level=debug msg="Error pulling image adminer:fastcgi, Error response from daemon: unauthorized: authentication required"
level=info msg="Unable to update container /adminer, err='Error response from daemon: unauthorized: authentication required'. Proceeding to next."

The fact that the rest of the requests are fine does rather not indicate some kind of rate limiting I suppose.
Maybe a specific registry server is having issues and only gets hit every now and then?

Edit: This also happens when specifying REPO_USER and REPO_PASS for the public registry, the error message also alternates between authentication and parsing error.

[waja]

This issue started on 23rd November on several systems here (Debian 9 systems). No docker, docker-compose or watchtower updates was deployed since then. It seems not to happen not every time and I would bet it's related to issues at docker hub.

[circa10a]

https://github.com/docker/hub-feedback/issues/1693

[Aerya]

Same in here...

Docker version 18.06.1-ce, build e68fc7a
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"

[mariotacke]

Same here, probably related to https://github.com/docker/hub-feedback/issues/1693 as @circa10a already mentioned.

[nightah]

This issue has been resolved as far as I can tell, haven't had it occur for several days.

Probably best to close this and track the upstream issue for a post-mortem.

[xbmcnut]

Yep, seems fixed. No more errors since Docker Hub made some tweaks.