watchtower 🚀 - Docker pull limits coming Nov 2, 2020

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

github-actions[bot] on 28 Oct 2020

Hey @Codelica,

Thanks for the heads up. I'm trying to get in contact with the dockerhub team to explore our options here. From the docker blog I get the impression that even a manifest fetch will count towards the rate limit, and in that case our options are kind of limited. Hopefully, we'll find a reasonable way around this however, ,given the popularity of this project.

Best,
Simme

simskij on 29 Oct 2020

I think the solution is to just adjust the default poll interval to something much longer than the current default of 5 minutes, and maybe some documentation on what counts as a docker pull so that users can adjust accordingly based on their usage.

TheCatLady on 29 Oct 2020

Perhaps the auth should be put on the getting started page of the docs

albertoxamin on 29 Oct 2020

I've gone through the docker client code, and wrote about my findings in https://github.com/containrrr/watchtower/discussions/668#discussioncomment-116021

The TL;DR is that it already handles this and that watchtower checking for image updates shouldn't count towards the rate limit, as only the SHA digest is fetched using an HTTP request type that's excluded from the rate limit.

simskij on 29 Oct 2020

👍3

I'm just started seeing a lot of rate limit errors 15 minutes ago, so I think this is an actual issue.

Unable to update container "/opt_watchtower_1": Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.

bolmstedt on 4 Nov 2020

👍13

Same here, my log is filled with rate limit errors

albertoxamin on 4 Nov 2020

It looks like something may have changed as I was getting lots of end of file errors till this morning (something I have been too busy to troubleshoot) to now getting the rate limits when trying to check for updates

Screenshot_20201104-123348

tombomb on 4 Nov 2020

👍5

Came here to say the exact same thing, logs are filling fast, I have to disable Watchtower.

SquareBeard on 4 Nov 2020

👍10

As an intermediate fix, we can manually set the interval argument. Docker will eventually go down to a limit of 100 requests per 6 hours for anonymous users. If you run c containers, Watchtower will do c / i checks per _second_ (on average), where i is the interval you pass. Thus, we need to make sure that 100 / 6 >= 60 * 60 * c / i, or equivalently that i >= 6 * 60 * 60 * c / 100.

In short, given your number of containers c, you need to pass the interval i = 216 * c, or better something a bit above that.

eikendev on 4 Nov 2020

👍8

@eikendev our investigations into this concluded that this shouldn't be the case unless there were actual changes. Unfortunately, this was only what we gathered from reading the source and talking to the Docker developers. We haven't actually been able to test this (and it seems to happen to users randomly).

If docker actually does a manifest pull every check, then we need to find another solution...

But yeah, set the interval higher if you encounter the limit, and I guess we need to do a wireshark capture to really get to the bottom of what is going on...

piksel on 4 Nov 2020

Just FYI, according to https://www.docker.com/increase-rate-limits, we are currently in the middle of the "temporary full enforcement window," which is why the errors are surfacing now:

Current Docker Hub Usage Limit Status (updated 11/2,4:00pm Pacific)
Unauthenticated requests: | 5,000 per six hours
Free tier requests: | 5,000 per six hours
Temporary full enforcement window (100 per six hours for unauthenticated requests, 200 per six hours for free accounts): November 4, 9am-12 noon Pacific Time.

TheCatLady on 4 Nov 2020

❤1

To all of you who are reaching the rate limit: Can you please let us know the version of your docker installation? ~~I've been running watchtower with a 1s interval now for a good ten minutes without hitting the limit.~~ Just reached it 😩

simskij on 4 Nov 2020

Docker version 19.03.13, build 4484c46d9d

Edit: Just saw your edit.

aveao on 4 Nov 2020

😄3 👍1

Ok, so this is apparently not a watchtower issue (not only at least). Instructing the docker engine to check for new images incurs the rate limit when using the unix socket directly as well:

❯ curl --unix-socket /var/run/docker.sock http://docker/images/create\?fromImage\=containrrr/watchtower\&tag\=latest -X POST                                                                                                                                                                                                                                                                                                                                                                                                                                                140ms
{"message":"toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"}

I have the latest image downloaded already. Will continue to look into this. Meanwhile, please extend your intervals to avoid running into the limit.

simskij on 4 Nov 2020

👀1

Just hit this issue as well, I didn't download any docker containers myself during the full enforcement limit, besides watchtower and seemingly still hit the limit (with <40 containers running)

Ornias1993 on 4 Nov 2020

The hard rate limit @TheCatLady was mentioning has now been lifted again. The rate limit for anonymous usage right now is 2500 per 6 hours, so you should still adjust your interval accordingly to limit the risk of being rate limited.

simskij on 4 Nov 2020

The hard rate limit @TheCatLady was mentioning has now been lifted again. The rate limit for anonymous usage right now is 2500 per 6 hours, so you should still adjust your interval accordingly to limit the risk of being rate limited.

Just the fact the temporary enforcement window is gone now, doesn't mean you where right.
You where clearly wrong in your assumption that Watchtower checks wouldn't count afaik.

Ornias1993 on 4 Nov 2020

The hard rate limit @TheCatLady was mentioning has now been lifted again. The rate limit for anonymous usage right now is 2500 per 6 hours, so you should still adjust your interval accordingly to limit the risk of being rate limited.

Just the fact the temporary enforcement window is gone now, doesn't mean you where right.
You where clearly wrong in your assumption that Watchtower checks wouldn't count afaik.

Err, @simskij literally commented that you should adjust the poll interval to avoid getting rate-limited...

TheCatLady on 4 Nov 2020

Err, @simskij literally commented that you should adjust the poll interval to avoid getting rate-limited...

He also not long ago pointed towards #668 , which was actually why I hit the rate limit in the first place.
(which wasn't too clearly rescinded as far as I can tell.)

So, if I understand correctly after reading this thread twice now, what he meant previously was the documented behavior of docker (on wich we previously based the assumption that watchtower wouldn't hit the limit) doesn't seem to be what is actually currently hapening?

edit
Basically: yes I understand correctly it seems. Resolved (see https://github.com/containrrr/watchtower/issues/669#issuecomment-721969780 )

Ornias1993 on 4 Nov 2020

The hard rate limit @TheCatLady was mentioning has now been lifted again. The rate limit for anonymous usage right now is 2500 per 6 hours, so you should still adjust your interval accordingly to limit the risk of being rate limited.

Just the fact the temporary enforcement window is gone now, doesn't mean you where right.

You where clearly wrong in your assumption that Watchtower checks wouldn't count afaik.

Thanks for the constructive feedback! To my knowledge, this is still an open-source project where we do as good as we can with the limited spare time we have. Pointing fingers is not helping.

If you have the need for better SLAs, I would suggest you to find another solution.

You get the service for free, which also means we're not leaving any guarantees what so ever in terms of stability.

The assumption, as you so elegantly describe it, was based on following the code in the docker engine repo as well as discussing the issue with docker's Head of Open Source.

Given we couldn't actually do any proper testing beforehand, my view has not changed; we did the best we could with the information we had.

We will of course try to solve this as soon as possible, but until then, I'll just repeat:

Adjust your poll interval or you will likely hit the rate limit.

simskij on 4 Nov 2020

👍12 ❤8

@Ornias1993 Basically, yes. There is a way to check without getting the whole manifest and is exempted from the rate limiting. The Docker source shows that it should check that before doing the actual manifest pull. But now that the rate limit is enforced, it seems like docker doesn't just do that. We still don't fully understand why that is.

piksel on 4 Nov 2020

Maybe I was a bit harsh, I'm not expecting an SLA or guaranteed stability obviously. I just was a bit struck by the lack of saying "sorry" when someone makes a mistake. Could be a cultural thing though.

Anyhow, @piksel Thats very very odd indeed...
So the question really is "what is rate limited" atm...

Ornias1993 on 4 Nov 2020

This PR in moby/moby aims to resolve the issue as there is an issue with how the docker engine caches manifest-lists.

https://github.com/moby/moby/pull/41607

However, we might need to investigate alternative solutions for this in the meantime as it will take a while before this makes it to a patch release and then finds its way out to all docker installations running watchtower.

simskij on 4 Nov 2020

👍3 😄1

Yeah, I guess. From the API's point of view, it's clear that it's the HEAD requests that are "free", and using such a request is enough to check if a tag has changed. If docker doesn't fix this upstream we might have to go another route.

Regarding the "sorry": anyone has been free to investigate how this would pan out. This is a community project and we try to make software that helps us by collaborating. We are of course sorry for the inconvenience this causes us users, but not for "getting it wrong".

piksel on 4 Nov 2020

👍7

We are of course sorry for the inconvenience this causes us users, but not for "getting it wrong".

Sums it up beautifully. Well expressed, @piksel!

@Ornias1993:

There definitely is a bit of stress involved when this kind of project goes south given the magnitude of it's usage, as you probably can imagine. 😅

Hours of following the code back and forth, trying to understand why the docker engine is not behaving as it should, knowing everyones log files continued to grow, clearly took its toll on my temper.

I apologize for interpreting your post as hostile if that wasn't your intention.

simskij on 4 Nov 2020

I apologize for interpreting your post as hostile if that wasn't your intention.

No hard/harsh feelings from my side... :)
Lets get this back on track!

Ornias1993 on 4 Nov 2020

I was wondering, is it possible currently to adjust/set poll interval to specific images ( haven't found such option in docs ).
For example do use both: public and private registries. So I would like to pull every 30secs for my private, but in public hub case set to 5mins to not hit the limit.
In case there is no such feature at the moment could this idea receive support from the community for implementation?

deividaspetraitis on 5 Nov 2020

👍1

I was wondering, is it possible currently to adjust/set poll interval to specific images ( haven't found such option in docs ).
For example do use both: public and private registries. So I would like to pull every 30secs for my private, but in public hub case set to 5mins to not hit the limit.
In case there is no such feature at the moment could this idea receive support from the community for implementation?

I know we have a feature request about it somewhere, but currently we lack the support I'm afraid. Sorry. 😩

simskij on 5 Nov 2020

https://github.com/tianon/dockerhub-public-proxy/blob/master/dockerhub-public-proxy.pl

This can be used to cache manifests... even to something like a free cloudfare.
One of the things it does is redirect the GET request for a manifest to a HEAD, then a GET to the cacheable/immutable, content addressed manifest url.

There are other solutions as well, but ultimately converting to a HEAD when using mutable tags is needed.

https://github.com/rpardini/docker-registry-proxy is another project that just uses nginx to cache all the things and sets cache expiration for mutable URLs. Has a fallback to use stale caches when hub returns a 429 (or is just down).

cpuguy83 on 5 Nov 2020

❤1 👍1

How to put credentials for docker to have bigger limit on every watchtower run?

lukjod on 6 Nov 2020

@lukjod Just login to docker cli:
docker login -u USERNAME

Ornias1993 on 6 Nov 2020

@Ornias1993 I think you meant @lukjod :smile:

lukapeschke on 6 Nov 2020

SHOOT.
I hate the fact github sometimes completely removes certain names from the dropdown.... >.<

Ornias1993 on 6 Nov 2020

thx
should i do it just once or with creation of watchtower docker image. Sorry not too familiar with docker

lukjod on 6 Nov 2020

Watchtower uses the host docker instance, so just once on the host ;)

Ornias1993 on 6 Nov 2020

👍1

@deividaspetraitis

I was wondering, is it possible currently to adjust/set poll interval to specific images ( haven't found such option in docs ).
For example do use both: public and private registries. So I would like to pull every 30secs for my private, but in public hub case set to 5mins to not hit the limit.
In case there is no such feature at the moment could this idea receive support from the community for implementation?

As a temporary workaround, you can run multiple Watchtower instances and define a monitoring scope for each one (basically setting which containers will belong to which scope). With this setup, you will be able to configure different intervals for every scope.

victorcmoura on 7 Nov 2020

👍3

Stuff is happening in https://github.com/containrrr/watchtower/pull/674

simskij on 9 Nov 2020

👍7 🎉1

Rate limit now down to 1,000 per six hours - https://www.docker.com/increase-rate-limits

Also:

Temporary full enforcement windows (100 per six hours for unauthenticated requests,
200 per six hours for free accounts): November 9, 9am-3pm Pacific time and
November 10, 3am-9am Pacific time.

ilike2burnthing on 9 Nov 2020

Hello all,
I'm not sure I get how the newly introduced rate limits could affect an aws pipeline that uses ecr and docker(not docker hub).
Could somebody explain why this is affecting us?

omar-dulaimi on 10 Nov 2020

@omar-dulaimi

Hello all,
I'm not sure I get how the newly introduced rate limits could affect an aws pipeline that uses ecr and docker(not docker hub).
Could somebody explain why this is affecting us?

watchtower`s image itself is published on public hub, I guess that's why.

deividaspetraitis on 10 Nov 2020

@deividaspetraitis So this means watchtower is being used internally with pull and push commands since we don't explicitly mention watchtower anywhere in our scripts.
Thanks.

omar-dulaimi on 10 Nov 2020

@omar-dulaimi I think you need to explain the issue you're experiencing, (and perhaps in another ticket, since, as you say, this doesn't seem to be the same issue).

piksel on 10 Nov 2020

Weirdly I've ran docker login and got the following config in /home/josh/.docker/config.json
```.json
"auths": {
"https://index.docker.io/v1/": {
"auth": "xxx"
}
}


I've then passed this in the `docker-compose.yml` file
```yaml
services:
  watchtower_apps:
    command: watchtower archivewarrior
    container_name: watchtower_apps
    environment:
      WATCHTOWER_CLEANUP: "true"
      WATCHTOWER_DEBUG: "true"
      WATCHTOWER_ROLLING_RESTART: "true"
    image: v2tec/watchtower
    restart: unless-stopped
    volumes:
    - /home/josh/.docker/config.json:/config.json:rw
    - /var/run/docker.sock:/var/run/docker.sock:rw
version: '3.8'

But am still getting the error Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading. This seems to indicate that the request is not being made authenticated? Anyone have any ideas?

joshghent on 10 Nov 2020

@joshghent Could you open a separate discussion for this? I know it's related, but most people track this for updates on the main issue.

piksel on 10 Nov 2020

👍2

@joshghent The message text is misleading, but I believe Docker Hub returns the same error response whether or not you're authenticated.

TheCatLady on 10 Nov 2020

@joshghent Could you open a separate discussion for this? I know it's related, but most people track this for updates on the main issue.

Will do, apologies for polluting the discussion. Thank you for a great project 👍

joshghent on 11 Nov 2020

❤1

any news when this will be solved? anything we can do?

nicx on 11 Nov 2020

@nicx https://github.com/containrrr/watchtower/issues/669#issuecomment-721885754

Currently the limit is 1000 per 6 hours, so set your interval to greater than or equal to 6 * 60 * 60 * c / 1000.

As for a fix, keep an eye on - https://github.com/containrrr/watchtower/pull/674

ilike2burnthing on 11 Nov 2020

thanks @ilike2burnthing for clarification. Just to be correct, I think you have a typo in your formula: It should be "100" instead of "1000", so the correct formula is 6 * 60 * 60 * c / 100

nicx on 11 Nov 2020

That's for when the limit is set to 100 per 6 hours, currently it's 1000 - https://www.docker.com/increase-rate-limits

ilike2burnthing on 11 Nov 2020

Rate limit now down to 500 per six hours.

Also:

Temporary full enforcement windows (100 per six hours for unauthenticated requests,
200 per six hours for free accounts): November 16, 3am-3pm Pacific time.

ilike2burnthing on 12 Nov 2020

Rate limit now down to 250 per six hours.

ilike2burnthing on 17 Nov 2020

I haven't pulled any image for a week and docker still refuses me to pull an image, how could i have exceeded the pull limit rate when i haven't even pulled

1-pro on 17 Nov 2020

@1-pro

I haven't pulled any image for a week and docker still refuses me to pull an image, how could i have exceeded the pull limit rate when i haven't even pulled

Even checks count, so if you have watchtower running with a tight interval, it will still hit the limit. #674 is on its way and will change this behavior for containers managed by watchtower.

simskij on 17 Nov 2020

you can run this to see the current rate limit of your IP without losing counts

TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" 2>/dev/null | jq -r .token) ;
curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1

you will see

Docker-Content-Digest: sha256:xxxxxxx
RateLimit-Limit: 250;w=21600
RateLimit-Remaining: 242;w=21600

rafipiccolo on 17 Nov 2020

👍3

Maybe I overlooked this info somewhere, but in the cases where a user has a paid Docker Hub account but Watchtower is pulling unauthenticated, is this how you'd configure Watchtower to use credentials when pulling? If so, is this the _recommended_ way to perform this configuration?

watchtower:
    image: containrrr/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /root/.docker/config.json:/config.json
    command: --interval 30
    environment:
      REPO_USER=${usernamePipelineVar}
      REPO_PASS=${passwordPipelineVar}

In some cases maybe this could be a $7 temporary workaround for at some folks who have production issues and aren't able to wait.

trademark18 on 17 Nov 2020

@trademark18 You should be able to just login using docker login and then mount your config.json in watchtower. Setting REPO_USER and REPO_PASS has the added negative impact of using those credentials for all registries, not only dockerhub, which for a lot of users wouldn't work as they might have private images on other registries.

simskij on 17 Nov 2020

👎1 👍1

Hello, thank you for your hard work on watchtower.

The problem with the rate limits at the moment is really annoying. On one of our servers we also reached the rate-limit through watchtower. I would like to increase the interval, but this is not possible, because we also have to poll images from a private registry regularly. (CD pipeline).

As a short term solution, would it be possible to give watchtower an additional environment variable to distinguish poll-intervals between private registries and the public docker hub registry?

e.g.
INTERVAL_PRIVATE=300
INTERVAL_PUBLIC=3600

Or as variable / label on the respective container
UPDATE_INTERVALL=300

Unfortunately I don't have any experience with go, so I can't offer any real support or implement it as a MR myself.

What do you guys think about it, are there others who now have the same problem?

Pjirlip on 18 Nov 2020

@Pjirlip current workaround:

I was wondering, is it possible currently to adjust/set poll interval to specific images ( haven't found such option in docs ).
For example do use both: public and private registries. So I would like to pull every 30secs for my private, but in public hub case set to 5mins to not hit the limit.
In case there is no such feature at the moment could this idea receive support from the community for implementation?

As a temporary workaround, you can run multiple Watchtower instances and define a monitoring scope for each one (basically setting which containers will belong to which scope). With this setup, you will be able to configure different intervals for every scope.

https://github.com/containrrr/watchtower/issues/669#issuecomment-723386580

ilike2burnthing on 18 Nov 2020

👍1

@ilike2burnthing Oh okay, thanks, i guess that's better than nothing :-)

Pjirlip on 18 Nov 2020

Its a bit sad, but i realized i'm better without watchtower.
i just run a cron every morning to update everything on my dev server (it can also be started from a docker container with https://github.com/rafipiccolo/dockercron inspired from https://github.com/mcuadros/ofelia)

if like me you have only docker compose project, watchtower is easily replaced with this script or similar.

cd /folder
docker-compose pull
docker-compose up -d
# then wait for healthcheck, example :
# docker inspect ${service} --format '{{ json . }}' | jq -r .State.Status
# then if its a private project its time to deploy to production
docker-compose push
# clean old images
docker image prune -af
# deploy to prod, exemple :
# ansible all -a docker-compose pull
# ansible all -a docker-compose up -d

i did my real script with nodejs so it's a bit different, but it is now way more powerful and complete than watchtower. At least for my usage.

rafipiccolo on 18 Nov 2020

@rafipiccolo You don't really need your workaround you can still use watchtower. Just configure it to run only once per day (set interval to 24hours) for example. That way you don't run into the rate limit and still get the automatic updates and notifications provided by watchtower.

fkrauthan on 19 Nov 2020

@trademark18 You should be able to just login using docker login and then mount your config.json in watchtower. Setting REPO_USER and REPO_PASS has the added negative impact of using those credentials for all registries, not only dockerhub, which for a lot of users wouldn't work as they might have private images on other registries.

I've bought a Docker Hub Pro Account and verified, that my unlimited API via

TOKEN=$(curl --user 'username:password' "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest

As I dont get any RateLimit-Limit or RateLimit-Remaining HTTP-Headers it seems to work.
So I did docker login and my config.json got updated.
Unfortunately watchtower does not recognise the https://index.docker.io/v1/ entry when my images are just referenced as repo/image:tag which defaults to index.docker.io.
Im still getting No credentials for influxdb:alpine in /config.json and then Error pulling image influxdb:alpine, Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Could you please change the watchtower code, tthat it uses https://index.docker.io/v1/ credentials, if no repository is given?
Im maintaining hundreds of containers and changing every docker-compose.yml to use full repository path isn't a valid option.

rkorn86 on 19 Nov 2020

@rkorn86 I may be in a similar situation. I have a pro account but I'm having trouble getting watchtower to use it. How did you determine that

watchtower does not recognise the https://index/docker/io/v1/ entry when my images are just referenced as repo/image:tag which defaults to index.docker.io

And if that's the case, is it possible just to change the config.json to use index.docker.io as the key?

trademark18 on 19 Nov 2020

When you just do docker login the generated key is https://index/docker/io/v1/. In my example I have a docker compose using just influxdb:alpine as image. Starting a watchtower container results in No credentials for influxdb:alpine in /config.json which determines, that watchtower does not recognize the default docker login. Hopefully it is clearer now.

rkorn86 on 19 Nov 2020

Hmm ok. I don't get the "No credentails" error, so maybe things are a little different. On mine I just get the standard "too many requests" message even if I already did docker login on the host. I also have a private image being updated by watchtower on ECR that apparently is working ok.

trademark18 on 19 Nov 2020

I have watchtowers debug mode enabled.

rkorn86 on 19 Nov 2020

The weird part is that I get errors on watchtower even with the mounted config.json, but it works ok when I run docker-compose pull manually. Anyone can explain why is this the case?

vdo on 19 Nov 2020

@rkorn86 So I turned on debug mode and I found that in my case, it was searching for "andyault" as a key in the auths. That's part of the name of my image (it's a derivative that works with ECR...may not be necessary any more, but this was developed a few years ago). So I changed the key from "https://index.docker.io/v1/" to what it expected...."andyault" but I'm not sure why it's looking for that. I thought that docker would assume Docker Hub if another registry name wasn't specified. I don't know why it thinks "andyault" is a registry name.

Here are the files showing that it worked once I changed the key to "andyault"

docker-compose.yml

watchtower:
    container_name: watchtower
    image: andyault/watchtower-ecr:armhf-latest

config.json

{
        "auths": {
                "andyault":  {
                        "auth": "redacted"
                }
        }
}

log (with debug enabled)

time="2020-11-19T20:05:00Z" level=debug msg="Pulling andyault/watchtower-ecr:armhf-latest for /watchtower"
time="2020-11-19T20:05:00Z" level=debug msg="Loaded auth credentials {    andyault  } from /config.json"
time="2020-11-19T20:05:01Z" level=debug msg="No new images found for /watchtower"
time="2020-11-19T20:05:01Z" level=debug msg="Scheduled next run: 2020-11-19 20:10:00 +0000 UTC"

Edit:
Looks like here in the code is where watchtower tries to determine the registry address. It just splits on "/" and then returns the first one.

https://github.com/containrrr/watchtower/blob/70523465701e83531592d33f5c991825f9f1bc14/pkg/registry/trust.go#L77-L87

I don't see where this would ever find the Docker Hub credentials under the expected name (from docker login) for an image with a name including a slash like mine andyault/watchtower-ecr:armhf-latest

trademark18 on 19 Nov 2020

👍2

It would be much better to split at /, use the first part. Split the first part again at : and check if the first part is a resolveable DNS or IP address. If not, its not a registry, but a repository and index.docker.io should be set as default. If it is a valid DNS or IP, than use this registry.
Examples:

private registry: my.registry.com:5000/repo/image:tag should resolve to my.registry.com:5000
private registry: another.registry.org/repo/image:tag should resolve to another.registry.org
docker hub: repo/image:tag should resolve to https://index.docker.io/v1/ which is the default entry in config.json after doing docker login
docker hub: index.docker.io/repo/image:tag should resolve to index.docker.io, because its explicitly given as registry, so a user should know, that just docker login will not work in this case. Instead the key in config.json has to be changed.

Did I miss something?

rkorn86 on 20 Nov 2020

@simskij Are we even on the right track here? I don't know enough about watchtower and docker (or even Go, for that matter) to know if this is the real issue or not. Appreciate any thoughts you may have.

My current plan is to roll out a fix to my config.json file that implements the workaround I wrote about in my previous comment. Obviously the concern is that if this is actually a bug in watchtower that will be patched (something like @rkorn86 just suggested, perhaps) then I'll have to switch everything back, as that would break the workaround.

trademark18 on 20 Nov 2020

Prefixing image names with the full dockeresque registry name won't break later, ie index.docker.io etc. However, as previously mentioned, there will be a watchtower fix for the rate limit part which is coming through #674. This will work regardless of whether the image names are fully qualified or not (famous last words).

simskij on 20 Nov 2020

@simskij Thanks for the quick respoonse. So if I'm reading this right, the following things are true:

I can change the image name to index.docker.io/andyault/watchtower-ecr:armhf-latest and then watchtower will look for the standard docker login-produced key in the config.json file.
When #674 is rolled out, the index.docker.io prefix in the compose file won't be necessary any more, but it won't break anything.

Update:
I tried adding index.docker.io but that didn't work because docker login creates a key of https://index.docker.io/v1/ which poses a problem due to the slashes. So watchtower parses the server to be index.docker.io but leaves off the /v1/ bit. So I guess I'm back to the workaround. It seems that watchtower is currently incapable of referencing the docker login credentials by any means.

trademark18 on 20 Nov 2020

bad news(. i noticed that limit reset when i reboot my local dev server with router reboot.

pavellzubkov on 20 Nov 2020

The rate limits of 100 container image requests per six hours for anonymous usage, and 200 container image requests per six hours for free Docker accounts are now in effect.

https://www.docker.com/increase-rate-limits

ilike2burnthing on 21 Nov 2020

The rate limits of 100 container image requests per six hours for anonymous usage, and 200 container image requests per six hours for free Docker accounts are now in effect.

https://www.docker.com/increase-rate-limits

Thats not the solution here. 100 api calls is fine if it does not burn them needlessly, as each container pull requires (for some unknown reason) 20 api calls

Motophan on 22 Nov 2020

Wasn't a solution, just an update. I've been giving them as the limits have changed over the last 2 weeks (see above).

ilike2burnthing on 22 Nov 2020

❤1 👍1

@ilike2burnthing Really appreciate it. Thanks!

simskij on 22 Nov 2020

👍1

Alright, so I think we're starting to arrive at something usable over in #674. Still requires a lot of testing as well as more automated tests. I'm sure I've introduced some regression somewhere that I haven't thought of 😅

I will likely not be able to continue on this until next weekend. So in the meantime, for those of you docker savvy enough, feel free to build it yourselves and test it out on non-production workloads.

Just for the sake of clarity, I repeat:

Only test #674 on non-production workloads

😅

To try it out, clone the repo, go to the project root and run:

$ go build -o watchtower . && docker build -f ./dockerfiles/Dockerfile . -t containrrr/watchtower:my-own-tag

simskij on 22 Nov 2020

🎉3

$ go build . -o watchtower && docker build -f ./dockerfiles/Dockerfile . -t containrrr/watchtower:my-own-tag

Thank you for the command @simskij. I just tried it but I had to omit -o watchtower because with that it complains about package -o is not in GOROOT.

Also, I got standard_init_linux.go:211: exec user process caused "no such file or directory" when launching the container, should I compile watchtower binary from alpine because the system in the container doesn't have glibc system? Am I right?

unixfox on 22 Nov 2020

@unixfox Try putting the dot at the end of the build command. I updated @simskij's comment.

piksel on 22 Nov 2020

👍1

That doesn't change anything because even without -o watchtower the correct binary gets created.

Anyway using this other Dockerfile Dockerfile.self-contained and changing WATCHTOWER_VERSION to fix/avoid-rate-limits fixed my issue.

unixfox on 22 Nov 2020

I don't know where I should submit a feedback for the PR #674 so I hope it's fine if it's here.

I just tested it and it works for some images but not all of them. I tested on blacklabelops/logrotate and openresty/openresty.

On logrotate image watchtower skipped the pull but not for openresty even though my Docker daemon already have the latest version of the image. I can reproduce the same issue on other images (if you want to have the names I can submit them here).

Here are the logs of the two images:

time="2020-11-22T19:57:19Z" level=debug msg="Trying to load authentication credentials." container=/logrotate_logrotate_1 image="blacklabelops/logrotate:latest"
time="2020-11-22T19:57:19Z" level=debug msg="No credentials for blacklabelops found" config_file=/config.json
time="2020-11-22T19:57:19Z" level=debug msg="Got image name: blacklabelops/logrotate:latest"
time="2020-11-22T19:57:19Z" level=debug msg="Checking if pull is needed" container=/logrotate_logrotate_1 image="blacklabelops/logrotate:latest"
time="2020-11-22T19:57:19Z" level=debug msg="Building challenge URL" url="https://index.docker.io/v2/"
time="2020-11-22T19:57:19Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2020-11-22T19:57:19Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2020-11-22T19:57:19Z" level=debug msg="Setting scope for auth token" image=blacklabelops/logrotate scope="repository:blacklabelops/logrotate:pull"
time="2020-11-22T19:57:19Z" level=debug msg="No credentials found."
time="2020-11-22T19:57:20Z" level=debug msg="Parsing image ref" host=index.docker.io image=blacklabelops/logrotate normalized="docker.io/blacklabelops/logrotate:latest" tag=latest
time="2020-11-22T19:57:20Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/blacklabelops/logrotate/manifests/latest"
time="2020-11-22T19:57:20Z" level=debug msg="Found a remote digest to compare with" remote="sha256:5ac65cf975fa4c7909f7b483e41a4a6c3d09932dcf92ca8618cb2a8d295aef47"
time="2020-11-22T19:57:20Z" level=debug msg=Comparing local="sha256:5ac65cf975fa4c7909f7b483e41a4a6c3d09932dcf92ca8618cb2a8d295aef47" remote="sha256:5ac65cf975fa4c7909f7b483e41a4a6c3d09932dcf92ca8618cb2a8d295aef47"
time="2020-11-22T19:57:20Z" level=debug msg="Found a match"
time="2020-11-22T19:57:20Z" level=debug msg="No pull needed. Skipping image."
time="2020-11-22T19:57:20Z" level=debug msg="No new images found for /logrotate_logrotate_1"
time="2020-11-22T19:57:20Z" level=debug msg="Trying to load authentication credentials." container=/webserver_nginx_1 image="openresty/openresty:alpine"
time="2020-11-22T19:57:20Z" level=debug msg="No credentials for openresty found" config_file=/config.json
time="2020-11-22T19:57:20Z" level=debug msg="Got image name: openresty/openresty:alpine"
time="2020-11-22T19:57:20Z" level=debug msg="Checking if pull is needed" container=/webserver_nginx_1 image="openresty/openresty:alpine"
time="2020-11-22T19:57:20Z" level=debug msg="Building challenge URL" url="https://index.docker.io/v2/"
time="2020-11-22T19:57:20Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2020-11-22T19:57:20Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2020-11-22T19:57:20Z" level=debug msg="Setting scope for auth token" image=openresty/openresty scope="repository:openresty/openresty:pull"
time="2020-11-22T19:57:20Z" level=debug msg="No credentials found."
time="2020-11-22T19:57:20Z" level=debug msg="Parsing image ref" host=index.docker.io image=openresty/openresty normalized="docker.io/openresty/openresty:alpine" tag=alpine
time="2020-11-22T19:57:20Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/openresty/openresty/manifests/alpine"
time="2020-11-22T19:57:21Z" level=debug msg="Found a remote digest to compare with" remote="sha256:b1acb1c9af187e70e07d444cf5c8ad94a67155db1ae89919028b455a0ed78382"
time="2020-11-22T19:57:21Z" level=debug msg=Comparing local="sha256:7dc0de0b6f267ab75e8c42c18922205f9775182e5f43af0518144847c0e6ff7c" remote="sha256:b1acb1c9af187e70e07d444cf5c8ad94a67155db1ae89919028b455a0ed78382"
time="2020-11-22T19:57:21Z" level=debug msg="Digests did not match, doing a pull."
time="2020-11-22T19:57:21Z" level=debug msg="Pulling image" container=/webserver_nginx_1 image="openresty/openresty:alpine"
time="2020-11-22T19:57:23Z" level=debug msg="No new images found for /webserver_nginx_1"

Output (filtered) of docker images -a:

REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
openresty/openresty           alpine              2bacf20c4b61        2 days ago          104MB
blacklabelops/logrotate       latest              1255623138c3        21 months ago       45.6MB

unixfox on 22 Nov 2020

👍1

Well, sha256:b1acb1c9af187e70e07d444cf5c8ad94a67155db1ae89919028b455a0ed78382 is the latest and was created 2 days ago.

Looks like your container was not running the latest image, even if you had pulled.
Is it possible you did docker pull but did not restart the container?

@simskij is this a scenario non taken into account?

New image for a running container is pulled manually.
Container is not restarted and keeps running with outdated image.
Watchtower compares the remote sha with the running local and finds difference.
Watchtower does not check with local docker daemon and re-pulls the same image.

Of course, watchtower using Docker daemon to pull images, it is possible that did no pull:

time="2020-11-22T19:57:21Z" level=debug msg="Pulling image" container=/webserver_nginx_1 image="openresty/openresty:alpine"
time="2020-11-22T19:57:23Z" level=debug msg="No new images found for /webserver_nginx_1"

which in this case the question is if a request that counts in the rate-limit is actually executed.

zoispag on 22 Nov 2020

Looks like your container was not running the latest image, even if you had pulled.
Is it possible you did docker pull but did not restart the container?

I can assure you that my container is running the latest image. I even tried:

docker-compose stop
docker-compose rm -f
docker-compose pull   
docker-compose up -d

According to the docker daemon when doing docker inspect I get this image ID: sha256:2bacf20c4b61fda243895f4d09be017e9c898a657114c9600ab00fd9e616df5a. That's really odd.

I can replicate the same bug on another Linux server like this:

docker run -d openresty/openresty:alpine
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower:fix-pull --run-once -d

I get the same issue, watchtower still want to pull the image even though the docker daemon have the latest image locally.

unixfox on 22 Nov 2020

My 2 cents : Imageid != digest.

rafipiccolo on 22 Nov 2020

👍3

My 2 cents : Imageid != digest.

Correct. The Digest is visible under RepoDigests, or through docker images --digests.

simskij on 22 Nov 2020

You are correct in your analysis however, @unixfox.

openresty/openresty:alpine should return sha256:7dc0de0b6f267ab75e8c42c18922205f9775182e5f43af0518144847c0e6ff7c from the remote, as that is the digest of the manifest list, which is what the docker engine will store.

To try it out yourself from the terminal:

$ TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:openresty/openresty:pull" | jq -r .token) 
$ curl -v -H "Authorization: Bearer $TOKEN" -H "Accept:application/vnd.docker.distribution.manifest.list.v2+json" https://index.docker.io/v2/openresty/openresty/manifests/alpine 2>&1

I think I might have to have another look at the Accept headers, as it's apparently not allowing manifest lists (multi-arch) currently.

simskij on 23 Nov 2020

Ok, the issues reported so far should be fixed now. Merging as soon as checks have passed. If you want to help test this feature, please switch to the latest-dev image and report all related errors here.

We'll likely never be able to hit 100% successful comparisons using HEAD, so we're falling back to the previous GET method in case we don't get a clear match to make sure we're not missing anything. If, however, we find some consistent fringe case we can accommodate for in the code, we should try to work it in.

simskij on 6 Dec 2020

🎉4

I have upgraded to the latest-dev. Is this what you were referring to for the 100% succesful HEAD comparison?

time="2020-12-06T12:29:04Z" level=info msg="Could not do a head request, falling back to regulara pull."

By the way, there's a spelling mistake :P

cdemi on 6 Dec 2020

😄1

I have upgraded to the latest-dev. Is this what you were referring to for the 100% succesful HEAD comparison?
time="2020-12-06T12:29:04Z" level=info msg="Could not do a head request, falling back to regulara pull."
By the way, there's a spelling mistake :P

Yeah, that is what you'll see when watchtower, for some reason, can't do a head request against the registry in question. Mind sharing some more details on that one so I can check it out? Would you expect this to work for the container in question?

If you enable debug logs using --debug, the reason will be printed as well.

Thanks for the spellcheck, I'll make sure to fix it. 🙇

simskij on 6 Dec 2020

Just using a signed in free account:

Log

2020-12-06T15:55:11+00 [I] Starting Watchtower and scheduling first run: 2020-12-06 15:55:12 +0000 UTC m=+1.795655720
2020-12-06T16:16:07+00 [I] Could not do a head request, falling back to regulara pull.
2020-12-06T16:16:12+00 [I] Could not do a head request, falling back to regulara pull.

[... ~200 more]

2020-12-06T16:23:58+00 [I] Could not do a head request, falling back to regulara pull.
2020-12-06T16:24:00+00 [I] Could not do a head request, falling back to regulara pull.
2020-12-06T16:24:01+00 [I] Unable to update container Autoheal: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.
2020-12-06T16:24:01+00 [I] Could not do a head request, falling back to regulara pull.
2020-12-06T16:24:03+00 [I] Unable to update container Sonarr3: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.
2020-12-06T16:24:03+00 [I] Could not do a head request, falling back to regulara pull.
2020-12-06T16:24:04+00 [I] Unable to update container PortainerCE: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.
2020-12-06T16:24:05+00 [I] Could not do a head request, falling back to regulara pull.
2020-12-06T16:24:06+00 [I] Unable to update container Watchtower: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.
2020-12-06T16:24:06+00 [I] Could not do a head request, falling back to regulara pull.
2020-12-06T16:24:08+00 [I] Unable to update container Jackettest: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.
2020-12-06T16:24:08+00 [I] Could not do a head request, falling back to regulara pull.
2020-12-06T16:24:09+00 [I] Unable to update container Jackett: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.
2020-12-06T16:24:09+00 [I] Could not do a head request, falling back to regulara pull.
2020-12-06T16:24:10+00 [I] Unable to update container Tautulli: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.
2020-12-06T16:24:10+00 [I] Could not do a head request, falling back to regulara pull.
2020-12-06T16:24:12+00 [I] Unable to update container Radarr3: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.
2020-12-06T16:24:12+00 [I] Could not do a head request, falling back to regulara pull.
2020-12-06T16:24:13+00 [I] Unable to update container TransmissionVPN: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.
2020-12-06T16:24:13+00 [I] Could not do a head request, falling back to regulara pull.
2020-12-06T16:24:15+00 [I] Unable to update container Autoheal: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.
2020-12-06T16:24:15+00 [I] Could not do a head request, falling back to regulara pull.

[... repeat ad infinitum]

Restored the interval for now and will repeat with debug logs once the block is lifted.

ilike2burnthing on 6 Dec 2020

While I wait for this to give me something useful, this is the typical response:

Log

2020-12-06T17:19:40+00 [D] Trying to load authentication credentials.
2020-12-06T17:19:40+00 [D] Loaded auth credentials for user redacted on registry linuxserver/jackett:development
2020-12-06T17:19:40+00 [D] Got image name: linuxserver/jackett:development
2020-12-06T17:19:40+00 [D] Credentials loaded
2020-12-06T17:19:40+00 [D] Checking if pull is needed 
                           container: /Jackett
                           image: linuxserver/jackett:development
2020-12-06T17:19:40+00 [D] Building challenge URL
                           URL: https://index.docker.io/v2/
2020-12-06T17:19:40+00 [D] Got response to challenge request
                           header: Bearer realm="https://auth.docker.io/token",service="registry.docker.io" 
                           status: 401 Unauthorized
2020-12-06T17:19:40+00 [D] Checking challenge header content
                           realm: https://auth.docker.io/token
                           service: registry.docker.io
2020-12-06T17:19:40+00 [D] Setting scope for auth token
                           image: linuxserver/jackett
                           scope: repository:linuxserver/jackett:pull
2020-12-06T17:19:40+00 [D] Credentials found.
                           credentials: blah
2020-12-06T17:19:40+00 [D] Parsing image ref
                           host: index.docker.io
                           image: linuxserver/jackett
                           normalized: docker.io/linuxserver/jackett:development
                           tag: development
2020-12-06T17:19:40+00 [D] Doing a HEAD request to fetch a digest
                           url: https://index.docker.io/v2/linuxserver/jackett/manifests/development
2020-12-06T17:19:41+00 [D] Skipped another update already running.
2020-12-06T17:19:41+00 [D] Scheduled next run: 2020-12-06 17:19:42 +0000 UTC
2020-12-06T17:19:41+00 [D] Found a remote digest to compare with
                           remote: sha256:aebebfb1006ddd398e1c239d225d8eec1ea31cc5ecfd1b746f77c1e211202160
2020-12-06T17:19:41+00 [D] Comparing
                           local: sha256:aebebfb1006ddd398e1c239d225d8eec1ea31cc5ecfd1b746f77c1e211202160
                           remote: sha256:aebebfb1006ddd398e1c239d225d8eec1ea31cc5ecfd1b746f77c1e211202160
2020-12-06T17:19:41+00 [D] Found a match
2020-12-06T17:19:41+00 [D] No pull needed. Skipping image.
2020-12-06T17:19:41+00 [D] No new images found for /Jackett

ilike2burnthing on 6 Dec 2020

The last part of the log you posted looks correct to me. Let's wait for the limit to be lifted for some more info.

Thanks for helping out! 🙏🏼

simskij on 6 Dec 2020

👍1

Failures:

Log 1

2020-12-06T17:38:09Z [D] Trying to load authentication credentials.
                         container: /Tautulli
                         image: linuxserver/tautulli:develop
2020-12-06T17:38:09Z [D] Loaded auth credentials for user redacted on registry linuxserver/tautulli:develop
2020-12-06T17:38:09Z [D] Got image name: linuxserver/tautulli:develop
2020-12-06T17:38:09Z [D] Credentials loaded
2020-12-06T17:38:09Z [D] Checking if pull is needed
                         container: /Tautulli
                         image: linuxserver/tautulli:develop
2020-12-06T17:38:09Z [D] Building challenge URL
                         URL: https://index.docker.io/v2/
2020-12-06T17:38:09Z [I] Could not do a head request, falling back to regulara pull.
2020-12-06T17:38:09Z [D] Reason: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 1.1.1.1:53: no such host
2020-12-06T17:38:09Z [D] Pulling image
                         container: /Tautulli
                         image: linuxserver/tautulli:develop
2020-12-06T17:38:10Z [D] Skipped another update already running.
2020-12-06T17:38:10Z [D] Scheduled next run: 2020-12-06 17:38:11 +0000 UTC
2020-12-06T17:38:11Z [D] Skipped another update already running.
2020-12-06T17:38:11Z [D] Scheduled next run: 2020-12-06 17:38:12 +0000 UTC
2020-12-06T17:38:11Z [D] Error pulling image linuxserver/tautulli:develop, Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
2020-12-06T17:38:11Z [I] Unable to update container \"/Tautulli\": Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.

and:

Log 2

2020-12-06T17:38:30Z [D] Trying to load authentication credentials.
                         container: /Sonarr3
                         image: linuxserver/sonarr:preview
2020-12-06T17:38:30Z [D] Loaded auth credentials for user redacted on registry linuxserver/sonarr:preview
2020-12-06T17:38:30Z [D] Got image name: linuxserver/sonarr:preview
2020-12-06T17:38:30Z [D] Credentials loaded
2020-12-06T17:38:30Z [D] Checking if pull is needed
                         container: /Sonarr3
                         image: linuxserver/sonarr:preview
2020-12-06T17:38:30Z [D] Building challenge URL
                         URL: https://index.docker.io/v2/
2020-12-06T17:38:30Z [D] Got response to challenge request
                         header: Bearer
                         realm: \"https://auth.docker.io/token\",service=\"registry.docker.io\"
                         status: 401 Unauthorized
2020-12-06T17:38:30Z [D] Checking challenge header content
                         realm: https://auth.docker.io/token
                         service: registry.docker.io
2020-12-06T17:38:30Z [D] Setting scope for auth token
                         image: linuxserver/sonarr
                         scope: repository:linuxserver/sonarr:pull
2020-12-06T17:38:30Z [D] Credentials found.
                         credentials: blah
2020-12-06T17:38:31Z [D] Skipped another update already running.
2020-12-06T17:38:31Z [D] Scheduled next run: 2020-12-06 17:38:32 +0000 UTC
2020-12-06T17:38:31Z [D] Parsing image ref
                         host: index.docker.io
                         image: linuxserver/sonarr
                         normalized: docker.io/linuxserver/sonarr:preview
                         tag: preview
2020-12-06T17:38:31Z [D] Doing a HEAD request to fetch a digest
                         URL: https://index.docker.io/v2/linuxserver/sonarr/manifests/preview
2020-12-06T17:38:31Z [I] Could not do a head request, falling back to regulara pull.
2020-12-06T17:38:31Z [D] Reason: Head \"https://index.docker.io/v2/linuxserver/sonarr/manifests/preview\": dial tcp: lookup index.docker.io on 1.1.1.1:53: no such host
2020-12-06T17:38:31Z [D] Pulling image
                         container: /Sonarr3
                         image: linuxserver/sonarr:preview
2020-12-06T17:38:32Z [D] Skipped another update already running.
2020-12-06T17:38:32Z [D] Scheduled next run: 2020-12-06 17:38:33 +0000 UTC
2020-12-06T17:38:32Z [D] Error pulling image linuxserver/sonarr:preview, Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
2020-12-06T17:38:32Z [I] Unable to update container \"/Sonarr3\": Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.

Same issue without using Cloudflare DNS

Also eventually got 'too many open files' TCP errors, but apparently I forgot to copy that before redeploying the container with an interval of 300 (so far so good).

Thanks for the edits @piksel, hopefully I've replicated the format.

ilike2burnthing on 6 Dec 2020

I made a tool for formatting the logs to make them a bit more readable, it's not strictly a requirement to do this, but it keeps these threads more manageable. Looks good!

Regarding the DNS error reported, does that happen every time? I can't think of any reason for the DNS lookup to fail...

piksel on 6 Dec 2020

Perchance is said tool public, or is it just for personal use?

Started with no DNS issues, then after ~15mins it showed up and was maybe 1 in 4 results. That increased to roughly every other result after ~30mins, with the 'too many open files' TCP errors appearing shortly after that at a similar rate.

EDIT: there were no issues with the network at the time, was able to access the web and my NAS locally just fine.

ilike2burnthing on 6 Dec 2020

I pushed it to piksel/logformatter but's really crude and I only barely spent long enough on it to actually get it working.

Perhaps there is reason to attempt at retrying the connection when a DNS error occurs? If it's actually a real connection issue, then the GET request should fail as well, so the failover could wait...

I guess you are trying to stress it to see what happens, but it seems like you are checking for updates more often than once a minute. I wonder if there is a bug somewhere with pending sockets not being cleaned up that is causing the TCP errors...

piksel on 6 Dec 2020

Crude is where I'm most comfortable. You should see some of the game fixes I've had to make; glorified batch scripts lol I'll give it a whirl, thanks!

I removed the interval (1020s) and for whatever reason it seemed to default to 1s, wasn't sure if that was intentional, or perhaps an issue on my end. I've been running it with an interval of 300s (I think that is supposed to be the default), and I'm yet to see any failures in the last 6hrs.

I'll remove and re-add Watchtower anew, without an interval, see if it was just some problem here.

EDIT: nope, back to 1s. Removed the container and image, added Watchtower again with just --cleanup --include-stopped --debug, as well as REPO_USER and REPO_PASS, otherwise it's vanilla. So unless there's something really obvious I'm missing (which is very possible), it's not me.

ilike2burnthing on 7 Dec 2020

Yeah, I totally misread the logs (shows how much formatting them helped 😁), it wasn't trying to update every minute, it was every second. The default SHOULD be 24h in latest-dev so something is clearly wrong here. ~Could you just write the full command line you use to start watchtower to make sure we understand what is going on?~

~Actually, please open an issue for this and post it there.~

I found the issue. #700

piksel on 7 Dec 2020

👍1

Fixed the interval issue. Goland tricked me. 😂

simskij on 8 Dec 2020

😄2

Damn, thought I had it sorted. 9.5hrs with an interval of 300 and I'm back to:

Log 1

Could not do a head request, falling back to regular pull.
Reason: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 1.0.0.1:53: dial udp 1.0.0.1:53: socket: too many open files

and:

Log 2

Could not do a head request, falling back to regular pull.
Reason: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 1.1.1.1:53: no such host

(165 of the former and 5 of the latter in past hour and a half, no successful checks in that time)

Also just spied a few:

Log 3

Could not do a head request, falling back to regular pull.
Reason: Get \"https://index.docker.io/v2/\": dial tcp 52.55.43.248:443: i/o timeout
Pulling image" container=/Watchtower image="containrrr/watchtower:latest-dev"
Error pulling image containrrr/watchtower:latest-dev, Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Unable to update container \"/Watchtower\": Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Proceeding to next.

(also 52.6.170.51:443, 34.200.7.11:443, and 34.204.125.5:443)

Apologies for the lack of formatting, I'm about to head to sleep here.

I'll bump the interval again, see if that helps any. Otherwise I'll post a new issue.

ilike2burnthing on 9 Dec 2020

Damn, thought I had it sorted. 9.5hrs with an interval of 300 and I'm back to:

Log 1
and:

Log 2
(165 of the former and 5 of the latter in past hour and a half, no successful checks in that time)

Also just spied a few:

Log 3
(also 52.6.170.51:443, 34.200.7.11:443, and 34.204.125.5:443)

Apologies for the lack of formatting, I'm about to head to sleep here.

I'll bump the interval again, see if that helps any. Otherwise I'll post a new issue.

Thanks!

That's very helpful! A fix for the file descriptor leakage will arrive with #705

simskij on 9 Dec 2020

👍1

Just to confirm that 3 days later with interval set to 300 and debug logs on, no issues.

I've removed --debug now, and I'll keep an eye on it, but all seems fine. Thanks all, nice work!

Update: 8 days, all good!

ilike2burnthing on 12 Dec 2020

🎉2 👍2 🚀1 ❤1

Seems like this is quite stable now?

I'd like to give all of you an opportunity to report any errors you've experienced with latest-dev which have not been addressed. If no blocker is identified until Sunday, I'll try to cut a release on Monday.

Thanks for your patience and for assisting in ironing out the kinks of this (quite substantial) change to how watchtower works!

simskij on 17 Dec 2020

🚀4 👍4 🎉1

This should be resolved by the v1.1.0 release we've just published. As this is a quite substantial change to how watchtower works, I'd really appreciate it if you could raise new tickets on any errors you experience that seem to be related.

Thanks a bunch for all your patience as we've tried to figure this out!

simskij on 20 Dec 2020

To avoid important communication to get lost in a closed issues no one monitors, I'll go ahead and lock this issue. If you want to continue the discussion, please open a new issue. Thank you! 🙏🏼

devbots-lock-issue[bot] on 20 Dec 2020

Watchtower: Docker pull limits coming Nov 2, 2020

Most helpful comment

Adjust your poll interval or you will likely hit the rate limit.

All 108 comments

Adjust your poll interval or you will likely hit the rate limit.

Only test #674 on non-production workloads

Related issues