Keda: Docker Hub's new policy limiting image retention

[tomkerkhove]

Docker Hub's new policy limiting image retention is going to throttle pulls for free accounts to 100 pulls / 6h for anonymous users and 200 for authenticated users:

Less impact is that they will remove images that are inactive for 6m.

Checking with CNCF what their guidance is but we might need to consider moving our images: https://github.com/cncf/foundation/issues/106

Related: https://www.docker.com/pricing/retentionfaq

[zroubalik]

Is this something that CNCF could cover?

[tomkerkhove]

They are looking at it in cncf/foundation#106

[BretFisher]

more info on technicals of pull limits, with more blog/faq coming next week https://docs.docker.com/docker-hub/download-rate-limit/

[zroubalik]

https://github.blog/2020-09-01-introducing-github-container-registry/

[turbaszek]

https://github.blog/2020-09-01-introducing-github-container-registry/

Exactly my thought @zroubalik

[tomkerkhove]

We can move but turns out we won't be impacted by Docker Hub changes as the pull limitations are on the puller side.

However, we can move everything here if we want to. If we do so, we should have a transition period to push to both places for now.

The real question is, other than the new DH policy, what would be the benefit of migrating?

[turbaszek]

The real question is, other than the new DH policy, what would be the benefit of migrating?

I would say that the only benefit is that "everything is in one place" and maybe pushing images on CI may work faster... but I do not think that this is worth a migration.

[tomkerkhove]

For CI I fully agree, official ones I'm not sure yet since DH is still a bit the place to be.

But if they give better analysis than DH I'm all in. I'll ask around.

[zroubalik]

We can move but turns out we won't be impacted by Docker Hub changes as the pull limitations are on the puller side.

I haven't followed the changes on Docker Hub deeply, so pardon my ignorance, but could you please elaborate on this?

[tomkerkhove]

There are two major changes:

• Image pull retention
• Image pull throttling

If an image tag is not pulled for x days on free account, it will be removed. I'd have to check what the exact amount is but I think it's 60 or 90 days but you can easily automate it to keep them alive.

Pull throttling will avoid people from pulling too fast. Initially the thought was that it would be based on total pull count on an image tag, but instead it's on the client side. So if I'm logged in to DH they will throttle based on that. If it's anonymous pulls they will throttle based on IP. This means that if one user pulls a lot, they will be throttled but not our other users which was my main concern.

[zroubalik]

Thanks for the explanation!

Pull throttling will avoid people from pulling too fast. Initially the thought was that it would be based on total pull count on an image tag...

That was my (false) understanding, got it now.

[zroubalik]

So agree, that we don't have to change the things that we have. We can migrate later if it is worth or needed.

[BretFisher]

Quick numbers on changes starting November 1st:

Retention Limits: If an image repository on a free account hasn't had a single pull of any tag in 6 months, the repository is deleted.

Pull Limits: Tracked by IP, Hub will limit pulls to 100 in the last 6 hours. 200 if the pull is authenticated.

If either of these are using a paid account (starts at $5/mo), limits are removed (unlimited retention and pulls).

[jaythamke]

Quick numbers on changes starting November 1st:

Retention Limits: If an image repository on a free account hasn't had a single pull of any tag in 6 months, the repository is deleted.

Pull Limits: Tracked by IP, Hub will limit pulls to 100 in the last 6 hours. 200 if the pull is authenticated.

If either of these are using a paid account (starts at $5/mo), limits are removed (unlimited retention and pulls).

Hi @BretFisher ,
Any idea - how to measure the number images pulled by a non paid docker account? Thanks
see - SO post

[BretFisher]

I don't believe Docker Hub provides pull stats to users beyond the "total pulls" estimate shown in the Web UI.

[jaythamke]

I don't believe Docker Hub provides pull stats to users beyond the "total pulls" estimate shown in the Web UI.

Yes, I have also checked the docker hub api spec but did not find any related to my specific need. I thought I might have missed something, hence asked. Thanks for reply.

[tomkerkhove]

Nope, they don't provide it indeed, however since the throttling is on the client side based on IP we are good to go for now.

[tomkerkhove]

One of the maintainers will have to fill in https://www.docker.com/community/open-source/application; happy to pick this up if need be @zroubalik @jeffhollan

[zroubalik]

@tomkerkhove go ahead if you can do it, please.

[tomkerkhove]

✔