Official-images: New image retention policy and status of official-images

[zakame]

As the Docker Terms of Service has been recently updated, there's also a new container image retention policy put in place. Of specific concern is that "inactive" images (i.e. images with no push/pull activity within the last 6 months) will be pruned.

Would this new policy (adversely) affect official-images as well?

[tianon]

I'm honestly not completely sure how (or if!) this new policy will affect Official Images. One major thing I don't think is clear in any of the announcements is whether these removals will be happening on a per-tag basis or a per-repository basis, and IMO the difference between those is pretty large (and would change how large I think the effect on our program potentially could be).

That being said, they do say that there will be warning notification before removals (as well as a new dashboard for viewing the "LRU" of images, for lack of a better term), so I think we've got some time and will have plenty of warning to ask them whether they can make adjustments if we think the effects on us will be heavy.

Additionally, I believe https://hub.docker.com/u/library is technically on a "paid" organization plan, so this FAQ entry should make at least that organization exempt:

How can I get unlimited retention on my container images?

Image retention limits only apply to Free individual and organization accounts. ...

However, I think the per-architecture organizations/repositories are a different story, and we'll have to watch those closely to see what changes we need to make (if any).

[zakame]

Thanks @tianon (for the response and for pinning!) Especially this point (emphasis mine:)

One major thing I don't think is clear in any of the announcements is whether these removals will be happening on a _per-tag_ basis or a _per-repository_ basis

This is _exactly_ the first thing I thought about upon hearing this announcement; I suppose we'll know more later when the pruning actually gets implemented.

[hairyhenderson]

whether these removals will be happening on a _per-tag_ basis or a _per-repository_ basis

@sudo-bmitch tweeted some great clarifications on this the other day:

https://twitter.com/sudo_bmitch/status/1293980220270280704?s=20

The image retention policy is based on images (repositories), not tags or layers. So for anyone with a lot of older versions of their image, it would be nice to cleanup some unused versions, but Docker won't do this as long as a current tag is being used.

So I think solely based on that, and regardless of whether library counts as free org or not, I would expect that most (if not all) official images are pushed/pulled much more often than every 6 months. And any that are not maybe shouldn't be official images? 😉

[zakame]

Thanks @hairyhenderson @sudo-bmitch, though to hear it first at Twitter (of all places) before on any official media/email is somewhat disquieting...

(On second reading of the Twitter thread though, it seems to be more an interpretation of the ToS and Pricing/FAQ pages; I'd rather await a more official statement from Docker Inc. if that's the case, though I appreciate taking the time to make a reasoned interpretation.)

[hairyhenderson]

it seems to be more an interpretation of the ToS and Pricing/FAQ pages; I'd rather await a more official statement from Docker Inc.

Right - Brandon isn't a Docker employee, so it's not official in that sense, however he is a Docker Captain, and as such he has _some_ access to certain information before the rest of the community.

But I do agree - it would be good to see a clarification at https://www.docker.com/pricing/retentionfaq with a precise definition of what exactly is meant by "image".

In practice, none of this is worth causing a huge fuss about, since the changes don't take effect until November.

I've logged https://github.com/docker/roadmap/issues/145 as well to ask for this officially 😉

[tianon]

It looks like https://www.docker.com/pricing/resource-consumption-updates has been updated to address this head-on:

Will Official Images be subject to the inactive image retention policy?
No. The inactive image retention policy is not applicable to Official Images. None of the images in “library” namespace will be deleted. Images that are published by Verified Publishers are also not subject to inactive image retention policy.

Does the inactive image retention policy apply to repositories, tags, or images?
The inactive image retention policy applies only to images within a repository that have not been pushed or pulled within the past 6 months, including unreferenced images or previous versions of a tag.

As an example, if the :latest tag is pulled, does it also prevent all previous versions of the tag from being deleted?
No. When the :latest tag is pulled, only the most recent version of :latest will be marked as active. The status of any previous versions of the tag will not change.

[zakame]

Thanks, that's a much better, and forceful, statement.

Also worth noting is the incoming changes to _image pull limits_: https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/ though as noted above, official-images pulls will be rate-limited according to the (anonymous/authenticated) user _pulling_, not by the _owner_ of official-images.

Let me close this as I think the basic questions have been addressed, but feel free to further discuss if needed.

[past-due]

Also worth noting is the incoming changes to _image pull limits_: https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/ though as noted above, official-images pulls will be rate-limited according to the (anonymous/authenticated) user _pulling_, not by the _owner_ of official-images.

@zakame Please see: https://www.docker.com/blog/expanded-support-for-open-source-software-projects/

To support the open source community, Docker has created a special program for Open Source projects to get continued free access and freedom from restrictions for their communities and their users. For the approved, non-commercial, open source projects, we are thrilled to announce that we will suspend data pull rate restrictions, where no egress restrictions will apply to any Docker users pulling images from the approved OSS namespaces.

Would it be possible to apply for this program for the namespaces arm64v8 (etc)? (Or maybe separate out anything that doesn't comply into other namespaces? We just hit the data pull rate restriction with arm64v8/ubuntu from CI, and it would seem reasonable for this to comply with the Open Source program, for example.)