Renovate: Research Docker Hub Rate limiting effects

[rarkins]

Background: https://www.docker.com/blog/scaling-dockers-business-to-serve-millions-more-developers-storage/

I already hit this on AWS CodeBuild for another project:

docker: error pulling image configuration: toomanyrequests: Too Many Requests. Please see https://docs.docker.com/docker-hub/download-rate-limit/.

It would seem important that any Renovate instance using binarySource=docker should authenticate first and also the impact of creating "sidecar" containers.

[viceice]

Should we check for hostrules and login before pulling our docker side-car images?

[viceice]

Maybe we need to signup for OSS project: https://forms.gle/vvKURDTYwok7Pc4r5

[viceice]

https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/

[rarkins]

I think any binarySource=docker users of Renovate (including the app) will need to be logged in with a paid account to be safe. Renovate can perhaps check for a host rule with docker.io before running any docker pull or docker run commands for the first time and log in with it if so.

[viceice]

Instead of persistant login we should use a custom credential mamager like https://gist.github.com/jasonk/480d87b49e4c8caf51932f184ff764b2 to pass auth when required 🤔

[viceice]

but we need to check, if docker cli is using the helper if no auth is required from registry

[rarkins]

Other concern: Ideally Renovate's docker identity shouldn't be running with write access to anything, or any private image access. i.e. its authentication should exist solely to overcome Docker pull rate limits for unauthenticated users.

[viceice]

the docker identity would always have write access to its own account namespace. so theroretically it can create new repos and push images and we can't suppress that.

[rarkins]

Sorry, was meaning the credentials it's running with. I hope Docker hub lets you create read only credentials.

[Chumper]

We have an internal artifactory mirror for public docker.
I would like to see a registryPrefix which I would set to my-internal-artifactory.corp.com so that the final image for the sidecars would be my-internal-artifactory.corp.com/renovate/npm.

I already took care of logging in there to run my-internal-artifactory.corp.com/renovate/renovate:slim so that would be the easiest solution for me.

[viceice]

Yeah, maybe it's time to add an option for registry prefix to customize our docker images

[viceice]

ref #6318

we can push our images to github too, as they can be pulled anonymously now.

https://github.blog/2020-09-01-introducing-github-container-registry/

[rarkins]

Docker finally followed up on our submission of https://www.docker.com/community/open-source/application

They write:

Docker remains highly committed to providing a platform where the non-commercial open source developers can continue collaborating, innovating and pushing this industry into new directions. For the approved, non-commercial, open source namespaces, Docker will suspend data pull rate restrictions, with no egress restrictions applying to any Docker users pulling images from those namespaces

There's a few marketing and attributions obligations here I'll need to double check are ok though.

[viceice]

I've got a mail this week about this, limits will be enabled on 02.11. if I remember correctly.

So we need an open source approval to get the renovate docker hub organization without pull limits?

[rarkins]

Yes, rate limiting on pulls will begin on Monday and I doubt they'll have whitelisted any repos by then, because it seems they'll require a legal agreement.

Until/unless such whitelisting happens, it's recommended that users utilizing binarySource=docker run Renovate in a container where (a) Docker is authenticated (e.g. "logged in"), and (b) the username for Docker is on a paid plan so that it gets unlimited pulls.

Still need to work out if our datasource Docker queries count as "pulls" or are unlimited.

[viceice]

I'm pretty sure they count our datasource queries. I've read somewhere they count metadata fetch too. 😕

[rarkins]

Useful article on registry mirrors: https://about.gitlab.com/blog/2020/10/30/mitigating-the-impact-of-docker-hub-pull-requests-limits/

[rarkins]

Update: the WhiteSource Renovate App - which uses binarySource=docker is now provisioned with a paying account so that it can pull its sibling containers from Docker Hub.

I have not noticed that our querying of Docker tags and SHAs as part of Renovate itself has been subject to rate limiting, although I had been concerned it might and still not 100% sure about it.