Cert-manager: Mirror to gcr.io or dockerhub

[blacs30]

Is your feature request related to a problem? Please describe.
If one image registry is down (like quay.io yesterday) images are unavailable.

Describe the solution you'd like
Have a mirror registry like gcr or docker hub

Describe alternatives you've considered
Mirroring images myself but that would be only a workaround and not so trusted by others as official images.

/kind feature

[meyskens]

/area deploy

[meyskens]

We have discussed this in our standup meeting. gcr.io is going to cost us a lot given the number of downloads we see on quay today. Docker hub seems to be a good 2nd option.

There are 2 strategies we could go here:

• change our release tooling to push to both places
• have a system take the Quay.io manifests and sync images to a mirror

We prefer the 2nd option so we can bring all old releases over as well.

[mvdan]

quay.io seems to be down again, for the sake of keeping record.

[alexellis]

Once you have a mirror on Docker Hub, could you provide a series of --set overrides or a custom values.yaml file for easy mitigation?

[meyskens]

We are looking into a few alternatives, probably also for hosting the main repo.
Currently we would love to move to gcr.io however it will cost a lot on bandwidth from our last calculations.
GitHub and Docker Hub also are options. For temporary setting up a repo we see a lot of different versions being used which makes it harder to push them up without rebuilding a year of releases.

Any other alternative suggestions to quay.io are welcome!

[mvdan]

For temporary setting up a repo we see a lot of different versions being used which makes it harder to push them up without rebuilding a year of releases.

Once quay is back up, could one not pull all the images from it, and push them to docker hub? You want to do that instead of rebuilding anyway, unless all of your versions can be built in a reproducible way (including the image itself).

[meyskens]

Our images are reproducible ;)

I do not say the following as cert-manager maintainer as i provide no guarantees https://github.com/meyskens/cert-manager/packages

[mvdan]

quay.io seems to be back up, for those following the thread. So about four hours of downtime in total.

[ttinkr]

Is there a mirror available yet?
We are trying to pull images from quay.io using kaniko and it seems quay.io only supports docker api v1, but kaniko only supports v2.

[meyskens]

@ttinkr not yet. Is weird that quay.io only supports v1 as we definitely use v2 specific features...

The idea here is to setup a tool that will mirror every x minutes/hours all images between quay.io and another registry. Which one is yet a question as Docker Hub now implements a lot of rate limits which could harm users and we did the math on GCR which will cost us (Jetstack) a lot of money (1.3 million pulls a day!) unless we have some kind of sponsorship. Not sure how the GitHub one has evolved since I last tried.

[blacs30]

I don't know if this tool would be a help for this
https://github.com/plexsystems/sinker

A tool to sync images from one container registry to another

[ttinkr]

@meyskens Alright, I will open a separate issue to maybe get help with that. Thanks!

[meyskens]

@blacs30 that looks good! we'll look into it!