Okd: [vSphere] Mirroring of images with ImageContentSourcePolicy not working ?

[jomeier]

Hi,

I try to mirror an image from quay to our inhouse private registry: https://docs.openshift.com/container-platform/4.4/openshift_images/image-configuration.html

original: quay.io/openshift-pipeline/openshift-pipelines-operator:0.10.7
mirrored: docker.rsint.net/rs_ruscloud-dit-public/okd4-operatorhub/operators/openshift-pipeline-openshift-pipelines-operator:0.10.7

I manually pulled the original image, tagged it to our inhouse mirror registry and pushed it there (no imagepullsecret necessary!).

I found the resource ImageContentSourcePolicy that seems to assist in mirroring images. I created this one here:

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  creationTimestamp: '2020-06-23T06:57:51Z'
  generation: 7
  managedFields:
    - apiVersion: operator.openshift.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
        'f:spec': {}
      manager: oc
      operation: Update
      time: '2020-06-23T06:57:51Z'
    - apiVersion: operator.openshift.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          'f:repositoryDigestMirrors': {}
      manager: Mozilla
      operation: Update
      time: '2020-06-24T06:54:35Z'
  name: okd4-operatorhub-catalog-sources
spec:
  repositoryDigestMirrors:
    - mirrors:
        - >-
          docker.rsint.net/rs_ruscloud-dit-public/okd4-operatorhub/operators/openshift-pipeline-openshift-pipelines-operator
      source: quay.io/openshift-pipeline/openshift-pipelines-operator

After applying this resource to the cluster I saw that all nodes got updated for a short period of time. This is expected behaviour.

I sshed in one of the worker VMs and checked the file /etc/containers/registries.conf:

[core@chmu2w1 ~]$ sudo cat /etc/containers/registries.conf
unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]

[[registry]]
  prefix = ""
  location = "quay.io/openshift-pipeline/openshift-pipelines-operator"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "docker.rsint.net/rs_ruscloud-dit-public/okd4-operatorhub/operators/openshift-pipeline-openshift-pipelines-operator"

[[registry]]
  prefix = ""
  location = "quay.io/openshift/okd"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "docker.rsint.net/rs_ruscloud-dit-public/okd4/version-4.4.0-0.okd-2020-04-21-163702-beta4"

[[registry]]
  prefix = ""
  location = "quay.io/openshift/okd-content"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "docker.rsint.net/rs_ruscloud-dit-public/okd4/version-4.4.0-0.okd-2020-04-21-163702-beta4"

[[registry]]
  prefix = ""
  location = "registry.svc.ci.openshift.org/origin/4.5-2020-06-18-025306"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "docker.rsint.net/rs_ruscloud-dit-public/okd4/version-4.5.0-0.okd-2020-06-18-024306"

[[registry]]
  prefix = ""
  location = "registry.svc.ci.openshift.org/origin/release"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "docker.rsint.net/rs_ruscloud-dit-public/okd4/version-4.5.0-0.okd-2020-06-18-024306"

This also looks good.

I checked if I can pull the image with podman on that worker:

[core@chmu2w1 ~]$ sudo podman pull --log-level debug quay.io/openshift-pipeline/openshift-pipelines-operator:0.10.7
DEBU[0000] Found deprecated file /usr/share/containers/libpod.conf, please remove. Use /etc/containers/containers.conf to override defaults.
DEBU[0000] Reading configuration file "/usr/share/containers/libpod.conf"
DEBU[0000] Ignoring lipod.conf EventsLogger setting "journald". Use containers.conf if you want to change this setting and remove libpod.conf files.
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{{[] [] container-default [] host [CAP_AUDIT_WRITE CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_FSETID CAP_KILL CAP_MKNOD CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETFCAP CAP_SETGID CAP_SETPCAP CAP_SETUID CAP_SYS_CHROOT] [] [nproc=4194304:4194304]  [] [] [] true [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] false false false  private k8s-file -1 bridge false 2048 private /usr/share/containers/seccomp.json 65536k private host 65536} {false systemd [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] ctrl-p,ctrl-q true /var/run/libpod/events/events.log file [/usr/share/containers/oci/hooks.d] docker:// /pause k8s.gcr.io/pause:3.2 /usr/libexec/podman/catatonit shm   false 2048 crun map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] kata-fc:[/usr/bin/kata-fc] kata-qemu:[/usr/bin/kata-qemu] kata-runtime:[/usr/bin/kata-runtime] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] missing [] [crun runc] [crun] {false false false true true true}  false 3 /var/lib/containers/storage/libpod 10 /var/run/libpod /var/lib/containers/storage/volumes} {[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman /etc/cni/net.d/}}
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /var/run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] cached value indicated that metacopy is not being used
DEBU[0000] NewControl(/var/lib/containers/storage/overlay): nextProjectID = 2
DEBU[0000] cached value indicated that native-diff is usable
DEBU[0000] backingFs=xfs, projectQuotaSupported=true, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] using runtime "/usr/bin/crun"
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument
WARN[0000] Error initializing configured OCI runtime kata-runtime: no valid executable found for OCI runtime kata-runtime: invalid argument
WARN[0000] Error initializing configured OCI runtime kata-qemu: no valid executable found for OCI runtime kata-qemu: invalid argument
WARN[0000] Error initializing configured OCI runtime kata-fc: no valid executable found for OCI runtime kata-fc: invalid argument
DEBU[0000] using runtime "/usr/bin/runc"
INFO[0000] Found CNI network crio (type=bridge) at /etc/cni/net.d/100-crio-bridge.conf
INFO[0000] Found CNI network 200-loopback.conf (type=loopback) at /etc/cni/net.d/200-loopback.conf
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist
WARN[0000] Default CNI network name podman is unchangeable
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/openshift-pipeline/openshift-pipelines-operator:0.10.7"
Trying to pull quay.io/openshift-pipeline/openshift-pipelines-operator:0.10.7...
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Trying to access "quay.io/openshift-pipeline/openshift-pipelines-operator:0.10.7"
DEBU[0000] Credentials not found
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000]  Using "default-docker" configuration
DEBU[0000]  No signature storage configuration found for quay.io/openshift-pipeline/openshift-pipelines-operator:0.10.7
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io
DEBU[0000] GET https://quay.io/v2/
DEBU[0030] Ping https://quay.io/v2/ err Get "https://quay.io/v2/": dial tcp 52.202.225.67:443: i/o timeout (&url.Error{Op:"Get", URL:"https://quay.io/v2/", Err:(*net.OpError)(0xc0002b0730)})
DEBU[0030] GET https://quay.io/v1/_ping
DEBU[0060] Ping https://quay.io/v1/_ping err Get "https://quay.io/v1/_ping": dial tcp 18.205.55.240:443: i/o timeout (&url.Error{Op:"Get", URL:"https://quay.io/v1/_ping", Err:(*net.OpError)(0xc0001507d0)})
DEBU[0060] Accessing "quay.io/openshift-pipeline/openshift-pipelines-operator:0.10.7" failed: error pinging docker registry quay.io: Get "https://quay.io/v2/": dial tcp 52.202.225.67:443: i/o timeout
DEBU[0060] Error pulling image ref //quay.io/openshift-pipeline/openshift-pipelines-operator:0.10.7: Error initializing source docker://quay.io/openshift-pipeline/openshift-pipelines-operator:0.10.7: error pinging docker registry quay.io: Get "https://quay.io/v2/": dial tcp 52.202.225.67:443: i/o timeout
  Get "https://quay.io/v2/": dial tcp 52.202.225.67:443: i/o timeout
ERRO[0060] error pulling image "quay.io/openshift-pipeline/openshift-pipelines-operator:0.10.7": unable to pull quay.io/openshift-pipeline/openshift-pipelines-operator:0.10.7: unable to pull image: Error initializing source docker://quay.io/openshift-pipeline/openshift-pipelines-operator:0.10.7: error pinging docker registry quay.io: Get "https://quay.io/v2/": dial tcp 52.202.225.67:443: i/o timeout

podman does not even try to pull the mirrored image from the mirror I defined in the ImageContentSourcePolicy.

What's wrong here?

Thanks and greetings,

Josef

Version: 4.5.0-0.okd-2020-06-18-025306

[vrutkovs]

You're pulling by tag:

quay.io/openshift-pipeline/openshift-pipelines-operator:0.10.7

but registries.conf has mirror-by-digest-only = true true.

0.10.7 (currently) resolves to sha256:ea29ea69ec12ad9396d9856569a819033a6166bf017a23eed5c6dc92549a8396, try pulling it by digest?

[jomeier]

Pulling by digest works.

But the real problem is that if I want to install a 3rd party application that uses images from the internet, that I have to change the manifest of the application (change image source from tag to digest).

My hope was, that the ImageContentSourcePolicy acts as a "symlink" to my internal mirror registry so I don't must change the original manifests.

This article https://access.redhat.com/solutions/4817401 says that in OCP 4.2/4.3 there is no support for tags which is a pity.

[jomeier]

If I set "mirror-by-digest-only" to false in /etc/containers/registries.conf I can pull the image with a tag instead of a digest !!!

Cool!

But is it possible to set this flag somewhere without editing /etc/containers/registries.conf manually?

[vrutkovs]

But is it possible to set this flag somewhere without editing /etc/containers/registries.conf manually?

I don't think so. Tags mirroring support is tricky from the security point of view, lets file an RFE for that and see if that gets accepted.

Current workaround would be overlaying a new registries.conf with a custom MachineConfig which has this setting enabled for particular repo

[jomeier]

Custom MachineConfig? This sounds interesting!

How can I create a MachineConfig that is applied automatically to all masters and workers?

I assume I can set own registry.conf files in /etc/containers/registry.conf.d directory. Or how would you do that?

[vrutkovs]

See https://github.com/openshift/machine-config-operator/blob/master/docs/MachineConfiguration.md

Not entirely sure if /etc/containers/registry.conf.d is supported in podman/crio

[ashaver]

I made exactly this change awhile back to allow mirroring by tag. I made two different machine configs 99-master and 99-worker (because each role needs a separate config). Each one changed the /etc/registries/registry.conf. It worked well, until it didn't.

If the ImageContentSourcePolicy was modifed (by someone changing the ImageContentSources), then that would alter the /etc/registries/registry.conf on each node's disk. That, in turn, would mark the node as degraded by the machine-config-operator. Then, the node would never be upgraded and rebooted.

So it worked in the short term. In the long term, it had the practical effect of permanently stopping upgrades and changes across the whole cluster.

[jomeier]

@ashaver:
Yes I know that this would break this workaround. We would deactivate ImageContentSource through RBAC.

We use ArgoCD to apply changes in the cluster so we do with the MachineConfig resource.

I hope this will work also on the long term. I'll prepare an enhancement to support tags in ImageContentSourcePolicy CR
nevertheless.

[vrutkovs]

@jomeier I believe this can be closed? ImageContentSource would always generate manifest-only mirror config, so one must manually intervene to mirror by tag name.

Would you mind creating an FAQ entry for this?

[kuzm1ch]

I also think that user should able to choose between mirror-by-digest-only = true or false in the ImageContentSource.

My use case is the same. There is a mirror in private registry for docker.io and cluster has access only to them. Change image sources for each 3rd party application is a kind of wasting time when is the same time you have a private mirror for them.

Also, this documentation is putting users into the wrong direction. There are no words that it works only for the digest.

https://docs.openshift.com/container-platform/4.2/openshift_images/image-configuration.html
As the last step, it asked you to pull image with tag latest to verify setup which always leads you to receive an expected error.

Pull an image to the node from the source and check if it is actually resolved by the mirror.
sh-4.2# podman pull --log-level=debug registry.access.redhat.com/ubi8/ubi-minimal