Crc: Importing insecure registries in CRC

[debu66er]

General information

• OS: macOS
• Hypervisor: hyperkit
• Did you run crc setup before starting it (Yes/No)? Yes

CRC version

crc version: 1.1.0+95966a9
OpenShift version: 4.2.2 (embedded in binary)

CRC status

CRC VM: Running
OpenShift: Stopped
Disk Usage: 16.53GB of 32.2GB (Inside the CRC VM)
Cache Usage: 21.25GB
Cache Directory: /Users/chemi/.crc/cache

CRC config

Nothing is returned

Host Operating System

ProductName: Mac OS X
ProductVersion: 10.15.1
BuildVersion: 19B88

Steps to reproduce

1. Expose the internal Image Registry: https://docs.openshift.com/container-platform/4.2/registry/securing-exposing-registry.html
2. Configure that external route as a insecure registry: https://docs.openshift.com/container-platform/4.2/openshift_images/image-configuration.html#images-configuration-insecure_image-configuration
3. Push an image to that internal registry
4. Deploy something using that pushed image (which is tagged with the external route, not the internal)

Expected

A pod should be able to pull the image and run.

Actual

When trying to pull the image from the image registry using the external route it fails with: Failed to pull image "default-route-openshift-image-registry.apps-crc.testing/default/testing": rpc error: code = Unknown desc = pinging docker registry returned: Get https://default-route-openshift-image-registry.apps-crc.testing/v2/: x509: certificate signed by unknown authority

Executing: oc edit image.config.openshift.io/cluster

apiVersion: config.openshift.io/v1
kind: Image
metadata:
  annotations:
    release.openshift.io/create-only: "true"
  creationTimestamp: "2019-10-30T08:32:57Z"
  generation: 2
  name: cluster
  resourceVersion: "282096"
  selfLink: /apis/config.openshift.io/v1/images/cluster
  uid: e001691f-faef-11e9-9c4b-5254005a2edf
spec:
  registrySources:
    insecureRegistries:
    - default-route-openshift-image-registry.apps-crc.testing
status:
  externalRegistryHostnames:
  - default-route-openshift-image-registry.apps-crc.testing
  internalRegistryHostname: image-registry.openshift-image-registry.svc:5000

You can see the insecureRegistries section.
I restarted CRC, and nothing changed.

So question is: Do we support insecure registries in CRC? If answer is yes, am I doing anything wrong?

Thanks!

[debu66er]

Investigating further this issue, I tried to get into the CRC VM to check if /host/etc/containers/registries.conf file existed and it was updated correctly.

I got inside the VM executing: sudo screen /Users/chemi/.crc/machines/crc/tty but a login is requested. What usr/pwd can I use? Thanks!

[debu66er]

Some more progress.
Reviewing closed issues I found out how to get into the VM with: oc debug node/crc-XXX

And checking the content of the documented file: sh-4.2# cat /host/etc/containers/registries.conf I get: unqualified-search-registries = ['registry.access.redhat.com', 'docker.io'] which obviously is not what I expected from what I read in OCP 4.2 docs: https://docs.openshift.com/container-platform/4.2/openshift_images/image-configuration.html#images-configuration-insecure_image-configuration

So it seems a bug or a different behavior of CRC, right?

[praveenkumar]

@debu66er I would do something following if I want to consume the images which I pushed to internal openshift registry.

$ oc new-project demo

$ docker pull busybox

$ docker tag busybox default-route-openshift-image-registry.apps-crc.testing/demo/mytestimage

$ docker login -u <developer/kubeadmin> -p $(oc whoami -t) default-route-openshift-image-registry.apps-crc.testing

$ docker push default-route-openshift-image-registry.apps-crc.testing/demo/mytestimage

$ oc get is
NAME          IMAGE REPOSITORY                                                           TAGS     UPDATED
mytestimage   default-route-openshift-image-registry.apps-crc.testing/demo/mytestimage   latest   About an hour ago

$ oc set image-lookup  --all

$ cat test.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: test
  name: test
spec:
  containers:
  - image: mytestimage
    name: test
    command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600']

$ oc create -f test.yaml

$ oc get events
LAST SEEN   TYPE     REASON      OBJECT     MESSAGE
6m4s        Normal   Scheduled   pod/test   Successfully assigned demo/test to crc-shdl4-master-0
5m56s       Normal   Pulling     pod/test   Pulling image "image-registry.openshift-image-registry.svc:5000/demo/mytestimage@sha256:679b1c1058c1f2dc59a3ee70eed986a88811c0205c8ceea57cec5f22d2c3fbb1"
5m56s       Normal   Pulled      pod/test   Successfully pulled image "image-registry.openshift-image-registry.svc:5000/demo/mytestimage@sha256:679b1c1058c1f2dc59a3ee70eed986a88811c0205c8ceea57cec5f22d2c3fbb1"
5m56s       Normal   Created     pod/test   Created container test
5m56s       Normal   Started     pod/test   Started container test

$ oc get pods
NAME   READY   STATUS    RESTARTS   AGE
test   1/1     Running   0          6m14s

Adding exposed internal registry to insecure doesn't make sense but if you really have some external insecure registry from which you want to pull images from then this is not out of the box supported in CRC, we are creating a doc for same and share soon.

[debu66er]

Thanks @praveenkumar!
Fully agree with your steps but I have a specific use case around Kabanero project. To be more precise Codewind and Appsody. Basically, when I execute: "appsody deploy --tag XXX --push" it will do following things:
1.- Create the docker image.
2.- Create a deployment yaml for Appsody operator (using the exposed URL of the registry plus the image name).
3.- Push the image to the CRC image registry.
4.- Create the deployment into CRC.
And there is where I have the issue as the final deployment has the exposed hostname as reference in the image name.
That's why I needed the access to that exposed and insecure endpoint.
Of course I can do the steps manually, and fix naming so everything works. But that breaks the Kabanero development environment I am trying to demonstrate.

BTW, in your sample, if the image name has not the full hostname doesn't go to Docker Hub? In my tests, it tries to go to Docker Hub instead of the internal image registry. How did it work that in your case?

[praveenkumar]

BTW, in your sample, if the image name has not the full hostname doesn't go to Docker Hub? In my tests, it tries to go to Docker Hub instead of the internal image registry. How did it work that in your case?

@debu66er You need to set oc set image-lookup --all for your imagestream to not query to docker hub I am hoping all the images are pushed to single namespace otherwise you need to do it for each namespace oc set image-lookup --all -n <namespace> now it will first look locally before go to upstream repo.

[debu66er]

Thanks for that hint. It could be a workaround but for some reason is not working...
I push the image:
Pushing image default-route-openshift-image-registry.apps-crc.testing/default/testing

I check the imagestream:

oc get is                 
NAME      IMAGE REPOSITORY                                                          TAGS     UPDATED
testing   default-route-openshift-image-registry.apps-crc.testing/default/testing   latest   47 seconds ago

I set the image lookup:

oc set image-lookup  --all
imagestream.image.openshift.io/testing image lookup updated

But when I deploy the deployment the pod shows this error:
Failed to pull image "testing": rpc error: code = Unknown desc = Error reading manifest latest in docker.io/library/testing: errors: denied: requested access to the resource is denied unauthorized: authentication required

Did you tested on CRC? Or in a standard OCP? Thanks!

[praveenkumar]

Did you tested on CRC? Or in a standard OCP? Thanks!

I tested this with latest CRC not with standard OCP, are you sure your deployment doesn't have a different namespace defined there? Try with a single pod like I did https://github.com/code-ready/crc/issues/821#issuecomment-555862714 here and also check the namespace in your deployment since you are putting the image in default namespace so I am hoping your deployment also should have namespace: default as part of metadata.

[debu66er]

Well, yes I use namespace default.
Potential problem here is that the deployment is created and deployed automatically by Appsody Operator. Perhaps that way of deploying is not affected by image-lookup directive?

Inspecting the generated and deployed YAML, it seems correct for me:

kind: Deployment
apiVersion: apps/v1
metadata:
  annotations:
    deployment.kubernetes.io/revision: '1'
    stack.appsody.dev/created: '2019-11-11T10:44:58+0000'
    kappnav.app.auto-create.name: testing
    kappnav.app.auto-create.labels-values: testing
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"appsody.dev/v1beta1","kind":"AppsodyApplication","metadata":{"annotations":{"image.opencontainers.org/created":"2019-11-22T07:35:28+01:00","stack.appsody.dev/configured":"appsody/nodejs-express:0.2","stack.appsody.dev/created":"2019-11-11T10:44:58+0000","stack.appsody.dev/revision":"dc03bec53c0bfa5b282e25c3359b6aea22609d70","stack.appsody.dev/tag":"appsody/nodejs-express:0.2.9"},"creationTimestamp":null,"labels":{"image.opencontainers.org/title":"testing","stack.appsody.dev/version":"0.2.9"},"name":"testing","namespace":"default"},"spec":{"applicationImage":"testing","createKnativeService":false,"expose":true,"livenessProbe":{"failureThreshold":12,"httpGet":{"path":"/live","port":3000},"initialDelaySeconds":5,"periodSeconds":2},"monitoring":{"labels":{"k8s-app":"testing"}},"readinessProbe":{"failureThreshold":12,"httpGet":{"path":"/ready","port":3000},"initialDelaySeconds":5,"periodSeconds":2,"timeoutSeconds":1},"service":{"annotations":{"prometheus.io/scrape":"true"},"port":3000,"type":"NodePort"},"stack":"nodejs-express","version":"1.0.0"},"status":{}}
    kappnav.app.auto-create.label: app.kubernetes.io/name
    kappnav.app.auto-create.version: 1.0.0
    stack.appsody.dev/revision: dc03bec53c0bfa5b282e25c3359b6aea22609d70
    stack.appsody.dev/configured: 'appsody/nodejs-express:0.2'
    kappnav.app.auto-create.kinds: 'Deployment, StatefulSet, Service, Route, Ingress, ConfigMap'
    image.opencontainers.org/created: '2019-11-22T07:35:28+01:00'
    stack.appsody.dev/tag: 'appsody/nodejs-express:0.2.9'
  selfLink: /apis/apps/v1/namespaces/default/deployments/testing
  resourceVersion: '337808'
  name: testing
  uid: 46e899fc-0cf2-11ea-927c-7200a50cbae2
  creationTimestamp: '2019-11-22T06:35:29Z'
  generation: 1
  namespace: default
  ownerReferences:
    - apiVersion: appsody.dev/v1beta1
      kind: AppsodyApplication
      name: testing
      uid: 46db61a6-0cf2-11ea-927c-7200a50cbae2
      controller: true
      blockOwnerDeletion: true
  labels:
    app.kubernetes.io/managed-by: appsody-operator
    app.kubernetes.io/name: testing
    app.kubernetes.io/version: 1.0.0
    image.opencontainers.org/title: testing
    kappnav.app.auto-create: 'true'
    stack.appsody.dev/id: nodejs-express
    stack.appsody.dev/version: 0.2.9
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: testing
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/managed-by: appsody-operator
        app.kubernetes.io/name: testing
        app.kubernetes.io/version: 1.0.0
        image.opencontainers.org/title: testing
        stack.appsody.dev/id: nodejs-express
        stack.appsody.dev/version: 0.2.9
      annotations:
        image.opencontainers.org/created: '2019-11-22T07:35:28+01:00'
        kubectl.kubernetes.io/last-applied-configuration: >
          {"apiVersion":"appsody.dev/v1beta1","kind":"AppsodyApplication","metadata":{"annotations":{"image.opencontainers.org/created":"2019-11-22T07:35:28+01:00","stack.appsody.dev/configured":"appsody/nodejs-express:0.2","stack.appsody.dev/created":"2019-11-11T10:44:58+0000","stack.appsody.dev/revision":"dc03bec53c0bfa5b282e25c3359b6aea22609d70","stack.appsody.dev/tag":"appsody/nodejs-express:0.2.9"},"creationTimestamp":null,"labels":{"image.opencontainers.org/title":"testing","stack.appsody.dev/version":"0.2.9"},"name":"testing","namespace":"default"},"spec":{"applicationImage":"testing","createKnativeService":false,"expose":true,"livenessProbe":{"failureThreshold":12,"httpGet":{"path":"/live","port":3000},"initialDelaySeconds":5,"periodSeconds":2},"monitoring":{"labels":{"k8s-app":"testing"}},"readinessProbe":{"failureThreshold":12,"httpGet":{"path":"/ready","port":3000},"initialDelaySeconds":5,"periodSeconds":2,"timeoutSeconds":1},"service":{"annotations":{"prometheus.io/scrape":"true"},"port":3000,"type":"NodePort"},"stack":"nodejs-express","version":"1.0.0"},"status":{}}
        stack.appsody.dev/configured: 'appsody/nodejs-express:0.2'
        stack.appsody.dev/created: '2019-11-11T10:44:58+0000'
        stack.appsody.dev/revision: dc03bec53c0bfa5b282e25c3359b6aea22609d70
        stack.appsody.dev/tag: 'appsody/nodejs-express:0.2.9'
    spec:
      containers:
        - resources: {}
          readinessProbe:
            httpGet:
              path: /ready
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 5
            timeoutSeconds: 1
            periodSeconds: 2
            successThreshold: 1
            failureThreshold: 12
          terminationMessagePath: /dev/termination-log
          name: app
          livenessProbe:
            httpGet:
              path: /live
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 5
            timeoutSeconds: 1
            periodSeconds: 2
            successThreshold: 1
            failureThreshold: 12
          ports:
            - name: 3000-tcp
              containerPort: 3000
              protocol: TCP
          imagePullPolicy: IfNotPresent
          terminationMessagePolicy: File
          image: testing
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: testing
      serviceAccount: testing
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
status:
  observedGeneration: 1
  replicas: 1
  updatedReplicas: 1
  unavailableReplicas: 1
  conditions:
    - type: Available
      status: 'False'
      lastUpdateTime: '2019-11-22T06:35:29Z'
      lastTransitionTime: '2019-11-22T06:35:29Z'
      reason: MinimumReplicasUnavailable
      message: Deployment does not have minimum availability.
    - type: Progressing
      status: 'True'
      lastUpdateTime: '2019-11-22T06:35:30Z'
      lastTransitionTime: '2019-11-22T06:35:29Z'
      reason: ReplicaSetUpdated
      message: ReplicaSet "testing-7b497f8bc" is progressing.

Namespace, image, etc... seem correct. Right? Do you see anything strange?

I will try an example manually as you did, although it is not useful for my use case. But I can double check if deployed things via operators are not affected by image-lookup.

In parallel, I understood you that there was another way to access untrusted image repositories different to the official way documented in OCP 4.2 docs. Do you have any URL or pointer I can read to try that way? Thanks!

[praveenkumar]

In parallel, I understood you that there was another way to access untrusted image repositories different to the official way documented in OCP 4.2 docs. Do you have any URL or pointer I can read to try that way? Thanks!

Check https://github.com/code-ready/crc/wiki/Adding-an-insecure-registry one.

[debu66er]

Check https://github.com/code-ready/crc/wiki/Adding-an-insecure-registry one.

Great, those steps worked fine. Issue now, is that authentication is required although it is the same internal repo but accessed via the external route. But this is a different question...

[praveenkumar]

those steps worked fine. Issue now, is that authentication is required although it is the same internal repo but accessed via the external route. But this is a different question...

You need to add a pull secret on that namespace and link it to default one, I will try to create another post on wiki for that but this is quite easy and you will find that info on the https://docs.openshift.com/container-platform/4.2/openshift_images/managing-images/using-image-pull-secrets.html docs

[debu66er]

Thanks a lot @praveenkumar!
Finally, investigating further appsody, I found out I could use two different parameters: --push-url and --pull-url, so I could push the image using external route but use inside the deployment yaml the internal one in the single appsody deploy command. ;-)

Anyway, I think https://github.com/code-ready/crc/wiki/Adding-an-insecure-registry URL answers the original issue: Untrusted registries can't be setup in CRC following standard OCP 4.2 docs but following the commented URL above it will work.

Do you want me to close the issue? Or should we maintain it opened until the official docs have this info included?

[praveenkumar]

Do you want me to close the issue? Or should we maintain it opened until the official docs have this info included?

@debu66er I am closing this issue and try to add a config option for crc if user want to include insecure registry in a more appropriate way.