Crc: [BUG] my-connect-cluster-connect-source:1.2.0 builder image 404

[deanpeterson]

I was able to start up a kafka cluster using the AMQ Streams operator in the new openshift 4 crc on my mac laptop. However, when I run
oc start-build my-connect-cluster-connect --from-dir=/Users/dpeterso/debezium, I get the following error:
The ImageStreamTag "my-connect-cluster-connect-source:1.2.0" is invalid: from: Error resolving ImageStreamTag my-connect-cluster-connect-source:1.2.0 in namespace es-rhug-demo: unable to find latest tagged image. I also attached an image showing a 404 error when drilling down to where the image reference is in the openshift 4 console of my project. Operators are new to me, is this a bug or am I just missing a step where I somehow have control over getting that image onto my machine?

General information

• OS: macOS
• Hypervisor: hyperkit
• Did you run crc setup before starting it (Yes/No)? Yes

CRC version

version: 1.0.0-beta.3-4.1.11+e97bf65

CRC status

CRC VM:          Running
OpenShift:       Running (v4.x)
Disk Usage:      12.82GB of 32.2GB (Inside the CRC VM)
Cache Usage:     10.48GB
Cache Directory: /Users/dpeterso/.crc/cache

CRC config

- bundle                             : /Users/dpeterso/.crc/crc_hyperkit_4.1.11.crcbundle
- cpus                               : 4
- memory                             : 8192
- nameserver                         : 
- pull-secret-file                   : 
- vm-driver                          : hyperkit
- warn-check-bundle-cached           : true

Host Operating System

ProductName:    Mac OS X
ProductVersion: 10.14.6
BuildVersion:   18G95

Steps to reproduce

1. Installed AMQ Streams operator
2. Started Kafka cluster
3. Followed instructions here: https://debezium.io/documentation/reference/0.9/operations/amq-streams.html
4. After downloading plugin, I ran the command in the instructions from step 3 above: oc start-build my-connect-cluster-connect --from-dir=/Users/dpeterso/debezium

Expected

The command would run successfully

Actual

The ImageStreamTag "my-connect-cluster-connect-source:1.2.0" is invalid: from: Error resolving ImageStreamTag my-connect-cluster-connect-source:1.2.0 in namespace es-rhug-demo: unable to find latest tagged image

Logs

INFO Checking if oc binary is cached              
DEBU oc binary already cached                     
INFO Checking if HyperKit is installed            
DEBU Checking if hyperkit is installed            
INFO Checking if crc-driver-hyperkit is installed 
DEBU Checking if crc-driver-hyperkit is installed 
DEBU crc-driver-hyperkit is already installed in /Users/dpeterso/.crc/bin/crc-driver-hyperkit 
INFO Checking file permissions for /etc/resolver/testing 
INFO Checking file permissions for /etc/hosts     
INFO Checking if CRC bundle is cached in '$HOME/.crc' 
Checking file: /Users/dpeterso/.crc/machines/crc/.crc-exist
Found binary path at /Users/dpeterso/.crc/bin/crc-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:59316
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(crc) Calling .GetBundleName
(crc) Calling .DriverName
(crc) Calling .GetState
(crc) DBG | exe=/Users/dpeterso/.crc/bin/crc-driver-hyperkit uid=0
(crc) DBG | hyperkit pid from json: 1572
Making call to close driver server
(crc) Calling .Close
Successfully made call to close driver server
Making call to close connection to plugin binary
CodeReady Containers instance is running

[deanpeterson]

I even tried following the documentation only using the cli here: https://access.redhat.com/documentation/en-us/red_hat_amq/7.4/html/using_amq_streams_on_openshift_container_platform/getting-started-str#downloads-str?extIdCarryOver=true&intcmp=701f2000001D1QWAA0&sc_cid=701f2000000u6QkAAI

I get the same error trying this command
oc start-build my-connect-cluster-connect --from-dir ./my-plugins:

The ImageStreamTag "my-connect-cluster-connect-source:1.2.0" is invalid: from: Error resolving ImageStreamTag my-connect-cluster-connect-source:1.2.0 in namespace es-rhug-debezium-demo: unable to find latest tagged image

[gbraad]

Do you also get this issue on another OpenShift 4.x cluster?

unable to find latest tagged image

sounds more like an issue with the images than with OpenShift or the provided images.

[deanpeterson]

I've tried it on mac and linux. Every which way. I'm thinking it's because they tagged the images wrong 7 days ago: https://access.redhat.com/containers/?tab=changeSummary&scrollTo=imageAdvisory#/registry.access.redhat.com/amq7/amq-streams-kafka-22/images/1.2.0-2

1.2.0-2 is not a valid tag. It should be 1.2.0 or 1.2.1, not 1.2.0-2. It doesn't seem openshift will allow 1.2.0-2, it cuts off the -2 when creating the imagestreamtag

[gbraad]

I'll ask if they forgot to update the documentation.

[deanpeterson]

Do you know of some kind of work around for this? I can't seem to point to the image at the registry.redhat.io location for amq7/amq-streams-kafka-22 to create the appropriate ImageStreamTag. It's actually easier to replicate this problem than what I listed above. All you have to do is spin up CRC, create a kafka cluster with the operator for amq streams, then try to create kafka connect s2i It will fail saying The ImageStreamTag "my-connect-cluster-connect-source:1.2.0" is invalid: from: Error resolving ImageStreamTag my-connect-cluster-connect-source:1.2.0 in namespace es-rhug-debezium-demo: unable to find latest tagged image

[gbraad]

CRC does not work specifically on making kafka or amq work on CRC, so this would be hard for us to resolve or do. We just do not have this bandwidth. Note that specific 'enabling' should not be needed ...

CRC provides a production-like OpenShift cluster, ... things should just work as they would on an actual cluster. The only difference is that we are resource constrained due to the notebook usecase and having to run all inside a single node cluster. As noted we do not have the bandwidth to test if Kafka, or any other application, would work on our cluster. For this to happen, I would like the relevant teams to test this and/or provide tests that we can run as part of integration.

I informed the team responsible for this and hopefully they will get back about this.

[deanpeterson]

It looks like I am unauthorized against the registry. I may have missed that in the instructions. What is the easiest way to make crc authorized to connect to the registry?

bash-3.2# oc get is -o yaml
apiVersion: v1
items:

• apiVersion: image.openshift.io/v1
  kind: ImageStream
  metadata:
  creationTimestamp: "2019-09-16T05:56:45Z"
  generation: 1
  labels:
  strimzi.io/cluster: my-connect-cluster
  strimzi.io/kind: KafkaConnectS2I
  strimzi.io/name: my-connect-cluster-connect
  name: my-connect-cluster-connect
  namespace: es-rhug-demo
  ownerReferences:
  
  
  
  • apiVersion: kafka.strimzi.io/v1beta1
    
    blockOwnerDeletion: false
    
    controller: false
    
    kind: KafkaConnectS2I
    
    name: my-connect-cluster
    
    uid: c3111459-d846-11e9-a46c-cec03ba5e24b
    
    resourceVersion: "180489"
    
    selfLink: /apis/image.openshift.io/v1/namespaces/es-rhug-demo/imagestreams/my-connect-cluster-connect
    
    uid: c3f8a7b5-d846-11e9-964f-0a580a800098
    
    spec:
    
    lookupPolicy:
    
    local: true
    
    status:
    
    dockerImageRepository: image-registry.openshift-image-registry.svc:5000/es-rhug-demo/my-connect-cluster-connect
    
    publicDockerImageRepository: default-route-openshift-image-registry.apps-crc.testing/es-rhug-demo/my-connect-cluster-connect
  
  
• apiVersion: image.openshift.io/v1
  kind: ImageStream
  metadata:
  creationTimestamp: "2019-09-16T05:56:45Z"
  generation: 2
  labels:
  strimzi.io/cluster: my-connect-cluster
  strimzi.io/kind: KafkaConnectS2I
  strimzi.io/name: my-connect-cluster-connect-source
  name: my-connect-cluster-connect-source
  namespace: es-rhug-demo
  ownerReferences:
  
  
  
  • apiVersion: kafka.strimzi.io/v1beta1
    
    blockOwnerDeletion: false
    
    controller: false
    
    kind: KafkaConnectS2I
    
    name: my-connect-cluster
    
    uid: c3111459-d846-11e9-a46c-cec03ba5e24b
    
    resourceVersion: "180592"
    
    selfLink: /apis/image.openshift.io/v1/namespaces/es-rhug-demo/imagestreams/my-connect-cluster-connect-source
    
    uid: c3e97469-d846-11e9-964f-0a580a800098
    
    spec:
    
    lookupPolicy:
    
    local: false
    
    tags:
  
  
  • annotations: null
    
    from:
    
    kind: DockerImage
    
    name: registry.redhat.io/amq7/amq-streams-kafka-22:1.2.0
    
    generation: 2
    
    importPolicy: {}
    
    name: 1.2.0
    
    referencePolicy:
    
    type: Source
    
    status:
    
    dockerImageRepository: image-registry.openshift-image-registry.svc:5000/es-rhug-demo/my-connect-cluster-connect-source
    
    publicDockerImageRepository: default-route-openshift-image-registry.apps-crc.testing/es-rhug-demo/my-connect-cluster-connect-source
    
    tags:
  
  
  • conditions:
    
    
    
    
    
    • generation: 2
      
      
      lastTransitionTime: "2019-09-16T05:56:46Z"
      
      
      message: 'Internal error occurred: Get https://registry.redhat.io/v2/amq7/amq-streams-kafka-22/manifests/1.2.0:
      
      
      unauthorized: Please login to the Red Hat Registry using your Customer Portal
      
      
      credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531'
      
      
      reason: InternalError
      
      
      status: "False"
      
      
      type: ImportSuccess
      
      
      items: null
      
      
      tag: 1.2.0
      
      
      kind: List
      
      
      metadata:
      
      
      resourceVersion: ""
      
      
      selfLink: ""
    
    
    
  
  

[gbraad]

@praveenkumar shouldn't the pull-secret suffice?

[praveenkumar]

@deanpeterson pull secret which you used as part of the crc start should have the token for registry.redhat.io, like following one.

$ cat pullsecert | jq .
{
  "auths": {
    "cloud.openshift.com": {
      "auth": "xxx",
      "email": "<my_email>"
    },
    "quay.io": {
      "auth": "xxx",
      "email": "<my_email>"
    },
    "registry.connect.redhat.com": {
      "auth": "xxx",
      "email": "<my_email>"
    },
    "registry.redhat.io": {
      "auth": "xxx",
      "email": "<my_email>"
    }
  }
}

[deanpeterson]

@praveenkumar Others are saying I have to manually add this pull secret in ocp 4.

"OpenShift 4 for whatever reason signs you automatically into the Red Hat registry for regular images but not for ImageStreams. So will need to create a Docker pull secret in your namespace with the credentials for the Red Hat registry. For the S2I functionality, you will also need to link it to the builder service account."

I'm not sure how to do any of that. Any pointers?

[praveenkumar]

@deanpeterson if that is the case then try to do following and see if that works.

$ oc create secret docker-registry redhat-registry --docker-username=<user-name> --docker-password '<Password>' --docker-server=registry.redhat.io

$ oc get sa
NAME       SECRETS   AGE
builder    2         3h15m
default    2         3h15m
deployer   2         3h15m

$ oc secrets link builder redhat-registry

[deanpeterson]

@praveenkumar thanks for the info. Do I use the kubeadmin user:pass or my access.redhat.com user:pass? Or do I copy what is in my pull secret in the section accociated with registry.redhat.io?

[gbraad]

Also see:
http://lists.openshift.redhat.com/openshift-archives/users/2019-September/msg00015.html

On Mon, Sep 16, 2019, 22:24 deanpeterson [e-mail] wrote:

@praveenkumar https://github.com/praveenkumar thanks for the info. Do I
use the kubeadmin user:pass or my access.redhat.com user:pass?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/code-ready/crc/issues/594?email_source=notifications&email_token=AAAAOZTDFXUI4PURJAUCO3LQJ6JJVA5CNFSM4IWV2FPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6ZKHFQ#issuecomment-531800982,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAAOZRYY5H4LG2FUZTCMBDQJ6JJVANCNFSM4IWV2FPA
.

[deanpeterson]

@gbraad I read through the responses on that link. It doesn't look like a resolution has been reached. The OP is still left trying to figure things out.

[deanpeterson]

I received this from Fernando Lozano. I can only get things to work in the openshift namespace. If I try to spin up the amq streams kafka connect s2i in a project I create, I get the authorization errors:

Hi,

If you use image streams from the openshift namespace, any S2I build should "just work". The openshift namespace already includes a secret to access registry.redhat.io (it is the one you have to give to the installer) and its image streams are already configured to use that secret. As far as I tested CRC it works the same way as an AWS cluster you install yourself. I am able to run S2I builds on my CRC instance without any additional steps.

If you are not using one of the image streams from the openshift namespace, then you need to setup pull and build secrets on your project, or setup a shared image stream in a shared project just like the openshift name space comes by default.

Our training+certification course DO288 - Red Hat OpenShift Development I: Containerizing Applications tried to shed some light on the topic of private/authenticated registries, image streams, and builds. See chapter 05 for instructions to use your own builder image from a private registry and chapter 03 for more about using private registries.

• To create the secret that allows authentication to a registry, run this after podman login

$ oc create secret generic mysecret \
--from-file .dockerconfigjson=${XDG_RUNTIME_DIR}/containers/auth.json \
--type=kubernetes.io/dockerconfigjson

• This links the secret to the builder service account, that runs S2I builds.

$ oc secrets link builder mysecret

[]s, Fernando Lozano

[deanpeterson]

Still running into issues in the openshift namespace. When I kick off a build:

Caching blobs under "/var/cache/blobs".
Warning: Pull failed, retrying in 5s ...
Warning: Pull failed, retrying in 5s ...
Warning: Pull failed, retrying in 5s ...
error: build error: After retrying 2 times, Pull image still failed due to error: unable to retrieve auth token: invalid username/password

[deanpeterson]

I was able to get builds working by following Fernando's advice above:

$ oc create secret generic mysecret
--from-file .dockerconfigjson=${XDG_RUNTIME_DIR}/containers/auth.json
--type=kubernetes.io/dockerconfigjson

This links the secret to the builder service account, that runs S2I builds.

$ oc secrets link builder mysecret

auth.json is just my pull secret text file downloaded and renamed