Crc: [BUG] No OperatorHub Items Found

[Harald-U]

General information

• OS: Linux
• Hypervisor: KVM
• Did you run crc setup before starting it (Yes/No)/? Yes

CRC version

crc version: 1.8.0+0a318dc
OpenShift version: 4.3.8 (embedded in binary)

CRC status

# Put `crc status` output here

CRC config

CRC VM:          Running
OpenShift:       Running (v4.3.8)
Disk Usage:      12.86GB of 32.72GB (Inside the CRC VM)
Cache Usage:     11.8GB
Cache Directory: /home/uebele/.crc/cache

Host Operating System

NAME=Fedora
VERSION="30 (Thirty)"
ID=fedora
VERSION_ID=30
VERSION_CODENAME=""
PLATFORM_ID="platform:f30"
PRETTY_NAME="Fedora 30 (Thirty)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:30"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f30/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=30
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=30
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"

Steps to reproduce

1. Login to OpenShift console as kube:admin
2. Goto Operators -> OperatorHub
3. or using CLI:
4. oc login as kubeadmin
5. oc get opsrc -n openshift-marketplace

Expected

I expect to see the catalog, used to work in previous versions of CRC

Actual

Web Console: No OperatorHub Items Found

CLI:

$ oc get opsrc -n openshift-marketplace
NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS        MESSAGE                                                                                            AGE
certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=certified-operators: context deadline exceeded   19d
community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=community-operators: context deadline exceeded   19d
redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=redhat-operators: context deadline exceeded      19d

Logs

You can start crc with crc start --log-level debug to collect logs.
Please consider posting this on http://gist.github.com/ and post the link in the issue.

[praveenkumar]

@Harald-U Can you try to describe one of the operator and see what events it have or what causing them as context deadline exceeded this looks like the operator not able to fetch the details over the internet. Also can you try to deploy a sample application on this cluster?

$ oc describe opsrc certified-operators  -n openshift-marketplace

[cfergeau]

Are you running behind a proxy?

[Harald-U]

I ran the "oc describe opsrc certified-operators -n openshift-marketplace" about an hour ago, then went into a web meeting, and just ran it again. It is now giving results (showing packages), and the OperatorHub catalog in the console is now displaying the expected list.

Seems like it takes about an hour now to populate ... I never experienced this before in previous versions of CRC.

I guess you can close this issue. Thanks so much for your extremely quick responses!!!

[rcarrata]

I hit the same issue, after a fresh install of crc no operatorhub content appears:

$ crc version
crc version: 1.8.0+0a318dc
OpenShift version: 4.3.8 (embedded in binary)

The opsrc shows:

$ oc get opsrc -n openshift-marketplace
NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS        MESSAGE                                                                                            AGE
certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=certified-operators: context deadline exceeded   20d
community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=community-operators: context deadline exceeded   20d
redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=redhat-operators: context deadline exceeded      20d

The packagemanifests are neither showing any result:

$ oc get packagemanifests -n openshift-marketplace
No resources found in openshift-marketplace namespace.

I'll check after a while for reproduce the behaviour of @Harald-U.

[praveenkumar]

Seems like it takes about an hour now to populate ... I never experienced this before in previous versions of CRC.

@Harald-U Are you able to reproduce this each single time, we do have integration tests around marketplace where we deploy an etcd operator and we didn't observe any kind of issue during those tests.

[Harald-U]

@praveenkumar I tried it three times this morning (crc stop, crc delete, crc start) and the OperatorHub displayed its entry every time immediately after startup of CRC. Demo Effect? I'll try it again this afternoon, maybe its a overloaded server or network somewhere?

[Harald-U]

I tried it again in the afternoon (European time) and it still works ...

[hosie]

I am seeing similar problems ( running crc 1.8.0 on MacOS)
I have deleted my cluster and recreated. The MESSAGE field seems to flip between dial tcp .....: i/o timeout and context deadline exceeded
i.e...

NAMESPACE               NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS        MESSAGE                                                                                                         AGE
openshift-marketplace   certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=certified-operators: dial tcp 3.223.162.55:443: i/o timeout   21d
openshift-marketplace   community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=community-operators: dial tcp 18.208.7.119:443: i/o timeout   21d
openshift-marketplace   redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=redhat-operators: context deadline exceeded                   21d

And after a few minutes, I see the

NAMESPACE               NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS        MESSAGE                                                                                            AGE
openshift-marketplace   certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=certified-operators: context deadline exceeded   21d
openshift-marketplace   community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=community-operators: context deadline exceeded   21d
openshift-marketplace   redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=redhat-operators: context deadline exceeded      21d

and after another few minutes, I see

NAMESPACE               NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS        MESSAGE                                                                                                         AGE
openshift-marketplace   certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=certified-operators: dial tcp 3.223.162.55:443: i/o timeout   21d
openshift-marketplace   community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=community-operators: context deadline exceeded                21d
openshift-marketplace   redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=redhat-operators: dial tcp 54.165.181.134:443: i/o timeout    21d

I will leave it for a while to see if it recovers itself like it did for @Harald-U

[rcarrata]

@hosie same here. In my case, after some time, stills failing... How about your installation?

[hosie]

2 hours later and it is still failing.

[hosie]

After deleting the cluster and creating it again, it seems to be working now

NAMESPACE               NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS      MESSAGE                                       AGE
openshift-marketplace   certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Succeeded   The object has been successfully reconciled   22d
openshift-marketplace   community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Succeeded   The object has been successfully reconciled   22d
openshift-marketplace   redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Succeeded   The object has been successfully reconciled   22d

and the operator hub page on the console UI looks correct.

The only thing that might be different from before was that I was connected to my company's VPN when I created the cluster before. I disconnected from the VPN, waited about an hour and the cluster was still broken. When I deleted it and recreated while disconnected from the VPN, it seems to have worked. I am not sure if this is a coincidence or whether the VPN was causing networking issues.

[opuk]

I'm running in to this every time I launch CRC. Running behind proxy.
Looking at the deployments for the operators they don't have any *_PROXY environment variables set. When I set them manually everything kicks off as expected and operatorhub items are populated.

crc version: 1.9.0+a68b5e0
OpenShift version: 4.3.10 (embedded in binary)


> oc get opsrc
NAME                  TYPE          ENDPOINT              REGISTRY          DISPLAYNAME           PUBLISHER   STATUS    MESSAGE                                                                                                                           AGE
certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=certified-operators: dial tcp: lookup quay.io on 172.30.0.10:53: no such host   12d
community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=community-operators: dial tcp: lookup quay.io on 172.30.0.10:53: no such host   12d
redhat-operators      appregistry   https://quay.io/cnr   redhat-operators  Red Hat Operators     Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=redhat-operators: dial tcp: lookup quay.io on 172.30.0.10:53: no such host      12d

[cfergeau]

@opuk I'd prefer that you create a new issue for this, thanks!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.