Okd: Operators are missing in the Operator Hub Marketplace

[jomeier]

Hi,

it seems as if some operators are missing in the Operator Hub tab in the Web UI. Today we were searching for Elasticsearch, or logging... but didn't found anything.

We talked in Slack about that recently. But maybe its good to track it here.

Greetings,

Josef

[jomeier]

@vrutkovs: This issue is for missing operators with label „custom“.

[lallinger-arbeit]

I'm having the same issue when wanting to install the Noobaa operator. Installing the operatorhub.io catalogsource in a namespace fixes the problem, but then an other problem arises:
The Noobaa operator also automatically installs the AWS S3 operator, which is available in the community-operator catalogsource and the operatorhub.io catalogsource, but in two different versions (1.0.0 and 1.01).
When leaving the installPlanApproval to automatic this leads to automatic consecutive updates (from 1.0.0 to 1.01 to 1.0.0...) which in turn always create new serviceaccounts and rolebindings (about 2000 per night). Disabling automatic updates leads still to installplans automatically getting created (about 4000 per night). Example installPlan:

apiVersion: operators.coreos.com/v1alpha1
kind: InstallPlan
metadata:
  creationTimestamp: "2020-04-17T09:10:33Z"
  generateName: install-
  generation: 1
  name: install-9grr7
  namespace: noobaa
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: Subscription
    name: noobaa
    uid: fa1e86ee-a111-4a26-a61e-f503b793da08
  resourceVersion: "2546748"
  selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/noobaa/installplans/install-9grr7
  uid: c3d69d61-1171-49a1-9653-8e4cefc92b50
spec:
  approval: Manual
  approved: false
  clusterServiceVersionNames:
  - awss3operator.v1.0.1
  - awss3operator.v1.0.0
  generation: 4287

[vrutkovs]

Created a PR which adds a new "upstream-community-operators" which is supposed to be closer to what OperatorHub.io. This still doesn't have Noobah or Elasticsearch - not sure why these are not build and not available in any of the sources. That would be a question for OperatorHub.io folks - like @dmesser

but then an other problem arises

That's entirely different issue, report that in https://github.com/operator-framework/community-operators

[dmesser]

@vrutkovs is this still an issue? Catalog content is fetched from external API endpoints outside of your OKD cluster. So the issue may be that your cluster does not have access to those. Can you list the instances of CatalogSource and their status on your cluster?

@lallinger-arbeit Noobaa isn't shipped as a standalone Operator on OKD/OCP, since it's part of OpenShift Container Storage offering there.

[lallinger-arbeit]

@dmesser I could live with that, if the other issue gets sorted out/if it's supported to install the operatorhub.io catalog in a single namespace. I don't wan't to use the OpenShift Container Storage operator.

[vrutkovs]

Catalog content is fetched from external API endpoints outside of your OKD cluster

we're using "community" and "upstream-community" stream, and yet Elasticsearch (or Noobaa) are still no available in OKD's Operator Hub tab - but available on OperatorHub.io

Which other sources should we add to get all the upstream operators?

[lallinger-arbeit]

@vrutkovs have you tried adding the "custom" stream?

[vrutkovs]

What's a "custom" stream?

[lallinger-arbeit]

I think the stream is called "custom", like the others are called "community" and "upstream-community"

See here in my noobaa namespace where i have installed the operatorhub.io catalogsource it states "custom"

[dmesser]

@lallinger-arbeit it is not supported to install opertorhubio catalog on OKD because it contains content that may not work on OpenShift/OKD. OKDs community catalogs are supposed to be that. You are not seeing the Elastic Search Operator because it's part of the official Red Hat supported Operators on OCP. OKD being community may not include that.

[lallinger-arbeit]

@dmesser catalogsources comming with OKD 4 already have operators with a disclaimer saying they are not tested. So why shouldn't i be able to add other sources that are not tested?

[dmesser]

@lallinger-arbeit You can do that of course, it's just not something you can raise issues for.

[ibotty]

Maybe I am missing something. Are the upstream-community-operators from openshift/installer#3487 supposed to be installed on OKD? When (manually) adding the objects from the pull request, I get the following alert.

The cluster has custom OperatorSource/CatalogSourceConfig, which are deprecated from OCP 4.5. Upgrade to OCP 4.5 will not be possible unless those resources are removed

(which also uses OCP instead of OKD).

[kai-uwe-rommel]

I am also hitting this problem when I try to install "cluster-logging" to a cluster with OKD 4.4-beta5 following the OKD documentation:
https://docs.okd.io/latest/logging/cluster-logging-deploying.html

I think I am not experienced and trained enough to understand all of what you write above.
I have already tried installing the necessary catalogsource(s) but they remain in TRANSIENT_FAILURE state ... how to solve that?

Is there a way how I can still install cluster-logging on OKD?
And is that legal with OKD?

[RyuunoAelia]

Hello,

I do not believe that openshift/installer#3487 was a good move, since upstream-community-operators are, by definition, operators made for "vanilla" kubernetes, thus are not made to run under OpenShift extra security constraints.
A good example of that is the rook-ceph operator as present on upstream-community-operators, it will absolutely not work on OpenShift4, I have made some PRs to the project to get it as close as possible to "community-operators" but it won't get there in a forseeable future.

As for custom operator sources, they are heavily tied to Quay.io. You are forced to use the Application Registry from Quay to be "future-proof" the format of packaging operators itself is not even rolled-out and already in way to be deprecated. The target is moving fast and documentation is scarce. The best way to get the operators you want into OKD is by asking the maintainers of the project to get it into community-operators, this was done by many of the Red Hat supported operators, like Strimzi and others, so there is a good chance they will do it. Note: both upstream-community-operators and community-operators are folders in the same github repository, you might need to explain to the maintainers of the operator what is the difference and why they should care about it.

[vrutkovs]

upstream-community-operators are, by definition, operators made for "vanilla" kubernetes, thus are not made to run under OpenShift extra security constraints.

That sounds like bugs in respective operators. Default settings in OKD are not guaranteeing these would work, but merely provide access to those. Also, this setting can be easily reverted.

[RyuunoAelia]

That sounds like bugs in respective operators. Default settings in OKD are not guaranteeing these would work, but merely provide access to those. Also, this setting can be easily reverted.

If I transpose what you are writing here to the Linux Kernel, I read it like this: "If the ubuntu kernel does not work on a RHEL installation, you should rise a bug in the kernel because the bug is on their side. Both are using the same linux kernel so it should work".

The example I pointed before, rook-ceph, has sensible differencies between the vanilla kubernetes and openshift setups, due to some particular things OpenShift mandates, like some containers need to run privileged on OpenShift but not on kubernetes, things like that. This changes the CSV a bit between the two platforms, and thus we cannot run upstream-community-operator rook-ceph on OpenShift as-is.

[vrutkovs]

Clarified that with Daniel - operators from "upstream-" are being copied to "community-operators" if these are known to work with OKD/OCP. So installer PR would revert this change.

This ticket got derailed pretty far from original issue - I suggest it should be closed and more specific issue be opened (in okd or operatorhub issue trackers if necessary). @jomeier wdyt?

[jomeier]

@vrutkovs:
You are right. I'll close it. Thanks guys for clarifying that.