Che: Make it possible to override theia, theia endpoint and machine-exec images from a cr.yaml

[nickboldt]

Today, to change the value of theia [1], theia-endpoint [2], or machine-exec [3] images (eg., for offline or airgap, or branded version), you have to change them in the plugin registry, then rebuild the registry [4], then use that new image in a custom resource yaml [5].

It would be sweet if we could skip having to rebuild the plugin registry to point to a different set of images. In fact we NEED this for airgap/offline scenarios for deployment to just work OOTB w/o having to build a custom plugin registry image for each customer scenario.

[1] https://github.com/redhat-developer/codeready-workspaces/blob/master/dependencies/che-plugin-registry/v3/plugins/eclipse/che-theia/7.3.1/meta.yaml#L51
[2] https://github.com/redhat-developer/codeready-workspaces/blob/master/dependencies/che-plugin-registry/v3/plugins/eclipse/che-theia/7.3.1/meta.yaml#L72
[3] https://github.com/redhat-developer/codeready-workspaces/blob/master/dependencies/che-plugin-registry/v3/plugins/eclipse/che-machine-exec-plugin/7.3.1/meta.yaml#L27
[4] https://github.com/redhat-developer/codeready-workspaces/blob/master/dependencies/che-plugin-registry/build/dockerfiles/rhel.Dockerfile
[5] https://github.com/redhat-developer/codeready-workspaces-deprecated/blob/master/operator-installer/custom-resource.yaml#L14

[tsmaeder]

Treating this as P1 because of "airgapped" dependency.

[tsmaeder]

Assigning to "platform", becuase IMO, this should be possible independently of the installation method.

[ibuziuk]

@nickboldt is it a requirement for CRW 2.0? why building offline registries is not an option?
Usage of airgap mode is documented in the following PR - https://github.com/eclipse/che-docs/pull/868

[nickboldt]

My understanding of "airgap solution" is that we do not require customers to build their own plugin registry in support of airgap, because in CRW2 we provide an OOTB existing airgap-friendly one.

But if we're reversing that and saying "you want airgap, you have to build the image yourself", then sure, we're OK here. But it's another step in the doc. :)

[amisevsk]

Apart from offline cached .vsix and project files, the airgap options are available already; if you run the plugin registry with

docker run -it --rm -p 8080:8080 \
  -e CHE_SIDECAR_CONTAINERS_REGISTRY_URL=myregistry.io:9999 \
  -e CHE_SIDECAR_CONTAINERS_REGISTRY_ORGANIZATION=myorg \
  -e CHE_SIDECAR_CONTAINERS_REGISTRY_TAG=mytag \
  quay.io/eclipse/che-plugin-registry:nightly

all plugins will be updated to use these values. e.g., for redhat/java/latest you get:

spec:
  containers:
    - image: myregistry.io:9999/myorg/che-remote-plugin-runner-java8:mytag

in the operator, setting

airGapContainerRegistryHostname: "myregistry.io:9999"
airGapContainerRegistryOrganization: "myorg"

sets the first two above (with tag assumed to be the current release)

[skabashnyuk]

unsign platform team since they were not involved in airgap activities

[ibuziuk]

@nickboldt https://github.com/eclipse/che/issues/15052#issuecomment-548505132 Closing based on the comment. Feel free to reopen with details

[nickboldt]

We also need this so that we can test production-ready bits (which reference registry.redhat.io) from behind registry.stage.redhat.io or quay.io.

Otherwise all automated QE tests for CRW will require building a custom plugin registry to point at reg.s.rh.com or quay.io to resolve the theia, theia endpoint, and machine exec containers. Or they'll fail because theia, theia endpoint, and machine-exec don't yet exist in the RH container catalogue.

Sorry I didn't emphasize the QE test scenario rationale more clearly from the start here. Must have been vacation-brain setting in. :)

cc: @rhopp @dmytro-ndp

Note that we have to also rebuild all containers (and crwctl?) where hardcoded references to quay.io exist today. The problem is that as soon as we do that, we won't be able to TEST the containers as they'll be referencing images that don't yet exist outside of quay.io and registry.stage.redhat.io. I don't have a solution for this other than "QE signs off on the quay-based version, THEN we change it, and hope for the best". We should be able to override most of the 20 images after deploying (or using cr.yaml override when deploying) to point back to quay or registry.stage URLs, but not all of them...

Specifically, we can't use operator/chectl/crwctl at this point to point to a different machine-exec or theia or theia-endpoint, as those are hardcoded in the plugin registry and not exposed to che.properties or the operator.

[amisevsk]

PR submitted for downstream: https://github.com/redhat-developer/codeready-workspaces/pull/199

Before including this sort of thing upstream I'd prefer we plan it out more, because in the general case overriding these images is going to be fraught with caveats (the reason we moved to the theia remote runtime binary init container was to avoid issues when theia and sidecar versions didn't match).

[bryantson]

Tried this on CRW 2.0 with airGapContainerRegistryHostname set. Every other image overrided fine, but theia-endpoint-rhel8 still did not get override. Any solution?

[amisevsk]

@bryantson This was an issue that I think we explicitly fixed a little while ago -- I'll have to double check. Which image of the CRW plugin registry do you have deployed?

cc: @nickboldt

[nickboldt]

There's a quick workaround you can try, where you inject 1, 2 or 3 env vars into the running plugin registry. Here's how I do it to use the quay.io/crw versions of the theia, theia-endpoint, and machineexec images:

QP="quay.io/crw" # prefix containing repo & organization 
QS="rhel8:2.0" # suffix containing qualifier and tag
for e in \
  THEIA_IMAGE=${QP}/theia \
  THEIA_ENDPOINT_RUNTIME_BINARY_IMAGE=${QP}/theia-endpoint \
  MACHINE_EXEC_IMAGE=${QP}/machineexec; do
    oc set env deployment/plugin-registry CHE_PLUGIN_REGISTRY_${e}-${QS};
done

# verify it worked
oc get deployment plugin-registry -o=json | jq '.spec.template.spec.containers[0].env'

Ref: https://github.com/redhat-developer/codeready-workspaces-deprecated/tree/master/operator-installer#override-crw-containers-in-the-plugin-registry

[bryantson]

Thank you, @amisevsk. Thank you, @nickboldt. I will try this way first before rebuilding plugin registry.

[nickboldt]

There's the more global option too (which overrides ALL container refs in the plugin registry), but I suspect you're already doing that... as that's the official recommended way for setting up airgap support in plugin registry (w/o having to rebuild the whole thing to inject your customizations):

for e in \
  URL=quay.io \
  ORGANIZATION=crw \
  TAG=2.0; do
    oc set env deployment/plugin-registry CHE_SIDECAR_CONTAINERS_REGISTRY_${e};
done

# verify it worked
oc get deployment plugin-registry -o=json | jq '.spec.template.spec.containers[0].env'

https://github.com/redhat-developer/codeready-workspaces-deprecated/tree/master/operator-installer#override-all-containers-in-the-plugin-registry

[amisevsk]

The confusing part here is that setting airGapContainerRegistryHostname does set CHE_SIDECAR_CONTAINERS_REGISTRY_URL as you suggest. @bryantson which image of the registry is running in your environment, I can't seem to reproduce this.

[bryantson]

@amisevsk This is the image: codeready-workspaces/pluginregistry-rhel8:2.0

[amisevsk]

Hmm, I just tested that image, setting CHE_SIDECAR_CONTAINERS_REGISTRY_URL=TEST and it worked as expected. Could you share

1. The output of env | grep CHE in the registry container (e.g. via oc exec <pluginregistry pod> 'env | grep CHE')
2. The yaml for che-theia, located at <plugin-registry-url>/v3/plugins/eclipse/che-theia/latest

[bryantson]

Not exactly related but... Updating CheCluster does not load the change to the properties in CustomConfig sometime. I am not sure what is the best way to redeploy.

[amisevsk]

@bryantson if you are using the operator, then it should recreate any deleted resources automatically (so a fallback option is to delete the resource in question and let the operator recreate it).

If you mean that the Che server is not being rolled out to a new update when the custom config is modified, please file an issue since that sounds like a bug in the operator (maybe it recreates the configmap but does not trigger a rollout to pick up the changes). As a workaround, the easiest way to ensure changes are picked up by the Che server is to either rollout a new deployment or delete the Che server pod.