che-plugin-registry - Failed to load plugins/editors error

[InterestedInTechAndCake]

Summary

I have built a custom plugin registry using the clone of the 7.12.0 release of che-plugin-registry and simply added the vscode-bazel meta-yaml file in the directory v3/plugins/ms-vscode/bazel/0.3.0 (everything else untouched). I used the build.sh script to build the image:

./build.sh --registry eu.gcr.io/ --tag 7.12.0

Then pushed the built image to the GCP container registry and used helm to deploy it to the che namespace on kubernetes:

helm upgrade --install che-plugin-registry --debug --namespace che --set global.ingressDomain= --set chePluginRegistryImage=eu.gcr.io//che-plugin-registry:7.12.0 --set chePluginRegistryImagePullPolicy="IfNotPresent" deploy/kubernetes/che-plugin-registry

After that's started, I point the che configmap property: CHE_WORKSPACE_PLUGIN_REGISTRY_URL to the newly deployed che-plugin-registry-che. url.

I have tried both restarting my existing workspace and created a new one after the changes, although the vscode-bazel plugin (along with all the pre-exist ones) are all loaded when I started the workspace, when I go to the "configure workspace" page everytime it shows this error:

"Failed to load plugins" and "Failed to load editors"

and indeed the Plugins and Editors tabs show completely blank pages.

Could anyone help with resolving this issue please? I tried looking into the kubectl logs of all the containers in the che namepsace, none show anything obvious (I did a kubectl logs -f, reproduced the errors, none of the logs showed anything new when the error happened.).

[filipkroupa]

@InterestedInTechAndCake I suspect your case might be caused by issue #16103 There is a workaround described in _Additional context_

[ibuziuk]

@ericwill could you please take a look?

[ericwill]

@ericwill could you please take a look?

Yup will do.

[InterestedInTechAndCake]

Thank you @filipkroupa, I have tried the suggested workaround but unfortunately that didn't resolve this particular issue.

From the browser console output, I got this when the error occurred:

app-33a84c37c4.js:211446 Possibly unhandled rejection: {"data":"","status":304,"config":{"method":"GET","transformRequest":[null],"transformResponse":[null],"jsonpCallbackParam":"callback","url":"/api/permissions/workspace/all?instance=workspacewzitgf0ibpgkssnb","headers":{"Accept":"application/json, text/plain, /","Authorization":

And also:

app-33a84c37c4.js:116775 GET https://che-che.ctltools.com/dashboard/app/editor.worker.module.js 404

[InterestedInTechAndCake]

This also showed:

app-33a84c37c4.js:209441 GET https://che-plugin-registry-che./plugins/ net::ERR_CERT_AUTHORITY_INVALID

So perhaps this is the culpit?

also, google-chrome is showing this error the the "Failed to load plugins/editors" error happened:

ERROR:ssl_client_socket_impl.cc(941)] handshake failed; returned -1, SSL error code 1, net_error -202

[InterestedInTechAndCake]

The 304 error happens when I clicked on "Configure Workspace" the second time (the first time it showed the Failed to load plugins/editors error, but no 304). The ERR_CERT_AUTHORITY_INVALID error happened both times though.

[filipkroupa]

@InterestedInTechAndCake Based on your helm command you did not deploy the pluginRegistry with TLS, I expected you don't use it then. If your Che deployment is using TLS, you should provide it to your pluginRegistry as well. I think you should add parameter
--set chePluginRegistryIngressSecretName=che-tls, providing your secret has the default name che-tls. But I must say my experience comes from deploying my private built registry docker image and in order to deploy it with helm and TLS, I had to adjust some of the deployment yaml files. I'm not sure about the eu.gcr.io//che-plugin-registry:7.12.0 image.

[InterestedInTechAndCake]

Thanks @filipkroupa - I just tried to set chePluginRegistryIngressSecretName but still getting the same error.

When I use the browser and go to https://che-plugin-registry-che.mydomein/v3/plugins, it loads correctly and the certificate is shown as valid.

[InterestedInTechAndCake]

Not sure if this is related... when I set up the Che server, I noticed that LetsEncrypt only generates tls.crt and tls.key, but ca.crt is 0 bytes. When I do kubectl describe secret/che-tls -n che I got this:

Data

ca.crt: 0 bytes
tls.crt: 3558 bytes
tls.key: 1679 bytes

Subsequently "chectl server:start --installer=helm --platform=k8s --domain=ctltools.com --multiuser --tls" failed.

Then I tried "chectl server:start --installer=operator --platform=k8s --domain=ctltools.com --multiuser --tls" and it worked.

[InterestedInTechAndCake]

The deployed ingress doesn't have the tls spec added! So I added that manually and deploy the ingress rather than using helm. Now when I load my workspace, I got a different error:

Mixed Content: The page at 'https://che-che.mydomain/dashboard/#/' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://che-plugin-registry-che.mydomain/v3/plugins/'. This request has been blocked; the content must be served over HTTPS.

So there are requests from Che use http url rather than https. I have verified that the CHE_WORKSPACE_PLUGIN_REGISTRY_URL property deployed on cm/che is set to the https url.

I tried adding force-ssl-redirect to the ingress but still got the same "mixed content" error.

Also this error might be unrelated to the "Failed to load plugins" but responsible for the "Failed to load editors" error?

GET https://che-che.ctltools.com/dashboard/app/editor.worker.module.js 404

[InterestedInTechAndCake]

Just to verify that the root cause is indeed to do with the "Mixed Content:....This request has been blocked: the content must be served over HTTPS.", I temporarily allow insecure content for the che-che.mydomain site, and voila - the plugins and editors both loaded successfully.

So what's the best way to resolve this? I don't know where in Che server it is making the request via HTTP... as mentioned above the CHE_WORKSPACE_PLUGIN_REGISTRY_URL has been set to use HTTPS url.

[mmorhun]

@InterestedInTechAndCake seems you took a hard way. I would suggest you to start over and deploy Che using operator installer with default registry first. When everything is ready, just edit checluster custom resource named eclipse-che and change plugin registry image. Wait a moment while operator recreate the registry pod and that's it!
If you still want to continue the hard way, you should check consistence of endpoints. TLS termination happens on ingresses level by default whereas services by itself are http (if you use helm installer). You may go another way and make ingresses http, but services itself https. But if you mix this two approaches it will probably fail with the described by you error.

[InterestedInTechAndCake]

Thanks @mmorhun! I am afraid I have already tried the approach you mentioned - I did start it using the vanilla install using the operator installer with the default registry, it was all working. Then I created the che-plugin-registry image using the 7.12.0 clone and even without modifying anything I got the "Failed to load plugins" error. That was using HTTP, no TLS added.

Then I tried added TLS because it seems that the error could be related to that. The only thing I have changed was added the TLS section to the Ingress (not the service or the deployment). Same error.

[InterestedInTechAndCake]

Do I need to do a helm upgrade on che and --set che.workspace.pluginRegistryUrl to the new URL to make things consistent? So far I have only updated the checluster custom resource property CHE_WORKSPACE_PLUGIN_REGISTRY_URL.

Or do I need to update any values on the persistent volume?

[mmorhun]

@InterestedInTechAndCake it is hard to say what you should do now, because I don't know the current state of the cluster and resources in it. Sometimes tiny things matters.

But you can go the way I suggested you above. So:

1. Completely delete your installation of Che
2. Precreate required secrets (please see doc for reference)
3. Deploy Che with operator installer. After this step you should have Che working with default plugin registry.
4. Build you own Che plugins registry docker image and push it into a docker registry (to have the image accessible).
5. Edit checluster custom resource (using kubectl for example). Set your image from step 4 instead of default one, save changes.
6. Wait a moment and your registry will be deployed and working.

[InterestedInTechAndCake]

Same error!

I have followed the exact steps, deleted the old che namespace and recreated as instructed using the self signed cert, verified that it's all working, then used this to update the checluster:

apiVersion: org.eclipse.che/v1
kind: CheCluster
metadata:
name: eclipse-che
namespace: che
spec:
server:
pluginRegistryURL: "http://che-plugin-registry-che.mydomain"
customCheProperties:
CHE_WORKSPACE_PLUGIN__REGISTRY__URL: "http://che-plugin-registry-che.mydomain"

Che server pod restarted, and I refreshed my browser before loading the dashboard.

I have tried both https (with tls added to ingress for the che-plugin-registry) and http, both resulted the same mixed content error!

[InterestedInTechAndCake]

Please note that the plugin registry is serving successfully, once I opened the workspace all the plugins were loaded fine. The problem is in the Che dashboard, it is for some reasons referencing a mix of the http and https, it seems and that's persisted somewhere.

[mmorhun]

@InterestedInTechAndCake can you try to open dashboard from private browser tab?

[InterestedInTechAndCake]

@mmorhun Thanks - tried the incognito tab, still same error :-(

[mmorhun]

@InterestedInTechAndCake can you please share plugin registry ingress configuration?

[InterestedInTechAndCake]

Thank you so much for helping out @mmorhun . Here it is:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: che-plugin-registry
namespace: che
annotations:
kubernetes.io/ingress.class: 'nginx'
nginx.ingress.kubernetes.io/ssl-redirect: 'true'
spec:
rules:

• host: "che-plugin-registry-che.mydomain"
  http:
  paths:
  
  
  
  • path: /
    
    backend:
    
    serviceName: che-plugin-registry
    
    servicePort: 8080
    
    tls:
  
  
• hosts:
  
  
  
  • "*.mydomain"
    
    secretName: che-tls
  
  

[mmorhun]

@InterestedInTechAndCake looks like you are mixing approaches as I described above.
Do you have any reasons to change plugin registry URL? Do you need to host it outside of the cluster?
If that doesn't matter for you, please follow my instructions above and change only image of plugin registry in the custom resource. Che server shouldn't restart. Registry URL shouldn't be changed in that case. Restarts only the registry pod (because its image was changed).
You may go the way you started to go, but then you have to change some configs. At least add TLS termination to new ingress. But again, if you don't need to move the registry outside of the cluster I see no point in it. Except you want to play with configs :)

[InterestedInTechAndCake]

oops it looks like the yaml alignment was lost when the ingress spec was pasted above.

Thank you very much again @mmorhun! I have now tried to have a clean install and instead of deploying a separate container for the plugin-registry, I simply updated the image as you suggested - and the worked!

So what could be the root cause then? The main difference between the two approach is, it seems, deploying the registry as a separate container (in the same cluster, same namespace) vs just updating the image for the existing running one.

I followed the instructions provided here: https://github.com/eclipse/che-plugin-registry

[gattytto]

app-33a84c37c4.js:211446 Possibly unhandled rejection: {"data":"","status":304,"config":{"method":"GET","transformRequest":[null],"transformResponse":[null],"jsonpCallbackParam":"callback","url":"/api/permissions/workspace/all?instance=workspacewzitgf0ibpgkssnb","headers":{"Accept":"application/json, text/plain, _/_","Authorization":

And also:

app-33a84c37c4.js:116775 GET https://che-che.ctltools.com/dashboard/app/editor.worker.module.js 404

for the browser javascript request part, please collect as much info as you can about the body of the request and response within the browser console.

Please note that the plugin registry is serving successfully, once I opened the workspace all the plugins were loaded fine. The problem is in the Che dashboard, it is for some reasons referencing a mix of the http and https, it seems and that's persisted somewhere.

seems like:
for the web client you add exception to the certs (fixes warning, you can get i), your plugin initiator (POD) is working ok with https comm to plugin-registry so it gets the yaml and finds the vs-extention reference, downloads it and starts the workispace, but your che-server (POD) refuses to accept the validity or can't find or has odd behavior.

which brings me to:

Could anyone help with resolving this issue please? I tried looking into the kubectl logs of all the containers in the che namepsace, none show anything obvious (I did a kubectl logs -f, reproduced the errors, none of the logs showed anything new when the error happened.).

looking for ways to bringing up verbosity level of che-server I can see this for my non tls setup so you might have a https-debug one in your che-server? :
yaml { "containerPort": 8000, "name": "http-debug", "protocol": "TCP" },
also within "kubectl get pod/che-xxxx -n che -o yaml" spec you can see props like:
yaml { "name": "CHE_SELF__SIGNED__CERT" }, { "name": "CHE_GIT_SELF__SIGNED__CERT" }, { "name": "CHE_GIT_SELF__SIGNED__CERT__HOST" },

also you have the cert files mounts in the spec :
````yaml
"volumeMounts": [
{
"mountPath": "/public-certs",
"name": "che-public-certs"
},
{
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"name": "che-token-xxxxx",
"readOnly": true

````

also you can run "kubectl -it exec pod/che-xxxxxxx -n che -- /bin/bash" which will bring you to a shell in the che server pod. then you may see the env variables set or list the contents of /public-certs or other folders mounted in the pod:
````bash
1724@che-xxxxx-yyyyy:/$ env
KUBERNETES_SERVICE_PORT_HTTPS=443

this is just an example you get lots of output

````
there is a env variable you can set to true for the pod spec called "CHE_DEBUG_SERVER" which may be what you are looking for to a higher verbosity log level.

oops it looks like the yaml alignment was lost when the ingress spec was pasted above.

you can use

````yaml

to open yaml code in a comment and

````

to close it. then change yaml for bash/json :man_shrugging:

I hope some of this can be useful for you to find the source of the error.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.