Che: kubernetes "single-host" configuration fails to run new workspaces

[geiseri]

I am running a 3 node kubernets cluster running single-host mode with 6.19.0 and it works fine for normal projects. I updated to rc this weekend and found that trying to use the che-7 IDEs did not correctly load the workspace. I see the following error:

Error: Workspace doesn't have a server which matches with URL: http://che.geekcentral.local/?uid=978289

This is the stacktrace from the console:

che.geekcentral.local-1553541995836.log

This is the HAR of the loading the workspace:

che.geekcentral.local.zip

If you need any other logs I can try to extract/clean up the logs from the containers, but I was unsure of what ones to post. I did try the old style projects specificly the "Default Node 8 Stack" and it did work as it did in 6.19.0.

[sleshchenko]

The issue you faced is related to secure servers behind JWTProxy. I think it does not work with single-host mode. Are you able to use multi-host mode?

If no - then as workaround you can disable JWTProxy based secure servers for your workspace by specifying che.server.secure_exposer attribute with default value in workspace config. Note that it is potentially dangerous since Theia IDE does not support legacy default workspace auth mechanism and workspace servers may be called without any token if URL is known.

[geiseri]

The VPN we are behind doesn't support wildcards, and only works with names or ip addresses so it wouldn't work for production but if it helps narrow down the problem I can give logs. One suggestion from our sysadmin was that if we could go behind our nginx server on the edge we could enable wildcard SSL/dns there. We tried that in our testing with 6.19 on docker, but it didn't work at all.

[geiseri]

Also, we are using http, not https. Would this work if we used https instead?

[garagatyi]

Thanks for providing HAR of requests.
I found in HAR next request that when gets redirected looses path to a server https://gist.github.com/garagatyi/0fb187810cd1c0145ed93af0584d4ad8
Which is probably a bug in our JWT proxy implementation

[geiseri]

Any progress on this? It sounds like we could set up multihost on our edge ingress. It is running nginx, so I am not sure if che will like being behind two reverse proxies. Do you know if that would mess stuff up? They have a wildcard cert on that domain, so I think this also has a problem with che using external SSL.

[akervern]

I encountered the same issue with a fresh Helm installation on GKE.
I configured the chart to enable TLS (with letsencrypt), so single-host seems to be the only viable server strategy.

[slemeur]

@skabashnyuk : Are you aware of this issue ?

[skabashnyuk]

yes https://github.com/eclipse/che/issues/12988

[slemeur]

@skabashnyuk : Do you have a workaround for this? Are you planning to take this issue in your sprint soon?

[skabashnyuk]

Not in the plan at this moment. Planned "After GA" stage

[slemeur]

Is there a workaround for this?

[skabashnyuk]

I don't know. We don't know the cause of https://github.com/eclipse/che/issues/12988 yet.

[slemeur]

Switching the milestone to 7.0.0 as this is a must have.

[sleshchenko]

There are some issues with single-host and default-host servers strategies which use path-based rules for ingresses. It's related to rewrite-target to /, and I assume that it worked somehow with GWT IDE when it was hosted on Che Master but it does not work with Theia in a sidecar.
A brief explanation of the issue with rewrite-target and path-based rules:

1. Theia has Ingress that is configured to path rule like server3421-theia-idesmd/server-4400. So, user should use http://{HOST}/server3421-theia-idesmd/server-4400 to access Theia.
2. When a user opens Theia server the main script tries to download like cdn-resource.js, but Ingress rewrote target and Theia thinks that it is located at /, so it tries to download script from http://{HOST}/cdn-resource.js instead of right URL http://{HOST}/server3421-theia-idesmd/server-4400/cdn-resource.js

If we disable rewrite-target then we will have another issue when a client will try to download cdn-resource.js from right http://{HOST}/server3421-theia-idesmd/server-4400/cdn-resource.js BUT Theia server is not able to handle server3421-theia-idesmd/server-4400 prefix path.

The possible solutions for single-host and default-host should be investigated.

[l0rd]

@sleshchenko the problem is only cdn-resource.js? Maybe a workaround would be to copy the static resource in the che-server container so that tomcat is able to serve it?

[l0rd]

cc @davidfestal

[benoitf]

it's every resources

you can ask /w/h/a/t/e/v/e/r it will redirect to /

cdn.js is listed just because it's the first resource loaded

[l0rd]

@benoitf got it

[skabashnyuk]

@geiseri your original bug related to Che7 workspaces or Che6. Can you clarify the scope of your request? As far as I can see single-host more for Che7(Theia based) workspace never worked before.

[bmangaraj]

@skabashnyuk
I am also facing the same issue while loading the workspace.
Error: Workspace doesn't have a server which matches with URL: http://ip-address/?uid=645886

I am using Che7 and below command for installation:
sudo helm upgrade --install che --namespace che -f ./values/default-host.yaml --set global.ingressDomain=abc.com ./

Though I am giving the domain name in ingressDomain, I am not sure why the ip address is coming in that url in the above error.

I am able to see the dashboard, but after workspace creation, workspace loading fails by giving the above error

Please help if anyone have resolved this issue.

[skabashnyuk]

@bmangaraj at this moment for Che 7 the only one available workaround is to use multi-host https://github.com/eclipse/che/issues/13869#issuecomment-513751078

[slemeur]

@bmangaraj : Could you please explain the reason why you would like to deploy in single-host mode? Would the multi-host be convenient for you?

The solution for single-host might get tricky to get, so while we are looking for a solution and get this fixed, I'd recommend a deployment with multi-host mode for the time being.

[l0rd]

@slemeur I think the main problem is that when a user enable TLS, single-host and multi-user are enabled as well. So even if a user doesn't have any particular preference, single-host is automatically enabled.

[bmangaraj]

@slemeur
I tried multi user with Keycloak mode using nip.io

sudo helm upgrade --install che --force --namespace che -f ./values/multi-user.yaml --set global.ingressDomain=10.9.2.65.nip.io --set cheImage=eclipse/che-server:7.0.0-RC-2.0 ./

==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
che-ingress che-che.10.9.2.65.nip.io 80 0s
keycloak-ingress keycloak-che.10.9.2.65.nip.io 80 0s

Our corporate policy blocks nip.io url,
So I had to add nip.io urls into the exception list so that my network is allowing now, then che is starting.
Also I am able to load the workspace.

But Can't I use domain name or my IP address instead of nip.io in multi user keycloak mode So that I can run the che in http:// domain name or IP_ Address?
I don't want che-che or che-keycloak to be prefixed with the domain name.
For this I tried below approach. But using IP address instead of .nip.io by setting isHostBased = false does not work

sudo helm upgrade --install che --force --namespace che -f ./values/multi-user.yaml --set global.ingressDomain=<ip_address> --set isHostBased=false --set cheImage=eclipse/che-server:7.0.0-RC-2.0 ./

==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
che-ingress che-che. 80 1s
keycloak-ingress keycloak-che. 80 1s

Error:(Che is not starting)
Caused by: java.lang.RuntimeException: Exception while retrieving OpenId configuration
from endpoint: http://keycloak-che./auth/realms/che/.well-known/openid-configuration

[slemeur]

Moving to 7.1.0 milestone, as for 7.0.0 we will prioritize to address multi-host with TLS.

[slemeur]

Linking to: https://github.com/eclipse/che/issues/13986

[metlos]

So the fix for this has just been merged into che server master branch. It would be great if you could take it for a spin on your side @geiseri to make sure all is working as it should. The fix should be included in the next nightly build of the che server.

Please re-open this issue or file a new one if you find any further problems.

Thanks for reporting the issue!

[leader-us]

7.2.snapshort not resolved this problem yet ,I tried latest 7.2 snapshort ,also this problem

14496