Che: Cannot start workspace

[SkorikSergey]

Describe the bug

Workspace starting fails with Internal Server Error occurred error message.

Failed Jenkins build logs - https://ci.centos.org/view/Devtools/job/devtools-che-nightly-multiuser-stable-test/75/console
Part of che-server log:
2020-04-23 04:30:29,691[nio-8080-exec-4] [ERROR] [c.a.c.r.RuntimeExceptionMapper 47] - Internal Server Error occurred, error time: 2020-04-23 04:30:29 io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.30.0.1/apis/authorization.openshift.io/v1/namespaces/eclipse-che/roles/workspace-stop. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. roles.authorization.openshift.io "workspace-stop" is forbidden: User "system:serviceaccount:eclipse-che:che" cannot get roles.authorization.openshift.io in the namespace "eclipse-che": no RBAC policy matched. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:505) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:395) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:376) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:845) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:214) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:168) at org.eclipse.che.workspace.infrastructure.openshift.provision.OpenShiftStopWorkspaceRoleProvisioner.provision(OpenShiftStopWorkspaceRoleProvisioner.java:53) at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProjectFactory.getOrCreate(OpenShiftProjectFactory.java:100) at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:69) at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:31) at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startAsync(WorkspaceRuntimes.java:466) at org.eclipse.che.api.workspace.server.WorkspaceManager.startAsync(WorkspaceManager.java:488) at org.eclipse.che.api.workspace.server.WorkspaceManager.startWorkspace(WorkspaceManager.java:373) at org.eclipse.che.multiuser.resource.api.workspace.LimitsCheckingWorkspaceManager.startWorkspace(LimitsCheckingWorkspaceManager.java:132) at org.eclipse.che.api.workspace.server.WorkspaceService.startById(WorkspaceService.java:469) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140) at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60) at org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306) at org.everrest.core.impl.RequestDispatcher.invokeSubResourceMethod(RequestDispatcher.java:297) at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:233)

Che version

• [ ] latest
• [x] nightly
• [ ] other: please specify

Steps to reproduce

1. Login to Openshift by oc login command.
2. Download custom resource yaml file from https://raw.githubusercontent.com/eclipse/che-operator/master/deploy/crds/org_v1_che_cr.yaml and set tlsSupport property to false.
3. Deploy Che by chectl server:start -p=openshift --chenamespace=che --che-operator-cr-yaml=/tmp/custom-resource.yaml
4. Create and start workspace from any devfile

Runtime

• [ ] kubernetes (include output of kubectl version)
• [x] Openshift 3.11 (include output of oc version)
• [ ] minikube (include output of minikube version and kubectl version)
• [ ] minishift (include output of minishift version and oc version)
• [ ] docker-desktop + K8S (include output of docker version and kubectl version)
• [ ] other: (please specify)

Screenshots

Installation method

• [x] chectl (chectl/7.11.0 linux-x64 node-v10.19.0)
• [ ] che-operator
• [ ] minishift-addon
• [ ] I don't know

Environment

• [x] my computer
  
  
  
  • [ ] Windows
  
  
  • [x] Linux
  
  
  • [ ] macOS
  
  
• [ ] Cloud
  
  
  
  • [ ] Amazon
  
  
  • [ ] Azure
  
  
  • [ ] GCE
  
  
  • [ ] other (please specify)
  
  
• [ ] other: please specify

[ibuziuk]

@SkorikSergey could you clarify what is the precise installation method?

[dmytro-ndp]

t seems OpenShift specific problem, because it works on minikube it.

[ibuziuk]

@dmytro-ndp @SkorikSergey we need to understand why the CRW tests has passed during the PR check. Any ideas - https://github.com/eclipse/che/pull/16532 ?

[SkorikSergey]

It was started on minikube.

[ibuziuk]

crw-ci-test tests are started against minikube? no coverage of openshift during PR checks?

[SkorikSergey]

[ibuziuk]

ok, so no openshift specific tests are running at all during the PR check, right?
@rhopp is it expected?

[SkorikSergey]

You can start java selenium tests by [ci-test] comment. It doesn't start on PR automatically.

[ibuziuk]

@SkorikSergey so [ci-test] will do the verification against OpenShift?
also, please update the description with the precise installation command

[SkorikSergey]

@SkorikSergey so [ci-test] will do the verification against OpenShift?

Yes. It uses cico_pr_test script.

[ibuziuk]

hmmm.. a couple of things here:

chectl server:start -p=openshift --chenamespace=che --che-operator-cr-yaml=/tmp/custom-resource.yaml

• why the custom-resource yaml isused ?
• it looks like namespace detection does not work correctly

che was installed in the che namespace, but according to the logs the detected namespace was eclipse-che
@SkorikSergey could you please check the KUBERNETES_NAMESPACE env var in the che-server deployment?

[SkorikSergey]

custom-resource.yaml uses to disable tlsSupport property.

[ibuziuk]

@SkorikSergey could you please post tmp/custom-resource.yaml ?
My idea is that KUBERNETES_NAMESPACE / POD_NAMESPACE env vars are not set correctly and do not point to che namespace

[ibuziuk]

PR with hot-fix https://github.com/eclipse/che/pull/16725

[ibuziuk]

@SkorikSergey PR merged. I believe we can close this now