Che: Failed to run the workspace: "Server 'theia' in container 'theia-idead0' not available."

[alexeykazakov]

I'm trying to start a workspace (tried nodejs and spring boot) but get the following error:

Error: Failed to run the workspace: "Server 'theia' in container 'theia-idead0' not available."

Logs:
https://gist.github.com/alexeykazakov/1f0621d99cf88ee0c07ad2009a4e823e
https://gist.github.com/alexeykazakov/b18284a2431703028c4b2762d6f8984c
https://gist.github.com/alexeykazakov/dfda63119cf48b356f485b7823c9a63c
https://gist.github.com/alexeykazakov/c5f6d45312094f82902e0f9a2608af5f
https://gist.github.com/alexeykazakov/cdca8cc10f596fa905164aacbbf27fde

Events:
https://gist.github.com/alexeykazakov/449b33551843fa051e02702bcc810557

Che version

Che Operator 7.21.1 running on OCP 4.5
Configuration: https://github.com/codeready-toolchain/toolchain-infra/blob/master/config/operators/che/che_cluster.yaml

Additional context

Maybe it's related to the recent change in the operator configuration: https://github.com/codeready-toolchain/toolchain-infra/pull/35

[alexeykazakov]

Reverting single-host change fixed the problem at least for new users. My existing user is now in some broken state.

[ibuziuk]

@alexeykazakov can not reproduce on OSD v4 with the following configuration - https://raw.githubusercontent.com/codeready-toolchain/toolchain-infra/master/config/operators/che/che_cluster.yaml

[sparkoo]

@alexeykazakov do you have che-server logs ? And also events from Che server namespace.

[sleshchenko]

This PR https://github.com/eclipse/che/pull/18083 would help a lot in discovering the cause but I did not manage to finalize it.
Platform team - if you have a chance to apply suggested changes and test it, would be really cool.

[alexeykazakov]

When trying to reproduce then try deploy it without the single host thing first. Then set it to single host. Our setup is not a fresh installation as Ilya tried on OSD.

[alexeykazakov]

I just installed a fresh CRW 2.5.0 operator and faced the same issue.

The logs from the operator che server pod:

2020-11-17 04:33:06,757[nio-8080-exec-9]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 480]   - Starting workspace 'alkazako-osiotest/spring-boot-http-booster-y8atf' with id 'workspaceif7jc5fpsuvdl0nt' by user 'alkazako-osiotest'
2020-11-17 04:33:30,594[aceSharedPool-1]  [INFO ] [.i.k.KubernetesInternalRuntime 652]  - Debug workspace startup. Will watch the logs of 'workspaceif7jc5fpsuvdl0nt'
2020-11-17 04:36:55,149[aceSharedPool-1]  [WARN ] [.i.k.KubernetesInternalRuntime 255]  - Failed to start Kubernetes runtime of workspace workspaceif7jc5fpsuvdl0nt.
org.eclipse.che.api.workspace.server.spi.InfrastructureException: Server 'theia' in container 'theia-idebnv' not available.
    at org.eclipse.che.api.workspace.server.hc.ServerChecker$ServerCheckingTask.run(ServerChecker.java:122)
    at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
    at java.base/java.util.TimerThread.run(Timer.java:506)
2020-11-17 04:37:00,815[aceSharedPool-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 984]   - Workspace 'alkazako-osiotest:spring-boot-http-booster-y8atf' with id 'workspaceif7jc5fpsuvdl0nt' start failed

[ibuziuk]

@alexeykazakov could you provide the logs from the che-server pod?

[alexeykazakov]

Sorry, the logs above are from the Che server pod. Not from the operator pod as I said.