Rh-che: Number of running workspace on prod reported by osd monitor is incorrect

[ibuziuk]

che_workspace_status{area="workspace",status="RUNNING",} 593.0

In reality the number of running workspaces should be much lower. Need to investigate the reason of such a big number of workspaces marked as Running in DB. Potentaially, fixing https://github.com/redhat-developer/rh-che/issues/1175 might tackle this problem

[ibuziuk]

Number of running workspaces dropped from ~ 600 to ~100 but this figure still does not correlate with the info from zabbix. Further investigation is required

[amisevsk]

Should be resolved by PRs https://github.com/redhat-developer/rh-che/pull/1255, https://github.com/eclipse/che/pull/12748/, and https://github.com/eclipse/che/pull/12738. The issue, as far as I can tell comes from three things:

• We don't have useful logging for the runtimes consistency check task -- it only logs when it encounters an inconsistent runtime or error. It should log every time it runs and include the number of workspaces it is checking (PR https://github.com/eclipse/che/pull/12738)
• Workspace recovery can fail if an unchecked exception is thrown while recovering -- this means that the runtime cache does not get properly populated and so the consistency check does not check all workspaces (a good reason to log number of workspaces being consistency checked) (PR https://github.com/eclipse/che/pull/12748/)
• TenantDataProvider can fail because of unchecked exception, which can interfere with some workspace recovery, since KubernetesInternalRuntime cannot be obtained (PR https://github.com/redhat-developer/rh-che/pull/1255)

The basic reason for the discrepancy (I think -- it's hard to test) is that workspace recovery semi-silently fails:

From Kibana logs:

java.lang.IllegalArgumentException: Environment contains '2' machines with wsagent server. Machines names: '[web/main, web/mysql]'
    at org.eclipse.che.api.workspace.server.WsAgentMachineFinderUtil.getWsAgentServerMachine(WsAgentMachineFinderUtil.java:63)
    at org.eclipse.che.api.workspace.server.spi.provision.ProjectsVolumeForWsAgentProvisioner.provision(ProjectsVolumeForWsAgentProvisioner.java:51)
    at org.eclipse.che.api.workspace.server.spi.RuntimeInfrastructure.prepare(RuntimeInfrastructure.java:102)
    at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.recoverOne(WorkspaceRuntimes.java:675)
    at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$RecoverRuntimesTask.run(WorkspaceRuntimes.java:621)
    at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Once this happens the workspaces being checked in the consistency check do not include any workspaces coming after the workspace that causes the error above, and so most of the RUNNING workspaces are never checked for consistency.

As far as I can tell, the consistency check works as expected (there are successful log messages for inconsistent workspaces), so fixing the above should resolve our problem.

[ibuziuk]

Closing since there is pending PR that is going to be merged soon