Rh-che: Ephemeral workspace is failing to start due to "the node was low on resource ephemeral-storage"

[ibuziuk]

So, far was able to repro only once with che 7 ephemeral workspace:

[ibuziuk]

@l0rd @pbergene wdyt?

[pbergene]

Ephemeral storage isn't an infinite resource, so would have to scope out requirements vs what is available etc.

[ibuziuk]

@pbergene and what is the current state? how much ephemeral storage is available per node?

[Katka92]

This issue is monitored by our periodic tests and happens almost every day. The test is running Vert.x and Maven stack, the issue is reproducible for both of them. From time to time workspace failed on starting, but more often workspace is switched to offline while running a maven build or starting/testing LS server. In the events, we can see that the reason is low ephemeral storage.

[Katka92]

Today we've seen another issue than can probably be related to Evicted. The maven build and run tasks failed with Terminal failed to connect. See more in 'Output'.. There is an evicted event in events report.

[1] events: http://artifacts.ci.centos.org/devtools/rhche/devtools-rh-che-periodic-prod-2/1956/events_report.txt
[2] Screenshot from the test: http://artifacts.ci.centos.org/devtools/rhche/devtools-rh-che-periodic-prod-2/1956/e2e_report/report/RhChe_E2E_Java_Maven_test_Validation_of_workspace_build_and_run_Build_application/screenshot-Build_application.png

[ScrewTSW]

Seen today twice at production cluster 2 x2
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/2072/console
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/2073/console
and on production cluster 2a x1
https://ci.centos.org/view/Devtools/job/devtools-rh-che-periodic-prod-2aProd/1861/consoleFull
and on prod-preview cluster 2a x1
https://ci.centos.org/job/devtools-rh-che-periodic-prod-preview-2a/2074/console

[ScrewTSW]

Seen again on production cluster 1b x1
https://ci.centos.org/view/Devtools/job/devtools-rh-che-periodic-prod-1b/1873/console

[Katka92]

We are still observing this issue.

[ibuziuk]

Closing in favour of https://github.com/redhat-developer/rh-che/issues/1770