Rh-che: [Infra issue] Workspace switching to offline randomly - Evicted

Created on 4 Oct 2019 · 37Comments · Source: redhat-developer/rh-che

Offline state during application build caused test failure

Offline state during project tree expansion caused test failure

This issue happened before once, but we thought that it was just a fringe occurence.
This time it happened twice on 1b cluster

EDIT:

Confirmed @amisevsk 's suspicion that this issue is connected to the Eviction errors (https://github.com/redhat-developer/rh-che/issues/1523)
When looking into the events of the workspaces that have been switched to offline mode, we can see that the eviction happens once the workspace is already started, causing the offline message:

{
    "kind": "Event",
    "apiVersion": "v1",
    "metadata": {
        "name": "workspacevigbckpth2bhbelq.maven-6f6f66d5d8-pj8rf.15d66587ca08e376",
        "namespace": "osiotest2-che",
        "selfLink": "/api/v1/namespaces/osiotest2-che/events/workspacevigbckpth2bhbelq.maven-6f6f66d5d8-pj8rf.15d66587ca08e376",
        "uid": "64e3ef3e-053c-11ea-95d8-02e52a0be43d",
        "resourceVersion": "1954991043",
        "creationTimestamp": "2019-11-12T11:05:53Z",
        "annotations": {
            "offending_containers": "theia-idee7w,maven,che-machine-execgwt,vscode-javazjk",
            "offending_containers_usage": "58284Ki,280Ki,252Ki,48592Ki",
            "starved_resource": "ephemeral-storage"
        }
    },
    "involvedObject": {
        "kind": "Pod",
        "namespace": "osiotest2-che",
        "name": "workspacevigbckpth2bhbelq.maven-6f6f66d5d8-pj8rf",
        "uid": "2c3a4a09-053c-11ea-a788-0233cba325d9",
        "apiVersion": "v1",
        "resourceVersion": "1954990443"
    },
    "reason": "Evicted",
    "message": "The node was low on resource: ephemeral-storage. Container theia-idee7w was using 58284Ki, which exceeds its request of 0. Container maven was using 280Ki, which exceeds its request of 0. Container che-machine-execgwt was using 252Ki, which exceeds its request of 0. Container vscode-javazjk was using 48592Ki, which exceeds its request of 0. ",
    "source": {
        "component": "kubelet",
        "host": "ip-172-31-65-163.us-east-2.compute.internal"
    },
    "firstTimestamp": "2019-11-12T11:05:53Z",
    "lastTimestamp": "2019-11-12T11:05:53Z",
    "count": 1,
    "type": "Warning",
    "eventTime": null,
    "reportingComponent": "",
    "reportingInstance": ""
}

Example of an event.

kinperiodic-e2e-test tearhche-qe

Source

ScrewTSW

👍1

All 37 comments

Seen again, caused LS initialization tests failed.
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/1521/console

Katka92 on 7 Oct 2019

Workspace went off during LS initialization on cluster 2
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/1526/console

Katka92 on 8 Oct 2019

Happened today again during project tree expansion, tests couldn't open the java file on cluster 1a

ScrewTSW on 16 Oct 2019

Happened twice on 1a cluster during project tree expansion

ScrewTSW on 17 Oct 2019

Seen once on 1a

ScrewTSW on 18 Oct 2019

Seen again 1x on 2
on 1a workspace became offline during the execution of build command 2x
https://ci.centos.org/job/devtools-rh-che-periodic-prod-1a/1443/console
https://ci.centos.org/job/devtools-rh-che-periodic-prod-1a/1444/console

Katka92 on 29 Oct 2019

Happened yesterday twice on 2 production cluster

ScrewTSW on 30 Oct 2019

Happened 10x on 1a cluster
1x on 1b cluster
1x on 2a-production

ScrewTSW on 11 Nov 2019

happened again on production-2 1x
production-1a 2x
production-1b 1x

ScrewTSW on 12 Nov 2019

Happened 4 times on production-2 cluster

ScrewTSW on 13 Nov 2019

Happened again on production-2 1x

{
    "kind": "Event",
    "apiVersion": "v1",
    "metadata": {
        "name": "workspaceqccdnnwj7vweflkm.maven-5d7944694d-4dwk7.15ce0bfb0a8c0f48",
        "namespace": "osiotest1a-che",
        "selfLink": "/api/v1/namespaces/osiotest1a-che/events/workspaceqccdnnwj7vweflkm.maven-5d7944694d-4dwk7.15ce0bfb0a8c0f48",
        "uid": "4442531c-efdc-11e9-b0d7-121912317b4c",
        "resourceVersion": "750216911",
        "creationTimestamp": "2019-10-16T06:14:52Z",
        "annotations": {
            "offending_containers": "vscode-javalnq,maven,theia-ide3x8,che-machine-exec6kx",
            "offending_containers_usage": "680Ki,684Ki,684Ki,676Ki",
            "starved_resource": "ephemeral-storage"
        }
    },
    "involvedObject": {
        "kind": "Pod",
        "namespace": "osiotest1a-che",
        "name": "workspaceqccdnnwj7vweflkm.maven-5d7944694d-4dwk7",
        "uid": "2a1d1eab-efdc-11e9-a0c7-125c769592ce",
        "apiVersion": "v1",
        "resourceVersion": "750215929"
    },
    "reason": "Evicted",
    "message": "The node was low on resource: ephemeral-storage. Container vscode-javalnq was using 680Ki, which exceeds its request of 0. Container maven was using 684Ki, which exceeds its request of 0. Container theia-ide3x8 was using 684Ki, which exceeds its request of 0. Container che-machine-exec6kx was using 676Ki, which exceeds its request of 0. ",
    "source": {
        "component": "kubelet",
        "host": "ip-172-21-54-12.ec2.internal"
    },
    "firstTimestamp": "2019-10-16T06:14:52Z",
    "lastTimestamp": "2019-10-16T06:14:52Z",
    "count": 1,
    "type": "Warning",
    "eventTime": null,
    "reportingComponent": "",
    "reportingInstance": ""
}

ScrewTSW on 14 Nov 2019

@ScrewTSW do you have logs from che-master?

skabashnyuk on 14 Nov 2019

@skabashnyuk From what I've seen, nothing special is logged on the server side when this occurs, the pod is just restarted by OpenShift.

amisevsk on 14 Nov 2019

This issue was seen again on 1a cluster https://ci.centos.org/job/devtools-rh-che-periodic-prod-1b/1561/console and cluster 2 4x https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/1770/consoleFull

Katka92 on 18 Nov 2019

Seen again on cluster 2 1x.

@ibuziuk According to the message [1] it seems like problems with the size of ephemeral storage. Regarding discussion with Radim, it seems to be Hosted Che related. It happens quite a lot so maybe it would be worth discussing with service delivery. They should see some peaks in disk usage [2]. In this report [3] there are twelve of Evicted messages, so you can read through if you would like.

[1]
"reason": "Evicted", "message": "The node was low on resource: ephemeral-storage. Container vscode-javayen was using 48592Ki, which exceeds its request of 0. Container che-machine-execw1y was using 244Ki, which exceeds its request of 0. Container maven was using 248Ki, which exceeds its request of 0. Container theia-idedjt was using 58040Ki, which exceeds its request of 0. "

[2]
"reason": "Evicted", "message": "The node was low on resource: [DiskPressure]. "

[3] http://artifacts.ci.centos.org/devtools/rhche/devtools-rh-che-periodic-prod-2/1781/events_report.txt

Katka92 on 19 Nov 2019

Seen again on cluster 2 2x and 1x on 1a. On 1a one workspace was not started because of eviction.

Katka92 on 21 Nov 2019

👍1

I expect it's directly related to the workspace idling issue -- it seems to come back when there are a ton of unstopped workspaces running. E.g. I could reproduce consistently before but not directly after a rollout.

amisevsk on 21 Nov 2019

Seen again on cluster 2 2x and on cluster 1a 3x.
I'm not sure if it is directly related to the workspace idling issue as far as it seems that workspaces were killed at ~10:10 and some failures happened at ~14:56. I would say that the idling issue made it easily reproducible but it seems there are still problems with resources on ephemeral storage.

Katka92 on 22 Nov 2019

👍1

Seen again on production 2 x6
prod-preview 2a x2

{
    "kind": "Event",
    "apiVersion": "v1",
    "metadata": {
        "name": "workspacerodpsn7u1cmy3e02.maven-c4cf469b-6x69r.15d99e49065efbf8",
        "namespace": "osiotest2-che",
        "selfLink": "/api/v1/namespaces/osiotest2-che/events/workspacerodpsn7u1cmy3e02.maven-c4cf469b-6x69r.15d99e49065efbf8",
        "uid": "c43dfe4e-0d7b-11ea-95d8-02e52a0be43d",
        "resourceVersion": "1964177575",
        "creationTimestamp": "2019-11-22T22:59:41Z",
        "annotations": {
            "offending_containers": "theia-ide5vb,vscode-javadia,che-machine-execdtp,maven",
            "offending_containers_usage": "252Ki,244Ki,244Ki,248Ki",
            "starved_resource": "ephemeral-storage"
        }
    },
    "involvedObject": {
        "kind": "Pod",
        "namespace": "osiotest2-che",
        "name": "workspacerodpsn7u1cmy3e02.maven-c4cf469b-6x69r",
        "uid": "9d285659-0d7b-11ea-a788-0233cba325d9",
        "apiVersion": "v1",
        "resourceVersion": "1964177140"
    },
    "reason": "Evicted",
    "message": "The node was low on resource: ephemeral-storage. Container theia-ide5vb was using 252Ki, which exceeds its request of 0. Container vscode-javadia was using 244Ki, which exceeds its request of 0. Container che-machine-execdtp was using 244Ki, which exceeds its request of 0. Container maven was using 248Ki, which exceeds its request of 0. ",
    "source": {
        "component": "kubelet",
        "host": "ip-172-31-65-163.us-east-2.compute.internal"
    },
    "firstTimestamp": "2019-11-22T22:59:41Z",
    "lastTimestamp": "2019-11-22T22:59:41Z",
    "count": 1,
    "type": "Warning",
    "eventTime": null,
    "reportingComponent": "",
    "reportingInstance": ""
}

ScrewTSW on 25 Nov 2019

Happened once on 2a produciton cluster
https://ci.centos.org/view/Devtools/job/devtools-rh-che-periodic-prod-2aProd/1610/consoleFull

{
    "kind": "Event",
    "apiVersion": "v1",
    "metadata": {
        "name": "workspacesh5otjw0i4fivm0d.maven-574b8bbf8b-d6cv7.15da6d79c309ccb4",
        "namespace": "osiotest2aprod2-che",
        "selfLink": "/api/v1/namespaces/osiotest2aprod2-che/events/workspacesh5otjw0i4fivm0d.maven-574b8bbf8b-d6cv7.15da6d79c309ccb4",
        "uid": "2c9143fb-0f8e-11ea-8c1a-029a5a55534e",
        "resourceVersion": "1006097833",
        "creationTimestamp": "2019-11-25T14:16:29Z",
        "annotations": {
            "offending_containers": "vscode-javatct,maven,che-machine-execg94,theia-ide6ab",
            "offending_containers_usage": "692Ki,696Ki,692Ki,700Ki",
            "starved_resource": "ephemeral-storage"
        }
    },
    "involvedObject": {
        "kind": "Pod",
        "namespace": "osiotest2aprod2-che",
        "name": "workspacesh5otjw0i4fivm0d.maven-574b8bbf8b-d6cv7",
        "uid": "10f7714a-0f8e-11ea-8c1a-029a5a55534e",
        "apiVersion": "v1",
        "resourceVersion": "1006097091"
    },
    "reason": "Evicted",
    "message": "The node was low on resource: ephemeral-storage. Container vscode-javatct was using 692Ki, which exceeds its request of 0. Container maven was using 696Ki, which exceeds its request of 0. Container che-machine-execg94 was using 692Ki, which exceeds its request of 0. Container theia-ide6ab was using 700Ki, which exceeds its request of 0. ",
    "source": {
        "component": "kubelet",
        "host": "ip-172-31-68-88.us-east-2.compute.internal"
    },
    "firstTimestamp": "2019-11-25T14:16:29Z",
    "lastTimestamp": "2019-11-25T14:16:29Z",
    "count": 1,
    "type": "Warning",
    "eventTime": null,
    "reportingComponent": "",
    "reportingInstance": ""
}

ScrewTSW on 26 Nov 2019

Seen on production 2 x2
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/1834/consoleFull
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/1835/consoleFull

ScrewTSW on 28 Nov 2019

seen again on prod-preview 2a x1
https://ci.centos.org/job/devtools-rh-che-periodic-prod-preview-2a/1842/consoleFull

{
    "kind": "Event",
    "apiVersion": "v1",
    "metadata": {
        "name": "workspace4j2yyqaafd0s8m4d.maven-5c44f87d8d-hrqqw.15db75a4a6802ad5",
        "namespace": "osio-ci-preview-testper-che",
        "selfLink": "/api/v1/namespaces/osio-ci-preview-testper-che/events/workspace4j2yyqaafd0s8m4d.maven-5c44f87d8d-hrqqw.15db75a4a6802ad5",
        "uid": "7158eb5a-1232-11ea-8c1a-029a5a55534e",
        "resourceVersion": "1007879308",
        "creationTimestamp": "2019-11-28T22:57:24Z",
        "annotations": {
            "offending_containers": "che-machine-exec8sv,vscode-javacge,theia-ideuw8,maven",
            "offending_containers_usage": "692Ki,692Ki,700Ki,696Ki",
            "starved_resource": "ephemeral-storage"
        }
    },
    "involvedObject": {
        "kind": "Pod",
        "namespace": "osio-ci-preview-testper-che",
        "name": "workspace4j2yyqaafd0s8m4d.maven-5c44f87d8d-hrqqw",
        "uid": "589c8e04-1232-11ea-8c1a-029a5a55534e",
        "apiVersion": "v1",
        "resourceVersion": "1007878207"
    },
    "reason": "Evicted",
    "message": "The node was low on resource: ephemeral-storage. Container che-machine-exec8sv was using 692Ki, which exceeds its request of 0. Container vscode-javacge was using 692Ki, which exceeds its request of 0. Container theia-ideuw8 was using 700Ki, which exceeds its request of 0. Container maven was using 696Ki, which exceeds its request of 0. ",
    "source": {
        "component": "kubelet",
        "host": "ip-172-31-69-247.us-east-2.compute.internal"
    },
    "firstTimestamp": "2019-11-28T22:57:24Z",
    "lastTimestamp": "2019-11-28T22:57:24Z",
    "count": 1,
    "type": "Warning",
    "eventTime": null,
    "reportingComponent": "",
    "reportingInstance": ""
}

ScrewTSW on 29 Nov 2019

Seen again on cluster 2 2x, 1a 1x.

Katka92 on 2 Dec 2019

Seen again on cluster 2 2x.

Katka92 on 3 Dec 2019

Seen again on cluster 2 2x. Workspce not started due to Evicted state 1x on production 2a.
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2aProd/1662/console

Katka92 on 4 Dec 2019

Seen again on cluster 2 1x

Katka92 on 5 Dec 2019

Seen again on cluster 2 2x
Workspace was not start due to Evicted on cluster 2 2x

Katka92 on 6 Dec 2019

Seen again on cluster 1a 1x https://ci.centos.org/job/devtools-rh-che-periodic-prod-1a/1702/console

Katka92 on 11 Dec 2019

Seen again on cluster 2 1x

Katka92 on 13 Dec 2019

Seen again on cluster 2 3x, 2a 1x, 1a 3x.

Katka92 on 16 Dec 2019

Closing this issue in favor of https://github.com/redhat-developer/rh-che/issues/1372.
According to an agreement with @ibuziuk we will not do our standard daily comments to the PR 1372. If needed engineers will track that on Slack channel daily reports.
cc @ScrewTSW

Katka92 on 16 Dec 2019

I'm necroing this issue :D

Seen today again on production cluster 2 x1:
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/2291/console

Evicted event present in logs

ScrewTSW on 12 Feb 2020

😄1

Seen again on production cluster 2 x1:
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/2344/console

Evicted present in events

ScrewTSW on 21 Feb 2020

Not observed for 20 days. Closing.

Katka92 on 12 Mar 2020

Seen again on cluster 2 1x - workspace didn't start
https://ci.centos.org/job/devtools-rh-che-periodic-prod-2/2605/consoleFull