Rh-che: Unable to delete one of my workspaces on che.openshift.io

[benoitf]

Issue problem:
When I try to delete one of my workspace, I have error 500 (I can create/delete others)

{"message":"\nInternal Exception: org.postgresql.util.PSQLException: ERROR: update or delete on table \"workspace\" violates foreign key constraint \"fk_che_k8s_runtime_workspace\" on table \"che_k8s_runtime\"\n  Detail: Key (id)\u003d(workspace09a2fhjrewbl04jj) is still referenced from table \"che_k8s_runtime\".\nError Code: 0\nCall: DELETE FROM workspace WHERE (id \u003d ?)\n\tbind \u003d\u003e [1 parameter bound]\nQuery: DeleteObjectQuery(WorkspaceImpl{id\u003d\u0027workspace09a2fhjrewbl04jj\u0027, namespace\u003d\u0027fbenoit-1\u0027, name\u003d\u0027theia-plugins\u0027, config\u003dWorkspaceConfigImpl{id\u003d795718, name\u003d\u0027theia-plugins\u0027, description\u003d\u0027null\u0027, defaultEnv\u003d\u0027default\u0027, commands\u003d[], projects\u003d[], environments\u003d{default\u003dEnvironmentImpl{id\u003d795719, recipe\u003dRecipeImpl{type\u003d\u0027openshift\u0027, contentType\u003d\u0027application/x-yaml\u0027, content\u003d\u0027kind: List\nitems:\n - \n  apiVersion: v1\n  kind: Pod\n  metadata:\n   name: ws\n  spec:\n   containers:\n    - \n     image: mm4eche/theia-ocp\n     name: theia\n     resources:\n      limits:\n       memory: 1536Mi\n    - \n     image: eclipse/centos_jdk8\n     name: dev\n     resources:\n      limits:\n       memory: 512Mi\n\u0027, location\u003d\u0027null\u0027}, machines\u003d{ws/theia\u003dMachineConfigImpl{id\u003d795720, installers\u003d[], env\u003d{HOSTED_PLUGIN_HOSTNAME\u003d0.0.0.0}, attributes\u003d{memoryLimitBytes\u003d1610612736}, servers\u003d{theia\u003dServerConfigImpl{id\u003d795722, port\u003d\u00273000\u0027, protocol\u003d\u0027http\u0027, path\u003d\u0027null\u0027, attributes\u003d{type\u003dide}}, theia-dev\u003dServerConfigImpl{id\u003d795721, port\u003d\u00273030\u0027, protocol\u003d\u0027http\u0027, path\u003d\u0027null\u0027, attributes\u003d{type\u003dide-dev}}}, volumes\u003d{projects\u003dVolumeImpl{id\u003d795723, path\u003d\u0027/projects\u0027}}}, ws/dev\u003dMachineConfigImpl{id\u003d795724, installers\u003d[], env\u003d{}, attributes\u003d{memoryLimitBytes\u003d536870912}, servers\u003d{theia-dev\u003dServerConfigImpl{id\u003d795725, port\u003d\u00273030\u0027, protocol\u003d\u0027http\u0027, path\u003d\u0027null\u0027, attributes\u003d{}}}, volumes\u003d{projects\u003dVolumeImpl{id\u003d795726, path\u003d\u0027/projects\u0027}}}}}}}, isTemporary\u003dfalse, status\u003dnull, attributes\u003d{updated\u003d1529906452222, stoppedAbnormally\u003dfalse, stackId\u003dstackrb2i6pd9fckfvt0g, stopped\u003d1527761724904, created\u003d1526567127270}, runtime\u003dnull})"}

(sorry the error message is kind of uglified :-/

Red Hat Che version:
6.7.1
version: (help/about menu)

• [X] I can reproduce it on latest official image

Reproduction Steps:
Click on delete of my workspace

Runtime:

runtime used:

• [ ] minishift (include output of minishift version)
• [X] OpenShift.io
• [ ] Openshift Container Platform (include output of oc version)
• [ ] k8s (include output of kubectl version)

[ibuziuk]

@benoitf am I correct that this happens for one particular workspace, not all the workspaces ?

[benoitf]

@ibuziuk yes correct, only one

[garagatyi]

@sleshchenko @skabashnyuk Please, take a look.

[gorkem]

I have 2 such immortal workspaces on my account. Also I think the cleanup logic for cleaning up the PVC of the old data is not really working

[sleshchenko]

@benoitf I tried to simulate such case locally and for me, it's possible to remove such workspaces after rolling update of Che Server.
I prepared PR with cascade removing of Kubernetes runtime when a workspace is removed https://github.com/eclipse/che/pull/10244.
So, after merging it the corresponding error will be logged but workspace will be removed successfully.
Usually, it should not be possible to remove STARTING/RUNNING workspace, it would be nice to understand real cause. Maybe you remember why this workspace can be special?

[benoitf]

@sleshchenko AFAIk nothing special on this workspace.
I just try to cleanup old workspaces after some times.
So I can just say it was a workspace previously created (not a workspace that I created and tried to remove it few seconds after)

[ibuziuk]

@davidfestal got an idea that the workspace that can not be deleted now might be created on osio when there was some bug with che not able to start workspace which crashed and can not be deleted ever after.
@davidfestal could you please add more details ?

[sleshchenko]

@ibuziuk I have merged the PR with cascade removing of K8s runtime https://github.com/eclipse/che/pull/10244. I think you can close this issue after update OpenShift.io to next release, or maybe you need to backport this fix. WDYT?
P.S. Does someone analyze Che Server logs on OpenShift.io? Is there a chance that error related to cascade removing will be detected if it appears again?

[ibuziuk]

@sleshchenko looks like we still have this problem even after update to 6.10.0 - https://github.com/openshiftio/openshift.io/issues/4271

[benoitf]

I was able to remove them with 6.10.0 update !

[ibuziuk]

@sleshchenko sorry, false alarm ;-)

[sleshchenko]

@ibuziuk no problem.
Good to see that it is not reproducible anymore.