Che: Deployment is not cleaned up if a workspace start fails because of pod creation timeout

[sleshchenko]

Description

Deployment is not cleaned up if a workspace start fails because of pod creation timeout.

I faced an issue when a workspace is failed to start

But workspace deployment is not cleaned up

And then it is impossible to start workspace because of deployment already exists error

Reproduction Steps

1. Create a workspace from broken recipe where it is supposed that pod creation timeout happens
   
   
   

   Workspace is non-existing service account specified

{
      "environments": {
        "default": {
          "machines": {
            "ws/theia": {
              "attributes": {},
              "servers": {
                "theia": {
                  "protocol": "http",
                  "port": "3000",
                  "path": "/",
                  "attributes": {
                    "type": "ide"
                  }
                },
                "theia-dev": {
                  "protocol": "http",
                  "port": "3030",
                  "attributes": {
                    "type": "ide-dev"
                  }
                }
              },
              "volumes": {
                "projects": {
                  "path": "/projects"
                }
              },
              "installers": [],
              "env": {
                "HOSTED_PLUGIN_HOSTNAME": "0.0.0.0"
              }
            },
            "ws/dev": {
              "attributes": {},
              "servers": {},
              "volumes": {
                "projects": {
                  "path": "/projects"
                }
              },
              "installers": [],
              "env": {}
            }
          },
          "recipe": {
            "type": "openshift",
            "content": "---\nkind: List\nitems:\n-\n  apiVersion: v1\n  kind: Pod\n  metadata:\n    name: ws\n  spec:\n    containers:\n      -\n        image: eclipse/che-theia:0.3.14-nightly\n        name: theia\n      -\n        image: eclipse/che-dev:nightly\n        name: dev\n    serviceAccountName: non-existing",
            "contentType": "application/x-yaml"
          }
        }
      },
      "defaultEnv": "default",
      "name": "theia",
      "projects": [],
      "commands": []
    }

1. Start the created workspace. Start should fail because of pod creation timeout error.
2. Update the workspace with correct configuration
   
   
   

   Workspace is non-existing service account specified

{
      "environments": {
        "default": {
          "machines": {
            "ws/theia": {
              "attributes": {},
              "servers": {
                "theia": {
                  "protocol": "http",
                  "port": "3000",
                  "path": "/",
                  "attributes": {
                    "type": "ide"
                  }
                },
                "theia-dev": {
                  "protocol": "http",
                  "port": "3030",
                  "attributes": {
                    "type": "ide-dev"
                  }
                }
              },
              "volumes": {
                "projects": {
                  "path": "/projects"
                }
              },
              "installers": [],
              "env": {
                "HOSTED_PLUGIN_HOSTNAME": "0.0.0.0"
              }
            },
            "ws/dev": {
              "attributes": {},
              "servers": {},
              "volumes": {
                "projects": {
                  "path": "/projects"
                }
              },
              "installers": [],
              "env": {}
            }
          },
          "recipe": {
            "type": "openshift",
            "content": "---\nkind: List\nitems:\n-\n  apiVersion: v1\n  kind: Pod\n  metadata:\n    name: ws\n  spec:\n    containers:\n      -\n        image: eclipse/che-theia:0.3.14-nightly\n        name: theia\n      -\n        image: eclipse/che-dev:nightly\n        name: dev\n",
            "contentType": "application/x-yaml"
          }
        }
      },
      "defaultEnv": "default",
      "name": "theia",
      "projects": [],
      "commands": []
    }

1. Start the workspace again

Expected: The workspace is started.
Actual: The workspace failed to start because deployment already exists error.

I think it can be reproduced on Kubernetes and OpenShift infrastructures.

[garagatyi]

@amisevsk FYI

[ibuziuk]

@sleshchenko @amisevsk guys, what is the workaround for this issue (manual removal of the deployment) ?

[amisevsk]

@ibuziuk That is the only way to resolve it pre-PR #11253

Closing since #11253 is merged.