Kubernetes-client: Confusing message from kubernetes-client: object is being deleted: secrets "XXXX" already exists

[skabashnyuk]

During studying this https://github.com/eclipse/che/issues/16512 that might be related with upgrade of kubernetes-client from 4.1.0 to 4.9.0 https://github.com/eclipse/che-parent/pull/159 https://github.com/eclipse/che/pull/16345 we noticed very strange error.

aused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.30.0.1/api/v1/namespaces/skabashn/secrets. Message: object is being deleted: secrets "workspaceyy5nesnxw954tsbz-sshprivatekeys" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=secrets, name=workspaceyy5nesnxw954tsbz-sshprivatekeys, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=object is being deleted: secrets "workspaceyy5nesnxw954tsbz-sshprivatekeys" already exists, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}).
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:251)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:815)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:333)
    at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesSecrets.create(KubernetesSecrets.java:49)
    ... 10 common frames omitted

Any ideas?

[rohanKanojia]

@skabashnyuk : Hi Sergi, Could you please share some more details on how to reproduce this? Is it during createOrReplace() operation for a secret?

[skabashnyuk]

@rohanKanojia full stack trace.

2020-04-03 09:51:09,540[ceSharedPool-26]  [ERROR] [.i.k.KubernetesInternalRuntime 259]  - Failure executing: POST at: https://172.30.0.1/api/v1/namespaces/skabashn/secrets. Message: object is being deleted: secrets "workspaceyy5nesnxw954tsbz-sshprivatekeys" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=secrets, name=workspaceyy5nesnxw954tsbz-sshprivatekeys, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=object is being deleted: secrets "workspaceyy5nesnxw954tsbz-sshprivatekeys" already exists, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}).
org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInfrastructureException: Failure executing: POST at: https://172.30.0.1/api/v1/namespaces/skabashn/secrets. Message: object is being deleted: secrets "workspaceyy5nesnxw954tsbz-sshprivatekeys" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=secrets, name=workspaceyy5nesnxw954tsbz-sshprivatekeys, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=object is being deleted: secrets "workspaceyy5nesnxw954tsbz-sshprivatekeys" already exists, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}).
    at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesSecrets.create(KubernetesSecrets.java:51)
    at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftInternalRuntime.createSecrets(OpenShiftInternalRuntime.java:127)
    at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftInternalRuntime.startMachines(OpenShiftInternalRuntime.java:112)
    at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:222)
    at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141)
    at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:920)
    at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
    at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.30.0.1/api/v1/namespaces/skabashn/secrets. Message: object is being deleted: secrets "workspaceyy5nesnxw954tsbz-sshprivatekeys" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=secrets, name=workspaceyy5nesnxw954tsbz-sshprivatekeys, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=object is being deleted: secrets "workspaceyy5nesnxw954tsbz-sshprivatekeys" already exists, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}).
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:251)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:815)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:333)
    at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesSecrets.create(KubernetesSecrets.java:49)
    ... 10 common frames omitted

I reproduce it on Openshift dedicated when I simultaneously very quickly start/stop 5-6 Eclipse Che workspaces. My assumption that secrets deleted but not GCd and this happens when I try to create a secret with the same name again.

[skabashnyuk]

One more thing when I've added https://github.com/eclipse/che/pull/16540 .withPropagationPolicy("Foreground") to every delete operation - things start working as expected.

[rohanKanojia]

I tried reproducing this error with this sample of code but could not reproduce the exact error you faced. But as you pointed out, I also noticed that objects are not being garbage-collected.

public class SecretDeletion {
    public static void main(String[] args) {
        String namespace = "default";

        try (KubernetesClient client = new DefaultKubernetesClient()) {
            for (int i = 0; i < 20; i++) {
                Secret secret = getSecret();
                System.out.println("Creating secret");
                client.secrets().inNamespace(namespace).create(secret);

                System.out.println("Deleting secret");
                /*

                 *
                 */
                client.secrets().inNamespace(namespace).withLabel("foo", "bar").delete();
            }
        }
    }

    private static Secret getSecret() {
        return new SecretBuilder()
                .withNewMetadata()
                .withName("mysecret")
                .addToLabels("foo", "bar")
                .endMetadata()
                .withType("Opaque")
                .addToData("username", "YWRtaW4=")
                .addToData("password", "MWYyZDFlMmU2N2Rm")
                .build();
    }
}

Setting a breakpoint in OperationSupport shows that by default we're using orphanDependents=true which has been marked as deprecated since 1.7. When using plain kubectl to delete Secret, I see that they are using propagationPolicy=Background by default for garbage collection.

                  I0429 12:09:18.237610    9996 request.go:897] Request Body: {"propagationPolicy":"Background"}
                  I0429 12:09:18.237636    9996 round_trippers.go:383] DELETE https://api.rh-idev.openshift.com:443/api/v1/namespaces/rokumar/secrets/mysecret

I think we should align Kubernetes Client's behaviour with kubectl and use propagationPolicy=Background by default