Project: Namespace stuck in Terminating status

[antonu17]

Description

When I delete a namespace (kubectl delete namespace) running kubedb database(s) it stuck in Terminating state forever. Any further attempts to delete namespace are failing with:

Error from server (Conflict): Operation cannot be fulfilled on namespaces "<my_poor_namespace>": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

Might be somehow linked to #600 but I'm not sure about that.

Steps to reproduce

1. Create a namespace and switch to it:

kubectl create namespace test-mongo
kubectl config set-context --current --namespace=test-mongo

1. Create a mongodb resource:

cat <<EOF | kubectl apply -f -
apiVersion: kubedb.com/v1alpha1
kind: MongoDB
metadata:
  name: mongodb
spec:
  version: "4.1"
  storageType: Ephemeral
  terminationPolicy: WipeOut
EOF

1. Delete the namespace

kubectl delete namespace test-mongo

At this point command never releases terminal input, only if ^C pressed. A subsequent run of kubectl delete namespace test-mongo throws an error:

kubectl delete namespace test-mongo
Error from server (Conflict): Operation cannot be fulfilled on namespaces "test-mongo": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

Corresponding logs from kubedb-operator pod.

When MongoDB created:

I1113 08:27:32.778565       1 mongodb.go:58] Patching MongoDB test-mongo/mongodb with {"metadata":{"finalizers":null}}.
I1113 08:28:07.968461       1 mongodb.go:58] Patching MongoDB test-mongo/mongodb with {"metadata":{"finalizers":["kubedb.com"]}}.
I1113 08:28:08.048845       1 service.go:19] Creating Service test-mongo/mongodb-gvr.
I1113 08:28:08.086534       1 serviceaccount.go:19] Creating ServiceAccount test-mongo/mongodb.
I1113 08:28:08.111912       1 role.go:19] Creating Role test-mongo/mongodb.
I1113 08:28:08.176827       1 rolebinding.go:19] Creating RoleBinding test-mongo/mongodb.
I1113 08:28:08.240176       1 service.go:19] Creating Service test-mongo/mongodb.
I1113 08:28:08.282776       1 mongodb.go:58] Patching MongoDB test-mongo/mongodb with {"spec":{"databaseSecret":{"secretName":"mongodb-auth"}}}.
I1113 08:28:08.343091       1 statefulset.go:22] Creating StatefulSet test-mongo/mongodb.
I1113 08:28:08.363877       1 pdb.go:22] Creating PodDisruptionBudget test-mongo/mongodb.
I1113 08:28:27.290482       1 appbinding.go:23] Creating AppBinding test-mongo/mongodb.
I1113 08:28:27.399311       1 service.go:121] spec.monitor.agent is not coreos-operator or builtin.
I1113 08:28:27.476923       1 service.go:56] Patching Service test-mongo/mongodb-gvr with {"spec":{"$setElementOrder/ports":[{"port":27017}],"ports":[{"port":27017,"protocol":null,"targetPort":0}]}}.
I1113 08:28:27.559230       1 service.go:121] spec.monitor.agent is not coreos-operator or builtin.

When namespace deleted:

E1113 08:28:50.413372       1 workqueue.go:36] error in wiping out database.: error in getting db secret: secrets "mongodb-auth" not found
E1113 08:28:50.413389       1 worker.go:76] Failed to process key test-mongo/mongodb. Reason: error in wiping out database.: error in getting db secret: secrets "mongodb-auth" not found
I1113 08:28:50.413395       1 worker.go:80] Error syncing key test-mongo/mongodb: error in wiping out database.: error in getting db secret: secrets "mongodb-auth" not found
E1113 08:28:50.431910       1 workqueue.go:36] error in wiping out database.: error in getting db secret: secrets "mongodb-auth" not found
E1113 08:28:50.431922       1 worker.go:76] Failed to process key test-mongo/mongodb. Reason: error in wiping out database.: error in getting db secret: secrets "mongodb-auth" not found
I1113 08:28:50.431929       1 worker.go:80] Error syncing key test-mongo/mongodb: error in wiping out database.: error in getting db secret: secrets "mongodb-auth" not found
E1113 08:28:50.455542       1 workqueue.go:36] error in wiping out database.: error in getting db secret: secrets "mongodb-auth" not found
E1113 08:28:50.455554       1 worker.go:76] Failed to process key test-mongo/mongodb. Reason: error in wiping out database.: error in getting db secret: secrets "mongodb-auth" not found
I1113 08:28:50.455560       1 worker.go:80] Error syncing key test-mongo/mongodb: error in wiping out database.: error in getting db secret: secrets "mongodb-auth" not found
E1113 08:28:50.489266       1 workqueue.go:36] error in wiping out database.: error in getting db secret: secrets "mongodb-auth" not found
E1113 08:28:50.489277       1 worker.go:76] Failed to process key test-mongo/mongodb. Reason: error in wiping out database.: error in getting db secret: secrets "mongodb-auth" not found
I1113 08:28:50.489283       1 worker.go:80] Error syncing key test-mongo/mongodb: error in wiping out database.: error in getting db secret: secrets "mongodb-auth" not found

I noticed that when I delete mongodb resource, not the whole namespace, kubedb-operator is patching mongodb-auth for some Reason. There's a log entry I see if I do kubectl delete mongodb mongodb:

I1113 08:27:32.765332       1 unstructured.go:84] Patching /v1, Resource=secrets test-mongo/mongodb-auth with {"metadata":{"ownerReferences":[{"apiVersion":"kubedb.com/v1alpha1","blockOwnerDeletion":false,"kind":"MongoDB","name":"mongodb","uid":"de691522-05ee-11ea-89f7-06970e366122"}]}}.

[the-redback]

For workaround, edit the mongodb object and remove the Finalizer.

[antonu17]

For workaround, edit the mongodb object and remove the Finalizer.

Thanks a lot! Removing finalizer helped, but as you said it is a workaround. I believe kubedb-operator should not block database resources deletion even if the auth secret is gone.

Update:
Whoever might need this. A oneliner to remove finalizer on all mongodb resources in a current namespace:

kubectl patch $(kubectl get mg -o name) --type='merge' -p='{"metadata":{"finalizers":null}}'

[esomore]

I am getting this as well

[matlion]

I got the same problem with mysql.
I found another workaround:

kubectl patch crd/mysqls.kubedb.com -p '{"metadata":{"finalizers":[]}}' --type=merge

Question: What's the difference between removing finalizers from Custom Resource (see workaround from antonu17) and Custom Resource Definition (my workaround)?

[tamalsaha]

This issue has been resolved in recent releases in v2021.. . Please reopen if you still see this issue.