Trident: Cannot delete trident namespace

[superbrothers]

Describe the bug
After running tridentctl uninstall -n trident, I tried to delete trident namespace, but it failed due to finalizers of Trident custom resources.

$ tridentctl uninstall -n trident
$ kubectl delete ns trident --wait=false
$ kubectl get ns trident
NAME      STATUS        AGE
trident   Terminating   25m
$ kubectl get trident -n trident
NAME                                       VERSION
tridentversion.trident.netapp.io/trident   19.10.0

NAME                                     AGE
tridentnode.trident.netapp.io/minikube   25m
$ kubectl get -n trident tridentversions.trident.netapp.io -o yaml  -o jsonpath='{.items[*].metadata.finalizers}'
[trident.netapp.io]

Environment
Provide accurate information about the environment to help us reproduce the issue.

• Trident version: [e.g. 19.10] 19.10.0
• Trident installation flags used: [e.g. -d -n trident --use-custom-yaml] tridentctl install -n trident
• Container runtime: [e.g. Docker 19.03.1-CE]
• Kubernetes version: [e.g. 1.15.1] 1.16.5
• Kubernetes orchestrator: [e.g. OpenShift v3.11, Rancher v2.3.3] minikube version: v1.6.1
• Kubernetes enabled feature gates: [e.g. CSINodeInfo]
• OS: [e.g. RHEL 7.6, Ubuntu 16.04]
• NetApp backend types: [e.g. CVS for AWS, ONTAP AFF 9.5, HCI 1.7]
• Other:

To Reproduce
Steps to reproduce the behavior:

$ minikube start --kubernetes-version=v1.16.5
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:47:40Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:37:43Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
$ tridentctl version --client
+----------------+
| CLIENT VERSION |
+----------------+
| 19.10.0        |
+----------------+
$ tridentctl install -n trident
$ tridentctl version -n trident
+----------------+----------------+
| SERVER VERSION | CLIENT VERSION |
+----------------+----------------+
| 19.10.0        | 19.10.0        |
+----------------+----------------+
$ tridentctl uninstall -n trident
$ kubectl delete ns trident --wait=false
namespace "trident" deleted
$ kubectl get ns trident
NAME      STATUS        AGE
trident   Terminating   2m9s
$ kubectl get trident -n trident
NAME                                       VERSION
tridentversion.trident.netapp.io/trident   19.10.0

Expected behavior
Just running kubectl delete ns trident, trident namespace is deleted.

Additional context
Add any other context about the problem here.

[superbrothers]

The workaround for this issue is the below:

# Remove finalizers from all Trident custom resources
for obj in $(kubectl get trident -n trident -o name); do kubectl get -n trident "$obj" -o json | jq -r '.metadata.finalizers = null' | kubectl apply -f-; done
# Then, delete trident namespace
kubectl delete ns trident

[zakkg3]

I did:

 kubectl delete -n trident $(kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found  -n trident -o name)

didn't work,
then set the finalizers to null as you did solved the problem.

[gnarl]

From the Trident documentation, "To completely remove Trident, execute the tridentctl obliviate crd command. This will remove all CRD objects and undefine the CRDs. Trident will no longer manage any PVs it had already provisioned. Remember that Trident will need to be reconfigured from scratch after this."

Will using the tridentctl obliviate crd command solve the issue you are reporting?

[litsegaard]

Hi!
Just wondering if somebody found a remedy for this problem as I'm currently, when writing this, stuck in "namespace-removal mode". Issuing a kubectl edit command on the trident namespace and then setting "finalizers:" to null did NOT remedy the problem. I also tried the "for..." suggestion earlier in this thread, however no objects exists which are related to the trident namespace.

The cluster in question hosting this trident instance is a Rancher provisioned one:
Kubernetes v1.17.4
Docker v19.3.8
Trident 20.04 installed using the operator, not tridentctl

I remember I had exactly the same issue roughly a year ago, however during that time I used tridentctl.

Any suggestions on how to remedy this problem are very welcome.

Cheers

[clintonk]

Hi, @litsegaard. If the namespace won't go away and has no finalizers itself, then something must still be in it. If your intention is to remove all traces of Trident, you can modify your context so you are in the trident namespace, then run tridentctl obliviate crd.

[kodo651]

Hi!

That solved the problem. I needed to issue the command twice because first
time it got stuck. Nevertheless many thanks for your help!

Cheers

On Wed, 10 Jun 2020, 23:31 Clinton Knight, [e-mail] wrote:

Hi, @litsegaard https://github.com/litsegaard. If the namespace won't
go away and has no finalizers itself, then something must still be in it.
If your intention is to remove all traces of Trident, you can modify your
context so you are in the trident namespace, then run tridentctl
obliviate crd.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/NetApp/trident/issues/331#issuecomment-642278738, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABBHDTFNA33DAEJ3PEPQRUDRV73TBANCNFSM4KJOBRUQ
.

[gnarl]

Closing this issue as resolved as a solution was provided.