Crossplane: Unable to delete ClusterPackageInstall after deleting helm release

[tapanhalani]

Installation of crossplane:
Installed using helm chart using the following command:
helm install crossplane --namespace crossplane-system crossplane-alpha/crossplane --version 0.9.0 --set clusterStacks.azure.deploy=true --set clusterStacks.azure.version=v0.7.0 --devel

Current behaviour:
When trying to uninstall crossplane using the command:
helm uninstall crossplane --namespace crossplane-system

The release got deleted along with all crossplane and azure provider pods. But the following component is still not deleted:
kubectl get clusterstackinstall --all-namespaces NAMESPACE NAME READY SOURCE PACKAGE CRD AGE crossplane-system provider-azure True crossplane/provider-azure:v0.7.0 3d

When trying to manually delete it using kubectl, it returns the status that clusterstackinstall.stacks.crossplane.io "provider-azure" deleted , but the resource is still there.

This is hampering me from reinstalling the chart, as upon trying to re-install the chart, I get the following error:
manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: namespace: , name: stacks.stacks.crossplane.io, existing_kind: apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition, new_kind: apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition

Crossplane version: 0.9.0

System info:
Kubernetes cluster: Azure AKS cluster
kubernetes v1.17.0
helm v3.1.2

[hasheddan]

Hi @tapanhalani, thanks for opening this issue! I am surprised in this case that the ClusterStackInstall gets left behind, but I am guessing it may have a finalizer that did not get removed before the stack manager pod was terminated. I will investigate this scenario, but you should also be able to fix it in the short term by manually removing the finalizer on the ClusterStackInstall. Please let me know if this does not work for you and we will do some additional troubleshooting :)

[tapanhalani]

@hasheddan Perfect. Thank you for the quick response, and for the short term fix. It worked perfectly on my side. I used the following 2 commands to remove finalizer from clusterstackinstall and stack, after which I was able to successfully delete the pending CRD clusterstackinstall, stackinstall and stack:

kubectl patch clusterstackinstall provider-azure -p '{"metadata":{"finalizers":[]}}' --type=merge -n=crossplane-system

kubectl patch stack provider-azure -p '{"metadata":{"finalizers":[]}}' --type=merge -n=crossplane-system

Let me know if you need any more info from me while investigating this problem to find a permanent solution. Thanks.

[jbw976]

Thank you for confirming your commands and that it removed the lingering CR for you @tapanhalani

For future visitors to this issue, we also have this section addressed in the troubleshooting page of our docs: https://crossplane.io/docs/v0.10/reference/troubleshoot.html#deleting-a-resource-hangs

[matthewhembree]

I ran into this issue with 0.12.0. The uninstall instructions tell you to delete the secrets:
kubectl delete secret -n crossplane-system --all

but that also deletes the helm release information. (Helm3 stores that in k8s secrets). When you try to helm delete, there's no secret to tell helm what resources to delete.

Removing the finalizers on:

clusterpackageinstalls.packages.crossplane.io provider-aws
packages.packages.crossplane.io provider-aws

allows the namespace to finish terminating.

I'm not sure of the best way to amend the instructions. There might need to be different ways depending on the helm version.