Jx: Unable to `jx uninstall`

[hferentschik]

jx uninstall hangs when running against EKS cluster. The problems seem to be that the deletion of the _PersistentVolumeClaims_ gets stuck in Terminating state.

? This action will permanently delete Jenkins X from the Kubernetes context foo. Please type in the name of the context to confirm: foo
Removing installation of Jenkins X in team namespace jx
WARNING: Not deleting jx-prow because the release is not installed

error: [failed to uninstall the jenkins-x helm chart in namespace jx: [failed to run 'kubectl delete pvc --ignore-not-found -l jenkins.io/chart-release=jenkins-x --namespace jx --wait' command in directory '', output: 'persistentvolumeclaim "jenkins" deleted
persistentvolumeclaim "jenkins-x-chartmuseum" deleted
persistentvolumeclaim "jenkins-x-docker-registry" deleted
persistentvolumeclaim "jenkins-x-nexus" deleted

Manually running kubectl delete pvc --ignore-not-found -l jenkins.io/chart-release=jenkins-x --namespace jx also hangs.

Could affect GKE as well. Only verified against EKS so far.

[hferentschik]

Setting the finalizer for the PersistentVolumeClaim to _null_ seems to do the trick:

k get pvc | awk 'NR>1 {print $1}' | xargs -I claim kubectl patch pvc claim -p '{"metadata":{"finalizers":null}}'

After that the uninstall completes.

[hferentschik]

See also https://github.com/kubernetes/kubernetes/issues/77258

[hferentschik]

I think always deleting the PVCs might be a bad thing, so I suggest to

1. Introduce a new option to jx uninstall, --include-pvc which when set deletes the data otherwise not
2. For the case where we want to delete the data, we first path the PVC to set the finalizer to _null_

[mrageh]

Confirmed that the same issue also happens on GKE, so I don't think it's specific to EKS. Once I set the finalizer PVC to null, jx uninstall worked.

Unfortunately this is a bug in Kubernetes as described in the issue linked above.

@hferentschik Why do you think always deleting the PVCs a bad idea?

[hferentschik]

Confirmed that the same issue also happens on GKE, so I don't think it's specific to EKS. Once I set the finalizer PVC to null, jx uninstall worked.

+1

[hferentschik]

Why do you think always deleting the PVCs a bad idea?

Because they might contain data which one wants to keep or re-use. Having the additional flag makes it an active choice to clean also generated data.

[hferentschik]

@mrageh - here is a data point which I find relevant - https://medium.com/coworkforce/the-continued-adventures-of-jenkins-x-b5a64bd4407a. Maybejx uninstall was not directly involved, but it shows that for example deleting the Nexus PVC can cause problems or at least making recovery harder. If we knew that Velero backups would be working and properly documented one could recover this way, but for now I would feel better if PVC deletion is an active choice.

I am not so hung up about this though. Most importantly I want to get this command to work again at least in the original form it was intended.

[hferentschik]

Might be worth looking at #6991 and #6142 to see whether something can be done for them as well.