Kiali: Kiali CR finalizer not removed when deleting the operator

[clebs]

Describe the bug
The Kiali CR does not get its finalizer removed when deleting the Kiali Operator, therefore the CR stays in the cluster after the operator is gone.

It might be related to https://github.com/kiali/kiali/issues/2484, but it is happening consistently every time the operator is deleted.

Versions used
Kiali: 1.18.1
Istio: 1.4.7
Kubernetes flavour and version: 1.15 GKE.

To Reproduce
Steps to reproduce the behavior:

1. Install istio 1.4.7.
2. Create a Kiali CR.
3. Install the Kiali operator in version 1.18.1.
4. Delete the kiali operator.
5. Delete the Kiali CR
6. Check if the kiali CR is still there and its finalizers are also still there.

(edited) made clear that both the kiali CR and the operator are deleted, not only the operator.

Expected behavior
When deleting the Kiali operator, the kiali CR finalizer is removed and the CR is deleted.

[lucasponce]

Thanks @clebs I cc @jmazzitelli to check this one

[jmazzitelli]

Deleting an operator does not delete Kiali CRs. And you cannot blindly delete the Kiali Operator without first deleting all Kiali CRs it is managing. Deleting the Kiali Operator is effectively removing the object responsible for cleaning up/uninstalling all Kialis (that's what the finalizer is for).

Before deleting the Kiali Operator, you must remove all Kiali CRs:

kubectl delete kiali --all

This gives the opportunity for the operator to clean up all Kiali resources (which may live outside the namespace where Kiali pod is and almost certainly includes resources outside the namespace where the operator lives - so you must give it a chance to clean up otherwise you will have unused, obsolete Kiali resources leftover unless you do this).

If you installed via the https://kiali.io/getLatestKialiOperator script, you should be able to uninstall everything, including Kiali, via:

bash <(curl -L https://kiali.io/getLatestKialiOperator) --uninstall-existing-operator

[clebs]

Hi @jmazzitelli,

thanks for the quick response!
Sorry if this was not clear, I am creating and deleting both, the operator and the kiali CR.
The deletion of the kiali CRs does unfortunately not work because the finalizer is still in there when the delete hits it.

Regarding the deletion order, I am using Helm to create and delete it all, therefore I can't guarantee that the CR is deleted before the Operator always.

If it is a timing issue, do you have any suggestion on how to keep the order? Or would it be the right way to go that the operator manages the deletion (or just removal of finalizers) of the resources it manages before its own deletion?

[jmazzitelli]

Or would it be the right way to go that the operator manages the deletion (or just removal of finalizers) of the resources it manages before its own deletion?

That is the right way. Kubernetes operators manage the installation and removal of their operands. In this case, Kiali Operator is the one that manages and installs Kiali so you should allow the operator a chance to "do its thing" and remove all resources and update all finalizers during the removal process (and by "do its thing" - I point out that Kubernetes operators potentially do more than just create namespaced resources in the namespace where the CR is located. In Kiali operator's case, it has the potential of doing other things like create cluster-scoped resources and label namespaces that only get uninstalled when the Kiali operator has a chance to "does its thing"). But once that happens, then you can uninstall the operator. So it is important to note that you just removing an operator WILL NOT remove the CR you created or the full operand (that is the way Kubernetes operators are designed to work - you could have an operator install an operand, remove the operator, and the operand is still running. That is how it works with OLM and operator subscriptions, for exampe).

So, in short, if you want to purge "all things Kiali", you delete the Kiali CR and wait for the operator to finish removing Kiali; once that is done, then you remove the operator resources. Note that kubectl delete kiali --all should block until the operator finishes removing everything (IIRC).

[clebs]

Hi @jmazzitelli,

I implemented a pre-delete hook on helm to delete the Kiali CR before the chart is uninstalled and it is working now.

Still, IMHO, this should not be required for the kiali operator (and its managed resources) to be deleted, order should not matter in a distributed system.

[jmazzitelli]

FYI: some folks are working on a possible solution to the issue of ordered deletes: https://github.com/openshift/enhancements/pull/380