Trident: Trident objects removed by Kube Garbage Collector

[StevenBarre]

Describe the bug
The Kube Controller appears to be deciding that several of the objects created by the opertator have no parent and then deletes them.

Additionally, when the operator re-creates the objects, the DaemonSet pods can't connect to the main Trident pod until they get recreated. I suspect there is a race condition here?

Environment

• Trident version: 20.07.1
• Trident installation flags used:
  yaml spec: debug: false silenceAutosupport: true
• Container runtime: CRI-O
• Kubernetes version: v1.18.3+47c0e71
• Kubernetes orchestrator: OpenShift 4.5.13
• Kubernetes enabled feature gates: none
• OS: RHCOS
• NetApp backend types: ONTAP AFF

To Reproduce
I don't have a good reproduce yet, just wait a few days and it seems to happen. It is happening in 3 different clusters though.

Expected behavior
Don't need to re-create objects. If it does, they should come up and work.

Additional context
After the stuff is recreated by the operator, the DaemonSet pods just have this log over and over.

time="2020-10-26T16:55:47Z" level=error msg="could not read transactions" error="Unable to find key"

Deleting the pods and having the DS recreate them fixes the issue.

oc -n trident delete pod -l app=node.csi.trident.netapp.io

Logs from kube-controller-manager pod

I1025 23:55:40.307401       1 garbagecollector.go:404] processing item [trident.netapp.io/v1/TridentProvisioner, namespace: , name: trident, uid: 93eba1cc-d74d-4859-bf47-ee8a31e08f01]
I1025 23:55:40.340456       1 garbagecollector.go:404] processing item [apps/v1/Deployment, namespace: trident, name: trident-operator, uid: 98318349-b8a9-4920-922a-f940ced7c8a8]
I1025 23:55:40.343345       1 garbagecollector.go:447] object [apps/v1/Deployment, namespace: trident, name: trident-operator, uid: 98318349-b8a9-4920-922a-f940ced7c8a8]'s doesn't have an owner, continue on next item
I1025 23:55:41.711548       1 garbagecollector.go:404] processing item [v1/Service, namespace: trident, name: trident-csi, uid: dfacdb58-2179-4b56-adf1-74ab4da38e4c]
I1025 23:55:41.845046       1 garbagecollector.go:517] delete object [v1/Service, namespace: trident, name: trident-csi, uid: dfacdb58-2179-4b56-adf1-74ab4da38e4c] with propagation policy Background
I1025 23:55:43.212448       1 garbagecollector.go:404] processing item [v1/Secret, namespace: trident, name: trident-csi-token-8dl6w, uid: 95005014-ce13-48c7-bc65-ef53684616e1]
I1025 23:55:43.345699       1 garbagecollector.go:447] object [v1/Secret, namespace: trident, name: trident-csi-token-8dl6w, uid: 95005014-ce13-48c7-bc65-ef53684616e1]'s doesn't have an owner, continue on next item
I1025 23:55:43.887067       1 garbagecollector.go:404] processing item [v1/Secret, namespace: trident, name: deployer-token-vwgp6, uid: 760a0a42-3197-4cdb-931a-a452e6d0c1d0]
I1025 23:55:44.020480       1 garbagecollector.go:447] object [v1/Secret, namespace: trident, name: deployer-token-vwgp6, uid: 760a0a42-3197-4cdb-931a-a452e6d0c1d0]'s doesn't have an owner, continue on next item
I1025 23:55:45.818953       1 garbagecollector.go:404] processing item [apps/v1/DaemonSet, namespace: trident, name: trident-csi, uid: 8bc7fa84-96de-4121-9df3-900fc1b3cbdc]
I1025 23:55:45.826520       1 garbagecollector.go:404] processing item [apps/v1/Deployment, namespace: trident, name: trident-csi, uid: b92a606f-ae57-4f94-9fa5-54c84f56de24]
I1025 23:55:45.832401       1 garbagecollector.go:404] processing item [storage.k8s.io/v1/CSIDriver, namespace: , name: csi.trident.netapp.io, uid: 6b739a14-699a-40b2-86d5-fe7b4edc4d9b]
I1025 23:55:45.839335       1 garbagecollector.go:404] processing item [v1/Secret, namespace: trident, name: trident-csi, uid: 6f62231a-3b7d-4f20-8388-0ac4697c0d50]
I1025 23:55:45.952225       1 garbagecollector.go:517] delete object [apps/v1/DaemonSet, namespace: trident, name: trident-csi, uid: 8bc7fa84-96de-4121-9df3-900fc1b3cbdc] with propagation policy Background
I1025 23:55:45.959601       1 garbagecollector.go:517] delete object [apps/v1/Deployment,  namespace: trident, name: trident-csi, uid: b92a606f-ae57-4f94-9fa5-54c84f56de24] with propagation policy Background
I1025 23:55:45.966239       1 garbagecollector.go:517] delete object [storage.k8s.io/v1/CSIDriver, namespace: , name: csi.trident.netapp.io, uid: 6b739a14-699a-40b2-86d5-fe7b4edc4d9b] with propagation policy Background
I1025 23:55:45.971894       1 garbagecollector.go:517] delete object [v1/Secret, namespace: trident, name: trident-csi, uid: 6f62231a-3b7d-4f20-8388-0ac4697c0d50] with propagation policy Background
I1025 23:55:45.981137       1 garbagecollector.go:404] processing item [rbac.authorization.k8s.io/v1/ClusterRole, namespace: , name: trident-csi, uid: b63b7328-2654-49b9-9551-3971d284cb3a]
I1025 23:55:45.985750       1 garbagecollector.go:404] processing item [rbac.authorization.k8s.io/v1/ClusterRoleBinding, namespace: , name: trident-csi, uid: b01c8596-c161-4074-b4bb-21be635219fb]
I1025 23:55:45.992810       1 garbagecollector.go:404] processing item [security.openshift.io/v1/SecurityContextConstraints, namespace: , name: trident, uid: 3fb60888-d40e-4215-90f9-5bedc6764a28]
I1025 23:55:45.999338       1 garbagecollector.go:404] processing item [policy/v1beta1/PodSecurityPolicy, namespace: , name: tridentpods, uid: ef17eb4c-cde2-4386-877a-268efaed4e92]
I1025 23:55:46.006523       1 garbagecollector.go:404] processing item [v1/ServiceAccount, namespace: trident, name: trident-csi, uid: 7cb2e3d8-4202-4573-a29c-7eab3e6c8bd0]
I1025 23:55:46.011836       1 garbagecollector.go:404] processing item [discovery.k8s.io/v1beta1/EndpointSlice, namespace: trident, name: trident-csi-jcpqw, uid: 0fb059a3-6500-4c38-96bd-60cb4f257862]
I1025 23:55:46.088516       1 garbagecollector.go:404] processing item [apps/v1/ControllerRevision, namespace: trident, name: trident-csi-6c5fdf4657, uid: 92c65f8b-5cba-4c86-a160-89a97dc55116]
I1025 23:55:46.093675       1 deployment_controller.go:576] Deployment trident/trident-csi has been deleted

[mrobson]

Here are the details on why this bug occurs.

This maps back to the following upstream issue: https://github.com/kubernetes/kubernetes/issues/65200

Specifically 3: the child is cluster-scoped, and the ownerReference refers to a namespaced kind (since the namespace of the parent is ambiguous, it should be treated as not existing)

OCP had this issue with logging, which required us to remove the ownerRef from the cluster scoped resources, speifically the ClusterRole and ClusterRoleBinding: https://github.com/openshift/cluster-logging-operator/pull/713 / https://github.com/openshift/cluster-logging-operator/pull/718

In the case of trident, you have the same issue where the cluster scroped CRB and CR have an ownerRef to the namespaced TridentProvisioner.

```
ClusterRole
ownerReferences:

• apiVersion: trident.netapp.io/v1
  controller: true
  kind: TridentProvisioner
  name: trident
  uid: 2e0df0a5-0b40-4f9e-a356-a56c997b82d4

ClusterRoleBinding
ownerReferences:

• apiVersion: trident.netapp.io/v1
  controller: true
  kind: TridentProvisioner
  name: trident
  uid: 2e0df0a5-0b40-4f9e-a356-a56c997b82d4

[ntap-arorar]

@mrobson

Thank you for following up on this issue, and pointing links to the upstream issue and OCP PR.
Removing the OwnerRef on the Cluster scoped objects makes total sense, and I agree this is an anti-pattern that should be fixed in Trident Operator.

Do we know if the OCP cluster that @sbarre-esit is running has this OCP patch?

Also, there are a couple of behaviours in OCP that I still trying to understand:

1. TridentProvisioner CR itself is missing the namespace field, if the owner is removed from the cluster scoped objects, why does the garbage collector think the TridentProvisioner CR has no namespace or is cluster-scoped?
   I1025 23:55:40.307401 1 garbagecollector.go:404] processing item [trident.netapp.io/v1/TridentProvisioner, namespace: , name: trident, uid: 93eba1cc-d74d-4859-bf47-ee8a31e08f01]
2. Today, TridentProvisioner CR is a parent to both namespaced as well as cluster-scoped objects, so why does it delete namespaced objects?

Correct me if I am wrong, but looking at above OCP PR, GC should remove the ownerRef on the cluster scoped objects, and TridentProvisioner's cluster-scoped dependents should not have any ownerReference field, on the other hand, it's namespaced dependents should continue to maintain a solid parent-dependent relationship with TridentProvisioner CR and should not be deleted.

Would appreciate if you could shed some light on the observed behaviour.

Thank you!

[StevenBarre]

Do we know if the OCP cluster that @sbarre-esit is running has this OCP patch?

Yes, we have the patch to the ClusterLogging operator.

My uneducated understanding is that the GC cache gets poisoned if it reads the cluster-scoped objects first. It reads the CRB for example and sees it has an owner, and then puts the owner (TridentProvisioner) into the cache without a namespace. Then when the GC loads the namespaced objects (DaemonSet) it can't find a matching owner in the cache and deletes them.

[ntap-arorar]

Thanks @sbarre-esit

I see that as a possibility and root of all the problems, we will prioritize and fix the anti-pattern issue in a future release.

Regarding the second issue (operator re-creation fails to refresh secrets on deployment and daemonset pod), we have already merged the fix and will be available in 20.10.0 release. With this fix, even if ServiceAccount is created last, daemonset pods and deployment pods will be refreshed with the updated secrets to avoid the disruptions that you are running into.

[megabreit]

Quite similar, if not identical to #444 I didn't notice the garbage collector problem, but in the case I opened with Redhat, they found some problem with garbage collection.

[mrobson]

These types of GC issues will be correctly handled based on the work that is targeting 1.20: https://github.com/kubernetes/kubernetes/pull/92743

The fix here is still required because this use case, if applied correctly, would lead to immediate deletion to avoid these surprises in the future:

Use Case:
child that is cluster-scoped with owner reference to namespaced type in namespace B
should be deleted immediately
event should be logged in namespace kube-system with involvedObject of bad-child indicating the error
test both orders:
real parent added to graph, then bad child added to graph
bad child added to graph, virtual parent added to graph, then real parent added to graph

[liggitt]

https://github.com/kubernetes/kubernetes/pull/92743 is merged for 1.20.

We ended up handling the cluster-scoped child → namespaced owner scenario by treating the cluster-scoped child as having an unresolveable owner, and never garbage collecting it (just as if it had an ownerReference with an unknown kind), rather than being immediately deleted as having a verifiably absent owner.

[ntap-arorar]

@liggitt

Appreciate the update!

For the cluster-scoped child → namespaced owner scenario if an immediate deletion is not expected, the only way to identify if this indeed is a case is by looking at: kubectl get events -A --field-selector=reason=OwnerRefInvalidNamespace, as mentioned here?

[liggitt]

That event will be available in 1.20, yes. There's a follow-up item from the PR to provide a tool that can be run on any cluster to identify problematic ownerReferences, but there's not an availability date for that tool yet.

[liggitt]

To follow up, the tool that can run and identify invalid owner references is https://github.com/kubernetes-sigs/kubectl-check-ownerreferences