Trident: Invalid backend state - backend trident is not Online or Deleting - errors after power outage

[mac-chaffee]

Describe the bug
We experienced a power outage that affected parts of our NetApp cluster and the entire Kubernetes cluster. After power was restored, the NetApp and the kubernetes cluster both recovered, but we noticed persistent volumes were failing to mount.

The trident-main container was logging the following over and over:

time="2021-11-17T16:46:06Z" level=error msg="Invalid backend state." expectedState=online/deleting requestID=49edd01b-575f-4726-bc7e-cb9d4ddabb53 requestSource=CSI state=failed
time="2021-11-17T16:46:06Z" level=error msg="GRPC error: rpc error: code = Internal desc = backend trident is not Online or Deleting" requestID=49edd01b-575f-4726-bc7e-cb9d4ddabb53 requestSource=CSI

The issue was resolved by just restarting the CSI controller pod (the one with 6 containers in it). But it would be nice if Trident could automatically recover from this state.

Environment
Provide accurate information about the environment to help us reproduce the issue.

• Trident version: 21.07.2
• Trident installation flags used: None
• Container runtime: Containerd
• Kubernetes version: 1.21.5
• Kubernetes orchestrator: Kubespray/kubeadm
• Kubernetes enabled feature gates: n/a
• OS: RHEL 8
• NetApp backend types: ONTAP NAS, 9.8

To Reproduce
Shutdown the kubernetes cluster, then partially shutdown an HA NetApp appliance. Observe volumes failing to mount, and trident-main logging errors.

Expected behavior
Trident recovers when the power is restored to all components.

Additional context
Sorry for the minimal details for now. If you need any more logs, I think I can recover some of them.

[balaramesh]

Hello @mac-chaffee

Can you provide additional details about the power outage? Did the Kubernetes cluster come up before the NetApp cluster?

There are a couple of ways to address this:

1. Bounce the Trident controller pod (as you have already identified)
2. Update the backend config. Depending on how you have created a backend, please refer to the documentation here and here

[mac-chaffee]

Yes it looks like the Kubernetes cluster did come up about 1.5 hours before the NetApp cluster was fully up:

[machaffe@k8s-node15 ~]$ uptime
 16:57:55 up 15 days,  7:24,  1 user,  load average: 0.67, 0.72, 0.98

na-europa::> system node show -fields uptime
node         uptime
------------ -------------
na-europa-01 15 days 05:52
na-europa-02 15 days 05:52
na-europa-p03
             112 days 23:36
na-europa-p04
             113 days 00:01

Trident appears to have access to aggregates across all of those nodes, but only nodes 01 and 02 went down (the others were connected to generator power I assume).

I wonder if this is at all related to the fact that you have to update the backend config for other things as well, like volume move operations: https://netapp-trident.readthedocs.io/en/stable-v21.07/dag/kubernetes/integrating_trident.html#volume-move-operations

Would it be possible for Trident to automatically reload backends every so often to fix both of these issues?

[balaramesh]

@mac-chaffee yes. For backends created using TridentBackendConfig, reboots of an ONTAP controller are automatically detected and handled. I am assuming you have created this backend using tridentctl. Can you confirm?

Please take a look at the documentation to learn more about creating and managing backends with TridentBackendConfig. You can migrate existing backends that were created with tridentctl to be managed using this method.

[mac-chaffee]

@balaramesh oh nice! If we use a backend creating using TridentBackendConfig, would that also mean we don't need to refresh the backend when we add new aggregates to the SVM? https://netapp-trident.readthedocs.io/en/stable-v21.07/dag/kubernetes/integrating_trident.html#volume-move-operations

Or is that still required?

[ntap-arorar]

@mac-chaffee When Trident reboots the TridentBackendConfig (tbc) controller wants to bring the TridentBackend to an online state (monitored via lastOperationStatus field) thus it will retry establishing Trident's connection to the storage controller, which should solve the original problem encountered at the time of power outage.

TridentBackendConfig does not actively monitor changes on the storage controller so it would not detect changes such as new aggregate addition and such.

TridentBackendConfig provides a declarative way of creating/updating/deleting a backend without needing to use tridentctl.

[mac-chaffee]

I've migrated to using a TridentBackendConfig. Thanks for pointing that out!