Trident: Support for metrocluster

[hendrikland]

In a Metrocluster configuration, a switchover/site failover causes Trident to fail, e.g. you can't provision any new volumes. Apparently, Trident looks for the vServer name. However, that name changes in case of a Metrocluster switchover and a "-mc" suffix is added to the original vServer name. Once a switchback is issued, the name is reverted to the original one and then Trident starts working again. So dynamic volume provisioning fails while in switchover (either planned as part of a maintenance activity or unplanned in a real disaster event). Adding the "-mc" suffix is standard Ontap behaviour in a Metrocluster environment and shouldn't cause a Trident failure. Trident should assume that in a metrocluster switchover state, any vServer name with a "-mc" suffix is identical to the original vServer).

Logs during MCC switchover look like this:

time="2019-03-23T09:15:51Z" level=warning msg="Kubernetes frontend has no record of the updated storage class; instead it will try to create it." storageClass=blockip time="2019-03-23T09:15:51Z" level=error msg="Kubernetes frontend couldn't add the storage class: Trident initialization failed; problem initializing storage driver 'ontap-nas': error initializing ontap-nas driver: could not create Data ONTAP API client: error reading SVM details: API status: failed, Reason: Specified vserver not found, Code: 15698" storageClass=blockip storageClass_parameters="map[backendType:ontap-san fsType:xfs provisioningType:thin]" storageClass_provisioner=netapp.io/trident
time="2019-03-23T09:15:51Z" level=warning msg="Kubernetes frontend has no record of the updated storage class; instead it will try to create it." storageClass=premium time="2019-03-23T09:15:51Z" level=error msg="Kubernetes frontend couldn't add the storage class: Trident initialization failed; problem initializing storage driver 'ontap-nas': error initializing ontap-nas driver: could not create Data ONTAP API client: error reading SVM details: API status: failed, Reason: Specified vserver not found, Code: 15698" storageClass=premium storageClass_parameters="map[backendType:ontap-nas provisioningType:thin snapshots:true storagePools:ontapnas_172.19.6.45:aggr1_dpp501001]" storageClass_provisioner=netapp.io/trident
time="2019-03-23T09:15:51Z" level=warning msg="Kubernetes frontend has no record of the updated storage class; instead it will try to create it." storageClass=standard time="2019-03-23T09:15:51Z" level=error msg="Kubernetes frontend couldn't add the storage class: Trident initialization failed; problem initializing storage driver 'ontap-nas': error initializing ontap-nas driver: could not create Data ONTAP API client: error reading SVM details: API status: failed, Reason: Specified vserver not found, Code: 15698" storageClass=standard storageClass_parameters="map[snapshots:true storagePools:ontapnas_172.19.6.46:aggr1_dpp509004 backendType:ontap-nas provisioningType:thin]" storageClass_provisioner=netapp.io/trident

(I can provide access to a NetApp internal test lab to reproduce this if required.)

[jacobjohnanda]

I believe this could be solved by updating the backend with the new vserver name using the "tridentctl update backend -f ".
Could you share the setup details.

[si-heger]

Any update here? When can we expect this?

[gtehrani]

No, still tracking as an uncommitted roadmap item.

[sirlatrom]

I work for one of your customers, and we experienced this issue during an unplanned failover. It resulted in nearly half an hour of downtime due to troubleshooting and having to reconfigure and disable/enable the plugin on all nodes.

[rijmenantspj]

In my professional life, I have a customer that is running into a simmilar issue. Hence, is there any update on this matter; as this Issue has been "Open" for quite some time now?

Reading through the documentation, I'm left wondering whether it would be an option to update Trident's backend configuration so that:

• the "managementLIF" parameter is included;
• the "svm" parameter is left out.

The documentation states that the "svm" parameter's value is derived if a management LIF is specified. Thus, somewhat likely the "-mc" suffix in the Storage Virtual Machine's name would no longer matter in such scenario.

As I currently have no combination of a NetApp MetroCluster and Trident at my disposal, I would appreciate it if someone would be able to validate this.

[rijmenantspj]

In my professional life, I have a customer that is running into a simmilar issue. Hence, is there any update on this matter; as this Issue has been "Open" for quite some time now?

Reading through the documentation, I'm left wondering whether it would be an option to update Trident's backend configuration so that:

• the "managementLIF" parameter is included;
• the "svm" parameter is left out.

The documentation states that the "svm" parameter's value is derived if a management LIF is specified. Thus, somewhat likely the "-mc" suffix in the Storage Virtual Machine's name would no longer matter in such scenario.

As I currently have no combination of a NetApp MetroCluster and Trident at my disposal, I would appreciate it if someone would be able to validate this.

Update:
In the meantime, the end customer confirmed the above can be used as workaround. Leaving the "svm" parameter out of Trident's backend configuration indeed ensures the "-mc" suffix is derived while in Switchover. However each MetroCluster Switchover/Switchback still requires an update of de backend configuration (same file can be used).

The above will do the trick for our end customer for now. Also, I hope posting this back will help others. However, we'll still be looking forward to see this "enhancement" incorporated in a future version of Trident so that we no longer need to act manually upon MetroCluster Switchover/Switchback.

[Numblesix]

Hello,

any Update on this :) ?

[gnarl]

This issue is still being tracked as an uncommitted roadmap item. If this request is considered a high priority by you please contact your NetApp account team and let them know.

[ysakashita]

@gnarl
When we evaluate NetApp/Trident on Metro Cluster with NetApp Japan member, we occurred same problem.
So we can’t proceed to install Metro Cluster to our data center.
I hope that the issue fix.

[gnarl]

Hi @ysakashita,

The work to provide a change to detect and handle an MCC switchover isn't currently scheduled on the Trident roadmap. Please work with your NetApp account team to communicate the priority for this change.

[ysakashita]

@gnarl Sure. I understood that it was a priority issue of roadmap, not a technical's one. I already discussed the priority for our business impact with NetApp Japan account team.

[pkris]

Hi @gnarl
We have experienced a similar incident during a Metrocluster failover recently.
Does the Trident documentation state somewhere that connections will not survive such a scenario - I tried digging through the but was unable to find anything ?

[gnarl]

Hi @pkris,

Unfortunately I am unable to answer that question as Trident hasn't been qualified against MCC switchover. The recent activity on this GitHub Issue has put focus on MCC compatibility and we'll keep this issue updated when new information is available.

[gnarl]

Hi @pkris,

I have more information regarding MCC now. During a MCC switchover existing connections should not be affected. During a switchover all data and management LIFs are migrated to the ONTAP cluster that is now active. Per the above stated workaround the only thing that needs to be updated is the SVM name in the Trident backend config. This change is needed so that Trident can communicate with the MCC ONTAP cluster that is currently active.

Hope this helps.

[pkris]

Hi @gnarl

Thanks for the update.
So the current state of the Trident driver stack does not allow for client-side transparency to failover events in the upstream Metrocluster - would that be a fair conclusion with the current state of things?
Any thoughts on how a client cluster utilizing the Trident driver could become aware of the need to apply the workaround (updating the svm name in the backend config) - i.e. know that a failover has occured upstream?

Cheers

[gnarl]

Hi @pkris,

MCC switchover works transparently at the networking level. Anything that is currently mounted will continue to work after a switchover. The SVM name needs to be updated in the Trident Backend config so that any new communication from Trident (volume create, clone, resize, etc...) to the backend are sent to the currently active SVM.

We are investigating how to properly detect a MCC switchover in Trident.