Trident: find_multipaths should be explicitly set to no in multipath.conf to preserve functionality

[ffilippopoulos]

Describe the bug

We run trident from master in our development Kubernetes cluster.
We saw from a recent commit: https://github.com/NetApp/trident/commit/2eee92b2f9c29c237e32dab5ec3be86632f8baf0
that the only value that is not supported for find_multipaths field in multipath configuration is yes and the rest should be ok to use.

We recently updated multipath tools to:

multipath-tools v0.8.5 (11/09, 2020)

which changed the default configuration to:

$ sudo multipathd show config | grep find_multipaths 
        find_multipaths "strict"
        find_multipaths_timeout -10

With that configuration running we are not able to schedule and PVCs in our nodes.

Trident daemonset on our nodes can see the config fine:

time="2021-07-29T13:19:57Z" level=debug msg="Multipath find_multipaths value found." findMultipathsValue=strict requestID=3c018256-7202-4440-b4a9-f05f30b3e51e requestSource=CSI

but then it loops endlessly, since no path ever becomes available on the host, with the following logs:

time="2021-07-29T13:19:57Z" level=debug msg="Could not find multipath device for device." device=sdb requestID=3c018256-7202-4440-b4a9-f05f30b3e51e requestSource=CSI
time="2021-07-29T13:19:57Z" level=debug msg="<<<< osutils.findMultipathDeviceForDevice" device=sdb requestID=3c018256-7202-4440-b4a9-f05f30b3e51e requestSource=CSI
time="2021-07-29T13:19:57Z" level=debug msg=">>>> osutils.findMultipathDeviceForDevice" device=sdc requestID=3c018256-7202-4440-b4a9-f05f30b3e51e requestSource=CSI

From sudo multipath -v3 -d I can see the reason that this is not mapped by multipath as:

Jul 29 13:24:17 | sdi: orphan path, only one path

Explicitly setting find_multipaths no is enough to mitigate this.

Environment

• Trident version: latest master
• Container runtime: containerd://1.5.4
• Kubernetes version: v1.21.0
• OS: Flatcar Container Linux by Kinvolk 2905.2.0
• NetApp backend types: Ontap 9.9.1

To Reproduce
Try scheduling PVCs on nodes with find_multipaths "strict" configured

Expected behavior

If our case is correct and we need find_multipaths no to work without surprises, this should be added in the respective host preparation doc: https://github.com/NetApp/trident/blob/master/docs/kubernetes/operations/tasks/worker.rst at least, and maybe there could be a check in the code against values that do not work?

[ntap-arorar]

Thank you @ffilippopoulos for the feedback and for identifying the issue related to the default find_multipaths strict value on Flatcar Container Linux by Kinvolk 2905.2.0.
The documentation has been updated to reflect find_multipaths no recommendation.
Commit: https://github.com/NetApp/trident/commit/1db87c6cd2989728854c225375bf63d3500aa134