Rke: Add default mountpoint for CSI plugin watcher

[blaggacao]

On rke0.2.0rc5

https://github.com/kubernetes/kubernetes/pull/70494/files#diff-da30b0eaed0cc98a669162eecb7586d7R24
makes /var/lib/kubelet/plugins_registry required on the host for csi-plugins like this to work.

What currently happens:

Events:
  Type     Reason       Age                  From                Message
  ----     ------       ----                 ----                -------
  Normal   Scheduled    2m19s                default-scheduler   Successfully assigned default/csi-hostpathplugin-h2qx7 to 10.0.1.74
  Warning  FailedMount  16s                  kubelet, 10.0.1.74  Unable to mount volumes for pod "csi-hostpathplugin-h2qx7_default(7841035d-35fd-11e9-8fe5-5254008e9fd0)": timeout expired waiting for volumes to attach or mount for pod "default"/"csi-hostpathplugin-h2qx7". list of unmounted volumes=[registration-dir]. list of unattached volumes=[socket-dir mountpoint-dir registration-dir default-token-qpn2l]
  Warning  FailedMount  11s (x9 over 2m19s)  kubelet, 10.0.1.74  MountVolume.SetUp failed for volume "registration-dir" : hostPath type check failed: /var/lib/kubelet/plugins_registry is not a directory

Proposed action:
Implement a mechanism similar to /var/lib/kubelet/volumeplugins (Isn't that overlapping semantics after all?) to provide for the existence of such host directory OOTB

Presumed Workaround:
https://github.com/rancher/rke/pull/417/files (merged)

[blaggacao]

Somewhat around these lines: https://github.com/rancher/rke/pull/857/files#diff-2e9ef6d97acca90e8ed40f581b9f7264R385 of #857 comes to my mind as proposed action.
/cc @niusmallnan

[blaggacao]

Work around:

    kubelet:
      extra_binds:
        - "/var/lib/kubelet/plugins:/var/lib/kubelet/plugins"
        - "/var/lib/kubelet/plugins_registry:/var/lib/kubelet/plugins_registry"
        - "/var/lib/kubelet/pods:/var/lib/kubelet/pods:shared,z"

is confirmed to work around the issue.

Result:

Events:
  Type     Reason       Age                   From                Message
  ----     ------       ----                  ----                -------
  Warning  FailedMount  6m42s (x27 over 65m)  kubelet, 10.0.1.74  Unable to mount volumes for pod "csi-hostpathplugin-h2qx7_default(7841035d-35fd-11e9-8fe5-5254008e9fd0)": timeout expired waiting for volumes to attach or mount for pod "default"/"csi-hostpathplugin-h2qx7". list of unmounted volumes=[registration-dir]. list of unattached volumes=[socket-dir mountpoint-dir registration-dir default-token-qpn2l]
  Warning  FailedMount  2m24s (x40 over 67m)  kubelet, 10.0.1.74  MountVolume.SetUp failed for volume "registration-dir" : hostPath type check failed: /var/lib/kubelet/plugins_registry is not a directory
  Warning  FailedMount  21s                   kubelet, 10.0.1.74  MountVolume.SetUp failed for volume "default-token-qpn2l" : couldn't propagate object cache: timed out waiting for the condition
  Normal   Pulling      19s                   kubelet, 10.0.1.74  pulling image "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
  Normal   Pulled       15s                   kubelet, 10.0.1.74  Successfully pulled image "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
  Normal   Created      14s                   kubelet, 10.0.1.74  Created container
  Normal   Started      14s                   kubelet, 10.0.1.74  Started container
  Normal   Pulling      14s                   kubelet, 10.0.1.74  pulling image "quay.io/k8scsi/hostpathplugin:v1.0.1"
  Normal   Pulled       11s                   kubelet, 10.0.1.74  Successfully pulled image "quay.io/k8scsi/hostpathplugin:v1.0.1"
  Normal   Created      11s                   kubelet, 10.0.1.74  Created container
  Normal   Started      11s                   kubelet, 10.0.1.74  Started container

[lopf]

I can confirm this issue is still open with the latest Trident version v20.01.1. We're provisioning from Rancher 2.3.6 onto CentOS 7 and Docker-CE (19).

The suggested workaround (only one mount seems to be required) works:

    kubelet:
      extra_binds:
        - "/var/lib/kubelet/plugins_registry:/var/lib/kubelet/plugins_registry"