External-storage: Local Volume Provisioner requires 'mount --bind' after a mount is successfully mounted from inside pod

[killfill]

Hello all,

Im trying to use local volumes with the static provisioner, looks like a great thing!
But not sure if im missing something or not.

When creating a mount on the host, like this:
mount -t tmpfs vol /disks/vol

then the log shows this, and stays in that this 'state':

discovery.go:201] Path "/disks/vol" is not an actual mountpoint

When entering inside the pod where the static provioner is running in this same host, and doing
mount --bind /disks/vol /disks/vol
It goes through, creating the PV just fine.

As far as i can see, the provisioner reads /proc/mount
https://github.com/kubernetes/kubernetes/blob/master/pkg/util/mount/mount_linux.go#L49

Is it required to enter the pod's and doing 'mount --bind' manually the desired workflow?

If not do you think it would be a good idea to make the /proc/mount file path configurable so one can mount it from the host to the pod so that the info about mounts that has the host is exposed to the provisioner inside the pod? That way the manual binding is not needed anymore?

https://github.com/openshift/origin/blob/master/examples/prometheus/node-exporter.yaml#L53

Thanks!

[msau42]

Hi @killfill, are you running the local static provisioner with "HostToContainer" mount propagation, like in the example spec?

[killfill]

Hi @msau42,

Yes, i triple checked, and the manifest im using does include the "HostToContainer" value, like this:

          volumeMounts:
            - mountPath: /storage/local-disk
              name: local-disk
              mountPropagation: "HostToContainer"

But i noticed that when asking for that data to kubeapi, there is no registry about that.

$ kubectl get po -n storage local-volume-provisioner-njw2v -o json|grep -i prop
$

im using v1.10.1:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-13T22:27:55Z", GoVersion:"go1.9.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:14:26Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

I noticed, in the begining of the readme, it tells about some feature flags that need to be enabled, like
export KUBE_FEATURE_GATES="BlockVolume=true"

But i don't need blockvolume support this time, so i just ignored it.

Do you think Mount Propagation support, is the thing im missing out?

Thanks!!

[msau42]

You shouldn't need the BlockVolume feature gate, and the MountPropagation feature should be on by default in 1.10.

I think there may be an issue with your volumeMount config. The spec you pasted shows the mountPath as "/storage/local-disk", but the logs are referring to "/disks" as the discovery directory path. The provisioner spec is a bit tricky to craft and it requires the volumeMounts.mountPath to be the same as the configmap.MountDir.

[killfill]

@msau42 Yes sorry.

I changed the directory a couple of hours ago.
Just for clarity, here is the manifest im using:

https://pastebin.com/raw/sux5FuUE

The logs shows this, when i do a new mount in /storage/local-disk/vol-01, inside the Host:

E0723 17:29:32.362064 1 discovery.go:201] Path "/storage/local-disk/vol-01" is not an actual mountpoint

Docker mount options looks like this:

$ docker inspect k8s_provisioner_local-volume-provisioner-8wkrc_storage_9158a23b-8e97-11e8-b7d9-d8cb8a7c5bb1_2

<inside .HostConfig.Mounts>
           {
                "Type": "bind",
                "Source": "/storage/local-disk",
                "Destination": "/storage/local-disk",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },

It should be something like rslave ?

Thanks!

[msau42]

Yes, it should say rslave. So for some reason it seems like the mount propagation setting is not getting passed through. Can you double check the "feature-gate" arguments passed to your kube-apiserver and kubelets?

cc @jsafrane

[killfill]

OMG, kubelet has this:

--feature-gates=MountPropagation=false

8|

[msau42]

The MountPropagation feature originally changed the default mode for all volume mounts to rslave, which caused regressions in certain environments (maybe that's why it was disabled for you).

In 1.10.3, the default mode was changed back to private: https://github.com/kubernetes/kubernetes/pull/62462

[killfill]

Manage to turn that flag to true, and things seems to be working fine!

Thanks!!

[killfill]

Hi @msau42,

Just for clarity, and if it helps someone else that could be hitting a similar problem, im seeing this, when trying to make a pod mount the local-volume:

https://github.com/rancher/rke/issues/500

Should kubelet really have access to the directory where the mounts are?

[killfill]

Oh i take it back :P

I guess the 'defaults' kubelet deployment's run in the host, not as a container, so this is not really needed.

Thanks