Rke: Add support for Flexvolume driver

[yasker]

There are two issues need to be resolved for this:

1. kubelet container need to bind-mount a directory on the host and use it as the location for Flexvolume driver. For example (assuming the Flexvolume driver is at the default location, see https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md):

diff --git a/services/kubelet.go b/services/kubelet.go
index 22b36d9..5450609 100644
--- a/services/kubelet.go
+++ b/services/kubelet.go
@@ -56,6 +56,7 @@ func buildKubeletConfig(host *hosts.Host, kubeletService v3.KubeletService) (*co
        hostCfg := &container.HostConfig{
                Binds: []string{
                        "/etc/kubernetes:/etc/kubernetes",
+                       "/usr/libexec/kubernetes/kubelet-plugins:/usr/libexec/kubernetes/kubelet-plugins",
                        "/etc/cni:/etc/cni:ro",
                        "/opt/cni:/opt/cni:ro",
                        "/etc/resolv.conf:/etc/resolv.conf",

1. The k8s package used by RKE need to support bind-mount /host/dev to /dev in the container, in order to utilize the block devices on the host. A simple shell script in kubenetes-package can do this:

diff --git a/package/Dockerfile b/package/Dockerfile
index 5cea599..f7dffcd 100644
--- a/package/Dockerfile
+++ b/package/Dockerfile
@@ -48,7 +48,7 @@ ENV SSL_SCRIPT_COMMIT 98660ada3d800f653fc1f105771b5173f9d1a019
 RUN wget -O /usr/bin/update-rancher-ssl https://raw.githubusercontent.com/rancher/rancher/${SSL_SCRIPT_COMMIT}/server/bin/update-rancher-ssl && \
     chmod +x /usr/bin/update-rancher-ssl

-COPY runc-1.10 utils.sh entry.sh addons-update.sh kubectl kubelet kube-proxy kube-apiserver kube-controller-manager kube-scheduler /usr/bin/
+COPY runc-1.10 utils.sh entry.sh addons-update.sh kubectl kubelet kube-proxy kube-apiserver kube-controller-manager kube-scheduler mount-entry.sh /usr/bin/
 COPY kubernetes/cluster/addons /etc/kubernetes/addons/

 RUN curl -OL https://github.com/rancher/cli/releases/download/v0.6.1/rancher-linux-amd64-v0.6.1.tar.gz \
diff --git a/package/mount-entry.sh b/package/mount-entry.sh
new file mode 100755
index 0000000..461aa3e
--- /dev/null
+++ b/package/mount-entry.sh
@@ -0,0 +1,6 @@
+#!/bin/bash
+set -e -x
+
+mount --rbind /host/dev /dev
+
+exec "$@"

After fixing these two issues, we can make Longhorn driver (after updated the Flexvolume driver mount directory) works with RKE.

[annymsMthd]

What's the status of this and using Longhorn on a rke created cluster? I'm testing out different volume providers and I would love to help test this out.

[annymsMthd]

Actually I just checked and all the pieces are in place. I have longhorn running in my cluster now:)

[galal-hussein]

Tested with latest master:

My steps:

1- started kubelet with the following extra_binds:

services:
  kubelet:
    extra_binds:
       - /var/lib/kubelet/volumeplugins:/var/lib/kubelet/volumeplugins
       - /dev:/dev

2- created the longhorn stack

 kubectl create -f https://raw.githubusercontent.com/rancher/longhorn/v0.2/deploy/longhorn.yaml

3- create test volume

apiVersion: v1
kind: Pod
metadata:
  name: volume-test
  namespace: default
spec:
  containers:
  - name: volume-test
    image: nginx:stable-alpine
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: voll-longhorn
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: voll-longhorn
    flexVolume:
      driver: "rancher.io/longhorn"
      fsType: "ext4"
      options:
        size: "2Gi"
        numberOfReplicas: "3"
        staleReplicaTimeout: "20"
        fromBackup: ""

I was able to see the volume created and the pod started successfully.

[yasker]

@galal-hussein Can we add those steps to the doc? If we're not going to enable them by default.

[mateuszdrab]

Is there a way I can do this bind mount on v1.6.14?
I installed a fresh install of docker 17.03 and rancher v1.6.14 and I can't get my head round this - wanted to try Longhorn as so far I can't find a reliable storage solution for docker.

Edit: sneaky bind mount of /dev:/dev and /usr/libexec/kubernetes/kubelet-plugins: /usr/libexec/kubernetes/kubelet-plugins worked...

[jbw976]

Do users of RKE need to do anything special currently to use flexvolume plugins? If so, is there a pointer to the single place in the documentation for RKE where enabling flexvolume plugins is explained? Thanks!