Rke: kubelet cannot mount local volume

[lurch42]

RKE version:
v0.1.5

Docker version: (docker version,docker info preferred)
docker info:

Containers: 8
 Running: 7
 Paused: 0
 Stopped: 1
Images: 9
Server Version: 17.03.2-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.9.80-rancher
Operating System: RancherOS v1.3.0
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 3.86 GiB
Name: ros13
ID: CR2U:TFT5:PC5E:ZDYK:DLHT:6QTY:RWPX:W7RJ:IK6G:SN2H:G3EP:AN4A
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

Operating system and kernel: (cat /etc/os-release, uname -r preferred)
uname -r: 4.9.80-rancher

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
vm-ware esxi

cluster.yml file:

nodes:
  - address: 192.168.190.11
    user: rancher
    role:
    - controlplane
    - etcd
    - worker
    ssh_key_path: ./ssh-id
  - address: 192.168.190.12
    user: rancher
    role:
    - etcd
    - worker
    ssh_key_path: ./ssh-id
  - address: 192.168.190.13
    user: rancher
    role:
    - etcd
    - worker
    ssh_key_path: ./ssh-id

services:
  etcd:
  kube-api:
    service_cluster_ip_range: 10.35.0.0/16
    pod_security_policy: false
  kube-controller:
    cluster_cidr: 10.34.0.0/16
    service_cluster_ip_range: 10.35.0.0/16
  scheduler:
  kubelet:
    cluster_domain: cluster.local
    cluster_dns_server: 10.35.0.10
    infra_container_image: rancher/pause-amd64:3.1
    fail-swap-on: true
  kubeproxy:

network:
  plugin: flannel
  options:

# At the moment, the only authentication strategy supported is x509.
# You can optionally create additional SANs (hostnames or IPs) to add to
#  the API server PKI certificate. This is useful if you want to use a load balancer
#  for the control plane servers, for example.
authentication:
  strategy: x509

system_images:
  etcd: rancher/etcd:v3.0.17
  kubernetes: rancher/k8s:v1.10.0-rancher1-2
  alpine: alpine:latest
  nginx_proxy: rancher/rke-nginx-proxy:v0.1.1
  cert_downloader: rancher/rke-cert-deployer:v0.1.1
  kubernetes_services_sidecar: rancher/rke-service-sidekick:v0.1.2
  kubedns: rancher/k8s-dns-kube-dns-amd64:1.14.8
  dnsmasq: rancher/k8s-dns-dnsmasq-nanny-amd64:1.14.8
  kubedns_sidecar: rancher/k8s-dns-sidecar-amd64:1.14.8
  kubedns_autoscaler: rancher/cluster-proportional-autoscaler-amd64:1.0.0
  flannel: rancher/coreos-flannel:v0.9.1
  flannel_cni: rancher/coreos-flannel-cni:v0.2.0


ssh_key_path: ./ssh-id

# Kubernetes authorization mode
# Use `mode: rbac` to enable RBAC
# Use `mode: none` to disable authorization
authorization:
  mode: rbac

# If set to true, rke won't fail when unsupported Docker version is found
ignore_docker_version: false

kubernetes_version: v1.10.0-rancher1-2

# List of registry credentials, if you are using a Docker Hub registry,
# you can omit the `url` or set it to `docker.io`

# Currently only nginx ingress provider is supported.
# To disable ingress controller, set `provider: none`
# To enable ingress on specific nodes, use the node_selector, eg:
# nodes:
#   - address: example.com
#     user: ubuntu
#     role:
#     - role
#     hostname_override: node3
#     internal_address: 192.168.1.6
#     labels:
#       app: ingress
#
# ingress:
#   provider: nginx
#   node_selector:
#     app: ingress

ingress:
  provider: none

Steps to Reproduce:

• bring kubernetes cluster up (rke up -config cluster.yml)
• create a directory for the local volume on node 192.168.190.11:
  sudo mkdir /mnt/disk01/vol01
• create a stateful set having a local volume (kubectl apply -f local-volume.yaml):
  local-volume.txt

Results:
The pod local-volume-test-0 is scheduled to the right node 192.168.190.11, but the mount of the local volume fails with error special device /mnt/disk01/vol01 does not exist.

kubectl describe pod local-volume-test-0:

Name:           local-volume-test-0
Namespace:      default
Node:           192.168.190.11/192.168.190.11
Start Time:     Fri, 13 Apr 2018 12:13:48 +0200
Labels:         app=local-volume-test
                controller-revision-hash=local-volume-test-769667496d
                statefulset.kubernetes.io/pod-name=local-volume-test-0
Annotations:    <none>
Status:         Pending
IP:
Controlled By:  StatefulSet/local-volume-test
Containers:
  ubuntu:
    Container ID:
    Image:          ubuntu:16.04
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /local-volume-test-data from local-volume-test (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6hspb (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  local-volume-test:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  local-volume-test-local-volume-test-0
    ReadOnly:   false
  default-token-6hspb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-6hspb
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                From                     Message
  ----     ------                 ----               ----                     -------
  Warning  FailedScheduling       20s                default-scheduler        persistentvolumeclaim "local-volume-test-local-volume-test-0" not found
  Normal   FailedScheduling       20s (x2 over 20s)  default-scheduler        Volume binding started, waiting for completion
  Normal   Scheduled              19s                default-scheduler        Successfully assigned local-volume-test-0 to 192.168.190.11
  Normal   SuccessfulMountVolume  19s                kubelet, 192.168.190.11  MountVolume.SetUp succeeded for volume "default-token-6hspb"
  Warning  FailedMount            3s (x6 over 19s)   kubelet, 192.168.190.11  MountVolume.SetUp failed for volume "local-storage-test-1-vol01" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -o bind /mnt/disk01/vol01 /var/lib/kubelet/pods/5ac224a6-3f03-11e8-91d7-000c2995b64f/volumes/kubernetes.io~local-volume/local-storage-test-1-vol01
Output: mount: special device /mnt/disk01/vol01 does not exist

[ibuildthecloud]

Our kubelet runs in a container so it doesn't see all host paths by default. You need to bind in paths you might use. This is done as follows in the cluster.yml

services:
  kubelet:
    extra_binds:
       - /mnt:/mnt:rshared

Make sure you add rshared. Hopefully that works for you.

[lurch42]

Yes it works. Thank you!

[xaralis]

For the record, you have to run whole rke up for the changes to apply: rke up --config ./rancher-cluster.yml.

[ap1969]

Our kubelet runs in a container so it doesn't see all host paths by default. You need to bind in paths you might use. This is done as follows in the cluster.yml

services:
  kubelet:
    extra_binds:
       - /mnt:/mnt:rshared

Make sure you add rshared. Hopefully that works for you.

Thanks - that worked for me as well.

[cbluth]

i cant get this to work anymore, it seems like there is some sort of regression.
see my issue here: https://github.com/rancher/rke/issues/1913

does anyone have this working with a recent version of RKE and Kubernetes?