External-storage: [nfs-client] Unable to start nfs-client-provisionner

[jrabary]

We have a local k8s cluster and want to use an external NFS server as a persistance storage.
On every node of this cluster we are able to mount manually the NFS share with

mount -t nfs 10.8.26.123:/testnfs /media/testnfs/

Here are the configurations we use to deploy the nfs-client-provisioner

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 10.8.26.123
            - name: NFS_PATH
              value: /testnfs
      volumes:
        - name: nfs-client-root
          nfs:
            server: 10.8.26.123
            path: /testnfs

The deployment fails to create the nfs-client-provionner pod and outputs the following error

(combined from similar events): MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32 
Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient 
mount for /var/lib/kubelet/pods/4509e02c-b2d5-11e8-82cf-0cc47ae2265e/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs 10.8.26.123:/testnfs /var/lib/kubelet/pods/4509e02c-b2d5-11e8-82cf-0cc47ae2265e/volumes/kubernetes.io~nfs/nfs-client-root 
Output: Running scope as unit run-rc8cc3ae4b6b44d15a0ff40788877c617.scope. mount: wrong fs type, bad option, bad superblock on 10.8.26.123:/testnfs, missing codepage or helper program, or other error (for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program) 
In some cases useful info is found in syslog - try dmesg | tail or so.

Any idea on what might go wrong here ?

[wongma7]

Is k8s containerized?
/area nfs-client

[MrApe]

Same Problem here. Simply used the helm chart to install. It works on an ubuntu machine after installing nfs-common.

[jrabary]

Thanks! Installing nfs-common solves the problem.

[MrApe]

I created a pull request to fix this for containers: #1012

[jianglingxia]

I have accured the nfs-client-provisioner pod not running problem, but not the same with above,

the nfs-client-provisioner pod always restart like that:

[root@jlx-k8s1 exports]# kubectl get po -w
NAME READY STATUS RESTARTS AGE
iag-192.168.1.170 1/1 Running 6 23d
local-test-0 1/1 Running 0 1d
local-test-1 1/1 Running 0 1d
nfs-client-provisioner-78d857f9d-xbnwl 1/1 Running 1 37s
nfs-client-provisioner-78d857f9d-zwq67 0/1 Terminating 9 26m
nfs-client-provisioner-78d857f9d-zwq67 0/1 Terminating 9 27m
nfs-client-provisioner-78d857f9d-zwq67 0/1 Terminating 9 27m
nfs-client-provisioner-78d857f9d-xbnwl 0/1 Error 1 1m
nfs-client-provisioner-78d857f9d-xbnwl 0/1 CrashLoopBackOff 1 1m
nfs-client-provisioner-78d857f9d-xbnwl 1/1 Running 2 1m
nfs-client-provisioner-78d857f9d-xbnwl 0/1 Error 2 1m
nfs-client-provisioner-78d857f9d-xbnwl 0/1 CrashLoopBackOff 2 1m
nfs-client-provisioner-78d857f9d-xbnwl 1/1 Running 3 2m
nfs-client-provisioner-78d857f9d-xbnwl 0/1 Error 3 2m
nfs-client-provisioner-78d857f9d-xbnwl 0/1 CrashLoopBackOff 3 2m

the logs is:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m default-scheduler Successfully assigned nfs-client-provisioner-78d857f9d-xbnwl to 192.168.1.171
Normal SuccessfulMountVolume 38s kubelet, 192.168.1.171 MountVolume.SetUp succeeded for volume "nfs-client-provisioner-token-rgt7s"
Normal SuccessfulMountVolume 38s kubelet, 192.168.1.171 MountVolume.SetUp succeeded for volume "nfs-client-root"
Warning BackOff (x6 over ) kubelet, 192.168.1.171 Back-off restarting failed container
Normal Pulled (x5 over 38s) kubelet, 192.168.1.171 Container image "quay.io/external_storage/nfs-client-provisioner:latest" already present on machine
Normal Created (x5 over 38s) kubelet, 192.168.1.171 Created container
Normal Started (x5 over 38s) kubelet, 192.168.1.171 Started container

I1129 14:38:58.708494 2562 kuberuntime_manager.go:513] Container {Name:nfs-client-provisioner Image:quay.io/external_storage/nfs-client-provisioner:latest Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:PROVISIONER_NAME Value:fuseim.pri/ifs ValueFrom:nil} {Name:NFS_SERVER Value:192.168.1.170 ValueFrom:nil} {Name:NFS_PATH Value:/exports ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:nfs-client-root ReadOnly:false MountPath:/persistentvolumes SubPath: MountPropagation:} {Name:nfs-client-provisioner-token-rgt7s ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Never SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
I1129 14:38:58.708650 2562 kuberuntime_manager.go:757] checking backoff for container "nfs-client-provisioner" in pod "nfs-client-provisioner-78d857f9d-xbnwl_default(f2a8d94d-f3a0-11e8-b368-fa163e662d67)"

My step is
1) create a service account;
2)install nfs;
3)deploy the deployment name nfs-client-provisioner;
4) create a storageclass
can you help me with the problem? @wongma7 thanks!

[jianglingxia]

[root@jlx-k8s1 nfs]# kubectl logs nfs-client-provisioner-78d857f9d-g5j6p
F1129 11:51:22.332731 1 provisioner.go:180] Error getting server version: Get https://10.254.0.1:443/version?timeout=32s: dial tcp 10.254.0.1:443: i/o timeout

[geerlingguy]

In my case, nfs-common is already installed, but it's still giving the error:

E1229 21:02:56.043228       1 leaderelection.go:234] error retrieving resource lock default/fuseim.pri-nfs: endpoints "fuseim.pri-nfs" is forbidden: User "system:serviceaccount:default:nfs-client-provisioner" cannot get resource "endpoints" in API group "" in the namespace "default"

Repeated every 6 seconds.

[geerlingguy]

Well, in my case I figured out the problem—the RBAC manifest I was editing locally (and I _thought_ I was applying to the cluster) was not actually being applied. I had left the old NFS RBAC manifest file from the nfs and so the nfs-client provisioner was not being updated.

I reset the cluster, built it again from scratch, and made sure the nfs provisioner's RBAC manifest was not being applied (just the one from nfs-client), and now it's all magically working, yay!

[brunnels]

For anyone else running across the "mount: wrong fs type, bad option, bad superblock" error.

on the k8s node you must install nfs-utils for centos or nfs-common on ubuntu

[scphantm]

im getting errors too, i get this event

lastTransitionTime: '2019-03-29T19:22:24Z'
      lastUpdateTime: '2019-03-29T19:22:24Z'
      message: >-
        pods "nfs-client-provisioner-75c6bd6c85-" is forbidden: unable to
        validate against any security context constraint: [fsGroup: Invalid
        value: []int64{65534}: 65534 is not an allowed group spec.volumes[0]:
        Invalid value: "nfs": nfs volumes are not allowed to be used
        spec.containers[0].securityContext.securityContext.runAsUser: Invalid
        value: 65534: must be in the ranges: [1000000000, 1000009999]]
      reason: FailedCreate
      status: 'True'
      type: ReplicaFailure

all i did was run the helm (latest).

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[costa]

Umm, this issue is not rotten, actually, it is very much alive. While I'm not sure of the right way to fix this, the nfs-common dependency should be clearly listed in https://github.com/kubernetes-incubator/external-storage/blob/master/nfs/README.md for now. Thanks!

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[shanhu]

[root@jlx-k8s1 nfs]# kubectl logs nfs-client-provisioner-78d857f9d-g5j6p
F1129 11:51:22.332731 1 provisioner.go:180] Error getting server version: Get https://10.254.0.1:443/version?timeout=32s: dial tcp 10.254.0.1:443: i/o timeout

did u solved it ? how ? thanks!!!