k3s in docker, nfs-provisioner not work

[huapox]

k3s in docker(docker-compose up -d, cluster in docker, use containerd), nfs-provisioner not work:

1. use the org Dockerfile's image from stratch:
mount -t nfs host:/dir /local with err

2.use the diy image from alpine:
mount -t nfs host:/dir /local with err, but mount -t nfs4 works.

3.use the diy image from alpine:
apk add nfs-utils
mount -t nfs host:/dir /local; OKay

but when using nfs-provisoner pod in cluster with a POD, there be err:
nfs conn timeout.

when k3s agent run in vm/host(centos7), the nfs-provisoner works fine.

help wanted, thx.

[ibuildthecloud]

I don't know if this is your issue but running NFS in containers often has an issue do to clients getting NATd. You need to add insecure to your export in the /etc/exports. More in the in the exports man page.

      secure This option requires that requests originate on an Internet port
              less  than IPPORT_RESERVED (1024). This option is on by default.
              To turn it off, specify insecure.

[huapox]

@ibuildthecloud thx for reply.

1.I've tried with the issue you said by add inisecure to nfs-server, but got the same err:

Events:
  Type     Reason       Age               From                   Message
  ----     ------       ----              ----                   -------
  Normal   Scheduled    72s               default-scheduler      Successfully assigned kube-system/nfs-provisioner-01-67c89b7746-twflb to 96e868f0a65c
  Warning  FailedMount  8s (x8 over 72s)  kubelet, 96e868f0a65c  MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 255
Mounting command: mount
Mounting arguments: -t nfs 10.200.100.180:/data/k8s /var/lib/rancher/k3s/agent/kubelet/pods/4ed52e70-6822-11e9-a844-024202030002/volumes/kubernetes.io~nfs/nfs-client-root
Output: mount: mounting 10.200.100.180:/data/k8s on /var/lib/rancher/k3s/agent/kubelet/pods/4ed52e70-6822-11e9-a844-024202030002/volumes/kubernetes.io~nfs/nfs-client-root failed: Connection refused

2.And when i just run my diy img in the vm, it works:

docker run -it --rm --entrypoint=bash --privileged registry.cn-shenzhen.aliyuncs.com/k-spe/att-k3s
mkdir -p /mnt
mount -t nfs 10.200.100.180:/data/k8s /mnt

Then i start with port test in an alpine pod:

/ # nc -vz 10.200.100.180 111
10.200.100.180 (10.200.100.180:111) open
/ # ip a |grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 7.0.3.3/24 brd 7.0.3.255 scope global eth0
    inet6 fe80::7846:88ff:fee9:883d/64 scope link

it's open

3.Attach info:

• nfs-provisioner: jmgao1983/nfs-client-provisioner:latest

• nfs-server: ubuntu1604

cat /etc/exports
/data/k8s *(insecure,rw,sync,no_subtree_check,no_root_squash)

• vm: centos7, docker-ce 17.12

• net-info:
  nfs-net: 10.200.100.0/24
  vm-net: 172.17.0.0/24

[huapox]

new progress:
1.try other nfs-provisoner, all the same err:
(all latest version)
lizhenliang/nfs-client-provisioner
gmoney23/nfs-client-provisioner
angelnu/nfs-client-provisioner

2.just run an alpine pod with privilege, to test nfs mount:

apiVersion: v1
kind: Pod
metadata:
  name: test-alpine1
spec:
  containers:
  - command: ['sh', '-c']
    args:
    - tail -f /dev/null
    image: alpine
    imagePullPolicy: Always
    name: test
    securityContext:
      privileged: true
  restartPolicy: Always

apk add nfs-utils

/ # mount -t nfs 10.200.100.180:/data/k8s /mnt
/ # cd /mnt/
/mnt # ls
archived-devops-jk-pre01-jenkins-pvc-5a3e7fc8-5055-11e9-a3f0-0050568dd0ab

it's Okay.

3.src

https://github.com/kubernetes-incubator/external-storage/blob/master/nfs-client/cmd/nfs-client-provisioner/provisioner.go

what could the error be?

[erikwilson]

It sounds like it might be an issue with routing, network, or configuration, where when running a pod with privileged: true it has greater access to network.

Does it work okay with the default provided containerd?
What configuration are you using for the nfs-client-provisioner deployment?

[erikwilson]

Also, if you are using selinux there may be additional commands that need to run, such as setsebool -P virt_use_nfs on, from https://docs.okd.io/latest/install_config/storage_examples/shared_storage.html#sharing-an-nfs-pv-across-two-pods-ensuring-nfs-volume-access

[huapox]

It sounds like it might be an issue with routing, network, or configuration, where when running a pod with privileged: true it has greater access to network.

Does it work okay with the default provided containerd?
What configuration are you using for the nfs-client-provisioner deployment?

@erikwilson thx for reply.
I've alter the title of the issue, I run k3s in docker, with containerd in docker.

my nfs-client-provisioner deployment:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  namespace: kube-system

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: kube-system 
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

---
kind: Deployment
apiVersion: apps/v1beta1
metadata:
  name: nfs-provisioner-01
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-provisioner-01
  template:
    metadata:
      labels:
        app: nfs-provisioner-01
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          #image: quay.io/external_storage/nfs-client-provisioner:latest
          image: jmgao1983/nfs-client-provisioner:latest
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              # 
              value: nfs-provisioner-01
            - name: NFS_SERVER
              value: 10.200.100.180
            - name: NFS_PATH
              value: /data/k8s
      volumes:
        - name: nfs-client-root
          nfs:
            server: 10.200.100.180
            path: /data/k8s

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-dynamic-class
provisioner: nfs-provisioner-01

[huapox]

What's more, I just use the nfs PersistentVolume to test, with the same err:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: slow
  nfs:
    server: 10.200.100.180
    path: /data/k8s

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: "slow"
  resources:
    requests:
      storage: 1Gi

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nfs-busybox-pvc
spec:
  replicas: 1
  selector:
    matchLabels:
      name: nfs-busybox-pvc
  template:
    metadata:
      labels:
        name: nfs-busybox-pvc
    spec:
      containers:
      - image: busybox
        command:
          - sh
          - -c
          - 'while true; do date > /mnt/index.html; hostname >> /mnt/index.html; sleep 10m; done'
        imagePullPolicy: IfNotPresent
        name: busybox
        volumeMounts:
          - name: nfs
            mountPath: "/mnt"
      volumes:
      - name: nfs
        persistentVolumeClaim:
          claimName: nfs-pvc

[huapox]

Also, if you are using selinux there may be additional commands that need to run, such as setsebool -P virt_use_nfs on, from https://docs.okd.io/latest/install_config/storage_examples/shared_storage.html#sharing-an-nfs-pv-across-two-pods-ensuring-nfs-volume-access

selinux both in my nfs-server and the vm of k3s were disabled.

The PersistentVolume is key to me like to save jenkins's storage.

could anybody test this issue in your env with k3s in docker?

[huapox]

new progress, resolved:

the key is to run rpcbind in the docker-node.

k3s in docker with nfs usage, alter the image:

1.from alpine

2.apk add nfs-utils

3.rpcbind -f &

4. enjoy it.

[sjmiller609]

@huapox's instructions worked for me. Here is the Dockerfile:

https://github.com/sjmiller609/baremetal_kube/blob/3829589ab2b7747389f7f281fd1d246f851c0697/roles/container-linux-setup-for-k3s/files/Dockerfile#L1-L36

[xz64]

You can specify nfs version 4 mount in your PersistentVolumeClaim to also avoid this issue, since nfs4 doesn't require rpcbind. I tried this using k3d and it works.

apiVersion: v1
kind: PersistentVolumeClaim
...
spec:
  mountOptions: ["vers=4"]

If you need to disable client side caching you can also add "lookupcache=none" to the mountOptions list.