External-storage: NFS demo does not work with kubernetes 1.6.6

[dcowden]

I followed the NFS provisioner demo, and I'm having a [probably subtle] problem getting it working. I'm using k8s version 1.6.6:

Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6", GitCommit:"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2", GitTreeState:"clean", BuildDate:"2017-06-16T18:21:54Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}

all of the pods launch correctly, and the PV and PVCs are created as well:

NAME                              READY     STATUS    RESTARTS   AGE
nfs-provisioner-225303794-7cxh7   1/1       Running   0          5m
nfs-test-1nn61                    1/1       Running   0          4m

NAME      STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
nfs       Bound     pvc-524a260e-650b-11e7-b217-0800274d0770   1Mi        RWX           example-nfs    5m

NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM         STORAGECLASS   REASON    AGE
pvc-524a260e-650b-11e7-b217-0800274d0770   1Mi        RWX           Delete          Bound     default/nfs   example-nfs              5m

However, the busybox image is unable to write data into the mounted volume, with error "Invalid argument":

sh: can't create /mnt/index.html: Invalid argument
sh: can't create /mnt/index.html: Invalid argument
sh: can't create /mnt/index.html: Invalid argument
sh: can't create /mnt/index.html: Invalid argument
sh: can't create /mnt/index.html: Invalid argument

getting into the pod, the volume is mounted ( other non-relevant mounts removed for clarity):

/ # mount
10.100.228.116:/export/pvc-524a260e-650b-11e7-b217-0800274d0770 on /mnt type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.2.15,local_lock=none,addr=10.100.228.116)

but manually trying to do anything on that volume doesn't work:

/ # touch /mnt/foobar
touch: /mnt/foobar: Invalid argument

curiously, its also impossible to umount the volume too, even though i'm root. I was going to try to manually re-mount it:

/ # umount /mnt
umount: can't unmount /mnt: Operation not permitted

I have also tried manipulating the rootSquash and gid parameters on the provisioner with no luck. Does anyone have ideas on how this could be going wrong?

[shadycuz]

@wongma7 @dcowden I'm using 1.6.6 and I had 0 problems. I just followed the steps, which seems unclear at some point's but I think I will just make a PR for it.

Thanks,
Levi

[wongma7]

I couldn't reproduce this issue either locally nor via Travis CI. However, I don't doubt that it exists. There are a lot of potential culprits... I'd say (the version of) kubernetes is unlikely to be it here. It's probably a weird interaction between NFS-Ganesha and docker.

What storage driver is docker using (output of docker info | grep Storage)?

What happens if you try to mount the nfs server on the host itself, if possible, e.g. sudo mount $NFS_PROVISIONER_CONTAINER_IP:/export/pvc-524a260e-650b-11e7-b217-0800274d0770 /tmp/asdf? If you try to mount it with nfsv4 instead of 4.2 like sudo mount -t nfs4 $NFS_PROVISIONER_CONTAINER_IP:/export/pvc-524a260e-650b-11e7-b217-0800274d0770 /tmp/asdf ?

Root_squash is no_root_squash by default which should be fine. And it's also normal that you're not allowed to unmount it from within the container, that requires more privileges.

@shadycuz thanks for confirming; and yeah, I wrote everything myself and I'm hardly an experienced writer so please feel free to make clarifications :)

[shadycuz]

@wongma7 Well you did a great job. I'm brand new to kubernetes and never messed with PV's until today. Though I have some questions, but I will post them when I make a PR.

[dcowden]

hi guys thanks, let me try out a couple of these suggestions. In reading, my hunch is that it could be an nfs4 vs nfs3 thing, but i wasnt able to prove this.

[r7vme]

This is probably the reason https://github.com/nfs-ganesha/nfs-ganesha/issues/192

In my case, remounting with vers=4.1 helped. But still not clear how to make it work in kubernetes by default.

For one specific PV setting volume.beta.kubernetes.io/mount-options: "vers=4.1" works.

[wongma7]

Yes sorry, I’ve known about this for a bit but neglected to document. You
can try to use the mount option storage parameter in the storage class so
that all pvs get that set automatically

On Thu, Aug 24, 2017 at 11:03 AM Roman Sokolkov [e-mail]
wrote:

This is probably the reason nfs-ganesha/nfs-ganesha#192
https://github.com/nfs-ganesha/nfs-ganesha/issues/192

In my case, remounting with vers=4.1 helped. But still not clear how to
make it work in kubernetes by default.

For one specific PV setting volume.beta.kubernetes.io/mount-options:
"vers=4.1" works.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/kubernetes-incubator/external-storage/issues/223#issuecomment-324662372,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMgP-G-55fTVVw_uX00_QNoR5A0XtGNWks5sbZDRgaJpZM4OSQOj
.

[r7vme]

just found it ;) https://github.com/kubernetes-incubator/external-
storage/blob/master/nfs/docs/usage.md#parameters

Thanks

[abrenneke]

I could not get either NFS 4.1 or 4.0 working either unfortunately. A write as soon as the container started caused an Invalid Argument error and the container (redis) crashed. NFS 3 is working alright. (vers=3)

[adamcharnock]

It took me a while to figure out the above, but in the end my StorageClass looked like the following:

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: nfs
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: developerapp.com/nfs
parameters:
  mountOptions: "vers=4.1"

It seems there is also a new mountOptions field which has been introduced to StorageClasses recently (rather then as a field under parameters). This confused me for a bit, but hopefully the above will save someone some pain in future.

[simmessa]

@adamcharnock Thanks a lot man! Your suggestion about using the parameters / mountOptions field saved the day :)

[agasbzj]

@adamcharnock You saved my life! mountOptions: "vers=4.1" resolves everything! Thank you so much!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.