Kind: Mount Points from Dynamically Created PVC's Are Not Writeable by Non-Root Users

[Xtigyro]

What happened:

1. A PVC is provisioned.
2. It is dynamically bound to a hostPath volume by the KiND's provisioner.
3. A pod is created that mounts the PVC.
4. The process in the pod is running as runAsUser 65533 ("git-sync" user), with fsGroup 65533 too.
5. The process can not write to the PVC mount, since it is only writeable by root user.

Trying mode or defaultMode 0777 has no effect to the permissions of the mount folder.

Since we don't want to allow escalating privileges in the pod, we can't use the PVC mount at all.

What you expected to happen:
Some way of specifying in the PVC what UID / GID the hostPath should be owned by, so we can write to it.

How to reproduce it (as minimally and precisely as possible):

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc-1
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Mi
...
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod-1
  labels:
    svc: test-pvc-permissions
spec:
  volumes:
    - name: test-volume-1
      persistentVolumeClaim:
        claimName: test-pvc-1
  containers:
  - image: k8s.gcr.io/git-sync:v3.1.2
    name: notebook
    volumeMounts:
     - mountPath: /home/test
       name: test-volume-1
    command: ["/bin/sh", "-c", "touch /home/test/hi"]
  securityContext:
    fsGroup: 65533
    runAsUser: 65533
...

Works fine with root:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc-1
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Mi
...
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod-1
  labels:
    svc: test-pvc-permissions
spec:
  volumes:
    - name: test-volume-1
      persistentVolumeClaim:
        claimName: test-pvc-1
  containers:
  - image: k8s.gcr.io/git-sync:v3.1.2
    name: notebook
    volumeMounts:
     - mountPath: /home/test
       name: test-volume-1
    command: ["/bin/sh", "-c", "touch /home/test/hi"]
  securityContext:
    fsGroup: 0
    runAsUser: 0
...

[BenTheElder]

This is an issue with the k8s.io/hostPath provisioner built into Kubernetes. Unfortunately that is not going to change by design, so we'll need https://github.com/kubernetes-sigs/kind/issues/118.

For now if you're on AMD64 architecture, take a look at the workaround with rancher local in that thread.

See this previous issue for more details on the hostPath driver https://github.com/kubernetes-sigs/kind/issues/622#issuecomment-520603295

https://github.com/kubernetes-sigs/kind/issues/118 is tracking proper dynamic volume provisioning support

[BenTheElder]

The hostPath volumes do indeed require root and will not chmod by design upstream in Kubernetes for security reasons. That won't change. What can change is the default driver in kind.

We'd love to ship something else and are tracking in #118

[Xtigyro]

The hostPath volumes do indeed require root and will not chmod by design upstream in Kubernetes for security reasons. That won't change. What can change is the default driver in kind.

We'd love to ship something else and are tracking in #118

@BenTheElder Thank you very much for the nice and explanatory reply!

Do you know why it works in AWS EKS then - because of the AWS GP2 storage class and that the AWS EBS provisioner chmod-s it?

[BenTheElder]

Yes that's it's 🙂

On EKS they are not using the hostPath storage drivers for the default storage (I assume, and instead the GP2 and EBS provisioner).

Cloud providers can just use a Kubernetes storage implementation that works against their existing cloud storage, for kind we can't really do that.

Unfortunately most of the local storage options have one of the following problems:

• not multi node aware
• not multi platform
• no dynamic provisioning

The rancher option is pretty good, but we need to follow up on figuring out if we should ship it by default or not and how to best manage and configure it. It's fairly high in my backlog.

I'd appreciate feedback on that thread if you try it out 🙂

What I'd _really_ like to do is try to write a little CSI driver similar to hostPath, but that would take more time. There is an official CSI local storage driver that runs on kind, but I think the intent for that one is to only support single node clusters.

We should sync again with SIG storage about this

[BenTheElder]

hopeful re: https://github.com/kubernetes-sigs/kind/issues/118#issuecomment-528993520

[Xtigyro]

@BenTheElder I can confirm that the local-path provisioner from Rancher has been working quite well on a multi-node kINd cluster for me.

[monotek]

Just had the same problem with rabbitmq chart version 6.7.4.

Chart can be used anyway when "volumePermissions.enabled: true" is set. -> https://github.com/helm/charts/blob/master/stable/rabbitmq/values.yaml#L371

Whats weird to me is, that kind does not post any output and just times out (example: https://circleci.com/workflow-run/77844cea-87b3-4e84-95f9-7f34cf99a334). I had to ssh into the circle ci host to see that rabbitmq (which was installed as requirement) was in crashloopbackoff, because of the permission error.

[aojea]

Chart can be used anyway when "volumePermissions.enabled: true" is set. -> https://github.com/helm/charts/blob/master/stable/rabbitmq/values.yaml#L371

This is explained by Ben here https://github.com/kubernetes-sigs/kind/issues/830#issuecomment-526932268 , seems it's by design in kubernetes :man_shrugging:

Whats weird to me is, that kind does not post any output and just times out (example: https://circleci.com/workflow-run/77844cea-87b3-4e84-95f9-7f34cf99a334).

The link shows that kind worked perfectly, it doesn't do anything else once the k8s cluster is deployed. It seems you have several steps in your config to install helm and some charts

I had to ssh into the circle ci host to see that rabbitmq (which was installed as requirement) was in crashloopbackoff, because of the permission error.

you can add some checks using kubectl or helm status to your CI, the same way you would do with a real k8s cluster ... as I mentioned before kind only provides the k8s cluster :)

[monotek]

Right, the output problem could also be an issue with the chart-testing tool...

[BenTheElder]

Deduping with https://github.com/kubernetes-sigs/kind/issues/118

I'm not actively involved at the moment but monitoring progress (which is happening!) in the CSI ecosystem around local drivers, which seems like the most likely path forward.

[metal3d]

I'm sorry to comment here, but the problem in #118 is not exactly the same. I'm using my own gitea chart https://github.com/metal3d/charts and the "gitea" user uses its own uid/gid. When I write data on mounted volume /data that are not persisted. Each time the pod restarts, my data is lost (e.g. sqlite database)
The problem is (on my side) to write on PV with non-root user, it "seems" to work, but actually the data are not really persisted.
Should I need to use another storage class ? wich one is the best for "kind" cluster ?

[BenTheElder]

check https://github.com/kubernetes-sigs/kind/issues/118

i'm not sure which is best, the CSI driver is attractive, and users have reported some success with the rancher local storage option.

[BenTheElder]

your chart seems to have a PVC, which is #118, right now those are backed by k8s.io/hostpath which is not a good option, but available in the kubernetes tree as a local driver...

we absolutely must ship something else in the near future, but handling multi-node in particular is tricky and storage is in flux a bit upstream.

[Xtigyro]

I'm still using Rancher's solution. So far so good - incl. handling multi-node.