Strimzi-kafka-operator: [Bug] Cannot create directory '/var/lib/zookeeper': Permission denied

[zanoubia]

Describe the bug
creating the kafka resource not successful:
reason : zookeeper permission denied on /var/lib which is on a pv
Error: mkdir: cannot create directory '/var/lib/zookeeper/data': Permission denied

To Reproduce
Steps to reproduce the behavior:

1. Go to https://strimzi.io/docs/latest/#deploying-kafka-cluster-kubernetes-str
2. kubectl apply examples/kafka/kafka-persistent-single.yaml -n namespace
3. wait until pod is created then run kubectl logs -f my-cluster-zookeeper-0 -n namespace -c zookeeper
4. See error
5. running /opt/kafka/zookeeper_run.sh inside the sidecar container using kubectl exec gives a similar error

Expected behavior
create zookeeper pod and the kafka resource

Environment (please complete the following information):

• Strimzi version: 0.12.0
• Installation method: yaml
• Kubernetes cluster: Kubernetes 1.14.2
• Infrastructure: on perm kubespray + openebs

YAML files and logs
mkdir: cannot create directory '/var/lib/zookeeper/data': Permission denied
yaml file from strimzi doc

Attach or copy paste the custom resources you used to deploy the Kafka cluster and the relevant YAMLs created by the Cluster Operator.
Attach or copy and paste also the relevant logs.

To easily collect all YAMLs and logs, you can use our report script which will automatically collect all files and prepare a ZIP archive which can be easily attached to this issue.

Additional context
Add any other context about the problem here.
offline installation

[zanoubia]

I added

template:
  pod:
    securityContext:
      runAsUser: 0 

to zookeeper specs

Now having the same error with kafka container -_-

[scholzj]

Honestly, I have no experience with OpenEBS and KubeSpray. It sounds like it sets the wrong permissions to the mounted volumes. I think it might be more secure to try to configure the runAsGroup and fsGroup options to then setting the runAsUser: 0. But I think using the securityContext is the right way out of it. You should try to exec into the pod or run it in debug mode to see under which groups does your cluster mount the volumes.

[tpasternak]

Hello, how to configure runAsGroup? The documentation doesn't mention such a field https://strimzi.io/docs/master/

[scholzj]

@tpasternak Here is an example how cou can configure the securityContext which among other wraps the runAsUser and runAsGroup options: https://strimzi.io/docs/latest/full.html#con-customizing-pods-str

[tpasternak]

/var/lib/zookeeper seems to be owned by root:root with permissions 755, so runAsGroup does not change anything in this case

[kafka@nfs-strimzi-zookeeper-0 zookeeper]$ cat /etc/passwd | grep kafka
kafka:x:1001:0::/home/kafka:/bin/bash
[kafka@nfs-strimzi-zookeeper-0 zookeeper]$ stat /var/lib/zookeeper 
  File: '/var/lib/zookeeper'
  Size: 80          Blocks: 0          IO Block: 4096   directory
Device: 42h/66d Inode: 21888438    Links: 4
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-10-29 15:16:55.626725809 +0000
Modify: 2019-10-29 15:14:34.144181952 +0000
Change: 2019-10-29 15:14:34.144181952 +0000
 Birth: -
[kafka@nfs-strimzi-zookeeper-0 zookeeper]$ cat /proc/mounts | grep /var/lib/zookeeper
tmpfs /var/lib/zookeeper tmpfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0

[adityacs]

@zanoubia Could you please ping me your complete file of kafka cluster yaml? When I try to add securityContext template, operator doesn't schedule the kafka cluster to start. So, need to verify my configuration

[scholzj]

@adityacs The scheduling is done by Kubernetes. Not by Strimzi. You should try to check the events on the StatefulSet if it exists for example - that might have some insights. One of the options is for example that the security context you configured doesn't match your policy.

[2pk03]

Hi, I struggle here a bit, too. Zookeeper never comes up since it has no write permissions to the mounted volume, which is NFS. Yes, NFS is bad for kafka, nevertheless thats what IBM has. Did anyone else had the issue or do I simply something wrong here?

Here's the kafka-yaml:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: test-ibm
spec:
  template:
    pod:
      securityContext:
        runAsUser: 1001
        fsGroup: 0
  kafka:
    version: 2.3.0
    replicas: 3
    listeners:
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.3"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 150Gi
        deleteClaim: false
      storageClassName: "ibmc-file-bronze"
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 50Gi
      deleteClaim: false
      storageClassName: "ibmc-file-bronze"
  entityOperator:
    topicOperator: {}
    userOperator: {}

but the pod's SecurityContext never shows up:

     restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName:  test-ibm-zookeeper
      serviceAccount: test-ibm-zookeeper
      securityContext:
        fsGroup: 0
      schedulerName: default-scheduler

Any help to get the permissions correct is appreciated! We use strimzi/kafka:0.14.0-kafka-2.3.0

[adityacs]

@2pk03 I have still not solved this issue by using non root user. If you are ok to use root user. You can set runAsUser: 0 and it works fine.

[2pk03]

@adityacs Thanks for the hint, just tried but same:

spec:
  template:
    pod:
      securityContext:
        runAsUser: 0
        fsGroup: 0

The nodes does not pickup the runAsUser - do I something wrong?

[2pk03]

got it - runs now as root. Not a good solution, but better than nothing. For anybody who got the same issue, here my kafka.yaml as reference:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: ibm-test
spec:
  kafka:
    version: 2.3.0
    replicas: 3
    listeners:
      plain: {}
      tls: {}
    template:
      pod:
        securityContext:
          runAsUser: 0
          fsGroup: 0
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.3"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 150Gi
        deleteClaim: false
      storageClassName: "ibmc-file-bronze"
  zookeeper:
    template:
      pod:
        securityContext:
          runAsUser: 0
          fsGroup: 0
    replicas: 3
    storage:
      type: persistent-claim
      size: 50Gi
      deleteClaim: false
      storageClassName: "ibmc-file-bronze"
  entityOperator:
    topicOperator: {}
    userOperator: {}

thanks @adityacs for the hint and heads-up!

[adityacs]

@2pk03 Yeah using root would not be a good solution. Currently Strimzi doesn't support initContainers. If they add support for initContainers we can do something like below

initContainers:
- name: initcontainer # Or replace the name
  image: alpine:latest
  command: ["/bin/sh", "-c"]
  args:
    - chown <UID>:<GID> /mount; # Replace UID and GID with values from the Dockerfile
  volumeMounts:
  - name: volume # Or you can replace with any name
    mountPath: /mount # Must match the mount path in the args line

[2pk03]

@2pk03 Yeah using root would not be a good solution. Currently Strimzi doesn't support initContainers. If they add support for initContainers we can do something like below

initContainers:
- name: initcontainer # Or replace the name
  image: alpine:latest
  command: ["/bin/sh", "-c"]
  args:
    - chown <UID>:<GID> /mount; # Replace UID and GID with values from the Dockerfile
  volumeMounts:
  - name: volume # Or you can replace with any name
    mountPath: /mount # Must match the mount path in the args line

Ya, sing my song :) Maybe one of our engineers will have a look and contribute..

[scholzj]

Why does IBM have only NFS? That sounds a bit weird because there are many usecases where NFS does not work. Have you actually successfully run Kafka on NFS? BEcause I didn't. Sooner or later it will stop working. So I think it might be pointless to make NFS work. Can't you use for example local persistent volumes? Or some cloud native block storage?

As for the actual feature - so how are other applications in your environment handling storage? Ragrdless NFS or not, they need to be able to read and write tot he storage. So there has to be either a security context configuration which will work or all other applications have to run under root. What settings are your other applications using?

[JoshBargar]

Has anyone had success fixing this issue with a non-root user?

[morhook]

Same problem happened to me on minikube. Not sure why the first creation of the cluster DID work ok.

[scholzj]

@morhook What version of Minikube and Strimzi are you using? Have you tried to customize the security context?

[morhook]

I'm using minikube v1.13.0 on top of docker driver (I believe this might produce the PVC are erased after restart), and strimzi latest operator. I've fixed it locally with this commands (and a restart of minikube prior strimzi creation of kafkas):

minikube ssh
sudo su
echo '/var/tmp/hostpath-provisioner /tmp/hostpath-provisioner none defaults,bind 0 0' > /etc/fstab

[scholzj]

I tried it with v1.13.1 and do not have the problem. So maybe it is related to the driver.