Charts: [stable/prometheus] alertmanager is not able to write to /data

[0lmi]

Describe the bug
After installing the chart alertmanager is not able to write to /data partition due to lack of permissions.

Version of Helm and Kubernetes:
Kubernetes: v1.12.6-eks-d69f1b
Helm: v2.14.1

Which chart:
stable/prometheus

What happened:
After installing the chart alertmanager gets "permission denied" when it tries to write to /data

kubectl -n prometheus logs prometheus-alertmanager-75d4576948-mchgg --container=prometheus-alertmanager
...
level=error ts=2019-07-18T08:20:22.529168071Z caller=nflog.go:363 component=nflog msg="Running maintenance failed" err="open /data/nflog.7e064331362ecea3: permission denied"
level=info ts=2019-07-18T08:20:22.52951607Z caller=silence.go:377 component=silences msg="Running maintenance failed" err="open /data/silences.41447c071def3b3: permission denied"

kubectl -n prometheus exec -ti prometheus-alertmanager-75d4576948-mchgg --container=prometheus-alertmanager sh
/alertmanager $ ls -la /data
total 20
drwxr-xr-x    3 root     root          4096 Jul 17 11:50 .
drwxr-xr-x    1 root     root            52 Jul 17 11:50 ..
drwx------    2 root     root         16384 Jul 17 11:50 lost+found
/alertmanager $ whoami
nobody
/alertmanager $ grep nobody /etc/passwd
nobody:x:65534:65534:nobody:/home:/bin/false
/alertmanager $ touch /data/test-file
touch: /data/test-file: Permission denied

What you expected to happen:
During pod startup proper permissions has to be set, so /data is writable for alertmanager process.

How to reproduce it (as minimally and precisely as possible):
Run k8s on EKS and install the chart (with using default storage class).

Anything else we need to know:

[severity1]

is there a workaround for this?

[0lmi]

SSH to an instance where this pod is running and chown -R 65534: 65534 manually:

1. find proper container:

sudo docker ps | grep k8s_prometheus-alertmanager_prometheus-alertmanager
b3063efcef7c        ce3c87f17369                                                       "/bin/alertmanager -…"   2 weeks ago         Up 2 weeks                              k8s_prometheus-alertmanager_prometheus-alertmanager-87d596bb4-lwpq7_prometheus_bf9ce3f8-a960-11e9-b2bd-0e9f361a8286_0

1. find assigned volume

docker inspect -f '{{ .HostConfig.Binds }}' k8s_prometheus-alertmanager_prometheus-alertmanager-87d596bb4-lwpq7_prometheus_bf9ce3f8-a960-11e9-b2bd-0e9f361a8286_0 | sed 's/ /\n/g' | grep "/data"
/var/lib/kubelet/pods/bf9ce3f8-a960-11e9-b2bd-0e9f361a8286/volumes/kubernetes.io~aws-ebs/pvc-fce821b7-a888-11e9-86b8-0a9cb3bbd532:/data

1. change ownership

sudo chown 65534:65534 -R /var/lib/kubelet/pods/bf9ce3f8-a960-11e9-b2bd-0e9f361a8286/volumes/kubernetes.io~aws-ebs/pvc-fce821b7-a888-11e9-86b8-0a9cb3bbd532

or do the same but through k8s, it will be something like:

1. stop alert manager

kubectl -n prometheus edit deployments. prometheus-alertmanager

and set 'replicas:' to '0'

1. create pod

cat <<EOF > pod.yaml
kind: Pod
apiVersion: v1
metadata:
  name: tmp-pod
spec:
  containers:
  - name: tmp-pod
    image: ubuntu
    command: ["/bin/sh", "-c", "sleep 3600;"]
    volumeMounts:
      - mountPath: /data
        name: storage-volume
  restartPolicy: Never
  volumes:
  - name: storage-volume
    persistentVolumeClaim:
      claimName: prometheus-alertmanager
EOF
kubectl -n prometheus apply -f pod.yaml

1. exec to the pod

kubectl -n prometheus exec -ti tmp-pod bash

1. change ownership

chown -R 65534:65534 /data

1. cleanup: delete the pod and change replicas to 1

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[a7i]

is there a workaround for this?

Update chart to 8.15.0 or later.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.