Loki: Error installing Loki with persistent volume with a user without 'root' privileges

[mkenne11]

__Describe the bug__

When installing Loki using the helm chart (loki/loki) the loki pod is crashing (with the "CrashLoopBackOff" status) and the logs for the pod are:

level=info ts=2020-04-27T11:49:54.44769852Z caller=loki.go:156 msg=initialising module=server
level=info ts=2020-04-27T11:49:54.448582974Z caller=server.go:147 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2020-04-27T11:49:54.449088913Z caller=loki.go:156 msg=initialising module=runtime-config
level=info ts=2020-04-27T11:49:54.449385151Z caller=manager.go:109 msg="runtime config disabled: file not specified"
level=info ts=2020-04-27T11:49:54.449489039Z caller=loki.go:156 msg=initialising module=memberlist-kv
level=info ts=2020-04-27T11:49:54.449580612Z caller=loki.go:156 msg=initialising module=table-manager
level=error ts=2020-04-27T11:49:54.449902146Z caller=log.go:141 msg="error initializing bucket client" err="mkdir /data/loki: permission denied"

The storage volume (.data folder) is mapped to the persistant volume claim (pvc) pvc-loki, using the persistence.existingClaim value in the chart.

The loki-values.yaml file contains:

image:
  repository: grafana/loki
  tag: 1.4.1
persistence:
  enabled: true
  accessModes:
  - ReadWriteOnce
  size: 10Gi
  existingClaim: pvc-loki
  mountPath: "/data"
resources: {}
securityContext:
  fsGroup: 1000
  runAsGroup: 1000
  runAsNonRoot: true
  runAsUser: 1000

The pvc and persistent volume were created using the following resource definition file:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-loki
  labels:
    name: loki-storage
spec:
  storageClassName: local-path
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
     path: "/mnt/cluster_share/kube/data/loki"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-loki
  namespace: monitoring
spec:
  storageClassName: local-path
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  selector:
    matchLabels:
      name: loki-storage

The loki service account (with uid 1000 and gid 1000) has been made the owner of the folder mounted by the pvc using:

sudo chown -R 1000:1000 /mnt/cluster_share/kube/data/loki

Note. This appears to be related to #1834.

__To Reproduce__

Loki was installed using the chart command line and custom values file (loki-values.yaml):

helm upgrade --install monitoring-loki --namespace monitoring -f loki-values.yaml loki/loki

__Expected behavior__

The Loki deployment and service should start (with a status of Running).

__Environment__

I'm using the k3s distribution of Kubernetes v1.17.4 running on a Raspbian cluster.

Helm version 3.2.0 is used for the installation.

Screenshots, Promtail config, or terminal output
See details above for definition files.

[mkenne11]

I tried installing the helm chart using a Loki configuration running the container as the root user (with a uid of 0 and gid and 0) and installed and ran without any issues. The working loki-values.yaml file is below:

image:
  repository: grafana/loki
  tag: 1.4.1
persistence:
  enabled: true
  accessModes:
  - ReadWriteOnce
  size: 10Gi
  existingClaim: pvc-loki
resources: {}
securityContext:
  runAsGroup: 0
  runAsNonRoot: false
  runAsUser: 0

Ideally I'd rather not run as a root user for security reasons.

[adityacs]

@mkenne11 1.4.1 is missing loki user in the docker image. So, there is no user with uid 1000. Use the latest image and change your securityContext settings to match uid 10001

FYR: https://github.com/grafana/loki/blob/master/cmd/loki/Dockerfile#L18

[adityacs]

@slim-bean Not sure why this change was not picked in both releases 1.4 and 1.4.1

Non-root user docker image for Loki

[slim-bean]

1.4.0 was cut intentionally before the changes to make Loki run as non-root because we hadn't totally tested that and weren't comfortable putting it in the release.

I'm afraid for 1.4.x you will still need to run as root

[mkenne11]

Thanks for the assistance @adityacs and @slim-bean. I'll look out for updated releases for Loki running as a non-root user.

@slim-bean - I noticed the pod security policy in the Loki helm chart has readOnlyRootFilesystem set as true:
https://github.com/grafana/loki/blob/664537e152ce6e46c00d0941fcd7163ea5f04366/production/helm/loki/templates/podsecuritypolicy.yaml#L37

Would that limit access to the (root) file system as read-only when I run the Loki pod under the root user?

[adityacs]

@mkenne11 readOnlyRootFilesystem: true will make the root file system readonly for all users.
You can make it false if you are using 1.4.1. Since, loki directory is /loki

[mkenne11]

Thanks @adityacs I'll try that setting.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.