Fluentd-kubernetes-daemonset: Permission problem access /var/log/containers/*

[jfillman]

This issue must have come up before so apologies for not finding it.

My fluentd containers are now run as user '1000' and so don't have read access to /var/lib/docker/containers which is owned by root and has 0700 perms.

I get the log error mesg:

unreadable. It is excluded and would be examined next time.

I can't seem to figure out how to run my container as root or grant the necessary perms on the volume. Adding a securitycontext didn't work

I'm a kubernetes beginner so hand holding is much appreciated.

Here's my daemonset:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      securityContext:
        runAsNonRoot: false
        runAsUser: 0
      containers:
      - name: fluentd
        image: <privaterepo>/fluentd-syslog-ng
        securityContext:
          runAsNonRoot: false
          runAsUser: 0
        imagePullPolicy: Always
        env:
          - name: SYSLOG_SERVER
            value: "SYSLOGSERVER"
          - name: SYSLOG_PORT
            value: "516"
          - name: API_SERVER
            value: "https://FQDN"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: vartmp
          mountPath: /var/tmp
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: vartmp
        hostPath:
          path: /var/tmp
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

[flowstate]

This is also happening to me. I can run bash on the box as the same user (root) and cat all the logs. But I am also getting the 'log unreadable' error. Any chance we can get some help? It's really disappointing that there's been no feedback on this issue for 28 days.

[pirankashani]

Hi,
I had same problem and solved when I delete --lod-driver=journald in docker.conf file.
If you check the folder /var/log/containers it would be empty and that's why you receive this error message.

[mvernimmen]

Could you try this?

securityContext:
   readOnlyRootFilesystem: true

[edsiper]

Considering this ticket as solved due to previous answers.

[jiuchongxiao]

@jfillman did you solved it? and how solved? i have the same issue.

[xinghui-hou]

I have same problem .
flunt-bit output log

[2019/10/11 10:43:59] [error] [in_tail] read error, check permissions: /home/docker/containers/*/*.log
check the directory permissions 0755 is ok

[gotoworld]

same issue

[2020/02/26 03:53:13] [error] [in_tail] read error, check permissions: /data/ratelimit-logs/gateway_2.log
[2020/02/26 03:53:13] [error] [in_tail] read error, check permissions: /data/ratelimit-logs/gateway_ratelimit.log

[repeatedly]

This is not fluent-bit daemonset repository. If you have a problem with fluent-bit, post it to correct place.

[jpiper]

I had this problem and it was due to SELinux profiles