Fluentd-kubernetes-daemonset: Access denied to /var/log unless FLUENT_UID is set to 0

[rajarajanpsj]

When I deployed fluent/fluentd-kubernetes-daemonset:v0.12.43-debian-cloudwatch to an EKS cluster, I was constantly getting access denied errors in /var/log. It was resolved when FLUENT_UID environment variable is set to 0. In the DockerFile i do see that USER is set to ROOT and when I exec into the daemonset pod, I am able to create/read files in /var/log but however the fluentd process (or the child process?) is not able to create files under /var/log UNLESS I set this environment variable FLUENT_UID to "0". If this is required, should it not be default? what is the significance of this environment variable?

[repeatedly]

This is same with https://github.com/fluent/fluentd-kubernetes-daemonset/issues/172 .
We update base image and it seems to cause permission error.
if setting FLUENT_UID to 0 in Dockerfile resolve the problem, we should do it.

[mvernimmen]

Would that be equivalent to running fluentd as root within the container? If so, that has some security implications (some explanation here, but there are probably better references out there: https://medium.com/@mccode/processes-in-containers-should-not-run-as-root-2feae3f0df3b) because it makes the kubernetes node as a whole a lot less secure.
Would there perhaps be other ways to solve the same problem without needing uid 0, for example by writing to a different location (as suggested in #172 ) or some other means of doing this?

[vpistis]

same problem, solved with USER root and ENV FLUENT_UID 0

[repeatedly]

I'm not sure which is the best.
We want feedback from k8s users/experts.

[PierrickI3]

Setting the env var FLUENT_UID to "0" fixed it for me too

[vpistis]

@repeatedly IMHO if a linux process (or container) have to read from /var/log then he must have appropriate permissions, it must be root or equivalent.
Into /var/log maybe there is private informations and only a superuser can read it consciously.

[mbolek]

FLUENT_UID is actually only a partial solution it seems. In the /bin/entrypoint.sh:

cat entrypoint.sh 
#!/usr/bin/dumb-init /bin/sh

uid=${FLUENT_UID:-1000}

# check if a old fluent user exists and delete it
cat /etc/passwd | grep fluent
if [ $? -eq 0 ]; then
    deluser fluent
fi

# (re)add the fluent user with $FLUENT_UID
adduser -D -g '' -u ${uid} -h /home/fluent fluent

# chown home and data folder
chown -R fluent /home/fluent
chown -R fluent /fluentd

exec su-exec fluent "$@"

So if FLUENT_UID is set to 0, fluent user doesn't actually get created since the container starts with this:

adduser: uid '0' in use
chown: unknown user fluent
chown: unknown user fluent

and then continues running everything as root :)
What was the reasoning behind moving to fluent user in the 43 build? If security then ok, but since we need to read from /var/log/ that is mounted from Host I guess root-level access is still required

[dimm0]

FLUENT_UID is not helping...

[mbolek]

@dimm0 any additional info? This seems to work for me but by breaking not working around the issue :)

[dimm0]

I'm hitting this error https://github.com/fluent/fluentd-kubernetes-daemonset/issues/172, pods are crashlooping, setting FLUENT_UID to 0 is not helping, running pod as user 0 is not helping either. Not sure what other info to provide

[mbolek]

@dimm0 What is you pos_file location in the config file? Are you 100% sure it's running deployment with FLUENT_UID set to 0?

        env:
          - name: FLUENT_UID
            value: "0"

Are you using alpine or debian image?

[dimm0]

Here's my yaml: https://gitlab.com/ucsd-prp/prp_k8s_config/blob/master/fluentd/fluentd-daemonset-elasticsearch.yaml

I saw a PR with pos_file, which was denied. And there's no such setting in examples. What should I set one to?

[mbolek]

@dimm0 I think you are using the image from #180 (which is fluentd-0.12.33) that doesn't need the FLUENT_UID workaround as the fluentd in the container is running as root.
FLUENT_UID workaround is meant for the latest image i.e. fluent/fluentd-kubernetes-daemonset:v1.2.2-debian-elasticsearch so you could try running latest image with this variable set to 0

[dimm0]

Solved by adding imagePullPolicy: Always. It was using the old image

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions[bot]]

This issue was automatically closed because of stale in 30 days