Fluentd-kubernetes-daemonset: Lot of "error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied" messages with new docker image version of fluentd-kubernetes-deamonset:elasticsearch

[repeatedly]

Does anyone have comments for this proposal?
We want feedback from k8s users.

[deitch]

This bit me too. I think this kind of change is a breaking change and should go as a major number. I agree wholeheartedly, it should run as fluent and not as root. But it breaks a lot of things (blew up my cluster, thankfully caught in dev, not prod), and should be vetted properly. Definitely not a patch release kind of thing. :-)

[mbolek]

We were also bit by this issue, and used the FLUENT_UID set to 0 workaround but it doesn't seem like the best one.
I've prepared a simple solution for dbluxo's proposal, please see PR: https://github.com/fluent/fluentd-kubernetes-daemonset/pull/178

I did some ultra-basic testing and seems ok. I guess we could create a /fluentd/pos and set the default there?

[mbolek]

Nope, as discussed in the PR it's not a proper way to move forward

[pvanderlinden]

This should have been a major version bump indeed as it is a breaking change.
While I understand that it is better not to run as root, won't this have complications for log files created by root which should be send to elasticsearch by fluentd as well?

[WUMUXIAN]

I encountered the same issue using this image: fluent/fluentd-kubernetes-daemonset:v0.12-alpine-elasticsearch

[brian-vu]

Got this issue when deploy fluentd-es-rbac on k8s.

[hixichen]

Got this issue when deploy with latest version: fluent/fluentd-kubernetes-daemonset:elasticsearch:

kubectl apply -f fluentd-daemonset-elasticsearch-rbac.yaml.

[mcswaroop19]

yes, getting same error with kubectl apply -f fluentd-daemonset-elasticsearch-rbac.yaml.

[repeatedly]

Do you apply this?

https://github.com/fluent/fluentd-kubernetes-daemonset#run-as-root

[mcswaroop19]

Yeah, Was able to resolve by setting FLUENT_UID to 0. Thanks for the response.

[whereisaaron]

Gah, just been chasing massive error rates on our API servers and tracked the problem to fluentd.

Moving from running as root to non-root is a significant breaking change! And it was done as a patch release?? 😲

In the interim we have locked down to a stable release:
fluent/fluentd-kubernetes-daemonset@sha256:d1f1b13d10825aae3078856533feca2f4f7187ed79ed91edd74fbbb1d29a85c6

[repeatedly]

Hmm... v1 images run as root by default so it should not have this problem.
Do you use outdated v0.12 version? v1 has image version to avoid unexpected breaking changes but v0.12 doesn't have it...

[whereisaaron]

We have tried switching to v1 images several times @repeatedly, but on k8s at least they work fine and feed logs to CloudWatch for about 60 seconds then crash or just stop operating without error or explanation. So we always end up switching back to the (previously!) stable and reliable 0.12 version, which just works.

We can try again next time we update the build, see if that has changed with the latest release. Right now our git files all contain warnings about not updating # Version 1.2 does not work, crashes after a minute.

If the v1 images run as root by default, and the <=v0.12.33 images run as root by default, why make this change to the default just for v0.12.43? You could have at least included export FLUENT_UID=${FLUENT_UID:=0} in the entry script to maintain the default behavior whilst still enabling users to set a non-root UID if they wanted.

[tahirali-endurance]

We have an EKS cluster which has cluster wide POD security policy to restrict container running as root. We provided the security context in helm chart to use non-root user in fluentd.

helm install fluentd incubator/fluentd-cloudwatch \
--set awsRegion=us-east-1,rbac.create=true \
--set podSecurityContext.runAsUser=999 \
--set podSecurityContext.runAsGroup=999 \
--set logGroupName=kubernetes \
--set rbac.pspEnabled=true \
--set awsRegion="us-east-1"

We are still getting

"unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/fluentd-cluster-autoscaler.log.pos"

By reading docs, "To be able to write pos files to the host system, you'll need to run fluentd as root". Is it possible to run fluentd as non-root user in this case?