Fluentd-kubernetes-daemonset: Errno::EACCES: Permission denied @ rb_sysopen

[collimarco]

I have tried this file on a new DigitalOcean K8s cluster:
https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch-rbac.yaml

I am using Loz.io and I have configured all the env variables properly.

The problem is that the container crashes and these are the pod logs:

2019-06-10 16:44:01 +0000 [error]: unexpected error error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied @ rb_sysopen - /var/log/fluentd-containers.log.pos>
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/plugin/in_tail.rb:145:in `initialize'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/plugin/in_tail.rb:145:in `open'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/plugin/in_tail.rb:145:in `start'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:115:in `block in start'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:114:in `each'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:114:in `start'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/engine.rb:237:in `start'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/engine.rb:187:in `run'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:570:in `run_engine'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:162:in `block in start'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:366:in `main_process'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:339:in `block in supervise'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:338:in `fork'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:338:in `supervise'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:156:in `start'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
  2019-06-10 16:44:01 +0000 [error]: /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
  2019-06-10 16:44:01 +0000 [error]: /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/gems/fluentd-0.12.43/bin/fluentd:8:in `<top (required)>'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/bin/fluentd:23:in `load'
  2019-06-10 16:44:01 +0000 [error]: /fluentd/vendor/bundle/ruby/2.5.0/bin/fluentd:23:in `<main>'
2019-06-10 16:44:01 +0000 [info]: shutting down fluentd
2019-06-10 16:44:01 +0000 [info]: shutting down filter type="kubernetes_metadata" plugin_id="filter_kube_metadata"
2019-06-10 16:44:01 +0000 [info]: shutting down output type="elasticsearch" plugin_id="out_es"
2019-06-10 16:44:01 +0000 [info]: shutting down output type="null" plugin_id="object:2ae5f1a3f320"
2019-06-10 16:44:01 +0000 [info]: process finished code=0
2019-06-10 16:44:01 +0000 [warn]: process died within 1 second. exit.

Note: I am trying with the RBAC version, because the simpler version has another issue

[ademars94]

Best solution I was able to find for this was setting the FLUENT_UID environment variable to "0". I think they changed the default user to fluent instead of root (something like that), and setting this to 0 will fix the permissions issue for now.

[repeatedly]

Yes. README mentions it :)

https://github.com/fluent/fluentd-kubernetes-daemonset#run-as-root

[predragstosic]

I confirm we have the same problem on our project. We are advised by our security audit company to run all pods with non-root users. Hopefully this gets fixed soon.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions[bot]]

This issue was automatically closed because of stale in 30 days