Fluentd: Permission denied @ rb_file_s_stat

[JayTeli]

I am working on reading nginx access logs and passing it to AWS Kinesis.
But i am getting below error in td-agent log.

2019-05-29 14:23:54 +0530 [error]: #0 **unexpected error error_class=Errno::EACCES error="Permission denied @ rb_file_s_stat - /var/log/nginx/access.log"**
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/in_tail.rb:539:in `stat'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/in_tail.rb:539:in `on_notify'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/in_tail.rb:521:in `attach'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/in_tail.rb:270:in `setup_watcher'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/in_tail.rb:301:in `block in start_watchers'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/in_tail.rb:287:in `each'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/in_tail.rb:287:in `start_watchers'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/in_tail.rb:264:in `refresh_watchers'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/in_tail.rb:195:in `start'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:165:in `block in start'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:154:in `block (2 levels) in lifecycle'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:153:in `each'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:153:in `block in lifecycle'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:140:in `each'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:140:in `lifecycle'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:164:in `start'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/engine.rb:274:in `start'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/engine.rb:219:in `run'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:799:in `run_engine'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:549:in `block in run_worker'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:724:in `main_process'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:544:in `run_worker'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/bin/fluentd:8:in `<top (required)>'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `load'
2019-05-29 14:23:54 +0530 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `<main>'
2019-05-29 14:23:54 +0530 [error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ rb_file_s_stat - `/var/log/nginx/access.log"

Below is the config that i've used :

<source>
   type tail
   path /var/log/nginx/access.log
   pos_file /var/log/td-agent/nginx-access.pos
   tag td.staging.access
   format nginx
</source>

[JayTeli]

I've given 777 permission to /var/log/nginx/access.log
Also i've added 'td-agent' user to root group that owns those logs.
Still getting same error

[JayTeli]

I also read somewhere that run td-agent with root access..
So i used --user root --group root option in td-agent.service file and reloaded the daemon using systemctl daemon-reload and restart td-agent , it couldnt start and gave below error.

May 29 16:17:41 ip.ap-south-1.compute.internal systemd[1]: Unit td-agent.service entered failed state.
                May 29 16:17:41 ip.ap-south-1.compute.internal systemd[1]: td-agent.service failed.
                May 29 16:17:42 ip.ap-south-1.compute.internal sshd[17413]: Failed password for root from 218.92.0.208 port 64211 ssh2
                May 29 16:17:44 ip.ap-south-1.compute.internal sshd[17413]: Failed password for root from 218.92.0.208 port 64211 ssh2
                May 29 16:17:44 ip.ap-south-1.compute.internal sshd[17413]: Received disconnect from 218.92.0.208 port 64211:11:  [preauth]
                May 29 16:17:44 ip.ap-south-1.compute.internal sshd[17413]: Disconnected from 218.92.0.208 port 64211 [preauth]
                May 29 16:17:44 ip.ap-south-1.compute.internal sshd[17413]: PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.208  
                May 29 16:17:55 ip.ap-south-1.compute.internal dhclient[3086]: XMT: Solicit on eth0, interval 111140ms.

I think td-agent must run with user ''td-agent" and cannot run as root otherwise ssh2 fails.
So i reverted the changes but then what is causing the persmission issue to nginx access.log which has 777 permission?

[JayTeli]

Issue Resolved.
chown -R root:td-agent /var/log
chmod -R 775 /var/log/

[mmahkamov]

Changing ownership & access in /var/log is a bad idea.

/var/log/nginx is owned by the www-data user and by the adm group.

Just add td-agent to the adm group:

sudo usermod -aG adm td-agent