Telegraf: open /var/run/ceph: permission denied for ceph output plugin

[JohnHaan]

Bug report

Relevant telegraf.conf:

[[inputs.ceph]]
ceph_binary = "/usr/bin/ceph"
socket_dir = "/var/run/ceph"
mon_prefix = "ceph-mon"
osd_prefix = "ceph-osd"
socket_suffix = "asok"

System info:

Ubuntu 16.04
telegraf 1.0.0-beta3
ceph jewel 10.2.1
influxDB 0.9.6

Actual behavior:

2016/08/22 14:59:20 Starting Telegraf (version 1.0.0-beta3)
2016/08/22 14:59:20 Loaded outputs: influxdb
2016/08/22 14:59:20 Loaded inputs: ceph
2016/08/22 14:59:20 Tags enabled: host=ngcephmon01 user=ceph
2016/08/22 14:59:20 Agent Config: Interval:10s, Debug:false, Quiet:false, Hostname:"ngcephmon01", Flush Interval:10s
2016/08/22 14:59:30 ERROR in input [ceph]: failed to find sockets at path '/var/run/ceph': Failed to read socket directory '/ var/run/ceph': open /var/run/ceph: permission denied

Use case: [Why is this important (helps with prioritizing requests)]

[JohnHaan]

I resolve the problem that ceph group includes telegraf user.
But, this is work-around I think.

Moreover, after resolving this problem. I faces another issue below..

2016/08/22 15:30:32 Agent Config: Interval:10s, Debug:true, Quiet:false, Hostname:"ngcephmon01", Flush Interval:10s
2016/08/22 15:30:40 error reading from socket '/var/run/ceph/ceph-mon.ngcephmon01.asok': error running ceph dump: exit status 22

As I command manaully, it would worked.

root@ngcephmon01:~# ceph --admin-daemon /var/run/ceph/ceph-mon.ngcephmon01.asok perf dump
{
    "cluster": {
        "num_mon": 1,
        "num_mon_quorum": 1,
        "num_osd": 47,
        "num_osd_up": 47,
        "num_osd_in": 47,
        "osd_epoch": 53074,

I don't know What is problem..

[alahiff]

I also have just experienced this permissions issue. In my case adding the user telegraf to the ceph group didn't help as only the ceph user has permission to access the sockets. As a workaround I've started running Telegraf as user ceph (!), but with this the Ceph input plugin works successfully and we can at least see if all the metrics we require are available.

[JohnHaan]

Thank you alahiff!
I worked when I execute telegraf as a ceph user.

[sparrc]

@spjmurray @robinpercy-xm is it expected that the ceph plugin needs to be run under the ceph user in order to access the socket stats?

[spjmurray]

@sparrc You need superuser access pre-infernalis according to my production cluster. I can spin up a Jewel cluster in OpenStack to confirm the post-infernalis behaviour requires the ceph user, but as of infernalis ceph runs unprivileged as its own user, so I'd be unsurprised if the socket belonged to the process user. Worth noting is that there is a config switch where you_can_ run a root (to prevent having to chown all your objects post upgrade) just to make matters more interesting still :)

[sparrc]

I believe the conclusion here is that the telegraf user needs to be added to the ceph group. Please let me know if something else needs to be done

[runsisi]

i m running a Ceph Jewel cluster, i think adding telegraf to ceph group is not enough, e.g.,

[root@ceph192-33-1-9 ceph]# id telegraf
uid=992(telegraf) gid=987(telegraf) groups=987(telegraf),167(ceph)

[root@ceph192-33-1-9 ceph]# ll ceph-osd.14.asok 
srwxr-xr-x 1 ceph ceph 0 Oct  2  2016 ceph-osd.14.asok
[root@ceph192-33-1-9 ceph]# sudo -u telegraf ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok perf dump  
admin_socket: exception getting command descriptions: [Errno 13] Permission denied
admin_socket: exception getting command descriptions: [Errno 13] Permission denied

if i add w permission for ceph group, then everything works as expected, e.g.,

[root@ceph192-33-1-9 ceph]# chmod g+w ceph-osd.14.asok
[root@ceph192-33-1-9 ceph]# ll ceph-osd.14.asok 
srwxrwxr-x 1 ceph ceph 0 Oct  2  2016 ceph-osd.14.asok
[root@ceph192-33-1-9 ceph]# sudo -u telegraf ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok perf dump
{
    "AsyncMessenger::Worker-0": {
        "msgr_recv_messages": 48940,
        "msgr_send_messages": 48804,
        "msgr_send_messages_inline": 48643,
...

[bennco1]

Hello can anyone advise how exactly to have telegraf run as root user? I am experiencing similar issues. I am able to pull cpu, mem etc but nothing related to ceph.

2016/10/26 16:41:40 error reading from socket '/var/run/ceph/ceph-osd.112.asok': error running ceph dump: exit status 22
2016/10/26 16:41:40 error reading from socket '/var/run/ceph/ceph-osd.126.asok': error running ceph dump: exit status 22
2016/10/26 16:41:40 error reading from socket '/var/run/ceph/ceph-osd.141.asok': error running ceph dump: exit status 22
2016/10/26 16:41:40 error reading from socket '/var/run/ceph/ceph-osd.153.asok': error running ceph dump: exit status 22
2016/10/26 16:41:40 error reading from socket '/var/run/ceph/ceph-osd.211.asok': error running ceph dump: exit status 22

[spjmurray]

Ceph's permissions are the problem here. They should create the Unix socket as 775, try seeing what they say for starters

[sparrc]

since there have been so many questions about ceph permissions, there seems to be a need for the doc to be more informative. If anyone can edit this doc: https://github.com/influxdata/telegraf/blob/master/plugins/inputs/ceph/README.md with details about permissions (chmod commands to run, groups to add telegraf user to, etc.) & submit a PR I would appreciate it a lot.

[bennco1]

@spjmurray Thanks I did confirm perviously it's a permissions issue by executing the following sudo -u root ceph --admin-daemon /var/run/ceph/ceph-osd.87.asok perf dump, but I am unsure how to resolve it from a telegraf perspective.

[sparrc]

has anyone tried just sudo chmod 775 <socket_path>?

[runsisi]

@sparrc see my comment above.
i will create a PR on ceph to let the user to configure the admin socket file mode.

[sparrc]

@runsisi do you think you could add step-by-step instructions to https://github.com/influxdata/telegraf/blob/master/plugins/inputs/ceph/README.md?

You don't need to clone the repo, you can just edit the file in-browser and submit a PR 😬 🙏

[runsisi]

@sparrc OK, i will try :)

[JohnHaan]

It perfectly works!!

Thanks a lot!

2017-03-08 6:42 GMT+09:00 José Ricardo Fernandes [e-mail]:

telegraf runs with 'telegraf' user on systemd, so I fix this 'issue'
changing the "User" variable on "/lib/systemd/system/telegraf.service" to
user 'root', then execute his command "systemctl daemon-reload" and restart
the telegraf service.

Why not run telegraf with root user?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/influxdata/telegraf/issues/1657#issuecomment-284868625,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADlcfuWxI7gRs1EsKDRSQ86Ulp8T--L_ks5rjc8sgaJpZM4Jpiqy
.