Telegraf: passenger-status path and sudo add-on missing.

[mozdzyp]

Relevant telegraf.conf:

[[inputs.passenger]]
  command = "sudo passenger-status -v --show=xml"

System info:

Centos6
telegraf-1.10.1-1.x86_64

Issue:

User telegraf is a member of the sudoers and on its own is able to execute the command ( from shell ) :

sudo passenger-status -v --show=xml # with the correct xml output.

however with the original configuration as on the telegraf.conf above the telegraf log returns:

2019-03-25T09:51:50Z E! [inputs.passenger]: Error in plugin: exec: "passenger-status": executable file not found in $PATH
2019-03-25T09:51:50Z E! Error in plugin [inputs.passenger]: exec: "passenger-status": executable file not found in $PATH

while setting the configuration to : command = "sudo passenger-status -v --show=xml" returs:

2019-03-25T09:54:20Z E! Error in plugin [inputs.passenger]: exec: "passenger-status": executable file not found in $PATH
2019-03-25T09:54:20Z E! [inputs.passenger]: Error in plugin: exit status 1

setting the full path inb the command , eg : command="/usr/bin/sudo /opt/ruby/bin/passenger-status -v --show=xml" still returns the :

2019-03-25T10:00:30Z E! Error in plugin [inputs.passenger]: exec: "passenger-status": executable file not found in $PATH
2019-03-25T10:00:30Z E! [inputs.passenger]: Error in plugin: exit status 1

the config describers the passenger input path

but the variable path also returns errors as not known variable when defined:

2019-03-25T10:03:08Z E! [telegraf] Error running agent: Error parsing /etc/telegraf/telegraf.conf, line 24: field corresponding to `path' is not defined in `*passenger.passenger'

can the configuration be expanded into adding sudo and corrected to be able to parse the correct path?

[danielnelson]

Can you ensure that all Telegraf instances are stopped and then restart the service?

I believe Telegraf 1.10 only emits the error in this form:

E! [inputs.passenger]: Error in plugin: exit status 1

The error like this must be coming from an older version of Telegraf:

E! Error in plugin [inputs.passenger]: exec: "passenger-status": executable file not found in $PATH

[mozdzyp]

indeed, had an old process running , I have stopped and run again the updated version.
however setting it with the configuration to :

command = "passenger-status -v --show=xml"

however it still returns the output ( telegraf-1.10.1-1.x86_64 ) :

2019-03-26T13:15:30Z E! [inputs.passenger]: Error in plugin: exec: "passenger-status": executable file not found in $PATH

I was unable to set the path in the command or inside the passenger module :(

[glinton]

You should be able to specify an absolute path to passenger-status:

command = "/path/to/passenger-status -v --show=xml"

[mozdzyp]

setting the absolute path to the passenger-status wont work without sudo ( please see the subject of this bug reported ) :
config set to :
command = "/opt/app1/bin/passenger-status -v --show=xml"
will return:
2019-03-26T23:05:50Z E! [inputs.passenger]: Error in plugin: exit status 2

due to fact the telegraf user calling the script directly will not be able to execute it without sudo:

-bash-4.1$ whoami
telegraf
-bash-4.1$ /opt/appi1/bin/passenger-status  -v --show=xml
*** ERROR: You are not authorized to query the status for this Phusion Passenger instance. Please try again with 'sudo'.

setting the command with sudo has beeen described on the first comment ( does not work from the telegraf agent , but it works from command line) : /usr/bin/sudo /opt/ruby/bin/passenger-status -v --show=xml

[mozdzyp]

additionally passenger-status in version 5.0.10 or later does not allow to run it without sudo.
https://www.phusionpassenger.com/library/admin/standalone/overall_status_report.html#to-sudo-or-not-to-sudo

[danielnelson]

Can you double check this command, in both Telegraf and also from a shell:

[[inputs.passenger]]
  command = "sudo /opt/appi1/bin/passenger-status -v --show=xml"

[mozdzyp]

Can you double check this command, in both Telegraf and also from a shell:

[[inputs.passenger]]
  command = "sudo /opt/appi1/bin/passenger-status -v --show=xml"

I did test already. ( please see the first top post )
But OK , please see here another test:

config result with sudo in config ( command = "/usr/bin/sudo /opt/ruby/bin/passenger-status -v --show=xml" )

2019-03-27T09:23:20Z E! [inputs.passenger]: Error in plugin: exit status 1

config result with full sudo path in config ( command = "/usr/bin/sudo /opt/ruby/bin/passenger-status -v --show=xml"` )

2019-03-27T09:23:50Z E! [inputs.passenger]: Error in plugin: exit status 1

and finally sudo from shell:

-bash-4.1$ whoami
telegraf
-bash-4.1$ sudo /opt/appi1/bin/passenger-status  -v --show=xml
<?xml version="1.0" encoding="iso8859-1"?>
<info version="3">
   <passenger_version>5.0.7</passenger_version>
   <process_count>26</process_count>
   <max>100</max>
   <capacity_used>26</capacity_used>
   <get_wait_list_size>0</get_wait_list_size>
   <get_wait_list/>
   <supergroups>
.....

[danielnelson]

In your last comment you show the command as:

command = "/usr/bin/sudo /opt/ruby/bin/passenger-status -v --show=xml"

I just want to make sure the path is exactly the same as in the shell version:

command = "sudo /opt/appi1/bin/passenger-status -v --show=xml"

Try running it first in an interactive shell:

telegraf --input-filter=passenger --test

We should add some addition code to print out stderr when an error occurs.

[mozdzyp]

yes, the path on the shell and the command is the same. ( sorry was on purpose hiding it off from github)
the command : telegraf --input-filter=passenger --test works

passenger,host=xxxxx,passenger_version=5.0.7 capacity_used=26i,get_wait
_list_size=0i,max=100i,process_count=26i 1553680491000000000
passenger_supergroup,host=xxxxx,name=/opt//apps//nane\ (p
roduction) capacity_used=8i,get_wait_list_size=0i 1553680491000000000
....
...

[danielnelson]

Okay, sorry if this is repetitive but I just want to make sure we have the bug properly characterized. Now, if you restart the service and then check the logfile it is still failing?

sudo systemctl restart telegraf
journalctl -u telegraf

[mozdzyp]

Okay, sorry if this is repetitive but I just want to make sure we have the bug properly characterized. Now, if you restart the service and then check the logfile it is still failing?

sudo systemctl restart telegraf
journalctl -u telegraf

I am trying to do this on Centos6 ( with no systemctl )

[danielnelson]

I believe, but don't currently have a Centos6 VM so I'm not 100% sure, that it would be:

sudo service telegraf restart
vi /var/log/telegraf/telegraf.log

[mozdzyp]

[mozdzyp@centos6 ~]$ sudo service telegraf restart
telegraf process was stopped [ OK ]
Starting the process telegraf [ OK ]
telegraf process was started [ OK ]
[mozdzyp@centos6 ~]$ tail -f /var/log/telegraf/telegraf.log
2019-03-27T10:46:40Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-03-27T10:46:45Z I! [agent] Hang on, flushing any cached metrics before shutdown
2019-03-27T10:46:49Z I! Starting Telegraf 1.10.1
2019-03-27T10:46:49Z I! Loaded inputs: swap system netstat cpu disk passenger diskio kernel
2019-03-27T10:46:49Z I! Loaded aggregators: 
2019-03-27T10:46:49Z I! Loaded processors: 
2019-03-27T10:46:49Z I! Loaded outputs: influxdb
2019-03-27T10:46:49Z I! Tags enabled: host=centos6
2019-03-27T10:46:49Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"centos6", Flush Interval:10s
2019-03-27T10:46:50Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-03-27T10:47:00Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-03-27T10:47:10Z E! [inputs.passenger]: Error in plugin: exit status 1

[mozdzyp]

can someone replicate the issue ?

[jpellman]

I'm encountering the same issue on a CentOS 6 host running Telegraf 1.10.2 (git: HEAD 3303f5c3). I also am giving telegraf access to passenger-status via sudo. As with your case, the following commands work as expected:

04/29 11:16:42 [jsp2205@clio-nginx-prod1: /var/log/telegraf]
$ sudo -u telegraf /bin/bash -c '/usr/bin/sudo /opt/nginx/passenger/passenger-5.0.30/bin/passenger-status -v --show=xml' | head
<?xml version="1.0" encoding="iso8859-1"?>
<info version="3">
   <passenger_version>5.0.30</passenger_version>
   <group_count>2</group_count>
   <process_count>25</process_count>
   <max>60</max>
   <capacity_used>25</capacity_used>
   <get_wait_list_size>0</get_wait_list_size>
   <get_wait_list/>
   <supergroups>

And:

04/29 11:16:53 [jsp2205@clio-nginx-prod1: /var/log/telegraf]
$ telegraf --input-filter=passenger --test | head
2019-04-29T15:18:03Z I! Starting Telegraf 1.10.2
2019-04-29T15:18:03Z I! Using config file: /etc/telegraf/telegraf.conf
> passenger,host=xxxxx,passenger_version=5.0.30 capacity_used=25i,get_wait_list_size=0i,max=60i,process_count=25i 1556551083000000000
> passenger_supergroup,host=xxxxx,name=xxxxx capacity_used=2i,get_wait_list_size=0i 1556551083000000000

Error in the logs is the same:

04/29 11:18:03 [jsp2205@clio-nginx-prod1: /var/log/telegraf]
$ tail -f telegraf.log
2019-04-29T15:18:40Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T15:18:50Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T15:19:00Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T15:19:10Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T15:19:20Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T15:19:30Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T15:19:40Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T15:19:50Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T15:20:00Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T15:20:10Z E! [inputs.passenger]: Error in plugin: exit status 1

[jpellman]

Just a guess, but might this be due to how your parseCommand function is written? Based off what I can see, it looks like the code assumes that if you split the string representing the command using space as the delimiter the first item in an array will be the command name, and everything after that will be an argument.

For this particular case, @mozdzyp and I have command strings such that "sudo" would be considered the first item of the array (unless I'm grossly misinterpreting that code).

[jpellman]

While it's still on my mind, why not forego having the parseCommand function in the first place and replace this with something like:

out, err := exec.Command("sh", "-c", g.Command).Output()

[danielnelson]

The way we are doing command splitting here is definitely an issue, though I'm unsure if it is the cause of this particular error. If this were to be changed I would want the plugin to use an array of command arguments, but a backwards compatible use of github.com/kballard/go-shellquote would be reasonable as well. I wouldn't want to use sh though if we can avoid it due to the overhead involved.

Could you check the sudo log for clues? On my system it is written to /var/log/auth.log.

[jpellman]

After running the following:

04/29 15:14:21 [jsp2205@clio-nginx-prod1: /var/log]
$ sudo -u telegraf telegraf --input-filter=passenger --test

/var/log/secure (the CentOS equivalent of /var/log/auth.log) gives the following line:

Apr 29 15:13:54 clio-nginx-prod1 sudo: telegraf : TTY=pts/4 ; PWD=/var/log ; USER=root ; COMMAND=/opt/nginx/passenger/passenger-5.0.30/bin/passenger-status -v --show=xml

Nothing else though, and nothing that corresponds with the timestamps in the telegraf log:

2019-04-29T19:13:00Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T19:13:10Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T19:13:20Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T19:13:30Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T19:13:40Z E! [inputs.passenger]: Error in plugin: exit status 1
2019-04-29T19:13:50Z E! [inputs.passenger]: Error in plugin: exit status 1

[danielnelson]

Can you read over #5779 and see if you both might be having the same issue?

[jpellman]

The requiretty default totally seems to be the issue. Adding the following to a sudoers file got the passenger plugin to work as expected:

telegraf   ALL=(root)    NOPASSWD:/path/to/passenger-status
Defaults!/path/to/passenger-status !requiretty

Similarly, I could've just removed the requiretty default and that should've achieved the same thing.

[danielnelson]

@mozdzyp Let us know if this doesn't work on your system and we can reopen.