Telegraf: PCP Plugin Request

[charles-d-burton]

I think it would be very useful to have a plugin to ingest Performance Copilot data. PCP provides a lot of interesting time-series metrics on running systems.

[panda87]

:+1:

[sparrc]

Would it make more sense for PCP to directly be able to output data InfluxDB? Seems a little odd for a collector to collect from another collector :sweat_smile:

For reference, this is the API documentation of PCP (I think interacting with it will require CGO): http://www.pcp.io/doc/pcp-programmers-guide.pdf

And here is a PCP module for directly outputting PCP data to graphite, which would probably be pretty similar to outputting it to InfluxDB: https://github.com/performancecopilot/pcp/tree/master/src/pcp2graphite

[charles-d-burton]

That's a pretty good idea. Do you know if there are any good examples of using python to send data to InfluxDB? Looking through the pcp2graphite code it doesn't look to be too difficult to modify it to send data to Influx.

[sparrc]

@charles-d-burton This is the official influxdb python client, if that helps: https://github.com/influxdata/influxdb-python

[charles-d-burton]

That will do, looks like I just need to figure out what the tuples look like that are being sent to graphite and format them into json to send to influx. Shouldn't be too difficult(famous last words). I think you can close this if you like, unless you think there might still be value added to ingesting via telegraf. The main point I can think of is that telegraf might scale better. Looking at the pcp2graphite it appears that you have to run an instance of that for each host using a push mechanism or alternatively an instance for each pmlogger file. That could incur a lot of overhead when you're talking about a couple of hundred hosts which is my use case.

[sparrc]

I don't think it hurts to have both, so I'll leave this issue open, thanks @charles-d-burton

[skares]

+1

[hc000]

anyone on the influxdb team can help with this? I wrote some go app to collect from PCP. can potentially convert it to telegraf plugin.

[matthew-churcher]

To be clear the reason you might want this vs procstat scraping is. 1. A standard API and names for metrics. 2. Kernel level optimisation for data collection.

There does now appear to be influxdb sender for PCP. So that might be an option for so people. https://github.com/performancecopilot/pcp/tree/master/src/pcp2influxdb

The reasons you might still want to use telegraf, i guess is the family of other plugins.

[matthew-churcher]

Actually this kernel module. Is just scrapping process so the performance is comparable with telegraf. So the only advantage would be access to the standardised metric naming plus access to any metrics or plugins in PCP which telegraf doesn't yet directly implement.

[sjwang90]

Closing this issue if your workaround is fine. Feel free to reopen if desired.