Telegraf: Multiple Telegraf instance service support on windows

[ashuw018]

Directions

Hi,

I was looking out for way to have multiple instances of telegraf running on a single windows machine, in fact i have done my all of the testing of TICK stack this way only by triggering every telegraf instance from separate elevated CMD window.

1. First CMD windows was running C:\Program Files\telegraf\telegraf .exe - this was sending standard host metrics

2. Second CMD windows was running C:\Program Files\telegraf_ps\telegraf .exe - this was sending metrics using some custom powershell scripts

3. Third CMD windows was running C:\Program Files\telegraf_SQL\telegraf .exe - this was sending SQL metrics by connecting them

Everything was working as expected with it, however then i have decided to run all of these instances as windows service using given command. for first instance it is working fine. then when i tried setting up another instance i figured that the service install command do not offer a way to install another instance by using different name.
"C:\Program Files\Telegraf\telegraf.exe" -config C:\Program Files\telegraf\telegraf.conf

Then i tried several options to setup another instance but no success.
Tried setting nssm - service installed but does not start. it work if i disable original instance of telegraf.
Tried setting windows startup scheduled task but shockingly that is also not working.
Tried creating custom batch and set the scheduler target to use this batch file but that is also not working.

Tried renaming the telegraf.exe

Don`t know what is causing it as from separate CMD windows it works.

Is there any flag in service install command by which we can setup services using different name ?

System info:

Telegraf 1.5.1 (tried earlier versions as well)
Windows Server 2016
Windows 10

Steps to reproduce:

Install telegarf as service "C:\Program Files\Telegraf\telegraf.exe" -config C:\Program Files\telegraf\telegraf.conf

Then try to install telegarf as service using differnt name using nssm like tool.

Expected behavior:

1. Service install command should have option to configure multiple instances of telegarf using different name

2. Or at-least we should be able to setup multiple instances using other tools like nssm

Actual behavior:

Only first instance works

Use case: [Why is this important (helps with prioritizing requests)]

As telegraf has many inputs plugins, so it will be better to have ability to run multiple instances of it each of the instance will have its own set of output, inputs , logs, interval setting etc.

[kerams]

https://github.com/influxdata/telegraf/pull/2754#issuecomment-298774676 is the problem. Nonetheless, what you're trying to achieve is more trouble than it's worth in my opinion.

[ashuw018]

@kerams
Actually this is being requirement for me, as we have a SNMP device an we need to capture its details in Influxdb. telegraf was not working with this device over snmpv3, therefore we have identified one server which is being just behind that snmp device and can collect details using snmpv2. (as from local it is not risky to have it on snmpv2). but the problem is this device already have its telegraf installed which used to send host metrics. and we want different database and interval time to be used by snmp data.

Any clue what can be done as a workaround.

1. snmp data should go to snmp db at 30s interval with hostname as somesnmpdevice
2. host metrics should go to db hostmetrics at 10s interval with hostname as "servername"

Thanks.

[ashuw018]

@kerams also it might be sounding like trouble, but as i am previously used to of New Relic in which they have different agent for different purposes. so i was figuring if we can setup multiple instances of telegraf having different set of inputs, outputs, interval. this way same can be achieved

Thanks.

[kerams]

@danielnelson will correct me if I'm wrong, but I think one instance of Telegraf is able to send data to one database only. Unfortunately, until #2754 is merged, I don't see a solution to what you want to do.

[danielnelson]

You can configure Telegraf to handle this all from one instance, using the measurement filtering options to control where the metrics are routed. Also take a look at this multiple output example.

Another popular way to route metrics is by adding a tag on the input plugins and using that to select the outputs. Here is an example on the community site: https://community.influxdata.com/t/converting-collecd-binary-stream-to-json-with-telegraf-or-collecd-forwarding/1067/9

[ashuw018]

@danielnelson is there any way by which i can build a telegraf from source in such a way that at the time of service installation it uses some different name like "telegraf_snmp" i guess this way i will be able to install two instances of telegraf. 1. telegraf and 2. telegraf_snmp

Thanks.

[kerams]

@ashuw018 Download the latest nightly (link in README.md) and make use of the new parameter -console in conjunction with NSSM.exe. You can use NSSM to install Telegraf multiple times with different names. I'm currently installing the service like so: nssm.exe install Telegraf "c:\Program Files\Telegraf\telegraf.exe" "-console -config ""c:\Program Files\Telegraf\telegraf.conf"" --config-directory ""C:\Program Files\Telegraf\sub""".

[ashuw018]

@kerams Thanks this method helps to setup multiple instances of Telegraf. However i did not understand this "Telegraf\sub"

@danielnelson I am able setup second service instance of telegraf for getting SNMP data using above method. however it is throwing error "snmptranslate": executable file not found in %PATH%. but it is already there in path and can be executed from CMD directly. Also before creating second instance same thing (snmp plugin) was running successfully with an open CMD promt. so dont know why it gives me error if it runs through service and works perfect if it runs in open CMD prompt.

At first i thought i is happening with nightly build only. but for some time i have swapped the config file and added snmp input plugin to my first instance`s config and same error was there.

Thanks.

[kerams]

It allows you to load multiple config files. If you don't use it, you can safely remove the --config-directory part.

[ashuw018]

@kerams ohh Thats cool, i guess this way i will not need multiple instances on telegraf as it can pull multiple .conf. and each .conf file can have it separate input and output configured, am understanding it correct ?

[kerams]

All it does is collect all .conf files from that directory and combine it with the primary .conf file. It should be no different than putting everything in a single file. I have separate config files that define inputs for specific services like RabbitMQ or Redis that I drop into the sub folder where needed. This way you don't need to edit the main config.

[ashuw018]

@danielnelson Sorry for the noise, i figured it out. actually telegraf was running with local-system and path for net-snmp was added under user profile not under system. Now its working.

@kerams Thanks got it. --console method helped me.

[danielnelson]

Sounds like we can close this feature request then now that #2754 is merged.