Telegraf: Telegraf Configuration - Recommended approach for multiple .conf files?

[n1nj4888]

Hi There,

I run Telegraf docker (v1.11.5- Awesome utility by the way!) and originally had one big /etc/telegraf/telegraf.conf file with a single (InfluxDB) output and a few different input plugins (each sometimes with multiple agents) - Something like this structure of /etc/telegraf/telegraf.conf:

/etc/telegraf/telegraf.conf:

[Output Plugin - InfluxDB]
[Input Plugin - local metrics]
[Input Plugin - SNMP] - Agent IP1, Agent IP2, ..., Agent IPx
[Input Plugin - Docker] - Agent IP1
[Input Plugin - Docker] - Agent IP2
...

Now, since I understand that the above single config file causes Telegraf to run the file in series, I was seeing an issue where if Agent IPx was down this could cause the SNMP Input Plugin to timeout before it was able to contact/return results from Agent IPy. So I decided to break the config up into multiple .conf files as follows, all under /etc/telegraf/telegraf.d/ and this time with no main /etc/telegraf/telegraf.conf file:

/etc/telegraf/telegraf.d/telegraf-input-metrics.conf:

[Output Plugin - InfluxDB]
[Input Plugin - local metrics]

/etc/telegraf/telegraf.d/telegraf-snmp-AgentIP1.conf:

[Output Plugin - InfluxDB]
[Input Plugin - SNMP] - Agent IP1

...

/etc/telegraf/telegraf.d/telegraf-snmp-AgentIPX.conf:

[Output Plugin - InfluxDB]
[Input Plugin - SNMP] - Agent IPx

/etc/telegraf/telegraf.d/telegraf-docker-AgentIP1.conf:

[Output Plugin - InfluxDB]
[Input Plugin - Docker] - Agent IP1

/etc/telegraf/telegraf.d/telegraf-docker-AgentIP2.conf:

[Output Plugin - InfluxDB]
[Input Plugin - Docker] - Agent IP2

In doing this, I have a few questions:

(1) With Telegraf command configured to run as telegraf --config-directory /etc/telegraf/telegraf.d in the container, and with main /etc/telegraf/telegraf.conf file _(all split conf files under /etc/telegraf/telegraf.d/)_: I got the following error on starting the container - I assume this means that there must be a /etc/telegraf/telegraf.conf file, even if the --config-directory /etc/telegraf/telegraf.d switch is passed to the telegraf command in the container?

2019-08-30T01:01:52Z I! Starting Telegraf 1.11.5
2019-08-30T01:01:52Z E! [telegraf] Error running agent: No config file specified, and could not find one in $TELEGRAF_CONFIG_PATH, /root/.telegraf/telegraf.conf, or /etc/telegraf/telegraf.conf

(2) To work around the above, I moved one of the config files (/etc/telegraf/telegraf.d/telegraf-input-metrics.conf) to /etc/telegraf/telegraf.conf and was then able to start the application with the following output to the log:

2019-08-30T01:04:48Z I! Starting Telegraf 1.11.5
2019-08-30T01:04:48Z I! Using config file: /etc/telegraf/telegraf.conf
2019-08-30T01:04:49Z I! Loaded inputs: disk diskio kernel mem processes swap system cpu docker docker file file snmp snmp snmp snmp snmp snmp snmp snmp snmp snmp
2019-08-30T01:04:49Z I! Loaded aggregators: 
2019-08-30T01:04:49Z I! Loaded processors: 
2019-08-30T01:04:49Z I! Loaded outputs: influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb

I assume that the many "influxdb" outputs that are shown (last line above) are because each of my config files has the [Output Plugin - Influx DB] header therefore, is the recommended approach to:

Configure /etc/telegraf/telegraf.conf as just the main [Output Plugin - InfluxDB] and configure all the .conf files under /etc/telegraf/telegraf.d/ to just be [Input Plugin - X]?

If so, I assume this means that all the .conf files under /etc/telegraf/telegraf.d inherit the [Output Plugin - InfluxDB] from the main /etc/telegraf/telegraf.conf file so they know where to write the output to?

OR do I really still need to declare the [Output Plugin - InfluxDB] header at the top of each config file under /etc/telegraf/telegraf.d/?

[danielnelson]

The way to think about it is that all of the files: /etc/telegraf/telegraf.conf + everything in /etc/telegraf/telegraf.d are logically combined into one giant config file.

So there are a variety of way to do this, but here is what I recommend as a starting point:

• Delete or comment out all plugins in /etc/telegraf/telegraf.conf, setting only the [agent] table and [global_tags].
• Create one file for each input plugin and one file for each output plugin in /etc/telegraf/telegraf.d:
  
  
  
  • /etc/telegraf/telegraf.d/inputs.disk.conf
  
  
  • /etc/telegraf/telegraf.d/inputs.diskio.conf
  
  
  • /etc/telegraf/telegraf.d/inputs.kernel.conf
  
  
  • /etc/telegraf/telegraf.d/outputs.influxdb.conf
  
  

The /etc/telegraf/telegraf.conf file is required, though it could even be an empty file. This is something we want to change but it turns out to need more refactoring than you might imagine.

You will just have a single [[output.influxdb]] definition.

Feel free to group or split plugins in the telegraf.d directory as desired, for example maybe it's easier for you to keep all the default plugins in one inputs.default.conf file. Or perhaps you will want to split the snmp plugins by the type of device.

Let me know if you have more questions.

[n1nj4888]

Thanks for the info, I've now aligned the config accordingly...

[n1nj4888]

hi @danielnelson,

I've got the above working on an Ubuntu host but when using the official Telegraf Docker image on a Synology NAS, there doesn't seem to be a way through the Synology Docker GUI to change the telegraf command to include the --config-directory switch (i.e. to be "telegraf --config-directory /etc/telegraf/telegraf.d)... As such the Telegraf container only seems to monitor the /etc/telegraf/telegraf.conf file and no others (no extra conf files are picked up under /etc/telegraf or /etc/telegraf/telegraf.d)....

Is there a way to specify the --config-directory switch through either the container environment variables (which can be amended via the Synology Docker GUI) OR in the initial /etc/telegraf/telegraf.conf file itself to get Telegraf to also look in /etc/telegraf/telegraf.d/ ?

Thanks!

[n1nj4888]

I figured this out - Although there is no option to edit the telegraf container command on an existing Synology Telegraf container, the option is there to change the execution command when creating a new container.

Thanks!

[danielnelson]

Glad you have it going, #5656 is still on the plan to simply this a bit and perhaps we can improve the documentation on docker hub.