Beats: Module configuration variants

[ycombinator]

Motivation

Today modules come with exactly one default configuration file, modules.d/<module>.yml.disabled. When the module is enabled, this file is replaced by modules.d/<module>.yml.

Sometimes it might be useful to have a configuration file for a module that is a slight variant of what we provide today. For example, consider the elasticsearch module in Metricbeat. It's default module configuration file looks like this:

# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-elasticsearch.html

- module: elasticsearch
  #metricsets:
  #  - node
  #  - node_stats
  period: 10s
  hosts: ["http://localhost:9200"]
  #username: "user"
  #password: "secret"

However, if we want to configure this module for X-Pack Monitoring, we need the module configuration file to look like this:

# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-elasticsearch.html

- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["http://localhost:9200"] 
  #username: "user"
  #password: "secret"
  xpack.enabled: true 

Today, a user would have to metricbeat modules enable elasticsearch and then manually make the necessary changes to modules.d/elasticearch.yml to configure it properly for X-Pack Monitoring.

Proposal

We ship not only modules.d/elasticsearch.yml.disabled but also modules.d/elasticsearch-xpack.yml.disabled. If a user runs metricbeat modules enable elasticsearch the first disabled file is enabled. But if the user runs metricbeat modules enable elasticsearch-xpack the second disabled file is enabled.

This has the implication that module names cannot contain - but I think that's a reasonable limitation? Of course, we can choose another delimiter as well, as long as it works well across OSes.

[ycombinator]

@ruflin would love your thoughts on this, as well as from anyone else you feel should be included in this discussion. Even though the proposal above is motivated by the X-Pack monitoring use case (hence I used that example as well), I think we can make module configuration variants a more generic feature of beats?

[jsoriano]

@ycombinator great idea, and actually this already works:

# cp /etc/metricbeat/modules.d/elasticsearch.yml.disabled /etc/metricbeat/modules.d/elasticsearch-xpack.yml.disabled
# metricbeat modules list | grep elasticsearch
elasticsearch-xpack
elasticsearch
# metricbeat modules enable elasticsearch-xpack
Enabled elasticsearch-xpack
# metricbeat modules list
Enabled:
elasticsearch-xpack
...

This would be also a way to partially implement the "lightweight" modules I was thinking of to distribute for example jolokia configurations for specific java services.

Not sure though if there is something that prevents us to distribute configuration files with different names to actual code modules in code.

[ycombinator]

Thanks @jsoriano. Indeed, this is something we can already implement pretty easily, as you demonstrated. I guess my question is more about whether we should do this — would it be okay if I added a modules.d/elasticsearch-xpack.yml.disabled file to the repo? Just want to make sure it's not going to cause any issues or if there's a better UX to achieve this idea or something else?

[jsoriano]

As the files in modules.d are generated, I'd propose to add these "config flavours" to the modules directories, for example the one you mention would be added as module/elasticsearch/_meta/config-xpack.yml. Then the module collector should be adapted to generate all modules.d/{module}-{flavour}.yml.disabled files from module/{module}/_meta/config-{flavour}.yml.
Original configs would stay the same.

Edit: Other option if we want to be able to make completelly arbitrary names would be to generate modules.d/{variant}.yml.disabled from module/{module}/_meta/config.{variant}.yml, and in your case the file added would be in module/elasticsearch/_meta/config.elasticsearch-xpack.yml.

[ruflin]

I like the idea of module flavors. I'm pretty sure we can use this also in other places. Could we perhaps use a / to separate the flavors or :? Would be similar to docker. : could also be used to separate the version of a module if we have that at one stage.

When I read this an other additional idea popped up on my end. What if you could run the following:

metricbeat modules enable elasticsearch-xpack -P hosts=["my-config-host"] -P username="foo"

The idea is that we would not only enable the module config but also directly set some config options on it written to the file. Like this the user could fully configure a module with the command line. If it's a good idea to pass passwords through the command line is an other question. I picked -P (Param) just as an example, we should come up with a better name.

[exekias]

This sounds good to me in general. I'm wondering if some users may end up believing that there is an actual module named elasticsearch-xpack. As of today, all files under modules.d map to a module in Metricbeat.

Probably using the - is good enough as a notation for that?

[ycombinator]

@ruflin said:

Could we perhaps use a / to separate the flavors or :?

I'm okay with using either of these as a delimiter for the flavor in the CLI argument for specifying the flavor. However, when it comes to the corresponding config file on the filesystem, I think it's safest to go with - or _ in the filename. So we would basically look at the CLI argument and replace any : with - (as an example).

When I read this an other additional idea popped up on my end.
...

I like this idea! I would implement it in a separate PR though, as a follow up enhancement, especially since it's orthogonal to the module variants idea (the module args idea can be applied even to modules as they stand today). Also, there might some more discussion required here as well around idempotency, multiple config blocks of the same metricset, etc.

---

@exekias said:

I'm wondering if some users may end up believing that there is an actual module named elasticsearch-xpack.

Good question. One way to remove this confusion would be to make the variant more explicit in the CLI command, something like the following?

metricbeat modules enable elasticsearch --variant xpack

It requires more typing but it's clearer that the actual module here is still elasticsearch, not elasticsearch-xpack.

[EDIT] We could also support both CLI syntaxes, short hand (e.g. metricbeat modules enable elasticsearch:xpack) as well as long hand (e.g. as shown above).

[jsoriano]

Take into account that if we add additional flags or syntaxes we'd need to modify also metricbeat modules list and other commands to be consequent, we might also need to review filebeat to give a similar support to it, one may ask if all beats have support for "module variants".

What I liked of metricbeat modules enable elasticsearch-xpack is that it is simple. It maintains the interface we have now to enable and disable modules and we only have to adapt how we collect modules.d directory to support it, to distribute different configuration cases for the same module.

I think other modules that currently have complete commented out sections could also benefit from that (think for example on system-raid, system-fsstats or kubernetes-events).

Also think that hopefuly users end up using configuration management tools and just placing specific yaml files in the modules.d directories, I don't think we should think so much in advanced uses of metricbeat modules subcommands.

[ycombinator]

Taking into account the various syntaxes and their pros/cons discussed so far, let me try to propose something that hopefully is simple to implement but also less likely to cause confusion for users:

Consider a module, elasticsearch, with one variant, xpack.

In modules.d/ there will be two files: elasticsearch.yml.disabled (same as today) and elasticsearch.xpack.yml.disabled.

To enable the "normal" (i.e. no variant) module, the user would run:

metricbeat enable elasticsearch

This would cause modules.d/elasticsearch.yml.disabled to be moved to modules.d/elasticsearch.yml.

To enable the xpack variant of the module, the user would run:

metricbeat enable elasticsearch:xpack

This would cause modules.d/elasticsearch.xpack.yml.disabled to be moved to modules.d/elasticsearch.xpack.yml.

---

The reason I am going with @ruflin's proposal of using : as the delimiter in the CLI argument is because I think it's probably quite unnatural to have : in an actual module name. Comparatively, I can see a product or service that actually has - in its name and so we might want to name it's monitoring/logging module the same. I think using : in the CLI should also alleviate @exekias's concern to some extent?

The reason I am proposing using . as the delimiter in the variant module filename is that we already use it as a delimiter in module filenames! No need to introduce a new one.

And finally, this proposal keeps it relatively simple by not adding any more flags to the CLI, hopefully addressing @jsoriano's concerns.

Thoughts everyone?

[jsoriano]

I'd avoid the use of - in beats modules because - has special meaning in golang package imports, so I think it'd be safe to go on with this option.
We currently have a module with two words separated by underscore (php_fpm), even if we can discuss if this is something appropriate for go module names, it is not problematic and we could continue with this for other cases.

In any case I am fine with : too, but is there any reason to don't have : as the delimiter in the filename? So we wouldn't have to do any modification to modules subcommands to interpret the file names:

# cp /etc/filebeat/modules.d/elasticsearch.yml.disabled /etc/filebeat/modules.d/elasticsearch:xpack.yml.disabled 
# filebeat modules list | grep elasticsearch
elasticsearch
elasticsearch:xpack
# filebeat modules enable elasticsearch:xpack
Enabled elasticsearch:xpack
# filebeat modules disable elasticsearch:xpack
Disabled elasticsearch:xpack

(Notice that I was also checking that it works with filebeat too)

[ycombinator]

In any case I am fine with : too, but is there any reason to don't have : as the delimiter in the filename?

The only reason I was avoiding : in filenames is that POSIX specifies : as a path separator. From http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08_03:

PATH
This variable shall represent the sequence of path prefixes... The prefixes shall be separated by a ( ':' )...

Obviously that _should_ mean it's safe to use in _filenames_ but personally I felt if we could avoid it, there would be no chance for any ambiguity or confusion for users.

[ruflin]

@ph I want to bring your awareness to this issue as I think it also ties into the discussion around different inputs for 1 fileset in Filebeat.

For the ones not up-to-date on this discussion: How will an nginx/access fileset work if the data comes from journald, file or syslog? It will need different preprocessing and how do we let the user configure which input he wants for a module? I think the two problems do not necessarly conflict with each other but we should make sure when implementing the above long term it does not conflict with the second problem.