Beats: Add option for global kubernetes autodiscover

[jsoriano]

Currently, beats in kubernetes are thought to be run as a daemon set where each beat collects information from the pods running in the same node, and from the node itself.

Autodiscover is built on this premise and it only discovers pods running in the same node.

Running the beat in the same node as the observed pods is necessary for example in filebeat because it needs access to local files, but it doesn't need to be necessary in metricbeat modules, that could be connecting to network endpoints.

_(The kubernetes module is an exception to that, it collects information from the API and don't need to be associated to any node, and is actually recommended to be run from a single Deployment per cluster)_

The proposal is to add an option to remove this dependency between autodiscover and the node where the beat is running, so, when possible, pods are discovered from a different node. This would allow different deployment architectures in special cases:

• Monitoring services from an endpoint external to the cluster, this can be quite interesting for heartbeat for example.
• Collect metrics from a single metricbeat pod. For example to collect prometheus metrics from all the cluster from a single metricbeat pod.

[beckjim]

👍
This is actually a requirement for #10292 which is very similar to this one and is for collecting logs from the kubernetes API.

[exekias]

:+1: to doing this, I would add a new parameter to add_kubernetes_metadata and kubernetes autodiscover provider, how about scope, with possible values of node (default) or cluster?

[beckjim]

👍 yes, such a scope would be very helpful. Actually I can see the following scope/namespace patterns (obviously requiring appropriate permissions):

• node / all namespaces
• node / single namespace
• cluster / all namespaces
• cluster / single namespace

At the moment I use a simple hack to the source code to disable (comment out) the check for the current node to get the cluster scope working for me, because I urgently needed it.
Unfortunately I'm not quite sure about side effects. I'm not familiar with the k8s library being used ...

[exekias]

namespace scope is a fair request. I wonder how you would use a theoretical node+namespace request. I wonder if for this case it would be better just to use node scope and then filter on namespace.

[nithril]

It would make sense too for heartbeat. IMHO having a daemonset is not really necessary.

[exekias]

14738 added the scope parameter that should provide this feature starting with 7.6