Beats: Add integration tests for Kubernetes

[exekias]

We want to improve our coverage with integration tests for Kubernetes. We should explore our options and come up with a solution that allow us to run Filebeat & Metricbeat against a running cluster.

In order to simplify this we can start by targeting testing a single version of Kubernetes (latest) in Jenkins jobs. We can reuse this in the future for compatibility matrix based testing.

There is a bunch of pieces we can test for, I think these would be the most important:

• Metricbeat Kubernetes module

  • retrieving kubelet performance related metrics
  • apiserver, proxy, controllermanager and others
  • kube-state-metrics? (I would make this optional for the first iteration)

• Metricbeat autodiscover

  • Check that a module is launched when a pod is created
  • Check the module stops when the pod is deleted

• Filebeat autodiscover

  • Check that a container input launched when a pod is created
  • Check the input stops when the pod is deleted (after 60s of cleanup_time)
  • Test hints based autodiscover (ie a JSON output)

There are several paths we can follow to do this. For instance:

• leveraging GKE and run tests there, but this would mean we need to build & upload a docker image for each beat.
• Use kind to deploy a local cluster, run MB/FB from "outside", just the local binary. There are some previous experiences using it here: https://github.com/elastic/metricbeat-tests-poc/pull/100

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[blakerouse]

Once #12792 is completed testing for metric-server should also be added. If leveraging GKE is the approach metric-server comes already running.

Another option for the docker images with the cloud providers would be to spawn a GKE (or EKS, AKS) cluster, deploy a private registry into the cluster, then build the images and push them there, then creation of pods could pull from the registry added to the cluster.

[exekias]

I've revisited our current status and some things happened recently. Tests are already using Kind: https://github.com/elastic/beats/blob/master/.ci/scripts/kind-setup.sh

They also test against several versions: https://github.com/elastic/beats/blob/c71b424e11b7a80a2ebe62ae525727a21624a34a/Jenkinsfile#L587

It would be worth testing if this is enough already (ie it allows us to run stuff or retrieve logs)

[blakerouse]

@exekias Been looking into the current usage of kind. It is rather shallow, it is only verifying that the manifests are being deployed.

I think we need to extend this to at least verify that all the required pods are in a ready state, then on top of that verify that metrics are flowing. That probably means also deploying elasticsearch into the cluster to verify with.

It also seems that all of the kubernetes integration tests are disabled in metricbeat/module/kubernetes, I guess do to flakiness but I am unable to find an filled issue to show where and why this was done.

[blakerouse]

With the references PR above, the kubernetes module has good coverage in metricbeat. That still leaves adding the following:

• Container Logs (filebeat)
• Container Logs Autodiscovery (filebeat)
• Container Metrics Autodiscovery (metricbeat)
• Container Heartbeat Autodiscovery (heartbeat)