Beats: Move away from kubelet stats/summary

[exekias]

Kubelet stats/summary API will be deprecated in Kubernetes 1.16, all the details can be found here: https://github.com/kubernetes/kubernetes/issues/68522

We need to come up with a plan to move away from it while retrieving the same metrics in a backwards compatible way.

cc @odacremolbap, maybe you have some knowledge here

[exekias]

cc @andresrc @sorantis, we need to plan ahead for this one, as 1.16 is approaching

[odacremolbap]

I might be wrong but I was at a recent instrumentation meeting and I think those metrics are going away in 1.17.
Anyway, yep this is needed, and although we need to figure out our version policy (which is a matter that prevented the API revisiting from hitting 7.3, I would start with this tasks right away and set compatibility to the _latest versions_ (TBD).

That version policy matter is not trivial but might have some simple solutions that we can discuss at a different issue.

[odacremolbap]

by kubernetes 1.18 the alternative to the summary endpoint should be ready to use.
It won't be until 1.20 that it will be set to disabled by default, and users will need to take action and deal with the problem.

[exekias]

We aim to support the last 3 minor releases of k8s, with best effort for the rest, as these are the versions officially supported by the kubernetes community. @odacremolbap that makes me think we may need to support both the old and the new method for a while? (with some version detection).

[odacremolbap]

according to https://github.com/kubernetes/kubernetes/issues/68522

• 1.18 will set enable-cadvisor-endpoints to disabled
  afaik metricbeat is not using the kubelet's cadvisor API directly

• 1.19 will add/deprecate enable-container-monitoring-endpoints
  yet the summary API will be available

• 1.20 will completely remove cadvisor direct API

• 1.21 will set the summary API to disabled by default
  we should:

  • opt1: ask users to set it to true to be able to access cadvisor at the kubelet
  • opt2: ask users to install cadvisor as a daemonset, then point metricbeat to the new endpoints
  • opt3: we might decide to use a different solution to cadvisor if that makes sense

• 1.23 remove summary API

  • see opt2 and opt3 from 1.21

At this moment it looks like moving to cadvisor as a daemonset is our best choice.

[blakerouse]

I think the move to using https://github.com/kubernetes-sigs/metrics-server over pulling metrics directly from kubelet is the approach for the future.

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[ChrsMark]

Giving a heads-up on this, it seems that it is currently stalled on k8s side. We can keep tracking https://github.com/kubernetes/kubernetes/issues/68522 for updates.

[exekias]

Thank you @ChrsMark! I'm closing this, let's reopen if we see the need