Elasticsearch: DLS bit_set_cache usage reported from master

[henningandersen]

ES version 7.3+

7.3 introduced a new cache for document level security. As part of it, _xpack/usage was extended to have some stats on the DLS bit_set_cache. However, the usage action runs only on the master node, which will not have anything in the cache for pure master nodes.

We should ensure that diagnostics dumps contain the dls cache usage from all data nodes instead.

[elasticmachine]

Pinging @elastic/es-security

[tvernum]

This is a problem for pretty much all of the xpack usage action.

There's a lot of things in that response that can vary by node, however we only report on them on the master.
For security, most of what's in that response is actually node specific. Much of the time the master values are likely to be as good as any other node, but not always.
The DLS cache is an obvious example, but the realm caches are much more meaningful from a coordinating node than the master.

I suspect we should revist the xpack usage action in its entirety and change some sections to return per-node results.
However, the will affect diagnostics and telemetry, so we'll need to coordinate whatever we change.

[jasontedor]

We discussed this during our team fix-it meeting this morning. One question that we have is whether or not we really need to be tracking DLS bit set cache usage in the usage API? It feels more like it's being used here to report stats, not so much feature usage. If that's the case, maybe we need to consider how it can be that plugins expose additional stats to the node stats API, rather than twisting the X-Pack usage API around so that usage can be reported at the node level.