Cloud-on-k8s: Support APM Agent Config Management:

[pebrc]

Starting with 7.3 APM server includes agent config management which requires a Kibana connection. We could facilitate the establishment of that connection in a similar way as we do for the Elasticsearch output by allowing users to simply specify a reference to a Kibana resource.

We would also need to specify a Kibana user with the correct permissions

[simitt]

FYI starting with 7.5 Agent Config Management will be GA and supports backend and RUM agents.

[simitt]

In ESS agent configuration management is turned on by default. The mentioned default connection to Kibana would improve usability when making use of this feature.

[barkbay]

I started to work on this issue while I'm experimenting with the APM Server.

I guess it doesn't make sense to just have a Kibana reference in the APM Server specification ? IIUC Kibana and the APM Server must share the same Elasticsearch cluster.
But since every object in ECK can freely set its associations it means that a user may produce a weird topology like:

• APM Server is associated to Kibana1 and Elasticsearch1
• Kibana1 is associated to Elasticsearch2

It raises the question of how to deal correctly with the Elasticsearch reference:

• Should we infer the Elasticsearch reference/connection for the APM Server from the Kibana spec ?
• If both references _(Elasticsearch+Kibana)_ are specified in the APM specification then should we give precedence to the ones specified by the user ? But what if the 2 Elasticsearch clusters are not the same ? Should we still proceed, create the 2 required users in the 2 clusters but generate a warning ?

[simitt]

I guess it makes no sense to just have a Kibana reference in the APM Server specification ? IIUC Kibana and the APM Server must share the same Elasticsearch cluster.

Right now APM Server uses the Kibana connection only for querying information managed by Kibana. It does _not_ use the Kibana connection for any index pattern or dashboard setup. Therefore it doesn't directly care about the Elasticsearch instance used by Kibana.

A default reference to the started Kibana instance should be enough, no extra handling for underlying Elasticsearch instances is necessary (from an APM Server point of view). The default reference to ES can be overwritten via APM Server, the default Kibana reference also needs to support user configuration.

[barkbay]

A default reference to the started Kibana instance should be enough

I'm not sure that there is a "default reference", or maybe I didn't get what is it.
With ECK a user can freely create some associations between objects. As of today to create an association between an Elasticsearch cluster and an APM Server user have to create the 2 following objects:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: es-sample
spec:
  version: 7.5.0
  nodeSets:
  - name: default
    count: 3

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-sample
spec:
  version: 7.5.0
  count: 1
  elasticsearchRef:
    name: "es-sample"

Now let say that the user has created a Kibana instance somewhere referencing the Elasticsearch cluster:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kb-sample
spec:
  version: 7.5.0
  count: 1
  elasticsearchRef:
    name: "es-sample"

And let say that the user wants to add a connexion between the APM Server and Kibana, the idea would be to enhance the spec of the APM Server by adding a reference to Kibana in the APM spec, something like that:

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-sample
spec:
  version: 7.5.0
  count: 1
  elasticsearchRef:
    name: "es-sample"
  kibanaRef: # here
    name: "kb-sample"

My questions are:

• What if kb-sample is not referencing es-sample?
• Does it make sense to only specify kibanaRef ?

I don't think we have to answer these questions to have a first implementation. We must add something like kibanaRef in the APM Server specification, we can expect from the user to set both of them, without doing any extra check. It was some random thoughts about the possible ways a user can set elasticsearchRef and kibanaRef.

[simitt]

Sorry for not responding earlier, lost track of this.

What if kb-sample is not referencing es-sample?

IMO we don't need to safeguard against this. In theory users are free to define a Kibana instance pointing to whichever underlying ES instance when running APM Server on-prem. However, for APM UI to work properly you are right that Kibana and APM Server should point to the same Elasticsearch instance, as the information the server queries from Kibana will only be there when APM data exist. What I mean with _default_ is exactly what you show in the example above, providing a straight forward way for users to connect APM Server to an ES and Kibana instance also created by ECK.

Does it make sense to only specify kibanaRef ?

Users can configure outputs different than ES, e.g. using Kafka or logstash as direct output. Therefore only referencing Kibana can be a valid use case. It would be up to the users then to ensure their chosen output sends APM data to the same ES instance as referenced by Kibana. But I would say that is out of our control.

[pebrc]

One question that came up in connection with #2657 is what user should be used for APM agent config management and what privileges does that user need. I tried it with a superuser and that works of course but I assume we want a more restrictive role set.

[simitt]

The APM Server has automated tests for agent config management using a user that has the built in apm_user role. To my knowledge that is the minimum required privileges for the feature to be supported (from APM Server 7.5 on). @sqren can you confirm?

[sqren]

what user should be used for APM agent config management and what privileges does that user need

From a ui perspective the user only needs write access to the APM app in Kibana to be able to see and create configurations.

[barkbay]

@simitt @sqren I tried to used the apm_user user but it does not seem to work.
Even if I have this message in the APM Server logs:

beater/server.go:67     Starting apm-server [64e91c95329991c36b16ff94fd34ea75230c06c2
...
kibana/connecting_client.go:81  Successfully obtained connection to Kibana.

I can see this warning in the Kibana logs which suggests that something is wrong:

{"type":"log","@timestamp":"2020-05-18T08:30:50Z","tags":["warning","plugins","security","api-authorization"],"pid":7,"message":"User not authorized for \"/api/apm/settings/agent-configuration/search\": responding with 404"}

And the configuration is not propagated:

Could you give me more details about the admin role described here: https://www.elastic.co/guide/en/apm/server/current/feature-roles.html#kibana-user-privileges ?

Also do you have any suggestion if we had to design a e2e/"high level" test in ECK to check that the association between the APM Server and Kibana is successful ?

Thank you for your help !

[sqren]

@barkbay Thanks for the heads up - we might have to improve the docs here if they are not clear.

To access the endpoint /api/apm/settings/agent-configuration/search and search for agent configs the user does NOT need to have access to apm-* indices. Instead they will need to have read privileges to the APM app in Kibana. This is handled with the Kibana privileges:
https://www.elastic.co/guide/en/kibana/current/kibana-privileges.html#_base_privileges

Afaik there are no built-in users that handles this so for now the user must create a custom role that has read access to the APM app:

_Example_

@bmorelli25 Do you think we should update the docs to clarify this?

[simitt]

@bmorelli25 and I have spent some time last week identifying the roles and privileges and improving the docs; related issue and WIP PR.