Kibana: [Fleet] Allow to config Agent log level

[nchaulet]

Description

Related to https://github.com/elastic/beats/issues/20756
We want to allow an user to change the log level for an agent

We decided that we are going to create a new agent action for that, and to not have a concept of per agent policy.

Details

Kibana Change

We can use the existing route POST /api/fleet/agents/{agentId}/actions to create a new AgentAction to update the log level for an agent.

Agent action

{
  "type":  "SETTINGS",
  "data" : {
    "log_level": "debug"|"info"|"warning"|"error"
  }
}

When the agent receive this action it should update the log level, for itself and his subprocesses

@michalpristas can the agent send the log_level as part of the metadata?

UI change

@hbharding How do we want to present the ability to configure the log level for one Agent?

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

@ruflin @blakerouse WDYT?

[ruflin]

Lets assume the user shipped down the specific log level for an Agent. What happens if the Agent is restarted? Will it keep the log level? Where can a users see what log level an Agent is running on?

Lets make sure we align with https://www.elastic.co/guide/en/ecs/current/ecs-log.html on the allowed log levels / naming. I think mostly we already do but wanted to mention it.

[blakerouse]

I think this is acceptable, but I would prefer to make it more generic. We might add other settings to change per Agent and we should be able to update those using the same action type.

{
  "type":  "SETTINGS",
  "data" : {
    "log_level": "debug"|"info"|"warning"|"error"
  }
}

We will need to persist this change so that on restart the Agent starts back at this log level. At the moment we read the log setting from elastic-agent.yml but Agent doesn't update that file once running. It only updates fleet.yml. So I would think that we update fleet.yml with this log setting, and it would override the setting set it elastic-agent.yml.

[nchaulet]

I am good with having a more generic action SETTINGS 👍

[nchaulet]

@michalpristas do you think it's possible to add the current log_level to agent metadata ?

[hbharding]

Hey there. Happy to help with a design. A few questions first:

1) Why was it decided that this should be a per-agent action and not an agent policy setting? I don't have a strong opinion one way vs the other, mostly curious on rationale. I'm guessing that setting an agent log's level to debug at a policy level would generate an unnecessary/overwhelming number of log events that might impact performance.
2) Assuming this is an agent action, should we also support this as a bulk action? i.e. A user selects multiple agents from the Fleet agent table to change the log level instead of having to do this manually one by one. I ask because currently all of the actions that a user can take on the agent detail page are also presented on the agent list page and are available for bulk actions.
3) What is an agent's default log level?

From @ruflin and related to @nchaulet's last question to @michalpristas:

Where can a users see what log level an Agent is running on?

I think we would show this information on the agent detail page where we currently list some agent metadata information.

cc @mostlyjason for visibility

[nchaulet]

@hbharding really good questions.

1. I think due to the volume of data that can generate we want to allow it to set it per agent.
2. We probably do not need a bulk action, as I imagine the feature will be used, is I go to the agent detail page, I saw an error want more information set the log level to debug, get my logs, then I reset the log level to info
3. by default the agent is in info log level

[ph]

@nchaulet This sound good to me, FYI @mostlyjason

[michalpristas]

discussed offline: agent will send current log level as part of local metadata