Kibana: Support logs-* pattern in Logs UI

[ruflin]

The default of the logs UI is using the filebeat-* index pattern. With the new indexing strategy and the Elastic Agent, these indices will be logs-*, which should be added to the defaults.

Logs AC:

• logs-* is added to the default source configuration
• existing users who have filebeat-* in their source configuration get logs-* appended
• ML flow is checked to make sure we alert users when their source config has been changed so that they can recreate ML jobs

Notes:

• There might be a third pattern events-* and it should be discussed, if this is added too to the logs.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ruflin]

@weltenwort Any issue to add the above?

[weltenwort]

Should be a simple change to add it to the defaults. In order to provide a better UX we should add support for interpreting the dataset and namespace semantics both in the logs ui settings and in the logs stream.

[mostlyjason]

@mukeshelastic can we get this in 7.9? If we don't add it then users won't be able to see logs or metrics from the Agent.

[mostlyjason]

@sorantis Do we need to file a separate GitHub issue for metrics UI? We'd like to get this on the roadmap for 7.9. We want to add this pattern under Metrics > Settings > Metric indices by default. Otherwise, users will not have visibility to their metrics coming from the Agent.

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[sorantis]

@mostlyjason sure, I can see that @sgrodzicki has already adding it to our backlog.

[sgrodzicki]

I'd keep it in a single issue for now as we'll probably cover both at the same time.

[jasonrhodes]

In order to track across the milestones, I created a Metrics ticket #70088

[mostlyjason]

@sgrodzicki is it possible to get this in 7.9? only a few days left

[sgrodzicki]

@sgrodzicki is it possible to get this in 7.9? only a few days left

Yes, we have a PR open (https://github.com/elastic/kibana/pull/70245) and it just got approved, so it should be merged soon and will make it into 7.9.