Kibana: Runtime APM UI index configuration

[graphaelli]

Describe the feature:

The APM UI currently relies on kibana.yml settings for configuration of indices to use for APM data. This is a request to allow configuration after kibana has started, similar to what is available for the logs and infra uis.

We may also consider making this space-specific, so that different spaces can be set up to use different APM indices. That may simplify future work regarding spaces for the APM UI.

[elasticmachine]

Pinging @elastic/apm-ui

[graphaelli]

fyi @tbragin

[formgeist]

@elastic/apm-ui Could use some help understanding the implications of configuring the default indices in this way. What would need to happen for us to be able to support it? I'm currently considering where these settings might sit within Kibana, and will also be investigating how users might be able to configure these settings per space in Kibana.

[tbragin]

@graphaelli @formgeist We could start with Kibana Advanced Settings, and not make these space-specific? Similar to the following SIEM setting.

Alternative would be to put this into the APM app itself, like we did with the Logs and Infra UI source configuration. I believe these settings are space-specific.

@roncohen @alvarolobato What do we currently default to when selecting whether to make new functionality we build in Observability apps Kibana Space aware or not?

cc @nehaduggal

[sqren]

The APM UI currently relies on kibana.yml settings for configuration of indices to use for APM data. This is a request to allow configuration after kibana has started

As I understand this proposal the default settings should still come from kibana.yml but be changable at runtime? So if Kibana is restarted it will go back to kibana.yml settings? Or will the runtime changes be persisted somewhere?

Perhaps this overriding behaviour should be generic and built-in to Kibana. So any setting specified in kibana.yml can be changed at runtime, and not only specific APM settings.

[graphaelli]

As I understand this proposal the default settings should still come from kibana.yml but be changable at runtime?

Yes, at least initially for backwards compatibility in 7.x.

So if Kibana is restarted it will go back to kibana.yml settings? Or will the runtime changes be persisted somewhere?

I didn't originally dive into the implementation details but a look at infra-ui shows it stores the configuration as a saved object per space - /api/saved_objects/infrastructure-ui-source/$space. The object is created only when the configuration is updated.

[sqren]

@katrin-freihofner We have decided to move forward with this so I've moved it to the design board. There are probably other higher priority items so just whenever it makes sense for you to look into.

[katrin-freihofner]

If we decide for a per app settings page this could be solved by: https://github.com/elastic/kibana/issues/40236

[sqren]

I'm probably missing something but how is this issue related to #40236?

40236 is about re-designing the header, whereas this issue is about a new settings page.

[katrin-freihofner]

@sqren sorry for the confusion. This is what I'm working on in the header issue:

If we want a settings page per app, we could navigate to it by one of the links/tabs in this header.

[katrin-freihofner]

I'm not sure what exactly is needed from a design perspective to get this going?

[sqren]

I'm not sure what exactly is needed from a design perspective to get this going?

Currently the link to "Settings" takes the user to the agent configuration page. This issue is about creating a page where the user can configure which indices to use for apm (eg. apm-*-transaction-*)

Kibana has an "Advanced Settings Page":

Should it look like that? LMK if you want to zoom about this.

[katrin-freihofner]

As discussed via zoom with @sqren we need an APM specific settings page where the following can be configured:

apm_oss.sourcemapIndices
apm_oss.errorIndices
apm_oss.onboardingIndices
apm_oss.spanIndices
apm_oss.transactionIndices
apm_oss.metricsIndices
apm_oss.apmAgentConfigurationIndex

# additional ones we might consider:
xpack.apm.ui.maxTraceItems
xpack.apm.ui.transactionGroupBucketSize

[sqren]

@katrin-freihofner Sounds good 👍
Btw. I updated your list with a few more settings we may consider.

[katrin-freihofner]

First proposal:

on smaller screens, the panels would be underneath each other.

Other approaches that I tried:
Using tabs - but this would mean that there are tabs underneath of tabs.

Collapsing the panels:

Open questions:

• Is it ok to only save when the 'apply' button is clicked? And should we add a modal if a user tries to navigate somewhere else without saving?
• Which help info do we to show inline?
• Who can help with the description text?

[sqren]

I prefer the second approach. Instead of the tabs we can also try the side nav which is used by other plugins: https://elastic.github.io/eui/#/navigation/side-nav

I think the the apply button should be below the container like in the first example.

[dgieselaar]

I'm gonna vote for anything but accordions. IMHO using accordions for grouping forms is confusing UX. Second approach has my preference as well.

[sqren]

Is it ok to only save when the 'apply' button is clicked? And should we add a modal if a user tries to navigate somewhere else without saving?

I think in the first version it's okay to save when the user clicks apply (and not as they type). If they click away their unsaved changes are lost.

Who can help with the description text?

@bmorelli25 is normally good with this. I can also pitch in.

[sqren]

We might also want to show the value that's coming from kibana.yml, and are overwritten by the changes in the UI Settings.

[katrin-freihofner]

@sqren

We might also want to show the value that's coming from kibana.yml, and are overwritten by the changes in the UI Settings.

That would be great - we could add it as a placeholder text in the input field.

[sqren]

That would be great - we could add it as a placeholder text in the input field.

Yes, that or below the field as Default value: apm-*. That way it's also visible when it's being overwritten.

[sqren]

Since all the fields in "UI settings" are for indicies, we might call it "Index settings" instead.

[katrin-freihofner]

Should we additionally provide a Reset defaults option?

[sqren]

Should we additionally provide a Reset defaults option?

This would just clear all fields, right? So I'm not sure how useful that'll be.

[katrin-freihofner]

Yes. I guess it is fine for now - if we get more and more settings it might be helpful.

[bmorelli25]

Since "settings" is used in the top level nav, I think we should avoid using it in the sub-headings.

What about something like "APM indices", "Index patterns", or even just "indices" instead?

- Indices
- Agent configuration

[sqren]

Makes sense to avoid "settings".
Conceptually I like "index patterns" but since Kibana has something (very different) also called "Index patterns" we should probably stick to "Indices" or "UI indices".

[katrin-freihofner]

Design update:

• text updates according to discussion
• navigation update (removed the settings tab to fit the currently implemented approach)
  

@bmorelli25 could you please help me with the intro text on the top and the hints under each input field - I'm pretty sure we can do better than my mockup shows ;)

[bmorelli25]

Here's a starting point for a discussion on the intro text and input hints. @katrin-freihofner

The APM UI uses index patterns to specify the location of your APM indices.
If you've customized the index names that APM Server writes events to,
you may need to update these patterns for certain features in the APM UI to work.

Index pattern for matching indices that contain **source map** data. Overwrites `apm_oss.sourcemapIndices`
Index pattern for matching indices that contain **error** data. Overwrites `apm_oss.errorIndices`
Index pattern for matching indices that contain **onboarding** data. Overwrites `apm_oss.onboardingIndices`
Index pattern for matching indices that contain **span** data. Overwrites `apm_oss.spanIndices`
Index pattern for matching indices that contain **transaction** data. Overwrites `apm_oss.transactionIndices`
Index pattern for matching indices that contain **metrics** data. Overwrites `apm_oss.metricsIndices`
Index pattern for matching indices that contain **Agent configuration** data. Overwrites `apm_oss.apmAgentConfigurationIndex`

[sqren]

Thanks @bmorelli25. Building on top of that I'll also have a go:

• "location of your APM indices": Location makes me think of a file path or similar
• "for certain features in the APM UI to work": Nothing in APM UI will work if the index names has been changed beyond the matcher apm-*.
• "Index pattern for matching indices that contain": feels a little long. Maybe trim it?
• "Overwrites": has the connotation of being irreversible (and scary) even though the user is able to remove their value in the ui and then it falls back to the original value again. This temporary aspect makes me think "override" is a better term

Suggestion

The APM UI uses index patterns to query your APM indices. If you've customized the index names that APM Server writes events to, you may need to update these patterns for the APM UI to work.

**source map indices**. Overrides `apm_oss.sourcemapIndices`
**error indices**. Overrides `apm_oss.errorIndices`
**onboarding indices**. Overrides `apm_oss.onboardingIndices`
**span indices**. Overrides `apm_oss.spanIndices`
**transaction indices**. Overrides `apm_oss.transactionIndices`
**metrics indices**. Overrides `apm_oss.metricsIndices`
**Agent configuration indices**. Overrides `apm_oss.apmAgentConfigurationIndex`

[bmorelli25]

Nice, thanks @sqren! Can we just remove the bolded "source map indices", "error indices", etc., since they match the text above the input?

We could also mention kibana.yml and link to the documentation page. Something like this might work (where kibana.yml is the link).

---

The APM UI uses index patterns to query your APM indices. If you've customized the index names that APM Server writes events to, you may need to update these patterns for the APM UI to work. Settings here take precedence over those set in kibana.yml.

Source map indices
┌────────────┐
└────────────┘
Overrides apm_oss.sourcemapIndices.

Error indices
┌────────────┐
└────────────┘
Overrides apm_oss.errorIndices.

etc.

---

[sqren]

Good idea with link to kibana.yml docs 👍

It's a wrap from here.

[katrin-freihofner]

This leads us to:

Let me know what you think!

[sqren]

@katrin-freihofner LGTM 👍

One thing: when we read the value from kibana.yml it's not possible to see if it's set or not because there is a default value if it's not set (apm-*). So we'll always get a value back.

[katrin-freihofner]

Ok, thank you!