Kibana: Index Pattern overwritten when an index exists

[neptunian]

In the Ingest Manager we create index patterns (logs-*, events-*, metrics-*) of fields from every package when a user installs or uninstalls any package. Indices may not exist yet and ones that do are created from their own unique index templates based on the dataset and package it belongs to. This creates an issue where we get a lot of errors because no index exists when visiting a Dashboard. In an attempt to suppress these errors we created an index placeholder, logs-index_pattern_placeholder that would match the index pattern for logs-*, but what this did is delete our index pattern and replace it with whatever fields or mappings the placeholder index had, which is none except a few defaults. This seems to be happening because our fields don't pass a check for existence of the property readFromDocValues

Before creating placeholder index:

After creating placeholder index:

Ideally, we would like to not have to create this placeholder index, but after some discussion here (https://github.com/elastic/kibana/pull/61257), it was decided by App Arch to not hide errors when no index exists

I am trying to figure out whether it's correct behavior to overwrite the existing index pattern with fields from an index, and if so how would we get around this? Adding readFromDocValues to every field seems to prevent it from rebuilding the index pattern, but I'm not sure how to calculate the value or whether I should be adding this field.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[mattkime]

Ingest manager creates an index pattern object (sidestepping kibana index pattern logic?) An index exists and it has field mappings but no documents. Then a new index is created that matches the index pattern wildcard but with a dummy doc with no mappings. isFieldRefreshRequired is run for some unknown reason and we only see the small set of fields from the dummy doc.

[ruflin]

@exekias @urso This might be also interesting for you as it is something we did for a few years also with Beats.

[ph]

@mattkime I think our workaround is not ideal, could we sync on this to find how we could solve it in the short term? We would like to help to define the correct behavior but _index pattern_ are core to the behavior of Kibana we would like some guidance on this issue.

[neptunian]

@ruflin to get around the index pattern issues, can we create a first index if it doesn't exist for each dataset when a user installs/uninstalls a package. the same way beats does during setup? or even placeholders for each one like metrics-system.network-placeholder so it inherits the mappings? that way kibana can do the recreating it wants to do and it will have the mappings to do so because the indices exist? the templates would be created first so the indices inherit from them. It seems like it'd be easiest to just let kibana create the index patterns and keep the logic in one place.

[ruflin]

Kibana will not create the correct index pattern as in some cases we have things like formatter inside which Kibana does not know about. We could create an index for default namespace but we don't know if other namespaces will be used by agent in standalone.

I would prefer if we fix this at the core which is in Kibana.

[mattkime]

There are a couple of ways to resolve this. First @neptunian could simply supply the readFromDocValues field. It works, but IMO it continues a broken pattern of working.

I'd like to define a goal solution and then if we need to do something different in the short term we can resolve that separately.

Kibana should be the sole creator of index pattern objects otherwise Kibana has to make sense of what other processes have written. Its worked so far but its not ideal.

After speaking with @neptunian a bit more, I started to think of the problem in terms of the field caps api. What does it know and when? It needs an index to return results - and we want to create an index pattern before an index exists. But (perhaps) we have the info we need in index templates. The field caps api could potentially return data for an index that does not yet exist but matches index templates. Ingest manager could create the index pattern, add formatters and scripted field via kibana api before an index exists.

I would very much like to get rid of the logic that automatically refreshes index patterns. It makes index patterns difficult to reason about and I'm unsure how Beats might need this functionality. It also leads to errors if the user doesn't have the correct privileges. Perhaps being able to create an index pattern without an index would remove the need.

@ruflin I'm curious to hear your thoughts in particular.

[ruflin]

Kibana should be the sole creator of index pattern objects otherwise Kibana has to make sense of what other processes have written. Its worked so far but its not ideal.

Why is that? For me an index pattern is just a saved object like a dashboard. Using the saved object API, anyone should be able to create them.

My long term goal would be that we don't depend on index patterns at all and have all the info we need in the index template and then Kibana can read from it directly. But we are not there yet and will probably take some more time.

TBH I think on the Beats side so far we assume that if we load the index pattern manually, it will never be refreshed. Seems like this is not the case but didn't break many things in the last years :-) But I would leave Beats out of the discussion for now.

For me the main problem we have at the moment is that Kibana does a magic refresh because my assumption is, Kibana thinks it knows better what an index pattern should look like, but it is actually the Ingest Manager that knows best.

As a first step, could we just have a flag refresh: false on an index pattern and if it is set to false, Kibana will not touch it? There are many more things we should iterate on long term, but would this solve the issue short term?

Side note: I wonder why if readFromDocValues is set, Kibana decides not to refresh?

[neptunian]

My concern is that Kibana has a lot of logic that it's running through the fields to create the index patterns that works with it. If we want to create the index patterns externally, we need to know exactly what all that is so we can replicate that logic. We need documentation that describes how to correctly create the index patterns and what the appropriate fields are and do, such as "esTypes" and "readFromDocValues", instead of trying to read through the code. At least for now, having the "right" fields according to Kibana (such as readFromDocValues), does not create a refresh, so stopping the refresh is resolved in the short term with this already.

I think we need:

• documentation on how to create a kibana index pattern so anyone can create them and not rely on kibana
• stop considering it an error if no index exists but the index pattern does exist (https://github.com/elastic/kibana/issues/62343). it seems like this is considered an error because kibana assumes it should always create the index patterns based on index template mappings passed onto existing indices

[mattkime]

I think my next step on this topic is to determine if any current elastic processes are creating index pattern fields without readFromDocValues - @ruflin do you know where I should check?

[neptunian]

I only know of Beats @mattkime

[ruflin]

Same, only know of Beats and indirectly apm-server. @simitt Does apm-server / APM UI still load an index pattern or did you completely get rid of it?

[simitt]

The APM Server uses libbeat logic to create the Kibana index pattern and then we sync them to Kibana, so the APM Server index pattern is actually bundled with and loaded from Kibana and can be found at src/plugins/apm_oss/server/tutorial/index_pattern.json.

[mattkime]

The field list is no longer being cached - https://github.com/elastic/kibana/issues/78758

The index pattern will no longer be overwritten but there may be other consequences. That said, this should be a big step in the right direction.

[ruflin]

Curious what you had in mind when you mentioned "other consequences". Any pointers?

[mattkime]

@ruflin The main thing I think of is that I'm not sure how dashboards responds when fields don't yet exist.

[neptunian]

@ruflin The main thing I think of is that I'm not sure how dashboards responds when fields don't yet exist.

Will test this

[neptunian]

It still gets the error that some fields don't exist so we will need to leave the index placeholder to continue repressing that.

[mattkime]

@neptunian - Can you file a bug? I think this is something we should handle gracefully instead of throwing unactionable errors.