Kibana: Discover - allow use without saved index pattern

[mattkime]

Using Discover currently requires a saved index pattern. This simplifies how users can interact with their data but is an unnecessary impediment. Discover could still use the index patterns api but work with unsaved index patterns. This likely introduces more complexity in the UI for how we select our data. Opening discussion to see how we might define a way to move forward.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[sqren]

We've talked a little about this before but is it possible for Discover to generating a fake index pattern on the fly by calling the field capabilities API for the specified index (eg. apm-*)?
That's what we do in APM when we need to pass index patterns to services like kql. It works pretty great and means we don't have to worry about whether the index pattern exists or not.

[mattkime]

@sqren You can create an index pattern without saving it. This was possible before as well but the API was so unclear that this was impossible to determine.

[sqren]

You can create an index pattern without saving it. This was possible before as well but the API was so unclear that this was impossible to determine.

Is this the fields capability API you are referring to? Either way, I don't mind which approach is used as long as solutions can simply link to /app/discover?indicies=my_custom_index-* without seeing error messages for missing index patterns :)

[ppisljar]

in the UI currently you can select the index pattern. I suggest to extend that dropdown with the capability to type in a custom pattern

[timroes]

@ppisljar I think that is a reasonable idea, as soon as we can request "index patterns on the fly" without the user having created them in advance. Do we have an issue we can crosslink here for this?

[ppisljar]

as mentioned above we do have index-patterns on the fly ready

[mattkime]

I was talking with @stacey-gammon and I think we might want to pause 'on the fly' index patterns at this point. There's not much you can do with them since they can't be referenced by other saved objects or other kibana apps.

Regarding @sqren 's use case, I think we could provide a function gets an index pattern based on the index pattern string OR creates it if it doesn't exist. I thiiiink that would work for the case you describe.

It would be best to provide some groundwork for 'on the fly' index patterns to ensure they operate as expected in all the places that persisted index patterns already work.

[ppisljar]

I am confused here, can you better explain: ?

There's not much you can do with them since they can't be referenced by other saved objects or other kibana apps.

I think there is a lot you can do with them and i don't see the point of other saved objects not being able to reference them ? that could be said about our visualizations by value (embeddables by value in general). Also in every kibana app you are doing a search but you are not persisting that search (no saved search, you just store searchsource config inside saved visualization directly). So its like saying we can't have visualizations that are not based on saved searches .....

[legrego]

I think we could provide a function gets an index pattern based on the index pattern string OR creates it if it doesn't exist. I thiiiink that would work for the case you describe.

This will only work if the user is authorized to create index patterns. Just something to keep in mind.

[sqren]

I think we might want to pause 'on the fly' index patterns at this point. There's not much you can do with them since they can't be referenced by other saved objects or other kibana apps. [...] think we could provide a function gets an index pattern based on the index pattern string OR creates it if it doesn't exist.

I'm not sure I follow this either. Is there a specific problem with retrieving the on-the-fly index pattern from field cap api? Why is it better to actually persist it - the user might not want to persist it (or might not have permissions to do it).

[mattkime]

the user might not want to persist it (or might not have permissions to do it).

I think it would be good to fully describe this use case and make sure that its real rather than theoretical. The goal of wider index pattern api adoption is more consistent inter application functionality and I'm not certain how 'on the fly' index patterns get us there.

That said, I'm all ears. I just want to make sure we're providing good tools.

[stacey-gammon]

This will only work if the user is authorized to create index patterns. Just something to keep in mind.

I wonder what admins intend by forbidding index pattern creation. Why should someone be able to create a visualization based on an "on the fly" index pattern, but not create and persist that index pattern? It almost feels like we would be sidestepping admin's intention by allowing users to create on the fly index patterns but not persist them as their own saved object (technically they would be persisted as part of a saved search if Discover supported these "on the fly" versions).

I'm not opposed to on the fly index patterns but it feels like we don't have a clear vision around them and by prematurely exposing them based on "we think they might be useful" has the potential to complicate other features. And please correct me if there is a vision around them that I am just unaware of.

It's possible these "on the fly" index patterns could be the solution to improving runtime field scoping (we don't want every analyst adding a bunch of random runtime fields and then there are way too many fields to wade through). I just think there are many tech considerations to flesh out. The same ones we have for object level runtime fields. If I save a discover view with an "on the fly" index pattern and then add it to a dashboard, will autocomplete show me fields from it? Does discover actually support "on the fly" right now? Are there tests around it?

I suggested Matt remove the API until these questions have answers, assuming that no one is relying on it. Sounds like whether APM is relying on it is still TBD.

[ppisljar]

i think the reasons (and concerns) are same/similar to other time to visualize efforts.