Kibana: Filters are applied to index patterns they were not created for

[lukasolson]

To reproduce:

1. Create multiple index patterns
2. Go to discover, and create a filter
3. Switch to another index pattern

Notice that the filter remains, and no documents show up (unless you have the same field configured in the new index pattern and there are docs with the same value for which you filtered).

This affects visualize and dashboard as well. It is especially apparent on dashboards with visualizations from different index patterns.

If you save a search/visualization/dashboard with a filter, you can go into the saved object editor and see the corresponding JSON that is stored. If you look at the filter, you'll see that there is actually an index associated with the filter. However, when the queries against Elasticsearch are run, the filter will be applied across all of your index patterns, regardless of whether or not they match the associated index.

This brings up an interesting question: Is this behavior intended? It could be useful if multiple index patterns have a similar structure. There are a variety of use cases which might benefit from the behavior as-is.

However, there are also issues with the current behavior:

• If the filter is applied across all index patterns, why store the index as part of the filter?
• If the index patterns vary widely in structure, as stated above, no documents will show up from the index patterns to which the filter doesn't apply (which may or may not be intended).
• If the index patterns do have the same field, but the field is mapped differently, this can cause errors.

Other known issues caused by this underlying issue: https://github.com/elastic/kibana/issues/9387

As far as I see it, there are a few proposals to fix this:

1. Only apply the filter to the index pattern for which it was originally created.
2. Apply the filter across all index patterns which contain the field for which the filter was created.
3. Leave the behavior as-is, but remove the index as part of the filter. (We would have to discuss a solution for the issue linked to above.)

[thomasneirynck]

applying the filter across all index patterns makes sense when the domain of the filter is consistent across all index patterns.

I think for both time-filters and geo-filters this is the case. We already treat time as a first class filter in Kibana. We do not treat spatial filters that way. One thing that complicates this is that index-patterns do not have a "default geo-field", unlike time-based index patterns.

I don't think this is the case for any other field. Matching name and field-type are probably not sufficient to claim that the fields contain the same kind of values.

So I think I would gravitate to a solution like (1), but with a special case for geo and time-based filters.

[weltenwort]

While this applies to all parts of Kibana that use the global filter bar (see #4909 for another example), the impact is probably biggest in the dashboard.

Only applying the filter from the global filter bar to some parts of a dashboard could be very confusing unless we find a way to indicate it clearly in the UI. In general I would try to minimize surprises for the user in order to not break the user's mental model - which usually means avoiding special cases. Given that the query bar applies to all parts of the dashboard and that we're moving towards a unification of the query and filter bar, I would be cautious about introducing more complex filter semantics.

[stacey-gammon]

Also related: https://github.com/elastic/kibana/issues/6373

The person in that issue suggests applying the filter to every index by default , but allowing an override.

Only apply the filter to the index pattern for which it was originally created.

I like this option, but if we made it the special case scenario, not the default, it would keep everything working the same as it does now (so no surprises to our users) but allow it to be configurable when necessary.

I agree with @weltenwort that some visual indication would be nice. Maybe another icon added to this row that can be checked on and off to say limit to the original index, or show for all indexes. An icon representing one of those would be tough though. [ ] i* ? No matter what we picked we'd probably want a way to explain what it meant.

[Bargs]

In general I would try to minimize surprises for the user in order to not break the user's mental model - which usually means avoiding special cases. Given that the query bar applies to all parts of the dashboard and that we're moving towards a unification of the query and filter bar, I would be cautious about introducing more complex filter semantics.

I agree pretty strongly with what Felix said here. The query and filter bars sit at the top of every app and apply to everything that sits below them. I'd find it very confusing if some of the filters started being conditionally applied to some panels on a dashboard but not others. And I don't think an extra icon would help, our filters already have a confusing number of icons/buttons as is.

One thing that might help is an ability to display and edit the query/filters that are attached to a particular panel. Right now it's impossible to see the query backing a visualization without clicking the edit button which takes you out of dashboard and into the visualize editor. If we could display a query/filter bar per panel, that would give us a very intuitive place to put new filters that don't apply to the entire dashboard. Then we just need to come up with a fairly easy way for the user to choose when they click a vis whether their filter should be applied globally, or to the vis they clicked on.

[stacey-gammon]

Yea, good ideas @Bargs. If we eventually have dashboards within dashboards, that would also help with being able to apply a filter to a subset of visualizations.

[Bargs]

Realized today we already have a setting related to this issue:

[lukasolson]

Awesome, looks like that's exactly what we want, and the issue I linked to in the original description has also been resolved. Closing this for now.