Incubator-superset: [SIP-49] Dashboard Filter Component

[graceguo-supercat]

[SIP] Proposal for Dashboard Filter Component

Motivation

This SIP is not about design or implementation. Instead I want to discuss the _requirements_ of dashboard filters: what functionalities the dashboard filters should offer. Based on this discussion, we will do architecture design and implementation plan.

_Similar discussions:_ [[SIP-9](https://github.com/apache/incubator-superset/issues/5925)][[SIP-25](https://github.com/apache/incubator-superset/issues/8452)]

Over the last a few years, we got many, many suggestions and feature requests to improve the usability and interactions for Superset dashboard filters. Currently dashboard filters are _charts_, when it is loaded each filter field is a query (select DISTINCT _column_name_ from _table_), and results are rendered as filter_box viz_type. It works good for most of use-cases, but still missed a few functionalities if compared with dashboard filter from other BI tools.

Proposed Change

I want to propose a solution to improve the functionalities for dashboard filters. Instead of using a separated slice component, I think dashboard should have build-in Filter component. It should carry over all the features from existed filter_box slice, understand properties that slice filter used (datasource, column, default values, sorting, .etc), and re-use dashboard specific properties, like filter scope and immune. All these filters related properties should be persistent in dashboard metadata.

For component UI/UX, we should follow SIP-34:
view mode: apply filter

add/edit filter config and scopes

New or Changed Public Interfaces

I list all dashboard filters related feature requests here, so that we can decide which ones should be implemented.

• Section 1: Current features should be kept:

  1. Filter indicators
  2. Filter scopes
  3. Share dashboard with snapshot of filter state
  4. Chart filter: user select a few rows in a table (or a piece in a pie chart etc), selected rows works like temporary filter and applied to other charts in the dashboard. This feature was lost during filter-scope implementation, but should bring it back.

• Section 2: Extra filter functionalities candidates:

  1. Allow user filter on metric (as well as column), similar to adhoc filter [[9502](https://github.com/apache/incubator-superset/issues/9502)]
  2. To support "less than" and "greater than" (or "BETWEEN"), current filter_box can only select one or more fixed values [[9974](https://github.com/apache/incubator-superset/issues/9974)]
  3. Non-editable filter box, or some options are not removable. [[10230](https://github.com/apache/incubator-superset/issues/10230)]
  4. Cascading filters: when user changes filter A, it will automatically trigger filter B loaded with new options, in a pre-defined sequence [[5664](https://github.com/apache/incubator-superset/pull/5664)][[10013](https://github.com/apache/incubator-superset/issues/10013)]
  5. Allow user to define static option values instead of running a query [[SIP-25](https://github.com/apache/incubator-superset/issues/8452)]
  6. Allow filter only apply to charts from same datasource (filter scope settings).
  7. Ensuring filter eligibility to the chart [[SIP-45](https://github.com/apache/incubator-superset/issues/9935)]

• Section 3: Extra intra-dashboard interaction:

  1. Time-based Visual Correlation between Charts on a Dashboard [[SIP-10](https://github.com/apache/incubator-superset/issues/5955)]
  2. Intra-dashboard interaction [[7512](https://github.com/apache/incubator-superset/issues/7512)]

Migration Plan and Compatibility

Currently Superset filters for dashboard are _slices_, and all filter properties are stored in slices table. After we implemented dashboard filter component, existed _filter slice_ should be converted to dashboard properties. We will offer a Superset db migration script, which will automatically convert filter_box properties into dashboard filter properties. But since each filter_box took some space in the dashboard, it will still need some manual work to remove filter_box from dashboard, and re-arrange dashboard layout to make it nice and clean.

Rejected Alternatives

Describe alternative approaches that were considered and rejected.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label #enhancement to this issue, with a confidence of 0.89. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[villebro]

I love this proposal! 😍

[michalmisiewicz]

How about not in / exclude filter ?

[graceguo-supercat]

How about not in / exclude filter ?

Good point! I added in the request list.

[amitmiran137]

If I understand this SIP correctly it is all about making a component which is responsible both Visually and logically to consolidate the set of filters that will be applied to all of the existing charts(slices) when fetching the data

so the component responsibilities are :

1. get a final distinctive list of filters that needs to be applied in this dashboard after merging filters from (query params, available filters from on the filter component UI, Chart plugins cross filtering, what else?)
2. apply filters to slices and send to BE by applying rules of filter scopes and by relevant datasources

is this correct?

Additionally there can be multiple filter components : global and one for each tab
and global need be consolidate into each of the tabs separately

[ckdarby]

@graceguo-supercat One of the things I just bumped into today is wanting the ability to not load charts _until_ the dashboard filter has been applied by the user.

[ckdarby]

@graceguo-supercat I wanted to provide additional feedback to this SIP & to the community to understand our frustration as a user of Superset with the disconnect between charts and the filters.

One of the major issues that we struggle with is we define our generic charts with nearly no filters and then apply the filters only from the dashboard perspective to allow all our charts to interactively be changed through the dashboard filter.

When we're making the individual charts the problem is the filter box is not tied into that chart yet. This means the individual charts when crafting them are running queries that basically are along the lines of, "fetch all data". For us, that is not doable because it is billions upon billions and PB kind of storage.

Once on the dashboard, the same issue happens when the filter box hasn't had values selected by the user yet but the charts try to run with the "defaults" on the filter box.

These queries usually have to be manually killed on Presto's side and then we apply the filter box from the dashboard which allows us to narrow into exactly the range we want to see with the specific partitions.

If there is an existing functionality to tackle this I'm all ears, the only thing we saw as a potential stop gap was custom viz plugin hacks.

[scriminaci]

This would be a huge improvement to the user experience. It would also make it possible to fix the filter while scrolling in the dashboard.

Another great thing should be to save filter combinations. Sometimes users create complex filter combinations and they would like to look at data by repeating a previous filter configuration.

[junlincc]

fix the filter while scrolling in the dashboard.

it's happening!

[fouadaabid]

Implementing these suggestions would be highly valuable to improve user experience

[junlincc]

12148

dashboard native filter and cascading filter are available for testing!! thanks in advance for feedback, comments, critics, love and support!!