Mui-datatables: Using onBlur to activate the filter instead of using onFilterChange

[hungbang]

I am using server-side filtering, I overrode onFilterChange to activate filter every time the filter list is changed, but the problem is every time the filter list is changed, the backend API will be called to update the table's data. It makes the layout quite lag and the backend is called too much. I want to filter every time the user leaves the field or maybe we will add the button in TableList called Submit Filter, as soon as the user enters all the field they need to filter then click the button to activate the filter.

| Tech | Version |
|--------------|---------|
| Material-UI | 3.9.2 |
| MUI-datatables | 2.8.0 |
| React | 16.8.5 |
| browser | Chrome |
| etc | |

[gabrielliwerant]

Right now, the way filtering and the callback work, it does not support server-side filtering. In my experience, filtering data is the exclusive domain of the UI, and I would expect it to operate only on what I'm looking at, at the moment, rather than change the data state first. Doing otherwise would tend to be very confusing from a user perspective. Maybe if I understood more about the specific needs of your project, I could empathize a bit more with your use case. Otherwise, I would consider this lower priority.

I do think there's probably a need at some point to enhance the callback API to make it more clear when they fire (e.g. onFilterChangeFinished).

[hungbang]

@gabrielliwerant thank you for your answer, I agree with you that we should have at least one event like onFilterChangeFinished or more simple solution is we will have one event called onCloseTableFilterList, as soon as the TableFilterList is closed we will call the API to reload data.

[mbniebergall]

I'm also looking for a solution to the same problem @hungbang . Having one or more new events to be able to call my search API once the user has completed typing in the search field would be very helpful. Either a new event is needed, or the option to have a 'Search' button to trigger the search would help solve the problem.

[hungbang]

@gabrielliwerant @mbniebergall
Regarding the user experience, I think we should open to the user decides when is the search or filter performed. of course, the real-time update will be really helpful, but in some specific situation, we should have an option such as :
disableAutoSearch: true or disableAutoFilter: true, when this flag is enabled, the button Search or Filter will be displayed.

[patorjk]

I also second either an onFilterClose event or an optional button to apply the filters (the latter is probably better but I'm open to either). I started integrating filters into my tables this past week and I have a few that are server side. I wound up just wrapping the call that does the AJAX in a debounce method (from within the onFilterChange method). It's slightly awkward because the loading lightbox I have for the table pops up behind filter dialog (though it didn't seem right for it to pop up in front of it).

The ability to filter results was something requested by users. For my own use case, the tables that are server-side can sometimes have over 100k items.

I'm in no rush for a solution as I don't go live with my app for a few months (I'm working on replacing an existing app) and what I have now seems to work fine. However, If a solution for this is decided I wouldn't mind taking a stab at implementing it.

Also, with regards to searching - I have that implemented in my server side tables and it works well, I just wrap the fetch call in a debounce method so that it doesn't make the AJAX call until the user has stopped typing. I'm not sure there's a need for special flag here. This could also be accomplished via a custom search component (a la the customSearchRender method).

[patorjk]

Thinking about this a little more, there might not need to be an API change. Instead, if serverSide was set to true, the Filter dialog could appear with an additional button (maybe called "Apply"), and it wouldn't add the filters unless it was pressed.

[nbrustein]

@gabrielliwerant you said that you don't understand a situation where server-side filtering would make sense, if I understand correctly, and I'm not sure that anyone ever really responded directly.

In our case, we have an admin UI where we are showing list of users. We have 200,000+ users, so we can never have the API return all of them. We set a hard limit for our API at 1000 users. If you are not using any filters at all, or if the filters you have selected would return more than 1000 records, we only return 1000. In that case we also show a message saying that there are more results on the server, and we suggest that the user use the filters to limit the results.

This works fine for us, because in most of the use-cases for this admin UI, the end user wants to filter the users in a way that limits the results to no more than a couple hundred. For example, our users are students, so the admin might limit to a particular class.

In addition, we have a search box. This is often used to search for a particular user by email or name. Since we can't pull down all of the users at once, using this search box has to trigger an API call.

So, this is a case where server-side filtering makes good sense, and I don't think it's confusing at all from the user's perspective.

[nbrustein]

Mostly just to summarize what people have already said clearly.

Problem: When using the serverSide option, onFilterChange and onTableChange are often called too much. In responding to them, you end up hammering your API with new requests that aren't really for what the user wants, since the user is not yet finished filling out their desired filters or typing in the search box.

Solutions

1. Apply button on the filter form and on the search input. These buttons could be attached to a new event (or new events, one for each). A developer could then respond to those events and only trigger an API call when the user indicates they are finished entering their filters. If this solution is implemented, it might also be necessary to make configurable the text on the button on the actual component used for the button, and to do these separately for the search box and the filter form.
2. onBlur events for the filter form and the search input.
3. debounce logic. It is currently possible for the developer to implement this herself, but it would also be possible to either add new events that only get triggered after a debounced delay, or to add configuration that allows for debouncing the existing events. (It might also be possible, rather than making this configuration, to provide a hook that debounces the table state changes.)

Would you be willing to consider PRs for any/all of these things if someone were to submit them?

[patorjk]

In my experience, filtering data is the exclusive domain of the UI, and I would expect it to operate only on what I'm looking at, at the moment, rather than change the data state first. Doing otherwise would tend to be very confusing from a user perspective.

Quick counter point: If I'm a user looking at a table and I see that it has 1000 entries, and I decide to filter that data, in my mind it doesn't matter if the UI has downloaded all 1000 rows or just the current page I'm looking at. I would expect the filter to act on all 1000 rows. Addditionally, many users aren't going to know the difference between client-side and server-side data, to them, they see what the table shows them. Using the server-side option just allows the developer to avoid having to download large amounts of data or to handle very large amounts of data efficiently.

[gabrielliwerant]

@nbrustein and @patorjk Thanks for your explanation and projected solutions. I understand the use case better. My thinking is that I like to prevent the possibility of inconsistency in the user experience, which is always possible when doing something like filtering client data via the server (since we can't guarantee the same data will return back, or at all). However, I understand it is not always possible or desirable to have all the data for filtering present on the client, and so I can understand how developers might want the option to take the risk to run it serverside.

So, here are my thoughts on some of the proposals, and how we might move forward:

1. As far as an apply button for search, something like this should already be possible to add via the customSearchRender option. The filter selection dialog does not have a custom render option however, so perhaps this is something we need.
2. I'm not sure onBlur would be that helpful because it would fundamentally alter how the filtering takes place. Right now, filtering occurs on the onChange event in the respective form fields. Using onBlur is a different experience, and it seems like it might not be clear to users that they need to release focus on a field in order to fire a search/filter. What if instead we had some kind async callback that could bypass client side fitlering/searching when serverside is true?
3. debounce sounds reasonable to me if we're going to allow serverside filtering, although maybe the async callback could serve the same purpose, or the two approaches could be combined?

Also, just wanted to point out that I do think it's possible to perform filtering serverside now, using the custom filterOptions with names and display. You would just have to use your own onChange or whatever functions you wish in your custom display component instead of the one sent through the options. The catch is that you'd need to implement this for every filter you provide for your users, so it's heavy do-it-yourself, but you would need to be supplying the logic yourself if you're going serverside anyway. Is there a reason this option is not good? Too much effort?

[patorjk]

As far as an apply button for search, something like this should already be possible to add via the customSearchRender option. The filter selection dialog does not have a custom render option however, so perhaps this is something we need.

I mostly agree with this point, though I think it would be beneficial to update the server side example to show a debounced search or one that uses a custom render (https://github.com/gregnb/mui-datatables/blob/master/examples/serverside-pagination/index.js).

A customFilterRender option would solve the problem for filters, but honestly, that seems like an unfair weight to put on the developer. I would hate to have to re-write that component just to get an apply button (search is a little different since it's a much simplier component).

I'm not sure onBlur would be that helpful because it would fundamentally alter how the filtering takes place. Right now, filtering occurs on the onChange event in the respective form fields. Using onBlur is a different experience, and it seems like it might not be clear to users that they need to release focus on a field in order to fire a search/filter. What if instead we had some kind async callback that could bypass client side fitlering/searching when serverside is true?

I'm not sure I understand what you're saying here. Right now filtering does not occur when serverSide is set to true. You can add filters, but the data in the table is not affected. This is good because if serverSide is true, I want to handle the filtering via my ajax calls.

debounce sounds reasonable to me if we're going to allow serverside filtering, although maybe the async callback could serve the same purpose, or the two approaches could be combined?

I'm doing debounce now in the onFilterChange event, I find it awkward for a few reasons:

• My lightbox that I have show up when data is being fetched has to pop up, and it pops up behind the Filter dialog.
• This may happen several times as the user plays around with selecting filter values.

It would be nice if there was an onFilterClose callback (that would be called when the filter popover closes), then I could put my AJAX code there.

[gabrielliwerant]

I'll see if I can clarify my thinking.

A customFilterRender option would solve the problem for filters, but honestly, that seems like an unfair weight to put on the developer. I would hate to have to re-write that component just to get an apply button (search is a little different since it's a much simplier component).

I agree. What I was thinking was just the dialog/modal part into which the filters reside. We could allow a custom filter modal, but keep the internal filter components. However, this might not be the best/clearest solution, just throwing it out there.

I'm not sure I understand what you're saying here. Right now filtering does not occur when serverSide is set to true. You can add filters, but the data in the table is not affected. This is good because if serverSide is true, I want to handle the filtering via my ajax calls.

I'm referring to how the filters fire on the client when serverSide: false. They don't use onBlur, so it would create an inconsistent experience to enable this method of firing serverside filter requests.

If understand the issue correctly, it's that filters currently don't have a good way to insert custom behavior when serverSide: true unless all the separate filter components are customized (which is burdensome). There's an internal onChange being used but it's separate from onFilterChange, which might be causing confusion, as I don't believe onFilterChange was intended for this purpose.

So, what I'm suggesting is that we allow a new onChange for filters (with perhaps a better name) which overrides the default in serverside situations. Then devs should be able to provide their own, which I think would handle the use case here. This point connects to the debounce issue, where we could offer to bundle debounce functionality in with this new onChange function. Alternatively we could leave it to the devs to implement as part of their own onChange functions, as you have done. I'm open to either case.

Does that make more sense?

[gabrielliwerant]

Also, the reason I'd rather see a new onChange for filters to fire serverside requests rather than something like an onClose for the filter dialog, would be to keep the default experience similar between clientside and serverside.

[patorjk]

Client-side and server-side tables will inherently have a slightly different UX. For example, search debounce - which I think we're both in agreement about - means that the table doesn't update with search results until the user is done typing. Users are be fine with this because they know that it takes time to search 100k rows as opposed to 100 rows. Filtering should have a similar, slightly altered UX to accommodate tables with large amounts of data.

So, what I'm suggesting is that we allow a new onChange for filters (with perhaps a better name) which overrides the default in serverside situations.

I'm not sure I follow. The onFilterChange function in TableFilter.js executes filterUpdate in MUIDataTable.js. All that function does is update the state with the new filter value and then call the user callback onFilterChange. I'm not sure there's any value is being able to override this function, as the controls in the TableFilter dialog are controlled components that rely on that function updating the state.

The components in TableFilter could be modified to use state values derived from props, but in this case, it would make sense to allow for an Apply button. I wouldn't see the value of another onChange function.

Also, just to be clear, here are the pain points in the current setup:

• AJAX calls can be expensive for these tables and may take several seconds.
• It's disruptive to have the table go into the loading state while the user is still picking filters (ex: imagine a table with 16 fields)

The default experience is achievable using onFilterChange in the current setup, I think what people want is a way to have a custom experience to avoid making extraneous AJAX calls.

[1] https://github.com/gregnb/mui-datatables/blob/master/src/MUIDataTable.js#L947

[gabrielliwerant]

Agreed UX will never be 100% the same, so it's more a question of how much sameness is desirable. I think switching to onBlur is too dissimilar, but I think we're on the same page about onBlur not being the best solution, correct me if I'm wrong.

Ok, so a new onChange isn't needed. You're right, on closer inspection, it wouldn't do anything new here. Which means. as you say, that onFilterChange is actually sufficient to add the desired functionality, including debounce. To help alleviate the pain of too many AJAX calls and make that more convenient for developers, we could build in optional debounce functionality to the onFilterChange event. How does that sound?

Regarding the disruptiveness of a loading state, I can see how that might happen. I can also see how it might be minimal. So the option to mitigate this is for another, distinct, optional means of initiating an API call when selecting filters, like a submit button for the filter dialog? I'd like to give that a little thought. I'm leaning toward a custom render option that would allow it if it's not too inconvenient. I'll look into it when I get some time, probably this weekend.

But if I've misunderstood anything, let me know.

[gabrielliwerant]

PR added to address some of the issues inherent to serverside filtering in this library https://github.com/gregnb/mui-datatables/pull/913. Please let me know what you think.

[patorjk]

PR visible here: https://codesandbox.io/s/github/gregnb/mui-datatables/tree/enhancement/issue-857-server-side-filters

I'm glad to see you did this. I hate render functions, and having used them a ton I don't think they're an optional solution for customization (though that's a different topic). 

Anyway, I was actually pretty alarmed by your resistance to this feature and I wound up forking the library to add it myself (nothing personal, but I can't say no to my boss when it comes to adding functionality). I would prefer not to be on a fork, though I will probably be on one for the foreseeable future as I have a lot I need to do. However, at some point in the future I would like to sync up and hopefully switch back to mui-datatables.

I checked out your PR, I have a couple of bits of feedback:

• If you press "Reset" the filters are cleared, even if you re-open the popup, existing filters are never brought back.
• The popover should close after hitting "Apply".
• It's unclear to me why I need to manage serverSideFilterList. Wouldn't this make more sense as a boolean option? (and the table would internally handle managing this list)?

In my fork, to allow for this functionality I added the following option:

filterPopoverOptions: {  mustConfirm: false,  confirmButtonLabel: "Submit"}

When the filter dialog pops up, if "mustConfirm" is false, everything acts the way it does now. If mustConfirm is true, then the filterList (as well as onFilterChange) aren't updated/triggered when the user changes the form's controls. Additionally, the user sees 3 buttons: Submit, Reset, Clear.

• Submit: will update the filterList (executing onFilterChange for each update) and then end it with an onFilterConfirm event, and then finally close the popover.
• Reset: takes the user back to the current filterList state.
• Clear: clears the filters (basically what "reset" does in your scenario).

These are just suggestions though. I think what you have would definitely be a net positive.

[gabrielliwerant]

@patorjk Thanks for the feedback!

I don't love render functions either, but what they have going for them is that they allow devs to come up with their own solutions, rather than restricting them to specific implementations. It's been my experience that as time moves forward for any given default feature, the requests for customization approach infinity. :) It makes sense, everyone has different ideas, different projects, different needs, etc. So for certain things, I think it makes the most sense to provide render functions so that devs aren't hemmed in any more than necessary. Though I suppose there's a lot of room to quibble about which things should be render functions and which shouldn't.

Anyway, I was actually pretty alarmed by your resistance...

What's key to me is the ability to empathize with a particular use case/user story, so I usually try to leave that pathway open for discussion. I'm not a huge fan of server side filtering, but you and others did convince me that there is a time and place for it. Also, the implementation of it in this library was never completed, so it does need some work.

If you press "Reset" the filters are cleared, even if you re-open the popup, existing filters are never brought back.

The example isn't intended to be comprehensive, as it's only faking requests and such, but I should probably also add something that demonstrates how one might deal with filter resets, so I'll work on that.

The popover should close after hitting "Apply".

The ability to close the filter dialog at will requires an additional set of changes, and the PR is already doing a lot, so that could come in a separate set of changes.

I happen to think that it's fine to keep it open, as this facilitates the use case where users want to see how their data changed and then make new changes based on that, as you can when you operate client side. Otherwise, you would have to re-open the dialog each time just to add a new filter, should you want to add them one at a time (especially important since server side filtering means users probably can't see all the data they're filtering on at any given time, meaning that they might be surprised by what turns up).

It's unclear to me why I need to manage serverSideFilterList. Wouldn't this make more sense as a boolean option? (and the table would internally handle managing this list)?

My first pass used a boolean for this, but it results in an edge case.

1. Select a set of filters (boolean false for chips to prevent them from immediately showing)
2. Apply the filters after a successful request (boolean true so that the chips will now show)
3. Select some additional filters (boolean is still true, so the chips will show automatically instead of waiting for success request, breaking with the previous experience).

There's no way around the above situation with booleans. We actually don't really need on or off, we need a list that only populates when the dev updates some data and uses only the updated data. That can't be done with filterList in the column options either since that's tracking both chips and the state of the filters in the dialog at the same time, hence we need something separate. It's not particularly elegant, but I don't see another solution that's nearly as easy to implement (like refactoring much of how filters work internally to separate out the dialog filters from the chip filters).

A "clear" option is a good suggestion, though I think it probably should be at the header rather than the footer, or at least for that to be an option, so perhaps that could be an addition for another time.

[patorjk]

but I don't see another solution that's nearly as easy to implement

What I did was have TableFilter manage it's own filterList:

// I'm using hooks, but this is also possible without them
const [filterList, setFilterList] = useState( cloneDeep(props.filterList) );

Then on the TableFilter component I added a key prop. This prop is a number that is incremented whenever setActiveIcon is called, so TableFilter is always rendered with the correct filterList when it's viewed.

Doing it this way I also needed to:

• Put if statements around the props.onFilterUpdate calls.
• Handle updating TableFilter's filterList the same way as the main filter list. To do this, I split MUIDataTable's filterUpdate code into 2 functions, and passed one of those to TableFilter. Example:

  const filterUpdate = (index, value, column, type) => {
    let newFilterList = filterList.slice(0);

    // this contains the filter switch code from MUIDataTable, it updates newFilterList
    props.updateFilterByType(newFilterList, index, value, column, type); 

    setFilterList(newFilterList);
  };

  const handleCheckboxChange = (index, value, column) => {
    filterUpdate(index, value, column, 'checkbox');

    if (props.filterPopoverOptions.mustConfirm !== true) {
      props.onFilterUpdate(index, value, column, 'checkbox');
    }
  };

Just an alternative idea.

[gabrielliwerant]

@patorjk Yes, that sounds similar to the alternative I was considering as well, but I went the other direction because I thought it might be easier and add a little more control over what would be displayed. There's already been a long delay in getting the improvements released, and I don't want to delay any more, so I'll go with what I have for now, but will consider reworking it depending on the community feedback.

[nmdhulipud]

@gabrielliwerant we are also using server-side filtering, and currently every filter change make individual api calls upon selection, and our plan is to come up with an Apply button and on click of it, have a filter call. As per the above conversation I see this requirement is set to low priority. Any new update on this if I am missing any ?

Thanks

[mbniebergall]

@nmdhulipud this was released in 2.12.0 (see https://github.com/gregnb/mui-datatables/releases/tag/2.12.0). I have already implemented using it on a project and performance is significantly improved. Thanks @gabrielliwerant !

[hungbang]

Nice @mbniebergall, I'm very appreciated and we should update the usage docs as well. It is able to help everyone know how to use the new features I think. Anyway, thank you very much for your implementation.