Mui-datatables: Experiencing massive lag when using 100k+ rows

[hwatersiv]

Expected Behavior

The table should be able to handle a large amount of data with the implementation of pagination in the table

Current Behavior

the table is nearly unusable With 8K+ entries and 7 columns

Steps to Reproduce (for bugs)

create a table with 8k+ entries with at least 7 columns

Your Environment

| Tech | Version |
|--------------|---------|
| Material-UI | 3.9.3 |
| MUI-datatables | 2.0.0 |
| React | 16.8.6 |
| browser | Chrome Version 73.0.3683.103 |
| etc | Typescript 3.4.5 |

[gabrielliwerant]

@hwatersiv I created an example table with 7 columns and over 8k rows, but did not notice any slow behavior while selecting rows, going through pagination of various sizes, searching, filtering, etc.

I created a branch to test this here: https://github.com/gregnb/mui-datatables/tree/large-data-test.

Please take a look and let me know if I'm missing something. Maybe if you could create an example where you see the issue? Maybe there's some smart caching happening with my example since I copy-pasted the same rows over and over. It's difficult to come across 8k rows of properly formatted data just lying around. 😛

[ideoclickVanessa]

I work with @hwatersiv so I can provide a bit more context:

We have pagination, sort, search, filter dropdowns enabled on the table.
We are using the Array of Objects feature (instead of the array of arrays).
One of the columns is a boolean that we have a custom render to a Material UI switch.
We are using a custom select toolbar that allows you to update the boolean field for multiple rows at a time.

[hwatersiv]

Update. 8k rows is not the issue any more the actual slow down occurs around the 100k mark.

here is a link to the gist:
https://codesandbox.io/embed/1qmkq7r7kj

and I know data is hard to come by so i have attached 100k lines of code you can replace the someData.js with but i recommend
NOT opening it in the browser you for sure will crash it

oneHundredK.zip

[gabrielliwerant]

@ideoclickVanessa @hwatersiv Ok, thank you, that helps a lot. I will look into it further this weekend. It's possible that good performance at 100k rows is pushing some limits that are beyond the scope of this library, but it's also possible that there is room for optimization.

[devloco]

Apologies for the "drive by" answer, but I was here looking at the open issues and this one caught my eye.

Anyway, I think you could set options.serverSide=true and then use the onTableChange function to pull the exact values needed for a page from your data array and feed them to mui-datatables. Thus the table would only ever be loaded with a number of rows equal to whatever your rowsPerPage is set to.

Just a thought...

By the way, thanks for this project, it is great!

[gabrielliwerant]

Certainly, one way to handle large sets of data is to use server side pagination, but I think what's at issue here is that there's a desire to work with large sets of data that are loaded into the table as this enables table-wide actions on all the data such as filtering, searching, downloading, etc.

I'm looking into data chunking, but it's time-consuming to work with such large data sets, as even loading it all into webpack takes some time.

[gabrielliwerant]

Just a quick update:

I have a couple of things I'm experimenting with to increase the speed of the table when there is a large amount of data, but some of them require a newer version of the material-ui library (which includes speed enhancements of its own), which breaks some things. I've traced down some of the errors, but it may actually require a fix in the material-ui library itself, so it will take time to isolate it and create minimal repo examples, etc.

[alexeykush]

any updates?

[gabrielliwerant]

Unfortunately, no. There's not likely to be a silver bullet here as performance depends on so many different factors. My plan was to at least provide an option that would allow data chunking which might have benefits on large data sets, but I haven't had the chance to work on it.

[khalilmarzouk]

any updates ?