Tabulator: addData() very slow when adding big chunks of data

[zeropage-io]

Hi all,
this is an awesome grid framework. Thank you!
I am having issues adding big chunks (thousands of new rows) of data using addData().
It is no problem to add hundreds of new rows. These are rendered quite fast.
But adding a thousand rows or more, say, in the middle of a table, is extremely slow incurring waiting times to be measured in minutes.
I tried progressiveRender (false and true, no difference) and the table height is fixed to 700px.
Is there a faster method of adding big chunks of data than using the addData()-method?
Any help is greatly appreciated!
Best wishes
Axel

[zeropage-io]

Any help greatly appreciated!

[olifolkerd]

Hey @zeropage-io ,

Could you post a copy of your table constructor so we can see how your table is setup.

You need to make sure you have defined the height property in the table to enable the virtual DOM which will dramatically speed up table rendering.

Cheers

Oli

[zeropage-io]

Hi Oli,

many thanks for your reply! I read the documentation quite thoroughly (as well as most of the issues here). ;-) The height property is already set: "I tried progressiveRender (false and true, no difference) and the table height is fixed to 700px."

Here's my constructor.

{
    height:700,
    layout:"fitColumns",
    columns: /* table col definition ommitted */,
    rowClick:function(e, row){
      $(".myloader").show( );
      if ( !row.getData().open ) {
        $("#main-table").tabulator( "addData", childs[ row.getData().id ], false, row.getIndex() );
        row.update({"open":1});
      } else {
        close_node( row.getData().id );
      }
      $("#main-table").tabulator("scrollToRow", row );
    },
    initialSort:[
      {column:"level", dir:"asc"},
      {column:"hub", dir:"desc"},
    ],
    progressiveRender:true,
    renderComplete:function(data){
      $(".myloader").hide( );
    }
}

I am emulating kind of a collapsible feature. If you click on a row it checks if that group is open, and if not, adds new data behind that row.
If the childs[] array contains only a couple of new rows all is blazingly fast. If I add one hundred new rows it takes 2-3 seconds (in Chrome). If I add a thousand new rows it takes a multitude of that.

Unfortunately, I am not able to track down where the bottleneck is? Is it the tabulator framework? Or the browser that takes (too much) time to render the DOM?

Is there any other way to insert big chunks of data that I might have overseen? Or is there a way to speed things up with addData()?

Your help is greatly appreciated
Best wishes
Axel

[olifolkerd]

Hey,

The progressive render features are no longer available in 3.0 onwards, they were replaced by the Virtual DOM.

Which version of Tabulator are you using?

Could you post a copy of your column definitions so i can see if there are any bottle necks there, also how many rows are you adding and how many columns are there, what formatters are you using, are there any mutatiors or editors or click events in your column definitions?

Cheers

Oli

[zeropage-io]

Hi Oli,
ok, here we go!

1) Tabulator version according to tabulator.js
/* Tabulator v3.3.1 (c) Oliver Folkerd */

2) Column definitions

[ {title:"ID",field:"id",sorter:"number",visible:false},
  {title:"Level",field:"level",sorter:"number",visible:false},
  {title:"Hub",field:"hub",sorter:"boolean",visible:false},
  {title:"",field:"open",
    formatter:function(cell, formatterParams){
      if ( cell.getData( ).level == 1  ||  cell.getData( ).hub == 0 )
        return "";
      else if ( cell.getValue( ) == 1 )
        return "<img src='img/close.png' />";
      else
        return "<img src='img/open.png' />";
    },
    visible:true,
    width:40,
    align:"right"
  },
  {title:"URL", field:"url",
   headerFilter:"input",
   headerFilterPlaceholder:" ",
   sorter:"string",
   width:"60%",
   align:"left",
   formatter:function(cell, formatterParams){
      var level = ( cell.getData( ).level - 1 ) * 30; // number of pixels to indent
      cell.getElement( ).css( {"padding-left":"" + level + "px"} );
      return "<a href='" + cell.getValue() + "' target='_blank'" + (cell.getData( ).hub?" style='font-weight:bold'":"") + "><img src=\"img/ext.png\" width=\"14\" height=\"11\" />" + cell.getValue() + "</a>";
    }
  },
{title:"Category Linkjuice<sup>*</sup>", field:"clj", headerFilter:"input", headerFilterPlaceholder:" ", sorter:"number",align:"left"},
{title:"Page Linkjuice<sup>*</sup>", field:"lj", headerFilter:"input", headerFilterPlaceholder:" ", sorter:"number",align:"left"},
{title:"#Incoming Links<sup>*</sup>", field:"il", headerFilter:"input", headerFilterPlaceholder:" ", sorter:"number",align:"left"}
]

3) Number of columns
Five visible, three invisible; cf. column definition.

4) How many rows added
This varies from only a couple of rows to a max of, currently, 4788. Adding these 4788 rows takes approx. 1,5 minutes.

5) Formatters used
Two custom made formatters: One in the hidden "open"-column and one in the URL column, please cf. the column definitions.

6) Mutators or editors
None.

7) Click events
A rowClick-event in the table constructor.

Ok, that's pretty much the whole code. :-)

Hope you can spot a potential bottleneck!

Many thanks in advance and best wishes
Axel

[zeropage-io]

Hi Oli,

I guess you receive a lot of cries for help like mine. Any chance you could look into this topic?

Many thanks in advance and best wishes
Axel

[olifolkerd]

Hey @zeropage-io

How are you loading the data into the table, is it from an array or from an ajax request?

[zeropage-io]

Hi Oli,
I am loading from an array. It looks like this:

childs[0] = [{"id":15741,"level":2,"hub":0,"open":0,"url":"https://domain.tld/zzz,"clj":"","lj":"3816,40","il":6940},{"id":15735,"level":2,"hub":1,"open":0,"url":"https://domain.tld/zzz,"clj":"4561,97","lj":"3816,40","il":6940},{"id":15734,"level":2,"hub":0,"open":0,"url":"https://domain.tld/zzz,"clj":"","lj":"3810,05","il":6940},{"id":15733,"level":2,"hub":0,"open":0,"url":"https://domain.tld/zzz,"clj":"","lj":"6,29","il":1},...]; childs[4] = [{"id":2906,"level":3,"hub":1,"open":0,"url":"https://domain.tld/zzz,"clj":"106912,86","lj":"7586,82","il":13881},{"id":2880,"level":3,"hub":1,"open":0,"url":"https://domain.tld/zzz,"clj":"87682,14","lj":"3770,40","il":6940},{"id":2861,"level":3,"hub":1,"open":0,"url":"https://domain.tld/zzz,"clj":"61061,16","lj":"3769,22","il":6940},{"id":2591,"level":3,"hub":1,"open":0,"url":"https://domain.tld/zzz,"clj":"823,62","lj":"675,52","il":9},{"id":1445,"level":3,"hub":1,"open":0,"url":"https://domain.tld/zzz,"clj":"149738,69","lj":"7587,83","il":13881},{"id":1087,"level":3,"hub":1,"open":0,"url":"https://domain.tld/zzz,"clj":"188858,54","lj":"7592,84","il":13881},{"id":312,"level":3,"hub":1,"open":0,"url":"https://domain.tld/zzz,"clj":"227391,10","lj":"7594,03","il":13881},{"id":188,"level":3,"hub":1,"open":0,"url":"https://domain.tld/zzz,"clj":"202703,65","lj":"7593,02","il":13881},{"id":5,"level":3,"hub":1,"open":0,"url":"https://domain.tld/zzz,"clj":"36803,10","lj":"0,00","il":0}]; childs[5] = [{"id":186,"level":4,"hub":1,"open":0,"url":"https://domain.tld/zzz,"clj":"0,04","lj":"0,02","il":2},{"id":184,"level":4,"hub":1,"open":0,"url":"https://domain.tld/zzz,"clj":"0,04","lj":"0,02","il":2},{"id":178,"level":4,"hub":1,"open":0,"url":"https://domain.tld/zzz,"clj":"0,07","lj":"0,00","il":0}, ...];
The array members can contain only a couple up to 4000+ rows (data sets).
Hope this helps!

Best wishes
Axel

[zeropage-io]

Hi Oli,
wish you a happy new year!! :-))
Did you have any chance to take a look into the performance issues?
Is there any more information that I can provide?
Best wishes
Axel

[olifolkerd]

Hey @zeropage-io

Happy new year to you too.

Sorry for the delay, things were a bit hectic over the festive season.

Which Tabulator function are you calling to set the data?

Cheers

Oli

[zeropage-io]

Hej Oli!
Good to hear from you! :-)
I am adding using addData. It's in this piece of code:
$("#main-table").tabulator( "addData", childs[ row.getData().id ], false, row.getIndex() );
Best wishes
Axell

[olifolkerd]

Hi @zeropage-io ,

If you are calling addData** 4000 times then that is hwy it is taking ages to load your data, the addData function causes a redraw of the table every time it is called so you are redrawing the table 10,000 times.

You need to call setData to add all the data in one go.

Cheers

Oli

[zeropage-io]

Hi Oli,
hm, there is no loop in my code. I use "add Data" since I need to add data behind specific rows. It is called once with its second parameter being an array of data (i.e. rows) and the fourth parameter being the row behind which the data shall be added.
My understanding of setData is that it always replaces all the data in the table and that will not work in my case.
So, I am not calling addData a couple of thousand times. Just once with a big array of up to a couple of thousand rows. So it's the call to addData that consumes so much processing time. Perhaps addData internally adds one row after another forcing the redraw after each row that you mentioned?
Best wishes
Axel

[zeropage-io]

Hi Oli,
I will not be able to solve this on my own. Unfortunately I am a complete noob with respect to profiling and optimising complex javascript code.
What I could derive from peeking into your code is that in function
RowManager.prototype.addRowActual = function (data, pos, index) {
there is a call to
this.renderTable();
Due to lack of profiling abilities I can not tell whether this call might be a potential bottleneck when adding a plethora of rows. But _if_ rendering each and every row immediately after it has been added turns out be a time consuming call, would it be possible to add a global option or a fourth parameter to "addData" to have all rows added first and only then render the table once?
Thanks for any help and best wishes
Axel

[olifolkerd]

Hey Axel,

The 3.4 release, coming out in February should resolve the issues with the addData function.

Cheers

Oli :)

[zeropage-io]

Hi Oli,
many thanks for the good news. Looking forward to release 3.4! :-)))
Best wishes
Axel

[GavinOG]

Just to note, also very interested in this issue. Adding by AddRow, or AddData is very slow, so I end up having to use setData, which loses table context, and makes things a bit messier.

[olifolkerd]

Hey All,

You will be happy to hear i have pushed a fix for this to the 3.4 branch, which will be officially released tomorrow!

Cheers

Oli :)

[Fealios]

I wanted to interject that I am using 4.2.3 of Tabulator in an angular application, trying to add around 3000 items (in an array) by using the tabulator.addData() method, and the page will hard crash if I do this. Any update on the progress of this issue? Or is there possibly an alternative way to inject a sizeable amount of data?

[olifolkerd]

Sounds like you arnt using the virtual dom

[Fealios]

@olifolkerd I have a height of 80vh currently.

    this.myTable = new Tabulator('#tabulator-div', {
      height: '80vh',
      selectable: true,
      layout: 'fitDataFill',
      pagination: 'local',
      paginationSize: 100,
      paginationSizeSelector: this.paginationSizes,
      columns: this.columnNames
    });

[AlexGII]

Hello, @olifolkerd I have the same issue, I use addData for periodically adding new data, and the amount of data can be huge < 1mm. I use v.4.2.7 and 4.7 result is the same bowser just stuck on adding a new portion of data. Will be very thankful for any advice.

table = new Tabulator("#table", { height: "100%", layout: "fitColumns", pagination: "local", paginationSize: 50, paginationSizeSelector: [50, 100, 200], groupToggleElement:"header", groupStartOpen: false, movableColumns: true, columns: [ //Define Table Columns {title: "Cluster", field: "clusterId", width: 90}, { title: "Date", field: "date", align: "left", sorter:"datetime", sorterParams:{ format:"DD-MM-YYYY HH:mm:ss", alignEmptyValues:"top", }, width: 150 }, {title: "Filters", field: "filters", width: 100} ] });

table.addData(data, true);

data is json that I receive periodically from rest resource.

thank a lot!

[mspanish]

Same problem, although I am adding via chunks using papa parse. I've set the chunks to 500KB, it should only be calling 200 total chunks using addData, and it freezes the app every time.

[olifolkerd]

Hey guys, this is a very old closed issue on a very old version of Tabulator. Any issues you may be having now are certainly unrelated to this issue.

If you think you have found an bug, please create a new issue using the bug template and provide a link to a JS fiddle that shows how your table is setup. It is almost impossible to help without seeing the way your table is configured.

Long loading times usually mean that no height has been set on the table so it has to render all the rows at once.

If you have a large number of rows in your table. You should look at using the progressive loading functionality that will automatically loaf more data into your table as the user scrolls. It will break up the transfer time needed to get data into the table so the browser won't be waiting for one massive Ajax call.

Cheers

Oli