Ngx-datatable: Server pagination not working

[SnakeB]

Pagination and Sorting should be handled through the server. To load my data from the backend I´m using the following method:

this.loadData('sortColumn', 'ASC', this.currentPage, this.options.limit)
        .subscribe(ratings => {
        this.rows = [];
        this.rows.push(...ratings.ratings);
        this.endRow = this.options.limit * (this.options.offset + 1);
        this.maxRows = ratings.recordsTotal;
        if (this.maxRows < this.options.limit) {
          this.endRow = this.maxRows;
        } else if (this.maxRows < this.endRow) {
          this.endRow = this.maxRows;
        }
        this.options.count = this.maxRows;
  }, error => this.errorMessage = <any>error);}

this.rows is the variable I´m binding to the data-table-template:

<datatable class="rating_table" [rows]='rows' [options]='options' (onRowClick)='onRowClick($event)' (onPageChange)='onPage($event)'>

My problem is when I´m loading the first page the data binding ( this.rows.push(...ratings.ratings)) works and in the table my ratings are shown, but when I´m loading the second page the new ratings are saved into the this.rows object but the table is not refreshing the ratings from page one are still visible.

[amcdnl]

Please make a demo in plunkr.

[juracy]

Same problem here, it looks like angular2-data-table is looking for options.offset + 1 in rows, instead of getting a completed new "buffer" starting from 0...

[amcdnl]

Offset + 1 is because offset is 0 based, for paging there is no 0. :)

[SnakeB]

I will prepare and post an example tomorrow

[SnakeB]

I prepared a small example to show the problem:
https://plnkr.co/edit/jTf0ospsvh5TzhOQtdQp?p=preview
I´m setting new rows every time the user switches from one page to another:

this.rows = [];
    if(pageNumber === 0){
      this.rows.push(...this.page1Data);
    }else{
      this.rows.push(...this.page2Data);
    }
    this.options.count = this.page1Data.length + this.page2Data.length;
    console.log(this.rows);`

if you look at the console the rows object is updated with the new data but the page is not refreshing. I hope you can help me.

[SnakeB]

Could you please help me with my problem

[robertbachmann]

I had a similar problem. The trick is to update the rows Array in a different way. See
https://plnkr.co/QfS4KyU7oNYCFXb2S22A

[juracy]

@amcdnl is there need for more info, yet ?

I think the problem is: the first line of grid must be the first element of array (row[0]) at least in server-side mode, regardless of the page or offset.

Thanks @robertbachmann for the workaround !

[il-santo]

Same problem here. IMHO the the right behavior should be the one pointed out by Juracy:
"the first line of grid must be the first element of array (row[0]) at least in server-side mode, regardless of the page or offset."

[vijayanandkannan]

Thanks @robertbachmann for the workaround. @amcdnl any update on this. I do have the same Issue. Also I would like to have Virtual scrolling from the server pagination. This will be Quite helpful when you browse from mobile (Similar to Gmail .. and others)

[tsquillario]

+1
I have server-side pagination working but I'm getting 'Cannot read property '$$expanded' of null' when trying to set rows = [].

I'm just realizing that the pagination expects you to insert the rows at the offset. Struggling to figure out how to properly reset the rows and essentially start over. The suggestion from @robertbachmann helped get the initial server-side working.

[IRCraziestTaxi]

When is this going to be addressed? It seems to me that server side pagination should not use the offset the way client side pagination does. The entire point of server side paging is to not have to store all results in memory as the workaround does.

Edit: I should edit to say that I don't wish to set all results to null as in the workaround because that results in the table collapsing vertically while new results are being fetched, which looks horrible.

Edit Edit: I implemented a solution that caches previous results to use as a placeholder while new results on the selected page are loading, but this is still a hefty workaround for an issue that doesn't make sense.

[dallinski]

Any update on when we can expect real server-side pagination?

[deeg]

I am currently investigating/working on server side pagination & virtual scroll problems (#353), and this seemed related.

The library does seem to cache all data, so once a page has loaded, it will remain in the rows array unless you manually clear it out like in the described workaround above.

The fix I was going to attempt was to use calculations to get the correct virtual scroll height, but try to continue leaving all of the data already loaded in the rows array because that is what it is doing now. See the conversation in #353.

Are people saying the correct behavior should be when loading new pages, the older pages should no longer be kept in memory and re-fetched when scrolling back to that page?

@java2kus, any thoughts on the discussion in this thread that might weigh into #353? Maybe this is what you were trying to convey from this comment and I just haven't put the pieces together yet.

[stefmabo]

I think $$index is the problem, when you click the second page or third page, etc. It doesn't add the $$index attribute to the objects on those selected pages. It only adds that attribute to the objects belonging to the first page.

Test it printing rows objects.

[PatrickHuetter]

Is there any progress on this? I think this is a critical bug.

[deeg]

@PatrickHuetter, this has not been fixed yet, and I haven't made any more progress on it has I have been busy at work.

There are some existing questions out from my last post. It might be more helpful to give input on those questions, so we can more quickly come to a solution.

@java2kus, I know you are very busy, but would you have a sec to look over the questions I listed above?

[deeg]

Okay I see now there may be two issues.

1. Server side pagination with vertical virtual scroll enabled
This is more the subject of what I was talking about in my previous posts. This is related to #353.

2. Server side pagination without vertical scroll enabled, using the paging mechanism

It seems even when fetching and setting the correct position in the rows array, the new data does not display on the server which is directly a duplicate of #486. It looks like if you mutate the array the change detection does not get fired. If you set the rows property to be a brand new array, then change detection picks up and the data will show correctly.

This is why the demo site for server pagination also sets a new array. I guess this could be considered a bug and when you mutate the array it should also update the data for the table. This would not address the issue of the table keeping more than one page or data in memory when you change pages, but rather all pages which have been loaded before.

@amcdnl, @java2kus, thoughts?

[chriseenberg]

Any development on this issue?
To be honest, i can't use the component until paging works - and i really like the look and the features of this component so i'd love to see it working.

[stefmabo]

When I show data list thought an API, it only loads data by offset and limit if the API supports pagination.
I think the implementation of this datatable is based giving JSON file.

If we need to call a API to show the data, the this.indexes.first would be always 0, because the data loaded by API always start in index 0.

Then I commented this line of code //first = Math.max(this.offset * this.pageSize, 0);
In my case, I never load the data using JSON file.

Go into the ngx-datatable app and edit updateIndexes()

body.component.ts

updateIndexes(): void {
    let first = 0;
    let last = 0;

    if (this.scrollbarV) {
      // Calculation of the first and last indexes will be based on where the
      // scrollY position would be at.  The last index would be the one
      // that shows up inside the view port the last.
      const height = parseInt(this.bodyHeight, 0);
      first = this.rowHeightsCache.getRowIndex(this.offsetY);
      last = this.rowHeightsCache.getRowIndex(height + this.offsetY) + 1;
    } else {
      //first = Math.max(this.offset * this.pageSize, 0); HERE!!!
      last = Math.min((first + this.pageSize), this.rowCount);
    }

    this.indexes = { first, last };
  }

Then in the component add this.

1. Variables

rows = [];
count: number = 0;
offset: number = 0;
limit: number = 10; //Whatever you want

2. ngOnInit()

ngOnInit() {
    this.fetch();
}

3. fetch()

fetch(event?) {
    if(event) this.offset = event.offset;
    this.callService.list('url', this.offset * this.limit, this.limit).subscribe(
      data => {
        this.count = data.total; //Web Service gives me the total
        this.rows = data.list; //Web Service gives me the data
      },
      null,
      () => {

      }
    );
}

4. ngx-datatable tag

<ngx-datatable
        class="material"
        ...
        (page)='fetch($event)'> <!--Here--> ...
</ngx-datatable>

Now it works using an API.

In my opinion, this datatable should have a flag to identify when we load data from JSON file or API that supports pagination.

[deeg]

@stefmabo, I believe this flag already exists. externalPaging input on the table. If it is false, all of the rows are loaded client side, and if it is true, you are fetching it from a server. Is this what you meant?

@amcdnl, I believe this can be closed.

I think the solution for this specific ticket is just do not mutate the rows array, set a new one.

[stefmabo]

@deeg client side and server side examples are using JSON files.

What happens when we load data from Web Service?

[deeg]

@stefmabo, I'm confused at what you are trying to do. Are you trying to fetch data from a web server but without pagination?

Just fetch it with a service outside of this library and pass in the data to a rows array. Make server side pagination false. I think this help is better suited for StackOverflow.

I load data (cases with and without server side pagination) from a server just fine.

[stefmabo]

@deeg Are you trying to fetch data from a web server but without pagination?
No, I need a pagination.

The problem is with indexes.
When I load the data using Web Service and I click second, third page, The rows are empty.
Just in the first page the data shows.

[bc3]

I have been struggling with this issue also, and i found a solution for now. I took the sample code from: https://github.com/swimlane/ngx-datatable/blob/master/demo/paging/paging-server.ts
and found that in case of server side paging with an API, there is actually 1 thing incorrect: where you loop over the rows, the sample uses i as indexer, which is correct in case of a json file with all data, however if you run it against an API, you usually get only the result set of a page, so changing the source code to the snippet below will make your server side paging on an API work

   var j = 0;
   for (let i = start; i < end; i++) {
        rows[i] = results[j];
                j++;
   }

[stefmabo]

@bc3 That workaround works, but the table jumps when you click next page, producing an visual error.
If you edit body.component.ts > updateIndexes()

And comment a line of code like this

updateIndexes(): void {
    let first = 0;
    let last = 0;

    if (this.scrollbarV) {
      // Calculation of the first and last indexes will be based on where the
      // scrollY position would be at.  The last index would be the one
      // that shows up inside the view port the last.
      const height = parseInt(this.bodyHeight, 0);
      first = this.rowHeightsCache.getRowIndex(this.offsetY);
      last = this.rowHeightsCache.getRowIndex(height + this.offsetY) + 1;
    } else {
      //first = Math.max(this.offset * this.pageSize, 0);  Comment this!!!!!!!!!!
      last = Math.min((first + this.pageSize), this.rowCount);
    }

    this.indexes = { first, last };
  }

//first = Math.max(this.offset * this.pageSize, 0); Comment this!!!!!!!!!!

The effect of jumping dissapears.

[amcdnl]

@stefmabo - can you PR this?

[bc3]

@stefmabo - that would indeed be a good idea, because i cannot change body.component.ts , because we work in a team with NPM_modules on each developer environment ..

[stefmabo]

I'm testing it before to do a PR.
I don't know if commenting that line of code, can affect other components, then I prefer check it and after that, I will do a PR.

For now, go into node_modules/@swimlane/ngx-datatable/release/index.js a find with ctrl+f updateIndexes method and the first line of code after else is:

first = Math.max(this.offset * this.pageSize, 0);

Comment that and go here: https://github.com/swimlane/ngx-datatable/issues/138#issuecomment-282727332

[psreeks]

@amcdnl Once this is fixed , can you also add this fix to 6.x.x version too? 7.0 version is not working for us, as we haven't upgraded to angular 4 yet

[psreeks]

@amcdnl any update on this?

[rob-balfre]

Is this actually fixed? Cause I've spent hours trying to get it working and I'm kinda peeved that I've only just found this thread. I'm using 7.2.1 with angular v4 and getting the exact same error still.

[vdurante]

Apparently it is still in the roadmap, not fixed yet

On Apr 8, 2017 1:57 AM, "Robert Balfre" [e-mail] wrote:

Is this actually fixed? Cause I've spent hours trying to get it working
and I'm kinda peeved that I've only just found this thread. I'm using 7.2.1
with angular v4 and getting the exact same error still.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/swimlane/ngx-datatable/issues/138#issuecomment-292695229,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFJqkDe6guOlGX5c8TE47QMg9-z0j9AFks5rtxO0gaJpZM4KC-73
.

[rob-balfre]

Maybe they should drop the demo then, its pretty misleading.

[deeg]

I am using sever side pagination without virtual scroll without problem. I have seen people have the index error when they are not setting the count prop correctly.

Are you having issues with sever side paging without virtual scroll?

[vdurante]

Tecnically it is server side, but without pagination. But as everyone else
here, we expected to be 100% server side

On Apr 8, 2017 2:01 AM, "Robert Balfre" [e-mail] wrote:

Maybe they should drop the demo then, its pretty misleading.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/swimlane/ngx-datatable/issues/138#issuecomment-292695379,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFJqkMy9XnnbXiGqGbN9DWiRKu4r3Lv0ks5rtxSxgaJpZM4KC-73
.

[rob-balfre]

@deeg what is virtual scroll? And I set the count prop to the total count of items in the dataset (not the number of items returned on that page), is that correct?

[rob-balfre]

So I'm correct in thinking there there is currently NO support for server based pagination? Unless you return the entire dataset (which defeats the entire point of server based pagination for 99% of people). I'd really suggest making it much clearer on the demo of it's limitations.

[deeg]

@rob-balfre, that is incorrect, it works with server side pagination. You do not have to load all the data from the start, but rather only the first page.

The one issue which still does exist is once a page has been loaded, it is always kept in memory. Those rows are not always rendered in the dom, but once a page is fetched the data is always kept around and never refetched.

I am not at a computer so cannot provide an example tonight, but the sever side pagination demo is a good place to start. You need to make sure you are not mutating the rows array, but rather setting a new instance once you load a new page as the demo code does.

If you are still having issues I suggest you open a new issue with a reproducible plunker so someone can better assist you.

To reiterate, if you are using the footer paging mechanism you should have NO PROBLEM doing sever side pagination and not loading all the data at once...

[amcdnl]

So I'm correct in thinking there there is currently NO support for server based pagination? Unless you return the entire dataset (which defeats the entire point of server based pagination for 99% of people). I'd really suggest making it much clearer on the demo of it's limitations.

There is support for server pagination. My demo uses a JSON file for demonstration purposes, here is an example of server pagination.

  async page(offset, limit) {
    // example: offset = 1, limit = 5
    const nextData = await myService.get({ offset, limit });

    // response from sever
    // nextData = { count: 100, data: [ {}, {}, {}, {}, {} ] }

    // get index to insert
    const start = offset * limit;

    // add new results to rows
    this.myRows.splice(start, 0, ...nextData);
  }

In this demo, I make a request to my server api with the offset ( page number - 1 ) and limit ( page size ). The server responds with that dataset only. Then I calculate the first index of the new data and insert the data into that position. Also, make sure you have externalPaging turned on.

If you want to ensure you don't make a request to your server every time you scroll, you can implement some simple caching like this:

  async page(offset, limit) {
    // example: offset = 1, limit = 5

   // create a cache of limit w/ simple flag
   // cache = { 1: true, 2: true, 3: true }
   if(this.cache[limit]) return;

    // if no data found, go ahead and make request
    const nextData = await myService.get({ offset, limit });

    // response from sever
    // nextData = { count: 100, data: [ {}, {}, {}, {}, {} ] }

    // get index to insert
    const start = offset * limit;

    // add new results to rows
    this.myRows.splice(start, 0, ...nextData);
  }

In this example, I added a cache hash that stores if request has been made or not. If it has been made, do not make the request to the server, otherwise proceed to request and insert into the rows. This allows me to track this myself and implement cache busting if my needs require such.

Does this answer everyone's question regarding this matter? If not, please provide me some details and I'll be happy to help fix!

[rob-balfre]

@amcdnl thanks so much, really appreciate it. I'll give it a go today.

[rob-balfre]

No more error which is great, but only shows results on page one. Even though the new data is in there and correctly ordered. Even stranger if you just concat the old and new arrays on page change instead of splicing it works but that creates a massive array after a few pages.

Why does it even need the old data in there? Doesn't make sense for server based data.

[amcdnl]

@rob-balfre - can you show your implementation of pushing data in?

[rob-balfre]

@amcdnl Afraid not. I decided to just use my own table / pagination solution.
I wasn't doing anything unusual though. Just followed the example.

[alexweber]

I admittedly haven't tried this for a few weeks but last time I checked the server-side pagination wasn't working as expected, at least not with virtual scrolling enabled. I too had to go with a custom solution as a stop-gap. Feelsbadman :(

[psreeks]

@amcdnl , the solution @stefmabo provided above to comment out a certain line in node modules is working without any problems. As we work in a team project i can't edit something that is in node modules. Can we have the same fix in 6.3.0?

[daibertdiego]

The solution from @stefmabo stops the client-side pagination :(.
And in my software, i use both.

[amcdnl]

@rob-balfre @alexweber - Sorry to hear, would have gladly accepted a PR that we could have shared with the community and others.

@psreeks - As @daibertdiego states that will break client side.

Has anyone tried my suggestions above?

[alexweber]

@amcdnl I meant custom solution as in dropping this package entirely... If I had code to contribute on this matter you can be sure I woulda done it 👍

[efriandika]

I have tried @dcesiel solutions => #714 .
And It works properly.

[amcdnl]

Both server-side paging via pager and virtualization is released in 9.0!!

[svendpeter]

Hi,

I am using the datatable with angular 5 cli , together with Apollo and graphql .
I am trying to implement infinite scrolling .
While everytthing is fine and I am able to load the new data (I could display it in the browser using a normal html table), the rows of the datatable are not updated (refreshed) .

When I move to a different page and then come back, only the last 2 rows are updated , even if I fetch 10 new rows each time .

Can you kindly advise me on this problem ?

Thank you !

[mali208]

i cant believe this error still exist 👎 facing that error