Components: virtual-scroll: integrate with relevant existing components

[mmalerba]

Integrate virtual-scrolling as an optional add-on for relevant existing components, including:

• [ ] Accordion
• [ ] Autocomplete
• [ ] Datepicker
• [ ] Grid List
• [ ] List
• [ ] Select
• [ ] Table
• [ ] Tree
• [ ] Drag & Drop

Part of each integration should include adding docs examples of how to set it up

[ahsan]

Hi, thanks for the awesome virtual scroll feature.
Any ETA for mat-tree?

[shlomiassaf]

I managed to implement virtual scroll with the table, working pretty amazing out of the box with both fixed and auto strategies.

I used the original CdkVirtualScrollViewport component and provided my "version" of CdkVirtualForOf (which should actually be the CdkTable + measuring capabilities)

I added 1 more strategy - NoStrategy and built my own directives put on the table to control which strategy to use.

I had some issues, most of them were easily solved, the most difficult thing to expect is the handling of meta rows, which are header and footer rows.

Handling of headers and footers require special treatment because they are not part of the datasource and thus mess up the entire flow, once you have multiple header/footer rows it get's messy quickly.

I solved all issues with size measurements when working with multi-header/footer table.

The only issues i'm facing now is sticky rows, which does not work because the virtual table now has a container that "offsets" the sticky row so position top 0 is no longer 0 after it was translated by the parent.

Working around this should also be easy if I had access to the function that sets the offset - I dont because it's private in CdkVirtualScrollViewport

[kiwikern]

I think integrating it with the Grid list would also be a great feature.

[IlCallo]

@shlomiassaf can you share the code you used or try a PR?
Virtual scroll is a thing in many are waiting to see integrated into MatTable component, a workaround while we wait for official support can be useful

[ahsan]

I needed Virtual Scroll in MatTree component, I hacked the MatList component to emulate behavior of MatTree as it already has the virtual scrolling capability. I did have to manage the indentations of individual nodes myself though.
Maybe a similar sort of approach can be used for MatTable component as well.

[shlomiassaf]

@IlCallo It would be difficult to share it right now, I need to clean some IP stuff from there.

The virtual scroll is also a part of a table component (on top of CdkTable) so it has some things that will not work as they are part of that table eco-system, things like global configuration, plugin integration etc...

I can confirm that it works, and works quite fast! for both Auto and Fixed size strategies.
I will try to extract it somehow, but I can't commit to a timeframe.

For now, I will try to describe how I did it:

This is my way of implementing it, there might be other ways!

First, the general layout we will use:

<cdk-virtual-scroll-viewport>
    <cdk-table></cdk-table>
</cdk-virtual-scroll-viewport>

The virtual scroll is external to the table.

Now we need to take care of 3 topics:

• Adjusting CdkVirtualScrollViewport to work with the table
• Providing our own version of CdkVirtualFor
• Adjusting strategies to work with the table

Adjusting CdkVirtualScrollViewport to work with the table

The general idea is to create a custom viewport component that inherits from CdkVirtualScrollViewport and apply minor adjustments so the table will work.

Our CdkVirtualScrollViewport will also control which strategy we use, replacing to our own table-targeted strategy when needed.

Providing our own version of CdkVirtualFor

We need to connect CdkVirtualScrollViewport with CdkTable, but CdkVirtualScrollViewport requires a CdkVirtualFor, which is a structural directive...

In general, CdkVirtualFor will render a subset (range) of rows from a DataSource and act upon changes in the datasource or range so it will always render the right subset.

CdkTable already does all of that, and some more...

We will use a simple class that mimics CdkVirtualFor while bridging the two components.

---

On a side-note, @mmalerba: CdkVirtualScrollViewport.attach(forOf: CdkVirtualForOf<any>): void; is probably narrow, forOf should proably be:

interface CdkVirtualScrollAdapter {
  dataStream: Observable<T[] | ReadonlyArray<T>>
  measureRangeSize(range: ListRange, orientation: 'horizontal' | 'vertical'): number;
}

---

Our CdkVirtualFor does 2 things:

Make sure sticky rows works

Because the table position is not static (transformed all over in the virtual viewport) the sticky rows
will not stick... we need to compensate for the transformation so their top root following the virtual top root offset. We do that by listening to offset changes (from CdkVirtualScrollViewport) and update the sticky rows with the offset.

Adjust the range to compensate for header/footer rows

Because the table might contain Header or Footer rows, we need to adjust the range accordingly. The virtual viewport is wrapping both header/footer rows and the actual table rows...

Our CdkVirtualFor adapter listens to renderedRangeStream changes from the viewport, and pass it on the the table (via CdkTable.viewChange.next(range)).

The viewport calculates the range using the strategy and it will return how many rows to render. This number is then used to extract rows from the data source. If we have a header row in view we need to reduce the range.

Adjusting strategies to work with the table

FixedSizeVirtualScrollStrategy works fine without changes, it's AutoSizeVirtualScrollStrategy that we need to amend.

The problem is not really in AutoSizeVirtualScrollStrategy but in CdkVirtualScrollViewport and how it emits the initial range, this is a known issue - See this comment by @mmalerba

To fix it I use a custom strategy that wraps AutoSizeVirtualScrollStrategy and ItemSizeAverager .
Basically, I use a TableItemSizeAverager that has access to the how many actual rows are rendered. When ItemSizeAverager.addSample() is called - if no rows are rendered it will use the default row height otherwise will work as is.

This could probably get solved differently, but because CdkVirtualScrollViewport has most of its logic methods private I had to go this way...

If this fix is not applied the average size will get very small values because it will get a large range of "rows" before rows are rendered... so the total height/rows will be small.

Hope it helps!!!

[shlomiassaf]

I managed to upload a small demo app I have for the table....

https://shlomiassaf.github.io/table-demo

Look at the "Demo" link on the left, it shows a large list with a virtual scroll (AUTO). You can set it to 100,000 rows about 23-24 columns.

The "All In One" link shows a virtual scroll with FIXED strategy.

It's a POC for all of you that want to use it.

Note that this is a quick and dirty demo app, expect bugs :)

[shlomiassaf]

OK, also managed to implement Drag and Drop using CdkDrag and my own version of CdkDropList.

See demo:

https://shlomiassaf.github.io/table-demo

It does both column and row d&d.

There is no "real" need to create a custom CdkDropList component, the material team just need to refactor it a bit so people can extend it (it's CDK after all)

Most of it is private and some functions are just big, if they port most to protected and split some functions (mainly _sortItem) I would be able to reuse it...

[CuriousDolphin]

it is possible to use virtual scroll with a grid list of element?.

[vrady]

OK, also managed to implement Drag and Drop using CdkDrag and my own version of CdkDropList.

See demo:

https://shlomiassaf.github.io/table-demo

It does both column and row d&d.

There is no "real" need to create a custom CdkDropList component, the material team just need to refactor it a bit so people can extend it (it's CDK after all)

Most of it is private and some functions are just big, if they port most to protected and split some functions (mainly _sortItem) I would be able to reuse it...

Hello @shlomiassaf , can you provide source code of your demo application? We need to figure out how to implement features that you show in demo. Thanks

[fusionUI]

I managed to upload a small demo app I have for the table....

https://shlomiassaf.github.io/table-demo

Look at the "Demo" link on the left, it shows a large list with a virtual scroll (AUTO). You can set it to 100,000 rows about 23-24 columns.

The "All In One" link shows a virtual scroll with FIXED strategy.

It's a POC for all of you that want to use it.

Note that this is a quick and dirty demo app, expect bugs :)

Hi,

It would be really great if you can share the code,

Thanks

[mmalerba]

@shlomiassaf Thanks for the detailed summary! That will be a great starting point for exploring integration with the table

[codestitch]

@shlomiassaf is there a way to check neg-table in your demo to experiment?

[nahgrin]

I got a basic version of virtual scroll working with the grid. I'll give a brief overview of what I did and try to come back and post a working example in a bit. I tried to follow what @shlomiassaf did and I ended up with a slightly different approach.

For the following code, I "borrowed" heavily from the material table examples. I'll try to describe what I did here and then just leave the code below to hopefully help answer any questions that my explanation leaves.

For the HTML, I wrapped the table element in the cdk-virtual-scroll-viewport as was suggested. However, I also had to modify the outlet for the row data so that it combined the cdkVirtualFor with the matRowDef. Instead of using the structural directive for the row, I expanded it out and kind of merged it with the cdkVirtualFor. Another important thing was that the datasource for the cdkVirtualFor is __not__ the same one that is feeding the table. The rows observable is basically the true observable of the data in the grid while the dataSource observable is a filtered version of the rows for the table.

I created my own strategy for dealing with the virtual scroll in the table and it's mostly just an exceedingly simplified version of the FixedSizeVirtualScrollStrategy from @angular/cdk/scrolling. The reason I did this was that the FixedSizeVirtualScrollStrategy was producing some really weird rendering errors where the table would routinely display elements in the table off by a certain index. I think that it was causing the cdkVirtualFor and the mat-table to fight each other for rendering or something, but I'm not informed enough to say for sure. Other than that problem, the FixedSizeVirtualScrollStrategy can just be dropped in and will work without concern.

The component stitches the data in the table and the strategy together and creates the separate dataSource observable for the table. Every time that the index of the scroll is updated it modifies the slice of the array so that the table only renders the piece of the table that should be in view.

That's basically what I've done and it is working pretty well for me. If anyone has any insight into getting the FixedSizeVirtualScrollStrategy, that would be wonderful.

table.component.html

<cdk-virtual-scroll-viewport [style.height.px]="gridHeight">
  <table mat-table [dataSource]="dataSource" class="mat-elevation-z8">

    <ng-container matColumnDef="position">
      <th mat-header-cell *matHeaderCellDef> No. </th>
      <td mat-cell *matCellDef="let element"> {{element.position}} </td>
    </ng-container>

    <ng-container matColumnDef="name">
      <th mat-header-cell *matHeaderCellDef> Name </th>
      <td mat-cell *matCellDef="let element"> {{element.name}} </td>
    </ng-container>

    <ng-container matColumnDef="weight">
      <th mat-header-cell *matHeaderCellDef> Weight </th>
      <td mat-cell *matCellDef="let element"> {{element.weight}} </td>
    </ng-container>

    <ng-container matColumnDef="symbol">
      <th mat-header-cell *matHeaderCellDef> Symbol </th>
      <td mat-cell *matCellDef="let element"> {{element.symbol}} </td>
    </ng-container>

    <tr mat-header-row *matHeaderRowDef="displayedColumns"></tr>
    <ng-template let-row matRowDef cdkVirtualFor [matRowDefColumns]="displayedColumns"[cdkVirtualForOf]="rows">
      <tr mat-row></tr>
    </ng-template>
  </table>
</cdk-virtual-scroll-viewport>

table-vs-strategy.service.ts

@Injectable()
export class TableVirtualScrollStrategy implements VirtualScrollStrategy {

  private scrollHeight!: number;
  private scrollHeader!: number;
  private readonly indexChange = new Subject<number>();

  private viewport: CdkVirtualScrollViewport;

  public scrolledIndexChange: Observable<number>;

  constructor() {
    this.scrolledIndexChange = this.indexChange.asObservable().pipe(distinctUntilChanged());
  }

  public attach(viewport: CdkVirtualScrollViewport): void {
    this.viewport = viewport;
    this.onDataLengthChanged();
    this.updateContent(viewport);
  }

  public detach(): void {
    // no-op
  }

  public onContentScrolled(): void {
    this.updateContent(this.viewport);
  }

  public onDataLengthChanged(): void {
    this.viewport.setTotalContentSize(this.viewport.getDataLength() * this.scrollHeight);
  }

  public onContentRendered(): void {
    // no-op
  }

  public onRenderedOffsetChanged(): void {
    // no-op
  }

  public scrollToIndex(index: number, behavior: ScrollBehavior): void {
    // no-op
  }

  public setScrollHeight(rowHeight: number, headerHeight: number) {
    this.scrollHeight = rowHeight;
    this.scrollHeader = headerHeight;
    this.updateContent(this.viewport);
  }

  private updateContent(viewport: CdkVirtualScrollViewport) {
    const newIndex = Math.max(0, Math.round((viewport.measureScrollOffset() - this.scrollHeader) / this.scrollHeight) - 2);
    viewport.setRenderedContentOffset(this.scrollHeight * newIndex);
    this.indexChange.next(
      Math.round((viewport.measureScrollOffset() - this.scrollHeader) / this.scrollHeight) + 1
    );
  }
}

table.component.ts

@Component({
  providers: [{
    provide: VIRTUAL_SCROLL_STRATEGY,
    useClass: TableVirtualScrollStrategy
  }],
  ...
})
export class TableComponent implements OnInit {

  // Manually set the amount of buffer and the height of the table elements
  static BUFFER_SIZE = 3;
  rowHeight = 48;
  headerHeight = 56;

  rows: Observable<Array<any>> = of(new Array(1000).fill({position: 1, name: 'Hydrogen', weight: 1.0079, symbol: 'H'}));

  displayedColumns: string[] = ['position', 'name', 'weight', 'symbol'];

  dataSource: Array<any>;

  gridHeight = 400;

  constructor(@Inject(VIRTUAL_SCROLL_STRATEGY) private readonly scrollStrategy: TableVirtualScrollStrategy) {}

  public ngOnInit() {
    const range = Math.ceil(this.gridHeight / this.rowHeight) + TableComponent.BUFFER_SIZE;
    this.scrollStrategy.setScrollHeight(this.rowHeight, this.headerHeight);

    this.dataSource = combineLatest([this.rows, this.scrollStrategy.scrolledIndexChange]).pipe(
      map((value: any) => {

        // Determine the start and end rendered range
        const start = Math.max(0, value[1] - GridComponent.BUFFER_SIZE);
        const end = Math.min(value[0].length, value[1] + range);

        // Update the datasource for the rendered range of data
        return value[0].slice(start, end);
      })
    );
  }
}

[garrettld]

@nahgrin I hope you don't mind, I've thrown your code into StackBlitz so that I could play around with it.

https://stackblitz.com/edit/nahgrin-virtual-scroll-table

There was a tiny bit of cleanup, but it pretty much just worked. Thanks so much!

[nahgrin]

@garrettld Awesome! Thanks a bunch for doing that!

[shlomiassaf]

@nahgrin Nice work!

The problem is in the header/footer rows, you need to take them into account when you calculate the range from the data source.

For example, if I have 5 header rows and 1000 items.
Let's say I can fit 10 rows in my viewport and for simplicity, there is no buffer.

When i'm in 0 scroll offset I can't return 10 rows because 5 are used by headers... same goes for footer rows.

You need to calculate the header/footer rows and their visible height in the view, remove that and use the height left to calculate what is the actual range.

Here's an example with 5 header rows that breaks it:

https://stackblitz.com/edit/nahgrin-virtual-scroll-table-wdlqlm?file=src%2Fapp%2Ftable%2Ftable.component.ts

Of course there are more things to take care of, sticky rows and AutoSize strategy...

You might want to refactor TableVirtualScrollStrategy into a directive. You can
put that directive on the table and then inject the table to it. You will have access to the table instance and the CdkVirtualScrollViewPort instance.

On that directive you can define all the heights... and you can also use a special input for the datasource, the directive will take that datasource and assign it to the table so it's a real plugin.

[fusionUI]

@nahgrin Tremendous work. Thanks for your efforts.

[nahgrin]

@shlomiassaf Good point about the headers. I tooled around a little more and moved some of the logic in towards the strategy (and removed a little bit of the hardcoding) and got a solution for headers:

https://stackblitz.com/edit/nahgrin-virtual-scroll-table-cvxa7v

I'm starting to get a distinct feeling that I'm reinventing the wheel, though, with the strategy. The FixedSizeVirtualScrollStrategy does basically everything the table needs, it's just off by a little bit off because of the headers. @mmalerba Would it be possible to add an offset field to the FixedSizeVirtualScrollStrategy for these sorts of use cases?

I'm not sure about sticky headers, but after tooling around a little and looking at the earlier example table, I'd imagine the table needs to apply a reverse transform to the headers to keep them pinned to the top. @shlomiassaf Did you use the display: flex table to specifically fix any of these problems?

[shlomiassaf]

@nahgrin For sticky you just need to compensate for the transformation done on cdk-virtual-scroll-content-wrapper.

For header rows, you just add the CSS property top with the negative value of the transform.
So if the transform is translateY(5196.05px) you need to set the css top: -5196.05px on each header.

For footer rows, you just add the CSS property bottom with the value of the transform.
So if the transform is translateY(5196.05px) you need to set the css bottom: 5196.05px on each footer.

[shlomiassaf]

@nahgrin I have to say that i'm still quite confused.

I believe that there are 2 rendering "engines" running.

1) The CdkTable
2) The CdkVirtualFor

    <ng-template let-row matRowDef cdkVirtualFor [matRowDefColumns]="displayedColumns" [cdkVirtualForOf]="rows">
      <tr mat-row></tr>
    </ng-template>

In this section of your template, you are passing a row template to CdkTable which will render it for every row in range.

But, you are also passing a template to CdkVirtualFor which will render it for every row in range.

The rendering of CdkTable will also add cells, the CdkVirtualFor rendering will just render an empty <tr>...

For every change in range the view port will notify both of them, causing a redundant diff operation in CdkVirtualFor.ngDoCheck() which is followed by a DOM update.

You can put breaking points in CdkVirtualFor and see how it works when you scroll and/or load a data source.

I'm not sure this design will scale...

This is why I built my own version of CdkVirtualFor so it won't double the work.

[danzrou]

Anyone found any workaround for using it inside <select> ?

[nahgrin]

@shlomiassaf I took a look at things and pulled the CdkVirtualFor out of the implementation. The only thing that I was using it for was to get the length of the data set, so it doesn't really do a lot for the virtual scrolling. It cleaned up the implementation in the table a little bit (and removed the need for the table component to know anything about the virtual scrolling), but I'm not seeing a lot of changes to the performance.

I created a separate stackblitz for comparison: https://stackblitz.com/edit/nahgrin-virtual-scroll-table-hampgc

[shlomiassaf]

You won't see it in this table...

If you have a huge one, with a lot of columns and rich content it might appear, anyway it wasn't supposed to be used like that so it's good you removed it.

The new implementation is much better, but you end up with something that will cause pain in the future.

You created a custom implementation for the scroll strategy that is completely different from the core strategies (fixed and auto-size).

When auto-size lands it will be difficult to use because you will have to again, rebuild it from scratch based on the cdk source code instead of just inheriting it and fixing here and there.

The TableVirtualScrollStrategy you built is actually a mix of the cdkVirtualFor and the FixedSizeVirtualScrollStrategy take code from both classes and mixing it up into one class.

Again, this is limiting because there are other strategies and you don't want to couple the cdkVirtualFor with it. Moreover, you don't want to re-write logical code written by others, if the logic changes you will need to follow.

Another thing I noticed is that the view-port is not used as intended, I didn't see any call to attach on the view port. It works but probably because you did things done in viewport.attach internally in the service...

[blue-seagull]

@shlomiassaf
I aggree with you, we need to

1. reuse viewport.attach
2. and not be tied to cdkVirtualFor

and the only clean solution for this is what you suggested earlier to @mmalerba:

CdkVirtualScrollViewport.attach(forOf: CdkVirtualScrollAdapter <any>):

interface CdkVirtualScrollAdapter {
  dataStream: Observable<T[] | ReadonlyArray<T>>
  measureRangeSize(range: ListRange, orientation: 'horizontal' | 'vertical'): number;
}

I will send a pull request for this.

[blue-seagull]

@shlomiassaf @nahgrin
I have sent the PR to add the scroll adapter: #14287, but the PR checks fail for an unrelated reason...

[jwrunner]

I managed to upload a small demo app I have for the table....
https://shlomiassaf.github.io/table-demo

@shlomiassaf If it's not too much trouble for you, I'd like to chime in with the others that it'd be so neat to see the code running your table-demo. Thanks!

[sebastien-savalle]

@ben-henoida Do you have a working example using the scroll adapter ?
@shlomiassaf : Can you share the code of your table demo ?

[iSerganov]

Hello gentlemen. Could you please let me know whether there are plans to make virtual scroll with server side pagination (dataSource) in nearest future? If no, could you please advise on possible workarounds? Thank you in advance.

[shlomiassaf]

Guys, extracting the code is a pain! sorry! I can't commit to a timeframe here, it's just a hell lot of work.

@mmalerba @andrewseguin

It seems that sticky positioning with virtual scroll is a HUGE pain!
I'v managed to position it correctly but on fast scrolling (whee, touchmove) it will go out of bounds and return once the viewport hit's a new update... making the sticky rows flicker...

It's not that simple to tame...

[IlCallo]

@ben-henoida
Any advancements on the PR?

[IlCallo]

@nahgrin
I played a bit with your last stackblitz and probably will do more in some weeks.
There seem to be some problems with sticky headers after a certain scroll threshold and they start moving to the bottom randomly.

[vytautas-pranskunas-]

Any progress on this?

[temaisgod]

Hi!

first thanks for this POST. Im create my Virtual Scroll Table with yours code and it's works!

But i have a problem:

My table has a filter by id input text. Im using MatTableDataSource Object in my 'row' Observable . The filter change the value of MatTableDataSource but only refresh data in the table when I move scroll bar.

my question is:

¿How could I bind DATA REFRESH to filter KEY UP ?

Thanks!

[temaisgod]

Hi!

first thanks for this POST. Im create my Virtual Scroll Table with yours code and it's works!

But i have a problem:

My table has a filter by id input text. Im using MatTableDataSource Object in my 'row' Observable . The filter change the value of MatTableDataSource but only refresh data in the table when I move scroll bar.

my question is:

¿How could I bind DATA REFRESH to filter KEY UP ?

Thanks!

Im resolve this problem using BehaviorSubject Object instead of Obervable in my variable data.

When apply filter, I call to .next() method with the data for this object emmit changes to table.

setData(){
     this.misDatos = new BehaviorSubject<Array<any>>(this.dataSource.filteredData);
     this.rows = this.misDatos.asObservable();
    this.misDatosObservable = combineLatest([this.rows, this.scrollStrategy.scrolledIndexChange]).pipe(
      map((value: any) => {

        // Determina el rango a extraer del array
        const start = Math.max(0, value[1] - TableComponentVS.BUFFER_SIZE);
        const end = Math.min(value[0].length, value[1] + this.range);

        // Extrae el rango
        return value[0].slice(start, end);
      })
    );
}

applyFilter(value: string) {
    this.dataSource.filter = value;
    this.misDatos.next(this.dataSource.filteredData);
    this.scrollStrategy.onContentScrolled();
}

[CRACKISH]

When is feature will be in box?

[Nevaan]

Hi,
Are there any plans of integration with mat-autocomplete, or anyone found any working solution for issue #13958 ?

[rabelloo]

I just spent a couple days hacking at this, after reading through all the amazingly helpful code and information here, and I just feel like I've reached an impasse.

I was able to successfully integrate mat-table and cdk-virtual-scroll-viewport, building off some of my old code and some of @nahgrin's.
However, as @shlomiassaf pointed out, sticky headers are definitely an issue.

Here's my stackblitz for anyone interested:
https://stackblitz.com/edit/mat-table-virtual-scroll

My implementation supports column filters, sorting, pagination and virtual-scroll.

I did rewrite the whole DataSource instead of extending from MatTableDataSource because I felt limited with the filters at one point, and my CoreTableVirtualScrollStrategy, like @nahgrin's, could also extend FixedSizeVirtualScrollStrategy, but as others said there's no point in doing so if they could all potentially change when getting the requested feature.

There's also one dirty instance of component composition by extending a CoreTable class, which I'm sure could be made better by a crazy complex component with TemplateRefs, structural directives and all that fun stuff.

Anyway, here's my problem with sticky headers:
One can definitely calculate the offset position of the viewport and compensate with transform: translateY() on the <th> elements.
However, flickering still happens and I'm not sure it can be avoided at all due to how the browser renders.

You can check my approach in example-table.component.ts, but essentially you subscribe to Viewport changes and do the math (I used a fixed magic number just for the purposes of testing this one out).
It works well until the point where the container transform starts to mismatch the <th>'s, which is when flickering starts.
Mind you, the header stays positioned correctly after scrolling (or it would, if you actually coded the math), but only after some delay.

I've seen other approaches to VirtualScrolling in tables like ag-grid and @shlomiassaf's table demo, but they all forego the <table> elements in favor of easily controllable <div>s which can render the header outside of the viewport - if only we could have tbody > viewport > tr...

Which finally brings me to my impasse: it seems like we can either care for HTML semantics and struggle with presentation, or just go for the easy and smooth implementation instead.

I even tried a different approach without cdk-virtual-scroll-viewport where the height of the <table> element would stay dynamically constant - if that makes sense - by using :after and :before pseudo elements that would resize according to scroll position, which involved about the same amount of math as the previously mentioned approach, but to no avail, flicker all the same - even worse I'd say.

Hopefully someone out there will be able to work this out, either with CSS or JS, maybe I'll even try again in the future.
For now I'm sticking with the slightly less user-friendly approach of paginating. Sticky headers with filters really are a must at my current project.

Anyway, I just felt like sharing my experience and my code in hopes it further helps people that come across this issue, like how other comments before mine helped me.

tl;dr: Could make mat-table virtually scrollable, but sticky headers flicker :/

[desdmit]

Hi,
I was able to implement sticky header with virtualization and infinite-scroll:
https://stackblitz.com/edit/nahgrin-virtual-scroll-table-3tx3mt

Any thoughts?

Thanks to @nahgrin

[rabelloo]

@desdmit I don't mean to rain on your parade, but this helps illustrate that we'd have to forego HTML semantics if we wanted an easy way of implementing sticky headers.
Not using the <table> elements also means column width is determined by flex and therefore they are all equally sized instead of adjusting to content, in which case I'd rather use a CSS grid instead.

Also, there's no need to use [cdkVirtualFor], [cdkTable] should be enough as a rendering engine.

I have forked my own code to incorporate your changes, as a proof of concept.
https://stackblitz.com/edit/mat-table-virtual-scroll-div

Still hoping that we get official support on this one.

[ThisIsIvan]

@rabelloo Thanks a lot for the implementation! It works great.

There are some TypeScript errors, which are easily fixable.
Apart from that there is a mistake in the select toggling of items. When the table is initialised with a large data set not all are selected after hitting the select all checkbox. That's fixable by replacing this.data with this.allData in data-source.ts::toggleAll() and data-source.ts::selectedAll().

The initialisation of the table with a large data set still takes a while unfortunately.

[rabelloo]

@ThisIsIvan That was actually a deliberate decision to select only visible items. It's rather controversial which behavior is expected by users, but we found that to be more intuitive, at least when paginating.

Still this stackblitz version has some flaws, like when toggling all on a page, then trying to toggle all on a diferent page, but I wasn't trying to reach for completion.

Anyway, glad you could easily alter it to your liking.

[shlomiassaf]

Hi guys,

Yes, sticky headers are an issue and probably will be due to the way they are implemented.

I did a lot of things to get around it and I was able to reduce it to minimal using some tricks and I thinks its ok now...

I also think most users will want the headers outside of the scrolling area so I support both mods as shown here: https://shlomiassaf.github.io/table-demo/table-demo#/features/sticky

As for <table> vs <div> I chose <div> because table might limit in complex scenarios / features.

By default all cells do get the same % but that's because mat-table doesn't have any size mechanism to control that.

I added that, and its possible to specify minWidth, maxWidth in absolute pixels and width in % or px and if not set the width will behave like in mat-table. This actually works great!

For example, with 10 columns, if minWidth: 150 is set for each we get 1500px minimum. If the table's width is 1000 the user will get 500 px HZ scrolling...

The user can choose his strategy here which I find great when I use the table...

In the demo site, there are actual demos (dedicated) with an action-bar that allow changing the width strategy

The menus just call table API's to make it work..

So <div> works OK I guess :)

[lujian98]

@rabelloo Thank you for the great implementation.
One question, is possible to use MatTableDataSource instead of use your custom CoreTableDataSource?

[rabelloo]

@lujian98 Glad you liked it!
Well, technically yes, but:

• For virtual scrolling you'd have to provide MatTableDataSource a MatPaginator instance and keep feeding it page index and size values (size can be constant);
• For this kind of custom filtering you'd have to provide your own filterPredicate and trigger emission by changing the filter property.
  I used to simply invert the value e.g. filter = `${filter === 'false'}`

Overall not too complex, just a little bit hacky which is why I chose to write my own, especially since I'm going to be working on more advanced filters in the future like Date, "contains in list of possible values", boolean and enumerable icons, etc.

You'd still want some kind of abstraction, probably, which I have in the form of CoreTable. That's the part of my code that I like the least, with how you extend from it and some things are just magic, but it definitely makes it pretty easy to reuse.
Feel free to write a component, directive, or whatever you think best to replace it.

[lujian98]

Thank you @rabelloo
The MatTableDataSource cannot be extends since the connect() { return this._renderData; } is different from the CoreTableDataSource connect() { return this.visibleData; }, which give the error message. Not sure why cannot override the parent function.

I may have to create my own TableDataSource. It's bad the Angular Material datatable not support virtual-scroll.

[rabelloo]

@lujian98 Oh I thought you meant to use a MatTableDataSource instance instead of a CoreTableDataSource instance, which you can definitely do.

If you mean to alter CoreTableDataSource and extend MatTableDataSource, you can remove most of the code - like filtering and pagination, rename conflicting properties that are left over and then implement the filter and MatPaginator overhaul that I mentioned.

[lujian98]

Thank you, @rabelloo Based on your code, I made a simplify code for virtual scroll only with Angular Material data table. So main issue here will be custom tableDataSource. Like what you did, build-in sort, filter will need to be added.
For the reference: https://github.com/lujian98/Angular-Material-Virtual-Scroll

[ThisIsIvan]

@rabelloo Do you see a way to improve the initialisation speed of the table given a large initial data set?
I have a list with multiple thousand entries. Your virtual scroll helps immensely in improving the scrolling speed, but initialising the table still takes a while.

[rabelloo]

@ThisIsIvan I think it comes down to memory allocation, so other than incrementally loading the data instead of all of it at once, I don't know if much can be done.

It shouldn't be too hard to subscribe to the viewport's observables like renderedRangeStream and fetch the next batch when getting close to the list's bottom.

If you want to get quick and dirty though, you can just setup an interval and load your batches that way, e.g. https://stackblitz.com/edit/mat-table-virtual-scroll-dirty-load?file=src/app/app.component.ts

I would recommend fetching on request tho, i.e. when the user scrolls to the bottom.

[ThisIsIvan]

Thanks for the response @rabelloo. I actually prefer the first method and already had something comparably implemented. I prefer it over fetching on request, because I feel it's a bad user experience to not see the full table length via the scroll bar, when you have a fixed amount of items.

That is of course not the case if you have an infinite long list. Thank you again!

[piernik]

@desdmit I'm trying to apply Your solution with CDK table (no material) but I cannot :/

https://stackblitz.com/edit/nahgrin-virtual-scroll-table-3tx3mt?file=src%2Fapp%2Ftable%2Ftable.component.html

You have there:

<ng-template let-row matRowDef cdkVirtualFor [matRowDefColumns]="displayedColumns" [cdkVirtualForOf]="rows">
      <mat-row></mat-row>
    </ng-template>

But how to translate it to plain cdk ?

<ng-template let-row cdkRowDef cdkVirtualFor [cdkRowDefColumns]="displayedColumns" [cdkVirtualForOf]="rows">
    <tr></tr>
</ng-template>

Is not working.

[cedvdb]

I vote for table semantics instead of divs. Display can always be changed to flex instead of table or row, so you can always get what you want visually and at least the html will be more readable.

[doronsever]

Can someone tell me how to calculate the _renderedContentOffset?
I'm trying to implement our own table version with cdk-virtual-scroll but I cannot set our header component to be sticky.
Then I found out that the CDK does a calculation of the content offset and if I set the header css top value to it's negative, it will fix it.

I looked at all the great example that was written here and tried to use the getOffsetToRenderedContentStart() function but it doesn't calculate the same value as _renderedContentOffset.
If this value could have been at least not private one it would be good as well...

Also, @rabelloo could you please explain how you have access to the viewPort property? I don't see any declaration of it

[RmilaWeb]

I Have Added Virtual scroll to my drag and drop and it causing problem see issue here #15457

[rabelloo]

@doronsever I'll assume you're asking about ExampleTableComponent since it's the only place viewport is not explicitly declared, in which case it is being inherited by the class it extends, CoreTable.

CoreTable is just a dirty way to avoid repeating code among all TableComponents, but alas component inheritance is not great as you can see. Essentially viewport comes from the Table's view, i.e. @ViewChild(CdkVirtualScrollViewport).

In regards to your _renderedContentOffset question, you could always access the private property if you want, even though there's no guarantee it will be available in future versions and I would not recommend it, e.g. (viewport as any)._renderedContentOffset or viewport['_renderedContentOffset'].

Regardless, I don't think using that private property would work either, gave it a try and couldn't make it work. If you do, please let me know how.

[piernik]

Can someone help with creating virtual scroll but for cdk-table ? I'd be very pleased :)

BTW. Do You think we can expect official support for that feature in version 8 ?

[IlCallo]

Based on previous statements from Angular team, they are fully focused on Ivy release and many Angular Material developers have been temporarly moved into the Angular Ivy team.
They also said that there will probably be really few new features before Ivy becomes the default renderer (Angular 9 - fall 2019).

For what I understood: expect all major feature requests to be delayed until beginning of 2020 🤷‍♂️
And then there are more high priority issues than this, so maybe even later.

[yantrab]

@piernik check my npm package or check the source code here.

[piernik]

@yantrab I'm trying to create virtual scroll with cdk table. Here is my code: https://stackblitz.com/edit/cdk-virtual-table

Myu problems:

1. I have 200 rows but it displays only 94 :/ Don't have clue why. Somehow this.viewport.setTotalContentSize is too small?

1. When I scroll back to top few first rows are not rendered

2. I don't know why You are using this line: const prevExtraData = start > (PAGESIZE / 2) ? (PAGESIZE / 2) : start;

[yantrab]

@piernik

1. just remove [itemSize]="50" from cdk-virtual-scroll-viewpor.
2. same as 1.
3. to show 25 items before the current , to prevent blank section in fast scrolling. if start row smaller than 25, than render all rows from start.

[omryoz]

@shlomiassaf how can I use your code?

[shlomiassaf]

HI guys!

I've release NGrid as a library as well as open sourced the code.

https://github.com/shlomiassaf/ngrid

There is a documentation site and a simple starter to kick things quickly.

If anyone would like to contribute it will be much appreciated.

Let's face it, it's up to the community for this to work!

Documentation is the KEY to push this.

Thanks all

[piernik]

@shlomiassaf I have to admit that it's great!
Making it bootstrap ready won't be a problem?

[shlomiassaf]

@piernik, what do you mean by bootstrap ready?

[piernik]

Theme: https://getbootstrap.com/

[shlomiassaf]

ohh, that bootstrap :)

Yes, of course, you can apply bootstrap style on it.

The basic grid does not have any material components in it (only CDK).

The idea is to have packages that add cell types like boolean, text, select, date, etc...
For each cell the package will include a view template and an edit template.

You can provide data to the templates (when defining columns) so it can be used for things like validation (min, max, etc...)

Of course it can also provide other things, like special headers (e.g. sorting in the header).

You might have a package adding a button at each header that exposes a menu to do things on the column (pin, hide, sort, filter by etc...), it's just a component.

Material components are added through @pebula/ngrid-material so adding this package gives you material like cells, the sorting via MatSort and more...

It's not complete yet, there is work to do, adding "edit" cells, context menu and header menus...

Lots of work on this grid, it has potential but I won't be able to push it without community help.

So to sum it up, just like @pebula/ngrid-material we can build @pebula/ngrid-bootstrap...

Thank you for the feedback 👍

[erksmaas]

@shlomiassaf, I am thoroughly impressed with what you've done! It's astounding what you've put together and how much is there and how well it all works in the first release!

[mrandreev]

@shlomiassaf great job! Thanks for sharing it with community.

But I would say its really huge library with a lot of customization, so its hard to figure out what is going on there and how to integrate virtual-scroll with material table. It will be great to have general solution how to achieve it with light material table.

Anyway at least we have alternative and source of code to play, so thanks :)

[shlomiassaf]

@mrandreev Thanks!

As I previously mentioned, it's really integrated into the grid and it requires some work to get it to work.

Because the material team did not fully integrate virtual scroll into the table (different components) there are a lot of features that does not work together.

For example, sticky rows & virtual scroll... You can read some of my previous comments in this thread with detailed info on things that can go wrong and some hints on how to get it done.

[jelbourn]

(I'm going through the highest voted issues today an commenting on their status)

We definitely plan on doing this, but the way most of the components are implemented today makes this a bit challenging. Components that need to know about their items (select, autocomplete, etc.) use @ContentChildren today. With virtual scrolling, though, that won't work. To make virtual-scrolling function, we'd have to do a larger rework on the components.

The data-table, though, uses a DataSource for its items, which makes this more straightforward. There still needs to be an abstraction layer to prevent people from getting the payload cost for virtual-scrolling (which is not tiny) for people using the table without that feature, but it will likely be the first thing that supports it. It's possible that we make some progress on this later in 2019 considering it's one of the higher priority feature requests on the backlog.

[literalpie]

I've also been looking into virtual scrolling in the table. Some of the POCs posted here have been really helpful, but sticky headers in the table were still causing problems. I found a solution, and thought I'd post it here so people can possibly use it.

The main idea is to use a "placeholder" row _instead of_ the cdk virtual viewport's default behavior of using a transform. This allows the table to largely work the same as usual - including the sticky header.

here is the stackblitz

[AshMcConnell]

@piernik

1. just remove [itemSize]="50" from cdk-virtual-scroll-viewpor.
2. same as 1.
3. to show 25 items before the current , to prevent blank section in fast scrolling. if start row smaller than 25, than render all rows from start.

I'm having similar problems, I tried removing itemSize, but I get: - Error: cdk-virtual-scroll-viewport requires the "itemSize" property to be set.. Am I missing something, or has the virtual viewport changed?

[AshMcConnell]

To answer my own question, you need to have the @angular/cdk-experimental installed and use the autosize directive on the cdk-virtual-scroll-viewport tag: -

e.g.
<cdk-virtual-scroll-viewport autosize fxFlex="0 0 100">

[IlCallo]

(This comment was an answer to a now deleted comment asking for code)

Please read all the thread 🤨

HI guys!

I've release NGrid as a library as well as open sourced the code.

https://github.com/shlomiassaf/ngrid

There is a documentation site and a simple starter to kick things quickly.

If anyone would like to contribute it will be much appreciated.

Let's face it, it's up to the community for this to work!

Documentation is the KEY to push this.

Thanks all

[nickwinger]

actually i don't want to use NGrid, i just tried it.
I want the workaround for pure material/cdk
now trying out this version:
https://stackblitz.com/edit/nahgrin-virtual-scroll-table-hampgc
(i read the whole thread...)

[simeyla]

How do all these examples work when using async pipe data on the page.

For example I may have an observable defined in the model where I want to show how long ago an order was placed. Maybe this updates every minute like this.

timeSinceOrder$: Observable<string> = Timers.runEveryMinute$.pipe(switchMap(() => ... ), tap(() => console.log('timeSinceOrder$ has run');

The sample stackblitz above that I played with did not unsubscribe from the observable when it was scrolled out of view. So after scrolling all the way down and up I was seeing hundreds of log entries instead of just the number of rows I have in view.

For a final feature complete scrollable table I would think this would be a necessity - but there's no other mentions of async in this whole thread. How can I get that to work?

[alexmacavei]

On my previous project and ended up using http://swimlane.github.io/ngx-datatable/#virtual-paging. I know it's kind of annoying to have to rely on 3rd party libs for this, but it seemed to work pretty well until this ends up in Angular Material. At least for data tables :)

[SaiSirishaS]

@nahgrin I have tried your code and it works fine for the basic table structure. I have requirement, when i edit a table row it should open up a Matdialog. The matdialog doesn't work properly when scrolling after a certain number of rows in the table.

[aalbericio]

mat-select needs support for this asap :(

[ahierro]

I need this feature for mat-table, please!!

[rahulyadav]

@shlomiassaf : Great work on the ng-grid, works super cool...
I was trying to implement the blocking scroll mode on cdk-virtual-scroll-viewport to prevent whitespace issues when scrolled too quickly.
In my solution I've used the fixed scroll strategy and added wheel event listener on the cdk-virtual-scroll-viewport to prevent the default scroll behaviour.
However, with this approach the scroll appear to be very sluggish and jumpy whereas your scroll implementation in ng-grid works seamlessly.

Can you give me some tips around how to get the block scrolling mode work smoothly?

[ahierro]

I ended up using primeng table component

[Splaktar]

I think that the UX of a select with virtual scroll is very questionable. If the user needs to pick from so many different options, then an autocomplete with virtual scroll would provide far better UX.

[doronsever]

There is a long waiting request to add a search inside the select list.
When the material guys will implement it, the virtual scroll inside a select list is a very reasonable request

[Splaktar]

@doronsever That's not currently a feature of the Material Design guidelines for menus, which is where the select menu (popup panel) behavior is defined.

The closest that I could find was the Editable Exposed Dropdown Menu example. However, this is designed to allow entering a value that isn't in the options set. It doesn't describe any filtering behavior (I think that's still TBD).

[camillewall]

I have tried both @rabelloo and @lujian98 as examples to implement virtual scrolling using the material CDK. But, they both have a problem I cannot figure out and it is around using a checkbox to select rows. I added a select checkbox to @lujian98 example. When the list first gets rendered everything is great I can click on the checkbox and the check appears immediately. Once scrolling starts and you go past the rows that are initially rendered then checking no longer happens when clicked, but when focus is lost. Any help would be appreciated.

[elimb]

@camillewall I had a similar issue using literalpie's initial solution. I ended up running setTimeout(()=>this.appRef.tick()) on row clicks which solved my checkbox issue.
I would think that there must be a better solution though.

[mike-lipin]

@camillewall I had a similar issue using literalpie's initial solution. I ended up running setTimeout(()=>this.appRef.tick()) on row clicks which solved my checkbox issue.
I would think that there must be a better solution though.

@camillewall there is another workaround. you may inject ChangeDetectorRef into the directive and force change detection on each click like this

constructor(private ref: ChangeDetectorRef) {}

@HostListener('click')
queueChangeDetection() {
    this.ref.markForCheck();
}

[camillewall]

Thank you @elimb and @mike-lipin for your help. My checkboxes are showing they are clicked without any delay.

[divyanshumehta]

Damm so many people worked on their own versions of the virtual scrolling version of mat-table, I thought angular team would have integrated the virtual scroll by now. Please lets get this feature soon

[leea10]

Hey everyone,

For those in need of a virtually scrolling tree, I put together a stackblitz demonstrating how I managed to work around the fact that cdk-tree does not yet support virtual scrolling out of the box

https://stackblitz.com/edit/angular-b5nkkd?file=src/app/app.component.html
(it shows a normal flat tree and a virtually scrolling one side by side)
Selling point is that if you already have a cdk flat tree working using the proper data source and tree control, this requires very little modification to get a nicely working virtual scroll without any brittle hacks.

And here's a stack overflow response I wrote with more detail for someone that was having this same exact problem: https://stackoverflow.com/questions/52606511/angular-material-cdk-tree-component-with-virtual-scroll/58846134#58846134

[kekel87]

Hi, new for this issue?

I currently have a mat-table with custom virtual scroll, I try all the proposed solutions here, but none works correctly with a mat-tooltip on an element of the table (after scroll).

[maranmaran]

I have a problem with this in a mat-menu. I have time though, does anyone have a workaround for mat-menu in the meantime perhaps?

[ARUS911]

almost 2 years ... (

[blogcraft]

Nothing yet?

[nate-knight]

Does anybody have a recommendation on how to get drag/drop to work with a virtually-scrolled CDK table? I have implemented a virtual scroll solution based off of the wonderful examples provided by @shlomiassaf @nahgrin and @rabelloo . The issue I face is that when dragging an item from the current "view" of the table, and then autoscrolling the table to a point where the table displays the next subset of data, the dragged item is lost and the table just continues to scroll. This seems to happen because these solutions directly replace the table's dataSource with a new subset of the total data. And one a new datasource on the table is set, the dragged item is lost because it was attached to the previous dataSource. With the CDK's *ngVirtualFor directive on lists of elements (not tables), the datasource is _not_ replaced, so the user can drag an item from one subset of data to the next subset. I'm not sure of a workaround for the cdk table use-case. Any insight and help is welcome!

[attilacsanyi]

Dear @nate-knight can you point to the wonderful solution of mat table with virtual scroll please based on @shlomiassaf @nahgrin and @rabelloo.
I have this example so far. Appreciate your help.

[JKremsner]

Thanks @attilacsanyi for the example but sorting still doesn't work as expected.

[TomaszKasowicz]

Hi All.

I recently started investigating on How to implement Virtual Scroll for Table with pages being dynamically loaded on scroll.
After reading through this whole thread and looking into examples I decided to use as less code as possible.

So based on example by @rabelloo and instructions on how to use data source with paging in scrolling (https://material.angular.io/cdk/scrolling/overview)
I created Data Source instance which

1. Uses MatTableDataSource to have all the data sorted (or filtered if needed)
2. Acts as CdkVirtualForOf for CdkVirtualScrollViewport.attach() so view port can calculate data size
3. Subscribes to range changes to trigger additional data fetches and data updates

The example code is available here:
https://stackblitz.com/edit/angular-qrbbwr

(Note: this one is without sticky header... I'll try to work on it later)

[Sharkww]

Dear @TomaszKasowicz, for the sticky header part, you should change the offset to negative like this

this.offset = this.viewPort.scrolledIndexChange.pipe( map(() => { return this.viewPort.getOffsetToRenderedContentStart() * -1 }), distinctUntilChanged(), tap(offset => console.log('SIC', offset)) );
However, there is a little bit lag for the sticky header

[julienboulay]

@TomaszKasowicz and @Sharkww , thank you for your help.
I've made a new version of the example code that fix the lag with the sticky header
https://stackblitz.com/edit/angular-table-virtual-scroll-sticky-headers

[ilapavuluri]

Hi Jelbourn,

Thanks for the response, I have more than 1000 records, without virtual scroll it is very slow, is there any alternate solution?

Appreciate your help.

Thanks
Ashok.

[GuillaumeUnice]

Hi guys,

I just want to know what the technical limitation to not enable a vitual scrolling in two-dimension on both axe (horizontal & vertical) at the same time?
And is there a workaround to use it with the CDK?

Thanks in advance

[ForTimeBeing]

Why hasn't angular integrated scroll within their mat-tables yet? Just seems strange to me they have pagation but no scroll. I am literally doing hack methods just to get angular table usable

edit: I am using mat-table and I have like 10-15 columns and 300+ rows. I added custom resizing, problem is scrolling or resizing was VERY laggy. To make up for this I swapped it to use viewport, only rendering the elements as they appear on screen. Using standard viewport broke the mat-table sorting

[tamtakoe]

I tryed to integrate virtual-scroll with mat-table in several ways and found that the first thing which we need is dynamic templates: https://github.com/angular/angular/issues/37960. It will allow us easy integrate any components to our custom component. For now we can merge controllers only (extend classes etc.), but have no adequate way to merge templates

[NitinMagdum]

any working examples with virtual scroll and mat-select or mat-autocomplete

[NitinMagdum]

mat-autocomplete with virtual scroll
https://stackblitz.com/edit/angular-htsgbw?file=src%2Fapp%2Fapp.component.html --- sometimes scrolling area showing blank space

[NitinMagdum]

mat-autocomplete with mat-select
https://stackblitz.com/edit/angular-gs4scp?file=src%2Fapp%2Fcdk-virtual-scroll-fixed-buffer-example.html
here select one item from dropdown then again click and scroll dont select any item click outside the box, the existing selected item is not visible/not bound to select box

[geraldpereira]

Hi guys,

I anyone is interested I wrote a blog post on how to use the virtual scroll with a flat tree: https://octoperf.com/blog/2020/08/13/angular10-material-tree-virtual-scroll/

The source code is visible here: https://github.com/OctoPerf/kraken/tree/master/frontend/projects/storage/src/lib/storage-tree

[scarabedore]

Hi, based on work from @julienboulay , I have written a table implementation with virtual scroll and expandable rows: https://stackblitz.com/edit/angular-demo-ua53y8?file=src/app/my-table/my-table.component.ts

[piernik]

@scarabedore will sticky header work ?

[scarabedore]

@scarabedore will sticky header work ?

Yes: in the template, replace the header row with

<tr mat-header-row *matHeaderRowDef="columnsToDisplay; sticky: true"></tr>

and add

.mat-header-row th {
  background-color: white;
}

to the styles.

[tamtakoe]

@scarabedore It works bad with sticky unfortunately. As with my attempts it has bounce. You can see it if fast scroll from bottom to top. Be sure it will have more terrible effect in real project. It looks like unfixable. The only way to implement sticky header is not put the header row into the virtual-scroll container and the only way to do it is to rewrite https://github.com/angular/components/blob/master/src/cdk/table/table.ts (because of Angular doesn't provide dynamic templates and has no this priority). It is difficult, but we can't adequate customize mat table simple way

[shlomiassaf]

Sticky with virtual scroll does not work because you must compensate to sticky position relative to the offset.

This is mostly effected when doing Vertical virtual scrolling with a top bound sticky row. The sticky row has top: 0 but as you scroll it goes away and you need to add to it so it will be visible within the viewport, the math is not that tricky.

However, in fast scrolls you get flickers sometimes.

Basically, there should be 3 modes in a grid

1) Fixed headers OUTSIDE of the scrolling area
2) Rows INSIDE the scrolling area behaving like data rows
3) Fixed rows INSIDE the scrolling area, with sticky mode to keep them "on top" data rows.

You can see it in action here:
https://shlomiassaf.github.io/ngrid/demos/complex-demo-1

[ievgennaida]

I have followed examples from geraldpereira of how to replace mat thee with cdk-virtual-scroll-viewport but unfortunately, something is wrong with preserving scroll state when items expanded toggled.

For the 1000 nodes it's working fine (performance) but when scrolled to the end state of the scroll is not preserved on expanding toggle.
There are some random scroll changes and tree is blinking.
This is not an issue in a middle or at the beginning of the list.

    <cdk-virtual-scroll-viewport class="mat-tree" itemSize="30" minBufferPx="500"
        maxBufferPx="800">
        <app-outline-node *cdkVirtualFor="let node of dataSource" [node]="node" class="mat-tree-node">
        </app-outline-node>
    </cdk-virtual-scroll-viewport>

[shlomiassaf]

@ievgennaida Your buffer is quite big which explain why it works at the start...

You need to manage the state of each tree (opened/closed) on the node and when the object pops into view you need to verify it toggles.

Also, both strategies provided by the CDK are not suitable for a list with expanding items.

The Fixed strategy is, well, fixed... so it wont work.

The Auto strategy is using math based around the average with small fixes to the average as more information streams in. This is not done on the row level but on the container level (performance), i.e. height for all rows / render row count.
Now, when you create a height that is based on the average it will work fine as long as the deviations of each item/row is somewhere within the standard deviation (i.e. not far from the average). If you have big bump, rows that have heights much larger then the average you get into trouble because the Auto logic can't fix the gaps to reflect in the average simply because this is how average works, it takes time for big changes to reflect in the average especially at the end of the list, where the average has a high score and each incoming item has less weight which means it effects the average less. I assume this is the case with a tree....

[ievgennaida]

@shlomiassaf thanks for the explanation, but the case is a bit different.
I am using a Fixed strategy. Node is 30px height. I am using the flat tree nodes approach, so it means that the expanded state is just more nodes to render, the height of the nodes remains const.

Also, the view is detached so I have to detect changes on the scroll.
All this works but makes some visual glitches, especially at the end of the list.

P.s. I would assume to take the height of the scroll viewer as min by default in other cases it makes no sense.

P.s.s I have seen two different virtualized implementations and one is used in angular is flickers when you are scrolling.
There is implementation without flickering for React:
https://codesandbox.io/s/github/tannerlinsley/react-virtual/tree/master/examples/fixed

[shlomiassaf]

@ievgennaida what do you mean by flickering?

[ievgennaida]

@shlomiassaf When you are scrolling fast, you can see that there is no content for a while:
Ex, your grid: https://shlomiassaf.github.io/ngrid/

There is another react implementation where they don't have this problem.

In any case, this is a minor problem comparing to the fact that the scroll position should be preserved perfectly when number of nodes are changed.

[shlomiassaf]

@ievgennaida This will happen in the react implementation you mentioned.

It's virtually impossible to make it go away, especially if you want to maintain proper frame rate.

This happens because the grid or any other component can't keep up with the pace of scrolling which is because one or more of the following:

• The scroll delta is high making the buffer irrelevant, skipping to rows which are not rendered
• The grid did not finish updating the rows (i.e. change detection in angular) but the scroll has moved on so when the grid finishes (and he browser updates the layout) we are in a new offset (virtual scroll) which shows "nothing" because the virtual scroll calculated the new position which is still not rendered.

Again, the browser is aiming to render at 60fps so scrolling is not blocked to the user which get's us to this issue.

You can limit the scrolling but it might create a bad experience.
I allow it in my grid (only for Desktop, not mobile).
You can check it here, choose "blocking" at the top.

Note that implementation has an impact here, making sure not to reflow, recalculate layout, etc..
but the user also has a great impact as well.

Creating a lot of "cells" with heavy calculations on change detection will make it "flicker".
Also, having big viewport (more rows to render) has a big effect

In your example, the react component is very simple, 5 cells with no logic and a small view port.
Put the same cell setup in my grid and you will rarely see flickering.

I still managed to make it flicker :)

https://user-images.githubusercontent.com/5377501/103115599-ec9ba380-466b-11eb-9859-fd305f0fa8e5.mov

In my grid, when I change the spacing (more rows in the viewport) I can clearly see the drop in the frame rate.
Same viewport, 25px row height vs 50 px row height, around 20% decrease in frame-rate and making cells do more work will reduce it exponentially!

[shlomiassaf]

Here is how blocking looks like, you can see the actual frame rate on the right

https://user-images.githubusercontent.com/5377501/103115850-cfb3a000-466c-11eb-8460-e0a76cdcbe61.mov

[ievgennaida]

@shlomiassaf ok, thanks for the input. In any case, this is really minor issue and should be just balanced to flicker less on the simple lists. I would guess that it's not convenient to set buffer min-height, should be never less than component height. Also I had problems with the preserved scroll position on the detached tree (flat list with padding) when nodes count are changed. But I will come back to this issue later.

[ylandau]

@scarabedore it doesn't work with data coming from an api?

thanks

[scarabedore]

It worked for me with data from an API. However as @tamtakoe mentioned, the sticky header keeps flickering when scrolling.

[garrettld]

As schlomiassaf has mentioned, there are a lot of moving parts here, and it's difficult to implement something that works under all scenarios.

I've come up with a solution that accounts for any number of sticky header rows and also fixes the flickering issue described in #21576. See stackblitz here: https://stackblitz.com/edit/virtual-scroll-cdk-table-test

My implementation has two directives and a custom virtual scroll viewport. The first directive translates the raw data source into a virtual data source by slicing the underlying array to match the range that the virtual scroll viewport wants to render. It also sets the top property on any sticky header rows to account for the translateY on the table's containing element.

The second directive applies a custom virtual scroll strategy. I started by copying the FixedSizeVirtualScrollStrategy and trying to make it account for additional offset created by the header rows, but I spent way too many hours trying to make it work without success. The logic around how it "expands" the rendered range is difficult to follow, and it would occasionally expand by ~8 items even though I only scrolled a few pixels and then immediately expand in the other direction on the next frame to end up at the right place. This caused jumpiness/jank in the table, and as soon as I swapped it out for a simpler approach it worked flawlessly. If we could define arbitrary begin/end offsets on the virtual scroll viewport to account for the static header and footer row height then theoretically it wouldn't be necessary to have a custom virtual scroll strategy.

The last piece is the custom virtual scroll viewport component that fixes the flickering of sticky header rows. See my comment here for more info on that: https://github.com/angular/components/issues/21576#issuecomment-766975065

Overall, this solution works with any number of sticky and non-sticky header rows, doesn't flicker, and seems performant enough for my use case. I haven't tested sticky columns or sticky footer rows, and I haven't tested other virtual scroll strategies. It also assumes that each item in the data source will render one row, which of course isn't true when multiTemplateDataRows is true.

The biggest problem I have with this approach is that the index-based context values on the row and cell defs aren't correct. For example, if your raw data source has 10000 items and you're rendering indices 105-155 and you want to use the index in your row or cell template, then the template context for the item at index 105 will see an index of 0 (zero), because the table doesn't know that there's 105 items before it. We essentially need the CdkTable itself to handle virtualization in order for those context properties to be "correct", or to at least be told what its starting index is so that it can account for it when building template contexts.

I know that the Angular team has said they'd like to do this, but with no timeline on it, I wonder if it's something that the community could work on? Now that I've spent the time to understand it, I can see a path towards a virtual CdkTable that doesn't suffer from the problems mentioned above. Would the Angular team be open to pull requests for this issue?

[developer-bala]

@IlCallo It would be difficult to share it right now, I need to clean some IP stuff from there.

The virtual scroll is also a part of a table component (on top of CdkTable) so it has some things that will not work as they are part of that table eco-system, things like global configuration, plugin integration etc...

I can confirm that it works, and works quite fast! for both Auto and Fixed size strategies.
I will try to extract it somehow, but I can't commit to a timeframe.

For now, I will try to describe how I did it:

This is my way of implementing it, there might be other ways!

First, the general layout we will use:

<cdk-virtual-scroll-viewport>
    <cdk-table></cdk-table>
</cdk-virtual-scroll-viewport>

The virtual scroll is external to the table.

Now we need to take care of 3 topics:

• Adjusting CdkVirtualScrollViewport to work with the table
• Providing our own version of CdkVirtualFor
• Adjusting strategies to work with the table

Adjusting CdkVirtualScrollViewport to work with the table

The general idea is to create a custom viewport component that inherits from CdkVirtualScrollViewport and apply minor adjustments so the table will work.

Our CdkVirtualScrollViewport will also control which strategy we use, replacing to our own table-targeted strategy when needed.

Providing our own version of CdkVirtualFor

We need to connect CdkVirtualScrollViewport with CdkTable, but CdkVirtualScrollViewport requires a CdkVirtualFor, which is a structural directive...

In general, CdkVirtualFor will render a subset (range) of rows from a DataSource and act upon changes in the datasource or range so it will always render the right subset.

CdkTable already does all of that, and some more...

We will use a simple class that mimics CdkVirtualFor while bridging the two components.

On a side-note, @mmalerba: CdkVirtualScrollViewport.attach(forOf: CdkVirtualForOf<any>): void; is probably narrow, forOf should proably be:

interface CdkVirtualScrollAdapter {
  dataStream: Observable<T[] | ReadonlyArray<T>>
  measureRangeSize(range: ListRange, orientation: 'horizontal' | 'vertical'): number;
}

Our CdkVirtualFor does 2 things:

Make sure sticky rows works

Because the table position is not static (transformed all over in the virtual viewport) the sticky rows
will not stick... we need to compensate for the transformation so their top root following the virtual top root offset. We do that by listening to offset changes (from CdkVirtualScrollViewport) and update the sticky rows with the offset.

Adjust the range to compensate for header/footer rows

Because the table might contain Header or Footer rows, we need to adjust the range accordingly. The virtual viewport is wrapping both header/footer rows and the actual table rows...

Our CdkVirtualFor adapter listens to renderedRangeStream changes from the viewport, and pass it on the the table (via CdkTable.viewChange.next(range)).

The viewport calculates the range using the strategy and it will return how many rows to render. This number is then used to extract rows from the data source. If we have a header row in view we need to reduce the range.

Adjusting strategies to work with the table

FixedSizeVirtualScrollStrategy works fine without changes, it's AutoSizeVirtualScrollStrategy that we need to amend.

The problem is not really in AutoSizeVirtualScrollStrategy but in CdkVirtualScrollViewport and how it emits the initial range, this is a known issue - See this comment by @mmalerba

To fix it I use a custom strategy that wraps AutoSizeVirtualScrollStrategy and ItemSizeAverager .
Basically, I use a TableItemSizeAverager that has access to the how many actual rows are rendered. When ItemSizeAverager.addSample() is called - if no rows are rendered it will use the default row height otherwise will work as is.

This could probably get solved differently, but because CdkVirtualScrollViewport has most of its logic methods private I had to go this way...

If this fix is not applied the average size will get very small values because it will get a large range of "rows" before rows are rendered... so the total height/rows will be small.

Hope it helps!!!

Hi @shlomiassaf

Im having the same problem you have described under Adjust the range to compensate for header/footer rows.
I have normal angular table with 5000 records to be displayed in the same page.

Where it has 10 header rows and first 3 header rows needs to sticky. When i scroll that header rows, i could not able to scroll down the header rows completely. When table data reaches viewport, the CdkVirtualScroll starts to render the next elements so the header rows are not scrolling up.

I need to scroll all the headers rows and make the first 3 rows sticky while the virtual scroll needs to work for table data.

I couldn't find any solution for this. Could you please help me and share the code to adjust the range to compensate for header/footer rows when we have many header rows.