Clarity: Provide an infinite scrolling option to the Datagrid

[jaffoneh]

Select one ... (check one with "x")

[ ] bug
[X] feature request

Expected behavior

The datagrid should provide a feature to allow infinite scrolling (in addition to pagination).

Actual behavior

Currently, we only offer pagination.

Environment details

• Angular version: 2.0.X
  2.1.X

• Clarity version:
  0.7.0

• Browser:
  All

[clementcur]

+1
We currently have a requirement on our project to support infinite scrolling in datagrids. Let us know if that feature is part of any roadmap yet.

[mathisscott]

This is not part of our roadmap at this time. We need the UX team to put together the interaction requirements. Still very early days on this.

[slrgriffin]

Any idea when this is going to move forward we are still anxiously waiting on this per @clementcur comment from Dec. 19. Thanks!

[mathisscott]

Per conversation with team, this needs a UXer on it for research before we can make recommendations.

[joshjohnsonux]

While a manual "load more" would likely work for many cases, I wanted to point out that pressure-based scrolling would likely provide a better overall experience.
https://www.npmjs.com/package/angular-ui-scroll

[reddolan]

@joshjohnson72 a forced load more experience would be worth looking into. Can you elaborate on what "pressure-based scrolling" would be? I looked at the link and didn't see anything by that name.

[joshjohnsonux]

Pressure-based just means that as you scroll to the bottom of the list, that action alone is what causes more results to be loaded and appended to the list. If done right (meaning user scrolls slow enough not to hit the very end and server returns results fast enough) it'll appear to the user that the results are already there and are just being scrolled into view. Watch the animated GIF under "how it works" to see.

[Jinnie]

I've worked on such a component in the past, so I can share some thoughts and things you may be careful about, or take decision about from the beginning.
How much data are you actually expecting to present? "Unlimited"?
What happens to the old data that goes offscreen? Shall it be kept, or "recycled"?
There are two things that are important on what I am more used to call "auto-paging". One is loading data in chunks from the server, and the other is rendering the data in chunks in the browser. They are not necessarily the same, as HTTP calls are expensive and you'd better load larger chunks less frequently, while rendering cycles should happen more often.
Also - a typical user will seldom be patiently scrolling down slowly. He'll more likely drag quickly to the end of the scroller, then drag again. And he'll be seeing placeholder rows or "loading..." labels all the time. That's why mail clients like Yahoo dropped this approach and changed it to fixed paging.
Modern browsers can handle tens of thousands, probably hundreds of thousands of lines of tabular data without choking too much, but for good performance, you should consider reclaiming the items that go offscreen too. Unless you are comfortable with the rendering capabilities of browsers today :)
Also, the interaction between the two types of paging - network and rendering - can be tricky, and you should consider a solid overwrap between networked pages.
Of course, if you only consider to load new data on scroll, things are much simpler.

[reddolan]

There also happens to be a fundamental difference in the goals behind datagrids and infinite scrolling. Datagrids are used for scanning and finding something. Large datasets should be filtered down. Infinite scrolling is most successful in casual consumption of content: reading, browsing, exploration. Successful infinite scrolling allows the user to casually find things of interest, they aren't specifically scrolling over and over looking for one specific thing. If they are looking for a specific item, searching and filtering are far more effective methods for finding something. Infinite scrolling by itself can be useful in some situations, but in the case of datagrids I am still having a tough time seeing how it would be more useful than filter and search.

[Jinnie]

Map exploration is a great example of good application of the infinite scrolling. That's where we took reference from, when trying our "auto paging table". Btw, it has been released with one of the latest versions of the VMware Server product, before it was discontinued. Possibly also with some version of the ESX, but I don't really remember if it was released, or we only made it work with it, but did not release it, as development of the flex client was started.
The interesting thing about the "recycling", for which we got inspiration from the maps, was that dom elements that went offscreen were not deleted, but moved to the bottom, and their content replaced. The idea behind this was that JS engines at this time were not known for their good garbage collectors. Some browsers performed better, others were too bad. So, deleting too much elements too often was leading to serious memory leaks.
Reusing the elements, instead of replacing, was reducing this effect.
I guess this is not the case now - so many generations of browsers later. Couldn't resist to share some historical tech details that are probably totally irrelevant today. Sorry for the offtopic. 😃

[mathisscott]

Not irrelevant at all @jinnie. Thanks for sharing that information!

[reddolan]

I know iOS reuses/recycles objects (List View), and I believe Android does the same thing too.

[joshjohnsonux]

@reddolan Per your previous comment about the fundamental difference in goals, I don't see it as a difference between datagrids and infinite scrolling as much as it's a difference between datagrids and scale. Infinite scrolling and pagination are just two mechanisms for dealing with scale, but at their core datagrids don't work well with large datasets, and users have to depend on reasonable filtering capabilities to keep datagrids useful.
Ideally, a datagrid that loads too many objects should prompt the user to reduce the dataset with a filter, but that's just not a well accepted pattern. Pagination acts as a poor man's filter by only showing one page at a time, but many things like selection and sorting break down quickly with the paginated model. Infinite scroll addresses those symptoms of a bad model, but it's more difficult to implement and still isn't as good as a robust filtering model.
So basically, I'm saying that datagrids at scale provide a poor user experience, but if you have to support it, doing so with infinite scroll is symptomatically better than with pagination.

[Jinnie]

@joshjohnson72, in my opinion, there is a very delicate but firm line, separating what is responsibility of the framework and what is responsibility of the user.
I don't think that Clarity should be concerned at all with data loading, auto paging, etc, but it definitely could enable the user to implement it himself, based on his own model and backend, by exposing some events related to the scroller change and position. I don't think it does that now, does it?

[reddolan]

@joshjohnson72 this article describes the pros and cons of infinite scrolling and pagination. https://uxplanet.org/ux-infinite-scrolling-vs-pagination-1030d29376f1

There are reasons why well established "large-data" patterns like Google Search still have pagination after almost two decades, it works. I am not arguing that Infinite Scroll doesn't have value, quite the opposite, that it is useful in certain use cases. Datagrid happens to be one use case where it falls short of being useful in many of the adaptations I have experienced. That said, I am not opposed to attempting to "crack the code" on Infinite Scrolling Datagrids but I have low expectations. We would have to solve for many issues, each potentially increasing the complexity of an already complex component.

One example is:

• scroll datagrid to a place
• navigate to a page
• navigate back to the datagrid.

To do this effectively, we would need to preserve their place. I am not aware of a way for Clarity to do that. The responsibility would lie with product teams to store that session info in some way. If they don't, then Clarity is seen to be the issue. There are probably other cases like this that need to be addressed.

[joshjohnsonux]

@reddolan Interesting write-up. It makes some assumptions that aren't always true, such as the fact that if I bookmark page 4 of a list, I can come back and see the same item, or that systems by their nature can return the full number of items over which the UI can paginate. He also misses the biggest downfalls to pagination, which are how to deal with selection that may span pages and how to deal with sorting in the context of a selection.

With regard to your example of navigating back to a page, while not likely a P0 requirement, having a method of getting back to a scrolled state is important, though there are a variety of ways that could be approached ranging from "an api on the datagrid to scroll either to an item by id or an item # based on a sort order" down to "keep the list alive in the DOM so it can be shown again". One is hard, the other is wasteful. Hopefully there's a place in between that meets our needs.

That said, if there's pushback on full infinite scroll, it might make sense to start by doing a manual "load more" style.

[mathisscott]

@reddolan
Is this task complete?

[reddolan]

Adding some additional feedback to Confluence. This can be closed out though @mathisscott

[bryanerayner]

I'm just starting to use Clarity, but is it possible for the pagination to have a dynamic page size?

It seems to me that if infinite scrolling can't be implemented right away, that might be a good stand-in. Presenting the user with an option to control the page size would allow them to see items at a faster rate, and potentially remove some of the frustration they might have from pagination.

[aaronfrost]

Bumping. Like many other people on this thread, I would like to see infinite/virtual scrolling instead of pagination.

Personally I hate websites that paginate tables. I started using clarity before I realized that you MUST paginate. @youdz can you shed some light on the status of this feature?

[youdz]

That's it. 😛

[youdz]

My previous comment was obviously a joke, but one that hits way too close to home for most people on the Clarity team. It shows exactly how many of us feel about infinite scrolling.

That said, you mention you hate pagination, I respond that I hate infinite scrolling, this is all subjective. That's not how we want things to happen in Clarity, so we actually have a full internal page of research with pros and cons for infinite scrolling, especially in the context of a datagrid. I'm not sure if we'll ever share the whole content knowing that it contains screenshots of VMware products, but I can give you the cliff notes.

Here are some considerations for infinite scrolling outside of the datagrid:
Pros:

• It makes discovery easier, like scrolling through images without knowing specifically what you are looking for.
• It keeps users engaged with content by minimizing interaction costs.
• There is a perception that content loads quicker, even if it doesn't.

Cons:

• Unless you spend time implementing url-bound scrolling (which no one ever does), you will lose your position when navigating away.
• The scrollbar size is misleading to the user, since it doesn't actually represent the remaining content.
• The lack of UI controls makes goal-oriented tasks seem more difficult, less straightforward, or prevents the user from estimating how long the task will take.

To put these in a more concrete context, you can think about Google Search and Google Images: the first is paginated, while the second scrolls infinitely. The first is goal-oriented, the second is more based around browsing and discovery with hard-to-narrow concepts.

Regarding datagrid specifically, the two main selling points are that selection across "pages" is more natural, and that you might gain screen real estate by dropping the footer altogether (but this would prevent the use of many other features tied to the footer, like selection).
But the data grid is a goal-oriented component at its core, whose goal is to allow the user to find something specific in the least amount of time. Infinite scrolling is generally a poor solution for this, filtering the data to a manageable size and sorting are often more efficient. In addition, when the user finally finds the item they're looking for, they might want to navigate to a different page and come back to where they are in the data grid after. Pagination allows them to jump directly to the correct page, rather than have to scroll through everything again.

Finally, and this is probably or of the more subjective points, we believe that it's very "easy" to push bad experiences onto users with a poorly implemented/integrated infinite scrolling. And we all know this happens on a daily basis because of deadlines, not enough developers, project managers pushing for a new feature, ... I personally would be willing to bet big that given 100 projects in the real world, preserving the position of an infinite-scrolling datagrid when the user clicks the "Back" button on their browser would almost always be one of the first features cut to meet a deadline. And that feature, as un-critical as it seems, is crucial to making the infinite scrolling experience go from plain frustrating to enjoyable. My point here is that the risk of delivering poor UX to the end-user by shipping this feature is much higher than with other features we've implemented.

All in all, we're not saying we don't want to do it: we simply believe the use cases are not as many as most people seem to think, and when we implement it we will have very precise recommendations on when to use it and when not to. Because of this, after considering the effort, risk and value of this feature, it never made it to our priorities list so far. And we actually want to improve our pagination to address some of the common issues people raise about it, so maybe this will be a reasonable compromise for now.

[joshjohnsonux]

While I don't want to pour gasoline on the flames of this debate, you point out the considerations "outside of the datagrid", which artificially limits the value infinite scroll, because two of the primary values for infinite scroll over pagination are a direct result of common datagrid features:

Column sorting: If a user is on the 3rd page of results and sees an item that's interesting and wants to see other items with the same value for a give column, the natural behaviour is to click the sort by in that column head. What happens? Is the user auto-magically moved to the 1st page of result and their context is lost? Are they left on the 3rd page of results and their context is preserved but the item they were looking at is lost?

Selection: If a user selects an item on page 1 of the results then pages through to page 4 of the results and selects another item, then performs a "delete" action, what happens? Is only the latter selection acted upon, because that's what's visible? Or is the action performed on an item that isn't currently visible? Or do you cop out and break the users workflow by prompting them to confirm the action affects items that aren't seen?

But your main objection seems to be that navigation to and from the datagrid would suffer under infinite scroll, as it's hard work to remember the scroll location. I stipulate that to be true... but, I would point out that the current system, has a failing too. It presupposes that data doesn't change often, because in reality what's remembered isn't the source object in the list, but rather the page it was on. There's an assumption that when you return to that same page of results your originating item is still going to be there. But what if the dataset is growing regularly and my item that was on page 3 a few minutes ago is now on page 4? As a user, I'm left confused and lost.

I fully agree that we shouldn't make it "easy" to push bad experiences onto users, but we shouldn't pretend that we're standing on a "shining hill" and not doing so already. (I'm looking at 'you' column filter) At the end of the day, there is a tradeoff to be made, and I fully acknowledge that it's easier to build a backend that returns paginated results and just throw them into a paged datagrid, so it makes sense to support that first. But let's not pretend that a paginated table (which was all the rage in 1995) will provide any sort of utopian user experience.

[mathisscott]

Hi @joshjohnson72
Thanks for the feedback. It is appreciated. I don't think anyone is standing on a "shining hill" here. @youdz was just sharing the results of our research on infinite scrolling which I think is well reasoned and fairly objective. Of course, he gives his opinion at the end as well. So there's that.

Frankly, I don't see why "infinite scrolling vs. pagination" warrants such ire in this day. It seems dogmatic to me. The same issues you raise for the datagrid would apply to infinite scrolling. There's no easy win in either case. Infinite scrolling is terrible if you have a good idea of what you're looking for within a huge dataset but don't know enough about it to do a meaningful search. Pagination is likewise heavy-handed with small datasets of only 3-4 pages or when applied to evergreen datasets that don't lend themselves to the concept of pages frozen in time – such as news feeds.

But the idea of pagination seems dated and people tend to not like it. And I would also propose that the limitations of infinite scrolling are often overlooked or downplayed because it seems newer.

As @youdz said, we aren't closing the door at all on infinite scrolling. I think we recognize that we will need to support it one day. But the ROI to build it right now isn't high enough for us to put it at the top of our queue.

[joshjohnsonux]

@mathisscott That's totally fair and even handed, as I know you always are. I just got the sense from @youdz reply that "infinite scroll has issues that people ignore" (which is true) and "pagination works just fine, making infinite scroll unnecessary" (which is where I would disagree). That combined with the comic taken out of context (paginated lists != books) compelled me to reply to something I knew I should stay out of. But it's good to hear that the door is still slightly ajar.

In any case, I'm 'taking my dog out of the fight' in the first place by trying to avoid the use of any and all datagrids wherever possible. An infinitely scrolling grid of cards shall rule them all! ;-)

[gnomeontherun]

This is a lengthy and complex issue. To date, there have been no technical proposals that make this accessible for all users, and that is why this issue has not been implemented. We won't commit to this as a change to Clarity Angular datagrid, but we know some workarounds have been created as well. I believe that our Clarity Core implementation of Datagrid will not prevent this capability.

The feature request here has been captured into our list and we’re going to take it into consideration as we develop Clarity Core capabilities. In an effort to clean up our backlog and focus our attention, I’m going to close this as captured in our feature requests. Please follow our development and releases to see when we release relevant components to make this possible. Future feature requests can be made in our GitHub Discussions.