Openrefine: Grid with inifinite scrolling instead of pagination

[Opennat]

With (or instead of) current pagination

http://nexts.github.io/Clusterize.js/

---

_This is a proposed Google Summer of Code project in 2020. If you are not planning to apply for an internship via GSoC, we kindly ask that you do not work on this task yet, in order to leave the floor to potential interns._

[jackyq2015]

From which aspects do you think the current implementation of Refine need
to be replaced? Or what's the benefit we can gain from the Clusterize.js
http://nexts.github.io/Clusterize.js/ you mentioned?

On Fri, Jul 10, 2015 at 7:32 PM, Opennat [email protected] wrote:

With (or instead of) current pagination

http://nexts.github.io/Clusterize.js/

—
Reply to this email directly or view it on GitHub
https://github.com/OpenRefine/OpenRefine/issues/1027.

[Opennat]

We work, I think as many of OpenRefine users, at large datasets, say, current project is 18.000.000+ rows with 17 columns. Even when you limit data with some filters/facets, there still many rows in result. Before applying any transforms over the results' cells, there is a good practice to look at these results: to add/delete/move/replace some data in some columns or some data depends on data in another column(s) and you need to apply another filter, and so on.

Now, we have ability to jump over pages in strictly predefined way: "X rows(records) before/after".

Yes, in our Refine we have already "tuned" data-table-view.js for

this._pageSize = 200;

and

var sizes = [ 100, 200, 500, 1000, 5000 ];

to get 200 rows in default result and to have ability to get more via filter menu.

But many rows in result table overloading and overheating browsers' DOM (browser lags getting worse as you increase count of results). So I think, that Clusterize or similar library will be very useful to avoid hard impact at DOM.

Plus, in current implementation, every time you click "previous/next" links (you must use mouse/touchpad for this), focus is getting lost from results table, and you need to use click to the table to be able of using PgUp/PgDown again or should use mouse to scrollbar (or use touchpad for this). Each of these movements is loss for faster and effective work.

And I did not said, that current pagination should be replaced, mind it as UI enhancement. Pressing "X" times on PgDown without clicking over pages is far faster and simple, that the way we have now.

So I think, with library, that I mentioned above, we could get almost free implementation of "infinitive scrolling page" of results (in limits of browser, sure), and all of the results will be sit in browser window always visible (in limits of Refine UI), so you dont need to scroll it in way, as it happens now.

P.S. Maybe it's time to change default limit of showable rows to 30-50, as minimum, instead of current 10? I dont think, that users who use Refine, using such small limit at all, may be in testing purposes only. Anyway, default limit of 50 rows would not affect anyone on 10 years or more modern computer.
P.P.S. Pardon about language, I hope you would get the idea.

[magdmartin]

Couple of comments:

• Refine haven't initially designed to support more than few millions rows project (I'll say up to 5M). On the top of the DOM limitation, the memory usage is getting. Some people investigated integration with Spark or orientdb.
• I personally don't navigate with the pagination when I have over 1000 records to see my data and rely mostly on facet.

Happy to see what we can do to have Refine support larger dataset.

[tfmorris]

While suggestions about implementation tactics are appreciated, much more valuable to the development team is a clear description of what you are doing, the problem(s) you are seeing, and how you'd like to see things improved.

Having said that:

• 18M rows / 300M cells is far above the design center for Refine, as Martin said
• scrolling vs pagination, while an interesting discussion, would be a significant UI change and development effort
• as you scale up to that size, relying on manual inspection of all cells isn't really feasible, so what's needed is more powerful tools to understand the impact of your transform

[Opennat]

I know about destination and limits of OpenRefine, and I don't complain of any part of it or pretend to provide "best usage reciept".
It was suggestion to make Refine more useful for anyone, who work on datasets even 10 times smaller, but have usability issues.

[thadguidry]

Support for infinite scrolling in our data grid will be added in next Phase 2 enhancements. See #1347

[akuckartz]

Please remove the ‘wontfix‘ label.

[HP-bit]

@Opennat @wetneb Hi I am interested to do this as a part of GSOC 2020.
Can you help me how to get started ?

[wetneb]

Welcome to the project @aryamanpuri! The best way to get started on this issue is to familiarize yourself with OpenRefine as a user, then try tackling a few good first issues. Once that is done, if you have specific questions about this issue, just ask!

[rajchandak]

Hi @wetneb & @ostephens, I'm looking forward to contributing to this project for GSOC 2020. I know i'm pretty late, but i've got lots of front-end dev experience and hope to get familiar with OpenRefine quickly so that I can have a draft proposal ready ASAP.

[wetneb]

Sure, here are the steps to get up to speed: https://github.com/OpenRefine/OpenRefine/wiki/Your-first-pull-request

[leoshrey]

@wetneb @ostephens Can you please help me locating the code for Pagination, sot that I can try to fix the issue.

[wetneb]

If you are a GSoC applicant, you should rather get a first contribution on a different issue (see the guide above to your first pull request).

If you are not a GSoC applicant, please do not work on this issue, since it will likely be worked on this summer by a GSoC student.

[leoshrey]

hi, @wetneb yes, I am a GSoC applicant and want to work for the issue.

[thadguidry]

Here's a transparent PNG file I made as a possible toggle icon for the Inifinite Scroll mode that could be placed on the extreme right of the viewpanel-header div (we can improve it and to have the enabled effect, we could border it with CSS or just alter the PNG file some more to have both modes side by side as we did with the Show/Hide left panel button in PR https://github.com/OpenRefine/OpenRefine/pull/2616 :