Contacts: Introduce lazy loading to shorten waiting time on open

[eppfel]

I open this issue, because I think this would be a huge improvement.
As discussed here: owncloud/contacts#176

Steps to reproduce

1. Create an adressbook with more than 300 contacts
2. Open the contacts app in your browser
3. Stare at mesmerizing loading animation, until you fall into a state of deep trance

[daniellandau]

The way I'm reading the spec (Example in https://tools.ietf.org/html/rfc6352#section-8.6.5, definition in https://tools.ietf.org/html/rfc6352#section-10.6) it seems that there isn't proper paging support in CardDav. You can ask for less results, but not results starting from the next page (that is, "give me results 1-10, then 11-20, etc").

I think the real thing we want is not lazy loading, but good performance and good UX in general. To that end we need to profile the app to see whether it's for example

1. Taking a long time to parse the vcard data → we can ask for minimal fields to render listing and make a backend request for more when user clicks on a contact
2. Taking a long time to render the list → we can render only the beginning of the list and render more when the user scrolls to the end or is idle.

If that fails to get results, we could (3) make the user click a button to fetch the full list if they want to or alternatively use the search to find contacts. (Talking about search, it took me reading the source code to realize that the search button is for contact search. I had always assumed it was a global search. Maybe it could be expanded and have a place holder text "Search contacts").

[eppfel]

That sounds reasonable to me. :+1:

[silverhook]

As someone with 1k+ contacts in the address book, running on an ARM server, I would very much like to see this happen :smiley:

[daniellandau]

So, results.

Problem

First of all, here's a full profile of initial load with ~1100 contacts, about ~25 of them having profile pictures.

The startup takes in total roughly 30 seconds with my i7 machine and the Chromium dev tools set to throttle transfer bandwidth at 10Mbits per second (the same I have at home). Most of the time (~21 seconds) is spent waiting for the whole addressbook with everyone's pictures downloading.

After that significant time is spent on other various things.

• Parsing the response and creating Contact objects, about 1.3 seconds

• Then some 5 seconds are spent just rendering the stuff. Both all the 1100 contacts to the list and the detail page of the first contact.

Fixes on a high level

1. Transferring images for lots of people is really expensive, don't do that before they are needed.
2. Rendering a big list (at least with Angular 1) is a really heavy operation (that also hangs the page while it's ongoing, preventing e.g. interacting with the search), don't render them all at once.
3. Rendering the detail page is also somewhat heavy, so don't do that if not specifically asked for via URL parameters. The user is probably unlikely to want the first contact anyways.
   (4. I didn't implement this, but it would be useful: keep actual local persistent state in localStorage to avoid doing such a long PROPFIND dance before actually getting around to getting the interesting information)

Details

1. In addressbook-query: address-data can contain a list of the fields that should be returned. I went with [ 'EMAIL', 'UID', 'CATEGORIES', 'FN', 'TEL', 'NICKNAME' ]. This seemed to me to be the minimal set of data to get for every contact for basic functionality like search and categories (groups). Immediately after first contactList controller run, we send an addressbook-multiget to get the full details, including pictures of the first 20 contacts, which is hopefully enough on most screen sizes to cover the initially visible contacts.

Then, always when the user finishes scrolling (debounced with 250ms), we multiget the visible contacts. We also get from the backend the selected contact on selection in order to make sure that it's a full contact even if the user clicked while still scrolling.

1. Use the limitTo filter to initially only render 25 contacts and window.setInterval the limit higher so that the contacts are rendered out little by little, and the browser/page never hangs. This way, the user sees things on the screen sooner, and can also make a search without waiting for the whole list to render.

2. Just remove the lines rendering the first contact.

Timeline after fixes

Initial load takes about 5.5 seconds. Some 2.5 seconds are taken by multiple sequential PROPFIND requests (and parsing their responses) and another ~2 seconds are still taken up by the REPORT with the reduced field set (including parsing thereof). If someone writes local caching for those (and we need to check that the backend supports the appropriate getctag or sync-token which it totally might, just not sure) the startup could be made super fast.

Pull requests

https://github.com/nextcloud/contacts/pull/88
https://github.com/nextcloud/3rdparty/pull/30

Why two pull requests? Sadly it turned out that the CardDav code in the backend was actually ignoring the field set inside address-data. The good news is that somebody already implemented the feature in SabreDav (https://github.com/fruux/sabre-dav/issues/889), but the bad news is that there doesn't seem to be a 3.2 series release incorporating that fix yet. The patch wasn't that large and applied cleanly on the 3rdparty source, so that's the reason for the pull request.

Naturally server needs to have the submodule reference updated as well, but I don't think a pull request adds any value for that task.

Final remarks

Thank you all who work on Nextcloud, I store most of my digital life in my personal instance.

[plutocrat]

A couple of thoughts on this. Is it possible to have the initial page loading only contacts beginning with 'A', and then an option to page through other contacts beginning with B, C, D etc.
Or maybe we don't even need to load any contacts. Maybe just a big search box (search name, search telephone, search city etc? Ideally both approaches!
I guess this depends how the contact data is stored ... if its stored in Vcard format, then ugh, I guess it might be necessary to deconstruct it into normal database fields, and then reconstruct it into vcf when needed.

[daniellandau]

It is, at least in theory. In practice, I already had reservations about whether it might be a good idea before:

• Going by initial letter definitely won't give a uniform distribution, and most probably not even a decent one
• One approach I tried was to ask for just the etags and then multiget contacts. The etag request took almost as long as a full request (I didn't have profile pictures in my test dataset at that point)

...but now I actually looked at the database, and it turns out carddata is stored in a single binary field.

[irgendwie]

Implemented by #88 - thanks again for this PR @daniellandau!
You :rocket:!