Maps: Display Contacts on Maps

[tacruc]

Planing of showing contacts on the map.

ToDo:

• [x] Caching Table for addresses to Cordinates (Address string), Lat, Lng, timelookedup
• [x] When to look up the results, We should query less then one time per second, so one demand is not possible => should be done in the backend in advanced, to not hit the one request per second limitation
• [x] Extend the navigation to show "suggestions for contacts"
  
  
  
  • [x] Api call for contactSuggestions
  
  
• [x] phrase coordinat form vCard in backend and frontend
• [x] make contact preview nice
• [ ] Delete Geo attribute from vCard
• [ ] Set geo attibute according to vCard version, in vCard v4 it should be an geo url.
• [ ] Check permission when setting Geo Attribute

Possible interesting vCard Attibutes

• ADR
  
  
  
  • Can be multiple, I guess all are interesting
  
  
• GEO
  
  
  
  • Different formates in v2,3 and v4
  
  

Problems

• it's currently possible to place a contact (edit its coordinates) even if it's from an address book shared with me with readonly permissions.

[eneiluj]

Yep, I think vCard GEO attribute should be used in priority if it's there. Then if there are multiple addresses, i guess one could be used arbitrarily.

There's more, considering we have an address, there might be multiple search result. We can reasonably assume it's ok to take the first result.

Photo metadata are generated by the file hook. How could we proceed for Contacts? Is there a hook when a contact is created/updated? Or should we access contact database on each Maps page load? It would be smart to make the address search just once and store the result coordinates in Maps DB :smile:.

A lot to think about.

[tacruc]

Well why would it be bedder to have it in extra table? There are hooks on carddav. But I think I would just read it from the db or an api.

[eneiluj]

So maps DB could act like a sort of cache: first time a contact is loaded from contact DB, data is generated and stored, next time it's just retrieved from Maps DB.

Is there a way to know if a contact was modified since a precise date?

[eneiluj]

Well why would it be bedder to have it in extra table?

To store potential coordinates resulting from geocoder search and avoid calling nominatim each time we load the contacts.

[tacruc]

We cloud maybe use
public OCP\Contacts\IManager::search($pattern, $searchProperties=array(), $options=array())

[eneiluj]

Nice to see the new contact controller.

About the contact base64 picture: no problem, <img src="BASE64_STRING"/> works fine.

About the geo information, do you mean it's supported by vCard v4 and not by v3?
Which version of vCard is supported/provided by Nextcloud API and contact storage?

You're right about not needing to store anything in Maps DB. If there is a geo field already there, we should use it.

About guessing the coordinates, my main concern is to be able to avoid doing the geocoder request multiple times for a contact. I mean, let's say we don't store the guessed coordinates until they are confirmed or edited by the user, it means we have to guess it again next time Maps page is loaded.

If we store them in the contact (server side), we need to remember that they have just been guessed and they are not definitive. If we don't store them in the contact, we could store them in Maps DB.

I like the idea of displaying contacts with guessed position with a different color to make it clear to the user that it might be wrong and that it can/should be edited.

Guessing picture coordinates seems trickier. Let's wait until the "device" part is implemented to think about that.

[tacruc]

About the geo information, do you mean it's supported by vCard v4 and not by v3?
Which version of vCard is supported/provided by Nextcloud API and contact storage?

https://en.wikipedia.org/wiki/VCard#Properties
The encoding is different between v3 and v4. Nextcloud supports both, and it is easy to adapt the filter to support v3, it is just not done jet.

About guessing the coordinates, my main concern is to be able to avoid doing the geocoder request multiple times for a contact. I mean, let's say we don't store the guessed coordinates until they are confirmed or edited by the user, it means we have to guess it again next time Maps page is loaded.

We cloud make a cacheing table in the maps app. This cloud be used for any geoinformation, request.

If we store them in the contact (server side), we need to remember that they have just been guessed and they are not definitive. If we don't store them in the contact, we could store them in Maps DB.

If we do it the right way, (what we should do) it changes the vCard and syncs to all devices. I wound't like it for a guessed not confirmed information. So I would prefer the caching table.

I like the idea of displaying contacts with guessed position with a different color to make it clear to the user that it might be wrong and that it can/should be edited.

what do you think about black and white and maybe 50% transparent?

Guessing picture coordinates seems trickier. Let's wait until the "device" part is implemented to think about that.

The Device is needed for that, but I think the logic is not to difficult I would do it than.

[tacruc]

I have thought again about the showing the address on the map.
The lookup of the coordinates should be done in the back end.
Therefore I would register Hooks on vCard write access events. A PR for this is created with https://github.com/nextcloud/server/pull/14973
The new addresses cloud be saved in a table and with each cronjob we cloud ask for x new address locations.

[eneiluj]

Yep, a vCard hook would be perfect to avoid overloading the server!

A cronjob sounds good but there still is a "problem" just after app install, when using Maps before the cronjob was triggered, there's not going to be any guessed contact coordinates yet.

BTW, do you know how to remove/unset a contact vCard property with NC API? I quickly gave up after failing because setting it to a null value does not work and API doc didn't help.

[tacruc]

The will be a time delay between the contact is added and it is visible on the map. It will be
TimeToNextCronjob + AddressesToLookup/lookupsPerCronjob*IntervalCronjob

But I guess we will not get bedder with that external provider.
The problem is that the PR for the Hooks is not merged, so it will be earliest available in NC17.

BTW, do you know how to remove/unset a contact vCard property with NC API? I quickly gave up after failing because setting it to a null value does not work and API doc didn't help.

no idea

[eneiluj]

The will be a time delay between the contact is added and it is visible on the map.

If there is a contact hook, there won't be any delay :smile:.
If there's no contact hook, the delay is acceptable IMHO.

no idea

So the "contact geo field reset" is sleeping in the code until we sort it out.

[tacruc]

So you would do the lookup directly on the vCard create event? But wouldn't that mean that initializing and uploading 200 contacts to the server will create 200 requests to the provider at once? Shouln't there be some delay mechanism?
I was thinking on adding them to a list table on the hook, and doing the lookup by the cronjob, to catch the initial upload of an hole addressbook.

[tacruc]

In https://github.com/nextcloud/server/pull/14973 is an example how we can register on vCard changes.

[anaqreon]

I spoke with @eneiluj about my idea for displaying contact by groups (motivated by a personal need for this feature). Conceptually this would be similar to the way Favorites groups can be individually displayed on the map; the UI would be the same, but the contact groups would be auto-populated based on the existing contact groups in the user's address books. Julien recommended that I bring up this idea to people like @jancborchardt and @tacruc to get their approval before spending time working on it.

[tacruc]

I like the idea. I remember a discussion about merging Groups, cycles, calDAV-Groups. So I was just wondering how this discussion went and what would be the right container to show in the maps app.

[jancborchardt]

@anaqreon yeah, good point to filter between groups. It could be possible to expand the "Contacts" entry and enable/disable certain groups then. By default, all would be shown, just like with Favorites.

[jancborchardt]

That said, before working on any more specific stuff, we need to make sure the basics work flawlessly. And that is simply displaying all contacts on the map. :)

[anaqreon]

Thanks for the feedback.

That said, before working on any more specific stuff, we need to make sure the basics work flawlessly. And that is simply displaying all contacts on the map. :)

Personally I have not seen any errors yet in the display of contacts on the map.

[eneiluj]

Hey. I removed filtering stuff in contactController.js and implemented contact group display. It's possible to toggle all groups and to zoom on a specific group. Groups are all enabled by default. Disabled groups are saved and restored on page reload. There is a "not grouped" group.

I did it on top of rework-eneiluj branch.

Feel free to try it.

[anaqreon]

I tested it out and it works great! I added two of three contacts to a group and left the third with no group, and it handled everything flawlessly. We'll want to add a zoom to bounds for each group in addition to zooming all contacts. I'll see if I can try that since I suspect it's cut and paste mostly.

[eneiluj]

It's already possible to zoom on one group's bounds.

[anaqreon]

I said it backwards. I meant zoom to all contacts. Here's an idea for doing that: https://github.com/nextcloud/maps/compare/rework-eneiluj...anaqreon:zoom-all-contacts?expand=1

[eneiluj]

Ow ok, I could have autofixed what you said :wink:.

It's actually much shorter and simpler to use the contact layer bounds:

var b = this.contactLayer.getBounds();

This way, it zooms on all enabled groups, i.e. all visible contacts.

If we want to zoom on all contacts (even those who are hidden because they're not in any enabled group), we can just change the condition on line 210 to:

if ((groupName === '0' && this.contactMarkers[i].data.groups.length === 0)
     || this.contactMarkers[i].data.groups.indexOf(groupName) !== -1
     || groupName === null
) {

I would say it makes more sense to only zoom on displayed contacts (enabled groups). What do you think?

[anaqreon]

Thanks for the detailed explanation. I agree with all your points, and this is a good example of the difference familiarity with a code base can make! It makes more sense like you said to have the "all contacts" zoom only zoom to those displayed, because (1) it would be confusing to zoom out to an area where no contact markers are visible, and (2) there is a "toggle all" button in the "Your contacts" control menu which makes it a trivial two-click process to display and zoom to all contacts.

Another thought: It would be convenient if there were a set of "testing" files for Favorites, Contacts, and possibly Devices and even Photos, so that we could compile a set of data that demonstrates these various features and helps find edge cases.

[eneiluj]

Yep it would be nice indeed. I'll add a set of test files to the repository when I'll implement some tests :grin:.

[anaqreon]

I made another attempt based on your suggestions: https://github.com/nextcloud/maps/compare/rework-eneiluj...anaqreon:zoom-all-contacts?expand=1

[eneiluj]

Thanks a lot, I made a mix between your code and mine. You had spotted a mistake in mine, nice catch.

If user zooms on a disabled group, it's automatically enabled.

Don't hesitate to make pull requests if you've implemented something.

[anaqreon]

If case anyone finds it useful, I created a repo that makes creating a Docker-based development environment super easy. You basically just clone the repo and run a script, and you get a local Nextcloud server running in Docker with the Maps app installed (running the latest dev branch). Then you can use your favorite IDE to develop code in the local ./maps/www folder and see the results by refreshing the browser.

This might also make it easier to import any test data such as a pre-populated database or Nextcloud user files needed to exercise or demonstrate some feature.

[w1n5l0w]

I just installed the rework-branch and on first look, I see the possibility to filter the displayed contacts on basis of their group/category, but not based on the address books the contacts are listed in. Is it possible or even planned to provide this feature?

For example, I have contacts separated in several address books, but sometimes tagged with the same category-name. In Maps they are all grouped under this category and I can't separate them as I like to do.

[jancborchardt]

@w1n5l0w this might be something for the future. But not for now as it’s already quite complicated. :)

We could have a thing in the 3-dot menu which would only show when you have more than 1 address book where you could choose between "Show all address books" and only picking the individual ones?

[w1n5l0w]

@w1n5l0w this might be something for the future. But not for now as it’s already quite complicated. :)

No doubt, something for the future!

We could have a thing in the 3-dot menu which would only show when you have more than 1 address book where you could choose between "Show all address books" and only picking the individual ones?

Sounds like a valid option for implementation for me.

[w1n5l0w]

If there are changes in the contacts db, how does maps refresh the contacts displayed? Is it possible to trigger a database refresh i.e. via occ?

Background: After I installed the rework-eneiluj branch Maps populated with contacts. Then I changed addresses, deleted some contacts and created new ones, but Maps didn't show any changes and stuck with the contacts of first population (for two days now). Is this a bug or a misconfiguration?

[eneiluj]

You can now use master branch to try things out. It's up to date and rework-eneiluj does not exist anymore.

Contacts are not properly stored in Maps DB. We store the coordinates of their addresses after having lookep up for them. The lookup process is triggered automatically with contact hooks and background jobs. Normally, if you change/add an address, it should have an immediate effect, I mean, it should be visible after reloading the app page.

But when you enable the app while already having a lot of contacts, once you load the app page, lookup process will be scheduled as a background job and be done progressively to avoid being blacklisted by Nominatim (the geo resolver service we use). So you'll have to wait a bit before seeing all your localized contacts on the map.

[jancborchardt]

I would add this to the 0.1 milestone since it’s working. Then we can open separate specific issues about stuff which is missing. What do you think?

[eneiluj]

Yyyyep!

[w1n5l0w]

Great. And thanks a lot for your work!