Server: Calendar and Contacts Trashbin

[georgehrke]

We already provide a trash bin for files, that allows users to restore deleted files.
I'd like to see something similar for calendars and contacts.

This also helps users who accidentally deleted events/contacts or an entire calendar/addressbook in their client.

cc @jancborchardt @tcitworld @schiessle

[georgehrke]

There are a few questions that need a prior discussion:

• Which app shall implement the features logic?
  
  
  
  • dav
  
  
  • a new dav_trashbin
  
  
  • calendar respectively contacts
  
  

I'd not like to see it in the latter, because you can use CalDAV and CardDAV without the Calendar and Contacts app. Therefore the trash bin should always work, no matter whether the web gui is enabled.

• Which app shall implement the GUI?
  
  
  
  • dav respectively dav_trashbin in the personal settings area
  
  
  • calendar respectively contacts
  
  

I'd actually go with calendar and contacts here. We can't do this with plain CalDAV/CardDAV. Therefore it's acceptable to having enable calendar respectively contacts before restoring imo.

I'd suggest to provide the trash bin via remote.php/dav/calendars_trashbin/ respectively remote.php/dav/addressbooks_trashbin/.

@schiessle I wasn't able to find the trash bin via remote.php/dav/. Does files_trashbin already use WebDAV for talking to the server?

[georgehrke]

However it would be of no use if the calendar/contacts app aren't here since you wouldn't be able to restore deleted calendars/contacts.

Yes, but there are the following scenarios:

Dav app takes care:
You delete a calendar by accident -> the calendar is moved to a trash bin by the dav app -> you enable the calendar app -> you restore the calendar

Calendar app takes care:
You delete a calendar by accident -> ¯_(ツ)_/¯

[tcitworld]

Totally agree with all of these. If I understand correctly, putting an item in the bin would mean moving the data from the oc_calendars table to oc_calendar_bin, while not touching oc_calendar_objects. Do the data from oc_dav_shares must remain too ?

[nickvergessen]

Couldn't we just create a flag, and some sort of hidden calendars/events which are not exposed, unless you propfind with a the special property? I think that would be very few lines to implement.

When you delete a calendar/event and it does not have the flag yet, we set the flag. If the flag already exists it's deleted from the system.

[georgehrke]

That would indeed be a good solution.
Instead of the flag I'd probably go for a timestamp, which is null by default.
This allows us to easily delete deleted calendars/events older than n days.

[georgehrke]

But there is actually one issues I'd like to discuss before implementing this:

When a calendar/addressbook (but also event/contact) is deleted, the uid should be available again.
What to do about this? Replace the uid (or uri for event/contact) with a random string?

Keep this random string when replacing or tell the client to send a new uid along the request?

The dav_shares table uses the calendar id and not the calendar uri, we don't need to change anything there.
The behavior should be the same as in the files app. Restoring a calendar/addressbook also restores all the shares.

[georgehrke]

Reminder for myself:
Do not return deleted calendars on allprop request

[nickvergessen]

When a calendar/addressbook (but also event/contact) is deleted, the uid should be available again.
What to do about this? Replace the uid (or uri for event/contact) with a random string?

As long as there are no duplicates on restoring, fine by me.

[georgehrke]

Yes, the db won't allow duplicate URIs.
https://github.com/nextcloud/server/blob/master/apps/dav/appinfo/database.xml#L375

[jancborchardt]

Regarding where to store it – definitely directly in the Calendar respectively the Contacts app. cc @Henni @irgendwie

[georgehrke]

Do you mean store or restore?

Please excuse my brevity and typos.
Sent from my mobile

On Oct 11, 2016, at 9:36 AM, Jan-Christoph Borchardt [email protected] wrote:

Regarding where to store it – definitely directly in the Calendar respectively the Contacts app. cc @Henni @irgendwie

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

[schiessle]

A bit late to the discussion, but I agree with the conclusion by @georgehrke. Think this is a good way forward.

[jancborchardt]

@georgehrke it was a reply to your previous question:

Which app shall implement the features logic?

And I think it should be the respective app. Definitely not a new calendar_trashbin app or something like that which needs to be separately enabled.

[georgehrke]

Please see https://github.com/nextcloud/server/issues/1662#issuecomment-252332641 on why I think this is a really bad idea.
You can use CalDAV and CardDAV without enabling the respective apps.

[Henni]

@jancborchardt this would mean that the calendar app has to be installed to offer a trashbin to all other clients.
I'd prefer to implement only the gui inside the respective apps (contacts & calendar) and implement the backend (functionality) inside a different app, maybe extend dav_trashbin or even create new apps (which are enabled by default).

[georgehrke]

I would really prefer to just put it in the dav app. This feature requires a Plugin for Sabre/Dav and at the moment there is no way for other apps to register sabre/Dav plugins.
All the other things like comments and tags are also in the dav app.

[jancborchardt]

Ok, putting it in the dav app seems to make most sense then. :)

[olantrust]

Can this be done for Tasks in the Tasks app too? Since that also uses Dav...

[nickvergessen]

@olantrust should be possible. cc @raimund-schluessler

[raimund-schluessler]

I created an issue: https://github.com/nextcloud/tasks/issues/29

[cornzy]

Hi, just came over this issue. Is there any progress?

I think it would be fine to separate this issue in at least two or three issues. The first step may be just having a trashbin for events. Second a trashbin for contacts (I do not know how much differeces there are compared to events). Third it seemst to me a bit more complicated having a trashbin for a whole calendar. Indeed I do not know whether this is realy necessary although it is nice to have. It is no common mistake that someone deletes accidently whole calendar.

Further for the event trashbin it would be useful to have it as extra calendar. That means you could view your trashbin calendar as every other calender (also sync it if you like). Deleting an event simply means to move the event from the origin calendar to the trashbin calender. Restore is moving back. Maybe it is necessary to remember the origin calender for automatically moving back. For me it would be enough to see the event and have the ability to move it back manually.

[lailai487]

Are we have any SQL command can recover calendar delete events???
I have a big trouble for this...lol

[georgehrke]

No, the calendar / contacts trash bin wasn’t implemented yet. I’m afraid gone is gone.

[lailai487]

OMG..... Thanks your answer.
Activity only log the event title?
No any startdate - enddate?
Maybe my boss will kill me...

[georgehrke]

Activity only log the event title?
No any startdate - enddate?

Only the title, yes
https://github.com/nextcloud/server/blob/ba3c608a00f3834f743de5a0cd82ddae3825d458/apps/dav/lib/CalDAV/Activity/Backend.php#L477

[tacruc]

I think if somebody takes a look at this she*he should think about logging the change to the activity app
nextcloud/server#3549.

[jancborchardt]

Folks, let's be civil here. Yes, a way to recover deleted Contacts & Calendars would be great. 2 options:

1. Can you help? Great! Dive into the code, communicate with the other developers, and let's make this real.
2. OR can you not help and just want to comment here: Do not. It's notifying everyone subscribed to this issue and wastes their time. And if your comment is negative, it will make people even less likely to implement what you want to have.

[MorrisJobke]

@georgehrke Nothing for 14, right? Move to 15 or backlog?

[georgehrke]

Backlog for now

[dokterbob]

I just accidentally deleted my contact list and I now lost all my contacts on both my phone and my computer. How is this even possible!?

[dokterbob]

Relevant: #3216

[natrius]

No news on that one i guess? Someone at my server mistakenly deleted all his contacts and something like this would be nice.

[dokterbob]

If you have time to mark a reaction as off-topic, at least also take the time to explain a little bit why an issue causing users to lose all their data unexpectedly is low enough a priority to go unresolved for about two years - while all sorts of new features do see implementation.

Also, please do point me to a place where I can see and try to possibly suggest priorities without being marked off-topic without any explanation provided. In other words: please allow us to be heard (without unnecessarily disrupting the technical discussion).

I've previously lost 3 days trying to get my contacts back and I suppose @natrius had a similar experience. Clicking the 'ignore' button on them is likely to trigger frustration. Instead, perhaps, you might edit the main issue such that users are immediately guided to a place where either their emotions/sense of priority may be expressed and heard or give them an overview of how they might help resolve the issue.

Thanks.

[natrius]

I guess because of https://github.com/nextcloud/server/issues/1662#issuecomment-368452011 and while its OK to do so it is still frustrating, to be honest, to see this issue open after more than 2 years. Will shut up now as the comment suggests.

[georgehrke]

Also, please do point me to a place where I can see and try to possibly suggest priorities without being marked off-topic without any explanation provided. In other words: please allow us to be heard (without unnecessarily disrupting the technical discussion).

There is no need to do that. If you want to show your support for a ticket, use GitHub reactions. We can judge the demand based on that.

And if it's still too slow in your opinion, contribute! This is an open source project and, looking at your profile, you are a programmer yourself. I'd be more than happy to give pointers where to start.

---

I will just quote @jancborchardt one last time here, but if these comments don't stop we will have to lock this conversation.

Folks, let's be civil here. Yes, a way to recover deleted Contacts & Calendars would be great. 2 options:

1. Can you help? Great! Dive into the code, communicate with the other developers, and let's make this real.
2. OR can you not help and just want to comment here: Do not. It's notifying everyone subscribed to this issue and wastes their time. And if your comment is negative, it will make people even less likely to implement what you want to have.

[dokterbob]

@georgehrke Thanks for your thoughtful response, to which I have two relevant comments - although they are meta in the sense that they relate to effective communication more than to the issue at hand, I would not know where else to put them - which is exactly related to my prior response.

1. Reactions are a suitable way to inform about tickets affective huge amount of users. However, there is no way to distinguish the degree to which users might be affected. Thousands of users might experience a mild nuisance, or ten users might run away from nextCloud forever.

   A long time user, having used ownCloud from the start, I am in fact considering alternatives given that an issue as severe as this one, however rare it's occurrence, does not seem to be considered a party-stopper.

   The current issue is exactly analogue to an issue in a backup system, which does not get found out until it's already too late.

2. Pointers where to start: yes! Please do! My time is, essentially, booked in terms of code but I might be able and willing to help pinpoint the issues, help on the specification etc. What helps a lot here, really, is creating a summary of the problem, proposed solutions and a roadmap to implementation right at the top of the issue. It makes it much more inviting for implementers.

3. Meta-discussion: it seems to be the community would benefit tremendously with having a place where frustrations can be voiced and discussed, past the technical level. Is there such a place? Please point me to it, it's much much much nicer than to just silence users politely enquiring about 6 months of inactivity on a legitimate issue.

[dokterbob]

In practical terms, I see a rigorous but expensive and a cheap solution. I strongly suggest implementing a cheap solution (disabling deleting of all contacts) until a rigorous solution can be implemented (perhaps give it another 3 years).

In the mean time, you'll save people days and days in frustration trying to find their friends and clients back and less people will walk away angrily into the arms of the closed source competitors with their infinite undo readily implemented.

Rigorous

Instead of the flag I'd probably go for a timestamp, which is null by default.
This allows us to easily delete deleted calendars/events older than n days.

I think a good way to implement this, as a pattern, is to work towards a full rollback capability. As it's small amounts of data, it would be technically cheap and the user experience would gain enormously: never ever ever lose appointments, contacts or tasks unintentionally again. Incidentally, this is also a pattern implemented by Google and the likes (perhaps you might implement a special privacy-friendly 'real' delete as well).

How this would work: you would store a transaction log of actions (which, I suppose, you already partly have as part of the sync' implementation). The items in the log would represent reversible mutations on the current state - which might or might not be serialised.

For example, an action could be:

1. Add a contact to group such and such.
2. Create a contact.
3. Delete a contact.
4. Delete a calendar.
5. Rename a calendar.

Now users would be able to view this log and revert to any previous state, simply by replaying the log in reverse.

As opposed to a 'small thing' which only affects users being bitten badly in the behind by this, as there is definitely the user expectation, it would be a great user-facing feature. This would also integrate well with the concept of versioned documents. In any case, it would provide consistency.

Cheap

Here I see two ways:

1. Simply disabling the DAV call to delete calendars or contact lists until there's a good backup solution implemented - but leaving this enabled in the web interface (potentially returning an error telling the users to go to the web interface). The loss of users when they loose their contacts is way worse than the gain for users being able to delete lists outside of the web-interface.

   This would have my strong preference as it's super easy to implement and it leaves space for a 'proper' solution.

2. Backup sync of calendars and contacts
   It seems there is already an undocumented feature of contacts backup. Just making sure this is enabled and documented, and making sure contacts aren't automatically deleted or at least can be reverted seems like a quick win.

[georgehrke]

I think a good way to implement this, as a pattern, is to work towards a full rollback capability. As it's small amounts of data, it would be technically cheap and the user experience would gain enormously: never ever ever lose appointments, contacts or tasks unintentionally again. Incidentally, this is also a pattern implemented by Google and the likes (perhaps you might implement a special privacy-friendly 'real' delete as well).

I like this idea and something similar to files_versioning should be our ultimate goal for calendars and address books as well. Unlike files_versioning, which just stores different copies of different versions, we can actually use proper text-based diffs, to decrease the amount of storage required.

How this would work: you would store a transaction log of actions (which, I suppose, you already partly have as part of the sync' implementation). The items in the log would represent reversible mutations on the current state - which might or might not be serialised.

Indeed. A transaction log is already stored in oc_calendarchanges and oc_addressbookchanges.
One approach would be to simply introduce two new tables. oc_calendar_versioning and oc_addressbook_versioning, that have a foreign key relation to the oc_*changes table and a text based diff. Combined with the information stored by the activity app, this would allow to easily show information like: "User 123 changed the cell phone number of contact abc to ..." and allow to revert this action.

As much as i like this idea, i would consider it two different features. Not only from a user perspective, but also from an implementation point of view.

The approach for this feature is still:

• add deletion_time columns to oc_addressbooks, oc_cards, oc_calendars, oc_calendarobjects.
• mark an object, which was moved to the trash bin, as deleted in oc_*changes, mark it as created on restore.
• implement a custom CalDAV / CardDAV based restore API for the calendar and contacts app.
• have a client-side implementation of the very same api in our cdav-library
• have a background-job, that deletes items from oc_addressbooks, oc_cards, oc_calendars, oc_calendarobjects after an admin-defined time (+ delete all versions)
• allow admins to completely opt-out of this feature and restore the current behaviour.

Simply disabling the DAV call to delete calendars or contact lists until there's a good backup solution implemented - but leaving this enabled in the web interface (potentially returning an error telling the users to go to the web interface). The loss of users when they loose their contacts is way worse than the gain for users being able to delete lists outside of the web-interface.
This would have my strong preference as it's super easy to implement and it leaves space for a 'proper' solution.

This is an absolute no-go for me. Being able to delete calendars and events/todos/journals is a fundamental feature and a CalDAV server that can't do that is broken imo. Additionally, the calendar app in the web interface does not make use of any special API, it's just using CalDAV as well.

Backup sync of calendars and contacts
It seems there is already an undocumented feature of contacts backup. Just making sure this is enabled and documented, and making sure contacts aren't automatically deleted or at least can be reverted seems like a quick win.

For a back up, it's enough to have a simple SQL backup. Creating regular backups of your database is frankly the job of every admin. (oc_calendars, oc_calendarobjects, oc_calendarchanges are the most important tables here)

[tcitworld]

For a back up, it's enough to have a simple SQL backup. Creating regular backups of your database is frankly the job of every admin. (oc_calendars, oc_calendarobjects, oc_calendarchanges are the most important tables here)

To be fair having these tables backuped is not enough if the rest of your Nextcloud installation is broken (you'll end up doing something like this).
It would be nice to have a calendars backup for contacts in the mobile apps just like there's a contacts backup in the (Android only?) app.

On the versioning topic, it seems exciting, but I'm kinda afraid of this being hard to handle properly.

[jancborchardt]

@georgehrke @tcitworld @rullzer @MorrisJobke maybe this is something to properly spec and plan at the next Contributor Week? (Or if not there, in a call or elsewhere.)

[simonsan]

Are there any updates on this?

[dbadura]

Will this feature be implemented?
I think that it should be high priority to fix this issue?