Client: Don't delete files when a backup on the server is restored

[jospoortvliet]

A simple and easy way of creating and restoring backups is using virtual machines or file system snapshots. Unfortunately, this is very likely to result in the deletion of all files on the client side which have been changed since the server snapshot was created!

It would be great to deal with that a little more gracefully.

What happens is this (from Olivier on IRC):

Server snapshot is restored. The client sees that the file are not on the server. And because the client knows that the file were there before from its database, it will remove the files localy as if they were deleted on the server.

Also it will download the files from the server erasing the ones in the client because what is on the server is considered new. The client has no way to know that there was a rollback. It just thinks the files were deleted or updated.

Dealing with it as end user/admin

You can prevent this from happening by deleting the local cache of the client before connecting to your ownCloud again. The client will then think all files locally are new, and after checking with the server, notice that they are newer than what is there and send them there, too (except for the ones with identical file size/name/change time).

Fixing the problem

A potential direction for solving this would be to either look better at timestamps (not just == but look if time stamps are older or newer too).

Another idea might be to have counter incremented whenever the client talks to the server, server-side or synchronized between client and server. When the counter suddenly drops, the client knows the server isn't who it used to be ;-)

Obviously both ideas above aren't mine but Olivier's ;-)

[tflidd]

This seems to be related to https://github.com/owncloud/core/issues/11621

Like you describe it, in the actual state, an admin has to inform all users/clients that he wants to restore a backup and that all files being uploaded since then (date of backup) will be automatically deleted.

[tflidd]

There was a slightly different case on the forum: https://forum.owncloud.org/viewtopic.php?t=26498&p=79941

Is it possible to make the client more "intelligent"? In this case an external drive was unmounted (unknown reason), owncloud detects an empty folder, the client "thinks" files were deleted and deletes all files on the client. In this case it would be great if the client asked if it should as well delete all files from the client.

[jospoortvliet]

@tflidd I guess the client doesn't even know the files are on external storage ... so it just sees a folder has disappeared and will synchronize that change. To make it warn, the client would have to learn what is external storage from the server, or the server needs a generic way of telling the client "hey, these files are at the moment not here, but they might come back so keep em around a bit longer".

I can actually see why that might be needed some day - not all external storage is equally reliable and if you are checking the server every few minutes, see it appear-disappear-appear-disappear and start to remove-re-download-remove-re-download files, that's not particularly smart ;-)

[tflidd]

It seems like you fixed something: https://forum.owncloud.org/viewtopic.php?f=29&t=29613#p93199

[guruz]

@tflidd Thanks for notifying..
@ogoffart has indeed improved the detection if server and client are "completely different"

[Dianafg76]

@guruz what are the steps to reproduce this issue?

[jospoortvliet]

@Dianafg76 see the first comment.

• run server in a Virtual Machine. Put data on it, sync with the client in a different VM or on the host. Make some changes on client, which sync back to the server.
• make a snapshot of the server in the Virtual Machine software.
• make some more changes with the client.
• restore the snapshot of the server (as in, run the client against that older snapshot).
• there should be no data loss (defined as "you don't lose the changes made on the client since the server backup was restored")

In reality, this probably will only work if a LOT of files have changed and the code by @ogoffart which detects large changes kicks in.

[guruz]

@jospoortvliet Wait, I thought this was if a file backup is restored without restoring the server oC DB.. ? (Which changes all ETags and FileIds..)
The thing you just described should work completely fine, it is as if you are offline for some time (independant of any server restore)

FYI @ogoffart

[jospoortvliet]

@guruz if that works fine, then great - if a sysadmin sets back the files from the server in the ownCloud data folder without also restoring the database, he/she does something entirely unsupported and (frankly) stupid. This was about restoring a snapshot of ownCloud to an earlier point in time, which would overwrite things which happened since then (and were preserved on the client).

[ogoffart]

@guruz the problem is that when you restore the backup, fileid stays the same, but etags are back to the state from which you restore the backup, and new files are gone. So the client will sync back to the state of the backup, loosing all the files that were changed inbetween

[ogoffart]

@dragotin had the idea in https://github.com/owncloud/client/issues/4088 to check the mtime of the modified files, and if they all send the files in the past, we could ask the user.

[guruz]

By "all files" you mean files that are modified, right? The ones we'd upload/download.
What about situations where client/server have a different time and we get false positives? This should not happen if files are always modified by client (OC-Mtime) but for arbitrary backends..?

[ogoffart]

Here is a part of the fix: https://github.com/owncloud/client/commit/1bb76f534391c6f3f0709a2297b9ae6700b46efd

The idea is that if there is at least two files that are going back in time, and no files that are going forward, then we ask the user if he really wants to overwrite the files.
This patch does the first step as discovering the problem, and replalace the download with conflirs, and the rename with upload.

I wanted to replace the download by downloading a conflict file followed by an upload, but that turned out to be lot of code for something that is almost never hapenning and quite hard to test.

@dragotin what's your opinion?

[MTRichards]

An idea, which might not work.
we keep a last sync success timestamp on the server and the client which get compared at sync.
When a new sync happens, the server keeps a time stamp for the sync interval, and then updates to the new time stamp.

Then, these things could happen:
1) we sync and all is normal, this sync client and the server timestamp of last sync match, all is well.
2) a client was off for weeks, or on a plane or something, and logs in again. Then the client sync time and the server last sync time match, should still work.
3) multiple sync clients are syncing the same account. When one logs in, the sync time on the server will be the same as this client last sync time, or slightly behind (because the other sync clients synced recently). But we know that this has to be within the sync interval window so we know if this is normal and all can sync happily.
4) 2 (or more) clients were syncing the same account, but one (or more) was off for a while while on a plane. The one that was off for a couple weeks comes back online, the client last success full sync is well behind the server sync time so this is normal behavior and all syncs fine.
5) a new sync client is tied to an account. It has no last sync time, so the sync is fresh from scratch, should work no problem.

FINALLY, after a backup is restored, the server sync time is now BEHIND a single client sync time by a substantial amount (more than a normal sync interval). Since this should NEVER happen in normal operation, we know a backup was performed.

We might also only need one server time stamp, which each desktop sync (and mobile) client keeping their own update timestamp at the END of a successful sync run.

[MTRichards]

Copied this into the public issue - Please comments on the idea there. https://github.com/owncloud/client/issues/2325

[ogoffart]

Even if the server had a timestamp, it needs then to be a different one for every client.
Different clients might sync different folders.
What do we do when we detect a backup recovery? We can re-upload files with a newer timestamp. but should we keep conflict files? Removed files will be re-added. Renamed files.directory will not be detected.

[rullzer]

The server side PR https://github.com/owncloud/core/pull/24054 will allow us to signal the client that there has been a backup.

@dragotin this was what we discussed in Nurnberg

[rullzer]

https://github.com/owncloud/core/pull/24054 is in.

[rullzer]

Ok writing down from how I understood this should work. ( @dragotin corret me if I'm wrong).

If the data-fingerprint changed (it is empty initially) then the clients know that a backup has been restored. Since there can be newer files (since after the backup) on the clients we do not want to sync the server state to all clients.

Instead what we do:

1. Halt sync
2. Figure out which files have different etags than in our local db
3. Create conflict files for those
4. Present a dialog with something like: "A backup on the server was restored. Some files on your local system might be newer than this backup. Please resolve al conflicts).
5. A possibility would be to dump a 'backup-collision.txt' or something where we also list all files.
6. Once all is well continue syncing.

[moscicki]

I think a backup scenario is just one of several of mismatching data-fingerprint.

The others are plain misconfiguration (talking to a different server behind DNS alias for example) or operational error on the server-side (e.g. incidentally removed the whole data directory or underlying storage system disconnected in a way that it not properly reported etc).

I would rather give a user a clear warning that something wrong is going on with the server (maybe hinting some possible reasons for this like a backup restore or misconfiguration) and give them two options how to proceed (for which I do not have good and catchy names):

• sync by merging local and remote changes (as if you dropped the local sync db), this may create extra conflict files
• sync as usual nontheless (and accept the risk of getting deletes you don't want)

It should be possible to postpone this decision and come back later and decide after a user understood what they want or need to do.

Note: this kinds of question was in the past already asked to the user who tried to synchronize existing content, but I cannot find it now in 2.1 client.

[MTRichards]

What is done on the server for 9.1:

• There is now a data fingerprint on the server
• Client needs to be updated to check for this random string on the root directory
• Clients store the string empty, string is only updated on the root - can be queried from a propfind, and can be added at no cost to the root
• If it changes on the server, the client stops syncing and compares the list of files, creates conflict files if they are different, and then creates a popup notification on the client that this has happened and the user needs to decide
• Needs to show the client which files that are conflicts are created (they are in the not synced tab)

[MTRichards]

Key point:

• THE ADMIN has to make a change to the random string on the server when a backup is restored.

[MTRichards]

@carlaschroder just verifying that this new change is documented (updating the string on root) is in fact changed when the admin applies a backup. Otherwise, this won't work.

[rullzer]

The command to run is: ./occ maintenance:data-fingerprint

[cmonteroluque]

Backup / restore from server overwrite issue

• Has a data fingerprint on the server
• Client needs to be updated to check for this random string
• Clients store the string empty, string is only updated on the root - can be queried from a propfind, and can be added at no cost to the root
• If it changes, the client stops syncing and compares the list of files, creates conflict files if they are different, and then creates a popup notification on the client that this has happened and the user needs to decide
• Needs to show the client which files that are conflicts are created (they are in the not synced tab)
• https://github.com/owncloud/enterprise/issues/966 - PR already merged

[davitol]

:+1: Tested in 2.3.0-nightly20160922 (build 3694) and WFM

[thetrial]

I am wondering … if this is closed 2016, it should be fixed? I recently had the (same) scenaria with 9.1.4 and client 2.3.0 or 2.3.1 (maybe even 2.3.2) that I had to replay a backup to my raspi, a month old. And it just deleted all files from the sync foldern that weren’t there a month before. Although it should have detected those files are newer than the stock situation on the server. This happened in May 2017.

I recently updated to 10.0.2 (client 2.3.2). Platform OSX 10.7.

[ogoffart]

Did you update the data-fingerprint after restoring the backup?

It is true that it is not well documented. => https://github.com/owncloud/documentation/issues/2360

[thetrial]

No, I did not? Indeed it is not well documented, the issue page leads to a 404.

So what is to do? I just restore the server back to the state two days ago. And then? Just do a »sudo -u www-data php occ maintenance:data-fingerprint«?

But if … isn’t it problematic that probably sync clients are waiting to see the server back again and start immediately to sync? A restored server does not start in maintenance mode?

[guruz]

@thetrial FYI the only documentation right now is https://github.com/owncloud/documentation/pull/2446/commits/94da318a7d8516d96cb14a6e92a127d0164a69cc but we hope to improve it.

But if … isn’t it problematic that probably sync clients are waiting to see the server back again and start immediately to sync? A restored server does not start in maintenance mode?

The maintenance mode is set in your config.php, not in the restored DB

[thetrial]

To be honest, I don’t get it :-) Maybe server restoring situations are too stressful situations. When I say restore, I don’t mean a file-by-file replay but a dd-based rollback to a snapshot of the whole server to day x-n. I did that yesterday again. And right from that snapshot (I guess OC 9.1.4) I updated to OC 10.0.2 before any client could connect. And, that’s interesting, this time something different happened: The new files were kept, not deleted. The newer files induced conflicts. Although it would be nice that new_er_ files are handled correct as new_er_ and so get updated towords the server. IMHO.

This time I did not set the occ command for fingerprints. BTW, I usually work in the CLI with occ and don’t set anything in config.php. That file just gets restored, afterwards I type into the CLI. When updating, as well.