Client: Improved Conflict Handling: Save Conflict Files to Server

[dragotin]

The clients conflict file handling is not optimal currently because we decided to always prefer the server version over the local changes. That makes us moving the local changed file aside with a conflict file name and download the server version in place.

For the user that means that the local changes of the file are never synced to the server. #3222 #4079

Under the assumption that we want to stick with the _server version wins_ policy we could improve this by supporting version branches in the servers versions app as described here: owncloud/core#23002

Once the client detects a conflict file, it uploads the current local version to the servers version app along with the ETag of the file's predecessor, which is known to the client. That way, the local file version would be saved on the server and data loss would be avoided.

This depends on owncloud/core#23002

The user experience would benefit even more from considering the version branches in #4556

@MTRichards @karlitschek @moscicki @jancborchardt @ogoffart @rullzer your opinions please.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[karlitschek]

Hmm. 99.99% of all users seems to be happy with the current conflict files handling.
But a good idea to consider of course.

[moscicki]

Is there a way to globally check if there are unsynced conflict files? Other than asking every user individually for logfiles or local directory listing ;-)

I think this would help to understand if there is a practical problem or not and maybe this could be a first step to tackling this issue. Would this information be stored in the syncdb journal file (in which case we could try to recall it to the server and analyze the whole bunch automatically)?

I tried to count 412 PUT responses in our http log files but then I realized it is only a subset of conflict cases (and I do not have any feeling how representative this subset is). So I still do not have a solid data point to reason about this issue for our system.

Dropbox does sync conflict files and probably there is a good reason to it. I am not sure if "99.99% of all users seems to be happy" is a good indicator in general because for example "99.99% of all users" did not notice the corruption problem that was actually a serious one (and statistically significant if you have above 10^7-10^8 files in the system).

[scolebrook]

I like this idea but I have a concern in how the average, IT illiterate user, would be able to understand what's happening. If you can have two versions of the same file on two users computers with the same name, that's very bad.

In the event of a conflict, I think a conflict file should be generated and the server copy sync down just as it is now (that tells the user something really, really needs their attention because they _always_ take notice of notifications and messages from the ownCloud app). But the client syncs the conflict file back to the server as a branched version.

This allows the switching of the branch easily. If the branch switches, then the local conflict file goes away as it then becomes a dupe.

One edge case that would need to be considered. User 1 makes an edit that results in a conflict file. A new branch is created. The branch is switched server side so now the official version is the previously conflicted version. User 2 has been offline all this time but has edited the file. They've been working on the now non-current branch. What happens when they come back online. Does their current version sync to the non-current branch and then get replaced by the current branch of the file? How does the software inform User 2 of what has happened? Do the other users get informed that there's now an additional version in the other branch?

Branching is a fairly easy concept to grasp as a developer but I think it's going to take a lot of explanation get that light bulb moment for many non technical users. This is not easy to do in user friendly way. It will take some special UI to make this intuitive.

LibreOffice Online when they implement live collaborative editing + richdocuments is a better solution to this problem IMHO. Avoid the creation of conflicts entirely, then you don't have to deal with them, branching, etc.

[ThaDaVos]

We are currently running into this problem.
If the client was shut off for like a day, because something happened, or the user accidently turned it off or windows didn't boot the client, when turning it back on, it just overrides all the newer files and generates conflict files.

Maybe instead of doing this, let the user decide who wins, server or client.
Like the QNap QSync client does, everytime there is a conflict it asks what to do, save server version, save client version or generate conflict file.
It also has the option to check a checkbox to have it do the chosen action everytime it has a conflict.

So simply said, let the user choose who wins, the server or the client instead of chosing it for them.

[michaelstingl]

@guruz @ogoffart Could we start with a lightweight approach? Just upload with filename and adding _conflict-$username-XXXX before extension? So everybody can see, there is something that needs to be resolved. I think this requires only minimal or no server-side changes. (Perhaps parameter in config.php and a capability /cc @PVince81 @DeepDiver1975 )

[CJR2]

From an end user perspective, everyone needs/must see that there is a conflicted version. Otherwise, folks will continue to work on their version of the document, rather than get with other document users to resolve the issue.

Agree with @michaelstingl.

Just upload with filename and adding _conflict-$username-XXXX before extension?

Users like our organization, who have used Dropbox in the past, are very used to this happening and have internal processes for resolving these issues between user versions if they do happen.

If the client was shut off for like a day because something happened, or the user accidentally turned it off or windows didn't boot the client, when turning it back on, it just overrides all the newer files and generates conflict files.

Our organization is a relatively large workgroup and have never encountered this issue.

I don't think we should guess what users must experience (99.99%). Instead, why not make this an admin decision? Perhaps make this a selection in the admin console, or if that's not feasible, as a client setting?

[michaelstingl]

Dropbox's approach is pretty straightforward:
https://www.dropbox.com/help/syncing-uploads/conflicted-copy

[dragotin]

I can not really read from the dropbox text above if they have the conflict files all on the server. We did not do that initially as we did not want to spread all conflicts that somebody might create for whatever reason to the harddisks of each and every user of a share. That would probably also confuse people.

I think the only good solution is to do it via the versions app.

[CJR2]

Yes, Dropbox puts all conflicted files on the server (see my original note).

The files are marked with the username as well, which helps in resolving the issues (see attached image at link: [https://dl.dropboxusercontent.com/u/1595301/Capture.JPG]

Think of the organizational disruption if you DON'T spread all conflicts to the server. First, they don't occur very often. When they do, vital information/revisions could be lost.

It is better to have the issue of conflicts than not (again, Dropbox seems to have the best implementation).

On an aside, the Dropbox client monitors if 2 or more users are in the same doc and warns if there's potential for conflict.

[moscicki]

The first function of a system that automatically handles files on behalf of the user is to protect the data. Local conflict file which is not uploaded to the server is a potential data loss. The larger the system the higher the probability of the data loss.

I would NOT bundle it with the versions app. At least not at the level of sync protocol. A PUT of a conflict file should be marked as such (e.g. using an extra header) and the server implementation should be free to handle it appropriately (e.g. via versions app). I can imagine other implementations, not via a version app but on a separate list on the server.

[dragotin]

@moscicki ok, true comment of course. PUTting the conflicted file with an extra header makes sense, but would that PUT go against the original name, or using the usual conflict name with time stamp?

[moscicki]

Good question. I'd rather use the original name because then the server can
identify the source file and apply some logic based on that (e.g transform
the name into conflict file pattern or store as version). I also think
that it could be a better idea to make the server have the last say about
the timestamp it wants to use (not the client, to avoid any possibility of
clashes). But obviously that changes the semantics of PUT quite a bit
because the resource specified in the URL is not the same as the one
actually updated (not sure if there are any other side effects of that in
general HTTP world.). It would also make for the server mandatory to handle
the special header, else it would be uploaded as a genuine file version (is
this a problem for current server implementation, e.g. conflict loops seen
in the past?).

I would imagine something like that:

PUT filename.ext
OC-CONFLICT: YYYYMMDD-HHMM

Would result in, for example, storing:
filename_conflict---.ext on the server

I would add USERNAME for conflicts in shared folders.

The question is: what to do if a user starts modifying the conflict file? I
think one should keep on uploading it. However, I would not download the
conflict files (the server could "hide" them from PROPFIND listing). In
this way we can avoid having conflicts of conflicts, loops etc.

Alternative of the above proposal is:

PUT filename_conflict-YYYYMMDD-HHMM-USERNAME.ext
OC-CONFLICT: filename.ext

In this one the server also has access to the original filename (what I
think is important).

On Wed, Jul 12, 2017 at 11:03 AM, Klaas Freitag [e-mail]
wrote:

@moscicki https://github.com/moscicki ok, true comment of course.
PUTting the conflicted file with an extra header makes sense, but would
that PUT go against the original name, or using the usual conflict name
with time stamp?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/owncloud/client/issues/4557#issuecomment-314701696,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAl9jTYJQ8pkFpzrvt4qGvmVOBAAr3ofks5sNIvygaJpZM4HslaE
.

--

Best regards,
Kuba

[ThaDaVos]

Isn't it better to let the user decide what to do when a conflict happens?
Like what the QSync client does, it shows a notification and has 4 options (can't remember them precisely):
1: override local version with servers version
2: override server version with local version
3: generate conflict file (local version is changed into a conflict file)
4: do nothing, keep both

This way the user can choose what happens and I think this is better.

[moscicki]

A user has a choice now: he can delete local conflict file (1), rename
local conflict file into original name (2), do nothing (3) and do nothing
(4).

A particular problem we are trying to solve in this ticket is that local
conflict file is not saved on the server so there is a potential data loss
if a user does nothing.

The sync client should take "data safe" decisions without user's attention
such as mandatory notification. Users usually do not pay attention to these
applets running in the background and also it is impractical/annoying if
notifications are popping up for a larger number of files. If, however,
such a notification was deemed nontheless useful it can be easily
implemented on top of proposed scheme as a convenience function on the
interface.

On Sat, Jul 15, 2017 at 8:22 AM, dvdbot [e-mail] wrote:

Isn't it better to let the user decide what to do when a conflict happens?
Like what the QSync client does, it shows a notification and has 4 options
(can't remember them precisely):
1: override local version with servers version
2: override server version with local version
3: generate conflict file (local version is changed into a conflict file)
4: do nothing, keep both

This way the user can choose what happens and I think this is better.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/owncloud/client/issues/4557#issuecomment-315513642,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAl9jXauPISFtUHhKwFbcvkTwz1Xg4Vrks5sOFq5gaJpZM4HslaE
.

--

Best regards,
Kuba

[CJR2]

Agree with @moscicki, the client should protect the intregity of both the base file and ensure the org does not lose productivity from work completed in a conflicted file but not visible to coworkers and managers.

I've used the dropbox applet that notifies of risk of conflict and 2 things:
1) You can turn off if annoying
2) It is really valuable; and don't find it triggered often. When it is triggered I take it very very seriously.

[moscicki]

Hello,

Any news on this?

kuba

[Agator82]

I agree with @CRJ2, any chance this feature might come available soon, this is the only issue preventing me from moving this onto a production machine.

[ckamm]

@moscicki @Agator82 It's still scheduled for 2.5. If urgent, we might make an experimental, unsupported trial available for 2.4 behind an environment variable flag. (this might be as easy as removing the ignore pattern for conflict files?)

[ckamm]

@moscicki @Agator82 FYI: https://github.com/owncloud/client/pull/6038

[CorneeldH]

Few things from the SURFdrive-community:

• Regarding the earlier mentioned 99.99%. This just ain't the case. We get a lot of angry users who complain that the current setup is 'unworkable'. They share files with colleagues, but their colleagues never receive their changes since the conflict file is only local.

• Making conflict files more visible and syncing them would already big help: Great that those things are in the works!

• Next step would be a 'Dropbox-style'-warning that they are not the only working in a file. Users often don't understand why a conflict-file is made or assume, incorrectly, that they are they only one working in the file.

[PVince81]

I'd also welcome having conflict file handling on the server side. It would open the door to more things than it seems. A while ago we talked about caching files from external storage locally on the OC server for faster access. Such caching could under some conditions run into conflict situations when both the cached version was locally modified and also the remote one. To resolve this, conflict files could be used as well.

For proper conflict resolution, it would be good to have conflict related metadata (webdav properties ?) and UI on the server.

[michaelstingl]

After 2.4.0 release, we will start to publish very early 2.5.0 pre-alpha builds. We could enable upload for conflict files there to get early user feedback…

[PVince81]

Talked with @ogoffart about first step: send headers for PUT upload and on final MOVE on chunked upload.

Header would contain original file name and timestamp.

The server will ignore this for now. We might need to store it somewhere at some point in the future once we start talking about conflict resolution in the web UI.

[ckamm]

@PVince81 Currently my wip code sends:

PUT path/to/base_conflict_testuser-1234.txt
OC-ConflictBasePath: path/to/base.txt
X-OC-Mtime: <mtime of conflict file> (as usual)

Tentative:
OC-ConflictBaseEtag: <etag of base file>
OC-ConflictBaseMtime: <mtime of base file>

It would make a lot of sense to also send the etag of the file version the conflict file is based on. The problem is that this is not always available currently. (the client could introduce a db table for conflicts though and start storing that information, makes sense to me). The same goes for the mtime of the base file version, which I assume is the timestamp you mention.

To make it symmetric and consistent this information about the conflict would need to be sent back to the client when the conflict file is downloaded. Otherwise you can get into a situation where client A creates a conflict, uploads it, client B downloads it, user on client B modifies it, the PUT from client B can't have all this meta info because client B doesn't know about it. (we can do it without that by using filename pattern matching; and the server not being guaranteed to receive the meta info on each PUT for the file)

[ckamm]

@PVince81 @ogoffart Also, I realized that due to the way the metadata on conflicts is stored on the client the following could happen:

• A conflict is created on A/foo, creating and uploading A/foo_conflict-1234
• A/foo is moved away or removed
• The conflict file A/foo_conflict-1234 is modified locally and uploaded again with OC-ConflictBaseFileName: foo.

It would be a serious amount of effort to make this safe against moves. What about we send fileid+etag instead of filename+etag?

[ckamm]

The current status is that the patch is merged and the behavior can be enabled though a server capability.

[julesmoretti]

@ckamm - Having the same issue here, running 2.4.0 build 8836 for mac.

Do you know which release will hopefully cover this fix?

Thank you for your support with this :)

[ckamm]

@julesmoretti The patch is available in 2.5.0 and onwards. This is currently controled by a server capability that has no configuration in the server yet.

[julesmoretti]

@ckamm also do you know this will also handle the There are unresolved conflicts. Click for details.

And upon clicking on details:

Issue
Conflict: Server version downloaded, local copy renamed and not uploaded.

Just to make sure :)

[ckamm]

@julesmoretti Yes, if conflict-uploading is enabled, the message will just speak about an unresolved conflict, and not say that the file was not uploaded. :)

[SamuAlfageme]

I think it's fair to close this one out. Has been broadly tested, improved and after https://github.com/owncloud/client/pull/6441 even documented.

There are also great concepts like https://github.com/owncloud/client/issues/5896 in the making. Someday we clients might as well get the formal concept of _conflicts_ we can use from the server.

Great job @ckamm! 👏

[guruz]

Uploading conflicts (experimental)

https://doc.owncloud.org/desktop/2.5/conflicts.html#uploading-conflicts-experimental

[guruz]

I forgot to paste the download link to 2.5.0 beta1.
Here it is: https://central.owncloud.org/t/desktop-sync-client-2-5-0-beta1-released/14667

[dragotin]

Hm, the documentation lacks an explanation what you can and should expect when enabling that feature. So I doubt many people will test.

[jnweiger]

linux mint tara

testpilotcloud version 2.5.0daily20180808 (build 10016)

Feature must be enabled through

• a server capability 'uploadConflictFiles'
• or through environment variable OWNCLOUD_UPLOAD_CONFLICT_FILES=1

Provoke a conflict by

• wait until sync finished. pause sync.
• modify file in the web gui
• modify file locally
• resume sync.

BAD: replacing the file on the server does not cause a conflict, it explodes: https://github.com/owncloud/client/issues/6701

PUT requests on a conflict should contain
PUT path/to/base_conflict_testuser-1234.txt
OC-ConflictBasePath: path/to/base.txt
X-OC-Mtime: (as usual)

Tentative:
OC-ConflictBaseEtag:
OC-ConflictBaseMtime:

OK: mitmproxy shows

Content-Type:           application/octet-stream
OC-Async:               1
OC-Checksum:            SHA1:4177821dbc55969573e5189bd757579af421a71a
OC-Chunk-Size:          10000000
OC-Conflict:            1
OC-ConflictBaseEtag:    4d9bd6cb0b8d2d090aa09b06dad4944f
OC-ConflictBaseFileId:  04400434oce9akr7u3v9
OC-ConflictBaseMtime:   1533747765
OC-Total-Length:        147
X-OC-Mtime:             1533747765

BAD: OC-ConflictBasePath is 'documented' in the issue, but missing.

Maybe server would do nice versioning for that in the future?
OK: 10.0.9 not yet.

Both original file and conflict file are on the server, and
must be rsynced down to other clients.
OK

Even older conflict files get uploaded once the upload conflicts feature is enabled
OK

BAD: username is missing in the conflict files that were generated before the feature is enabled.

BAD: The tab 'not synced' now talks about a nicely synced (conflict) file.

[jnweiger]

@dragotin people not testing stuff has many reasons. There is documentation, hooray!!!!. It could be worse!!! Actually I didn't read that snippet in the first place and discovered the environment variable in the source...

enhancement request: That behaviour should be made default asap! It is a huge improevement to have the conflict files on the server. Even if the server does not do any extra magic right now. It is a great feature!

[moscicki]

@jnweiger: your test can easily be automated with smashbox and thus effortlessly rerun at will until the end of world. we can have a look at this together.

[dragotin]

@jnweiger in your testcase where you modify both on server and client (to actually create the conflict), which version is the file with the original file name after the whole conflict sync happened?
It is probably still the server version, as it was before?

Maybe that really should be changed now and make it the latest version, ie the one that was modified last? Is that what user expect?

[ogoffart]

BAD: OC-ConflictBasePath is 'documented' in the issue, but missing

Can you please open an issue for that?

BAD: username is missing in the conflict files that were generated before the feature is enabled.

I suppose that's fine.

BAD: The tab 'not synced' now talks about a nicely synced (conflict) file.

Right.... Although it is still nice to have this information there since we want to act upon this.
There is code for this condition that explicitly add the conflict files there even though they are sync'ed.
Maybe a better UI is needed (another tab?)

---

@moscicki : We also have automated testing for this feature here: https://github.com/owncloud/client/blob/master/test/testsyncconflict.cpp But of course, this does not replace real testing on a real server.

[jnweiger]

@dragotin Yes in all cases of conflict tested, the version from the server was kept in place, versions from the local client were renamed.

Assuming nobody edits on the web interface, but all changes come through syncing clients, the client that syncs first would win. Which is the opposite of your suggestion "last client should win".

A strong reason for "last edit wins" would be if we could assume that later authors have seen, considered and rejected (or merged) earlier changes. In our case they cannot see each other until after the conflict.
A weaker reason is its simpler overall logic compared with 'first sync to server wins'.

The timestamp should be the modify timestamp of the file on the client. Not the moment when it got successfully synced to the server. Right?

[jnweiger]

@ogoffart right, those other 'BAD' items are acceptable for me too. Thanks to your review, one of them is now an issue: #6702.

[jnweiger]

Enhancement idea: environment variable OWNCLOUD_UPLOAD_CONFLICT_FILES=1 should be a config file setting.

[zippytiff]

Agreed, make this a config file setting and corresponding gui setting

ZT

[gessel]

There's a great little program called "allwaysync" that has a lovely conflict resolution tool. The key to the program's utility is that the conflict list is provided for in-program resolution. I have a folder with a lot of meta information modifications which generate conflicts with the server. The local files have the latest data, but they were all renamed. Now there's no straight-forward way to, say, select all of these conflicts and specify that the local file wins, please delete the copied down server file, un-rename the local file, and sync back to the server. That is, how about a UI tool that allows straight forward manual selection of which file should win and then executing that directive?

[gessel]

It seems that if you rename the conflict file to overwrite the original, it then takes precedence over the server file and the next sync will overwrite the server without generating a conflict report.. So the problem of deconflicting client precedence is only to bulk rename all the conflict files overwriting the copies downloaded from the sever like:

  find -iname *_conflict-2016* -type f -exec rename -n -f 's/_conflict-2016[0-9]{4}-[0-9]{6}//' {} \; 

as a dry run, then removing the "-n" flag to really do it like:

  find -iname *_conflict-2016* -type f -exec rename -f 's/_conflict-2016[0-9]{4}-[0-9]{6}//' {} \;

This will recurse through directories below where the command is executed, so be warned if some conflicts should be resolved with the server file taking precedence.

Note that all my conflicts seem to have "2016" as the prefix, YMMV on that. If the years vary, something like

 find -iname *_conflict-* -type f -exec rename -n -f 's/_conflict-[0-9]{8}-[0-9]{6}//' {} \;

should do the trick, as long as "_conflict-" isn't used

[gorn]

The worst thing in all this is, that there is really no SANE way to resolve the conflict. The only way which works is to open the server version file and the local one (renamed to conflict file) and copy the contents of the local file into the server one). This might be possible with simple text files, but it is nearly impossible with more complex data, like graphics files, databases, etc. etc.

This should work (and does NOT):

• delete the server file
• rename the local "conflict" file to the original name.

After these asctions synchronization should understand that the user wants to keep local changes and upload them to the server. However it does not work now. It sais, that the server responded that the file is LOCKED. I use client 2.5.1 on Linux Mint

[lowlyocean]

Am I missing an obvious "bulk resolve" to select local versions of all files currently in conflict? My server was down for a while; now all of applications are loading stale data from the server because my fresher local files have been renamed.

[ckamm]

@lowlyocean There's currently no bulk-resolve option. #5896 is an issue about providing more conflict resolution ui help but that's not yet scheduled or even fully thought through. Until then, check out the renaming console commands mentioned above.

The client tries hard to avoid creating unnecessary conflict files. For example it'll only ever do it in situations where the server reports a file change (i.e. the server file's etag differs from the locally recorded etag). If you still have information about the situation that caused conflict files to be created in a "pure backup" situation that'd be very much worth a report.

Optionally limiting client behavior for "pure backup" operation may be a worthwhile discussion to have.

[lowlyocean]

It concerns me that conflict resolution is proposed as a server-side task. I'm running on a raspberry pi and try to avoid the web interface as much as possible because it is slow to load. The more the client can do, the better.

If I'm not using nextCloud for collaboration, I shouldn't see any conflicts. It should be assumed my local copies are the most recent and back them up to the server.

[ckamm]

I agree that conflict resolution is usually a local task, that ticket is about giving users better support for resolving conflicts locally through the sync client.

[lowlyocean]

locally through the sync client Yes, this is good. I should clarify that I was concerned about suggestion in https://github.com/owncloud/client/issues/5896#issuecomment-391609389

Good work so far, I've noticed lots of improvements like reduced time for client scan for changes. Very much looking forward to DeltaSync feature