Client: Conflict file creation on bad network connections

[jancborchardt]

I just had a few days of bad connection and as a results a few conflict files of notes popped up. (During the time I basically exclusively edited stuff locally on my laptop so I wonder why there are conflict files, but I might be wrong.)

Anyway – I noticed a pattern that in basically all cases the _conflict files are the newer and correct ones. It seems that the algorithm assumes the server has the »right« version.

Instead I would propose that the latest / last modified file should get the »clean« name and the older one should get the _conflict suffix.

What do you think @dragotin @danimo @rullzer?

[dragotin]

No. I know its debatable, but we always had the policy _server wins_. I am not seeing that we are going to change that now.

[jturcotte]

Let's say you have userA and userB:

• userA modifies fileA (version 1) and sync, fileA(v1) is on the server
• userB modifies fileA meanwhile (version 2), and then syncs, and get a conflict. fileA(v1) is renamed to fileA_conflict and fileA(v2) is uploaded
• userA sync, and his file gets overwritten automatically on his local station. fileA(v1) is lost without anybody doing any conflict resolution. It's not really lost, but it's stored on userB's computer as fileA_conflict and if he doesn't notice the file is not available for userA to recover.

On the other hand, if the local file (the last one trying to reach the server) is the one renamed to _conflict like we currently do, this means that a human is forced to decide which file gets chosen as the right one, possibly integrating both changes in the result. I also think that it's better to let the server "win" to force this process to happen without losing any data.

[jturcotte]

The weird thing I think is that you get conflict files and that you don't care about what's on the server. If anybody has been updating those files, you shouldn't just throw away their modifications, both files would be "correct".

If there wasn't any modification, maybe the solution to your problem would be to make sure that we create conflict files only if there is conflicting data.

[jancborchardt]

If there wasn't any modification, maybe the solution to your problem would be to make sure that we create conflict files only if there is conflicting data.

Yeah, I guess that’s the main issue. Unfortunately I don’t know why there were conflict files to begin with. It was with 5 notes text files which I definitely only edited on my laptop. And the internet connection was very bad. So not sure how to help debug this further.

[guruz]

If there wasn't any modification, maybe the solution to your problem would be to make sure that we create conflict files only if there is conflicting data.

We check the file byte by byte.

@jancborchardt Would be good to see a diff of the files to reconstruct what happened.

[jturcotte]

If you're sure that no other computer could change this file on the server, it could be some situation that the file was uploaded on the server, but that due to the bad connection the resulting server etag wasn't saved in your laptop's client database (either the server kept the state before the PUT response was properly sent to the client and then the connection was cut, or the client received the PUT response and didn't keep the resulting etag because of a sync abort).

disclaimer: I don't know that code so well, just throwing in possible causes

[jancborchardt]

@guruz I diffed them and the »normal name« file was clearly the old one, and the _conflict file had all my additions.

@jturcotte yeah, pretty sure – I didn’t use the web interface and only have my Android connected to it. Neither online nor on my Android do I edit these text files.

[jturcotte]

@jancborchardt Have you been hitting the same issue again, that conflict files get created even though you're the only one working on it?

If it's the case above, I'm not sure what we could do, we usually use checksums to look if the local disk has the same file as the one on the server, but we'll only be able to verify that if the state that was uploaded on the server and not committed in the client database is the same as on the disk, and I guess that in your case you continued editing the file so the server state doesn't match anything in the present.

We could maybe store a history of propagated modtime+size+checksums and try to match the server state to this before tagging it as a conflict, but this could make the database grow quite a bit.

cc @ogoffart

[jancborchardt]

@jturcotte yup, in the case I reported at https://github.com/owncloud/client/issues/5965 it was definitely only me working on it. Might have been in a federated share, but no one else worked on the file in the meantime for sure.

Sure yeah, it would be great if the conflict file does not get created at all too. :)

[jturcotte]

Let's reopen it to track the case where conflict files shouldn't be created for past uploads of the same machine.

[jturcotte]

We need to confirm that this does happen with ownCloud servers as well, as this could be a server issue.

[ogoffart]

Here is what I think happens:

You are editing a file, the file is uploaded to the server. But the connection drops before we get a reply from the server with the new etag telling the file was properly uploaded (Byzantine failure). So the client cannot save the new etag in the DB. But for the server everything is fine.
You continue to edit the file and change the content of it.
Now on the next sync, the client sees the new etag from the server and things that's a new file on the server. So we got a conflict.

Solution: If it is a chunked upload, we would have an entry if the upload info table with the mtime and size of the previous file. We can compare the server file against that and "resolve the conflict". For smaller files unfortunately we have no way to know. I guess then we could add an entry for them in the upload info table anyway for this reason.

[floledermann]

I just wanted to chime in that this has been reported to happen both on slow/unreliable network connections and on very fast connections (i.e. gigabit ethernet local LAN, see some of the discussion in #3276)

[floledermann]

So do I read this correctly and a fix for this is in master, but will only be released with 2.5.0 (or later?)

[ogoffart]

@floledermann yes, 2.5.0. This patch is big and intrusive. Considering the issue has always been present, the fix won't be backported.

[floledermann]

@ogoffart is there a nightly build for Windows somewhere available of the 2.5.0 branch so I could test this in our institution? This has prevented adoption of OwnCloud so far, so if this would fix our issues this would be big news, but 2.5.0 seems to be a while away still...

[guruz]

@floledermann https://download.owncloud.com/desktop/daily/
Use at your own risk :-)
And please report if this works and improves your situation.

[ckamm]

Looks good to me