Client: Owncloud overwrites local files with outdated ones from the server

[byroncoetsee]

Somehow Owncloud decided (for the second time) that files on the server dated 20 days ago are more important than the current local ones and overwrote them. How can owncloud not notify the user? Maybe provide a way of recovering the files? Anything at all???

As a developer myself I'm both astounded and appalled at the fact that nothing was mentioned to the user of this ridiculous event.

[danimo]

Somehow Owncloud decided (for the second time) that files on the server dated 20 days ago are more important than the current local ones and overwrote them.

Sorry to hear that. Of course that should not happen. The most recently changed file should _always_ take precedence.

Maybe provide a way of recovering the files? Anything at all???

On the server there is the trash app that will allow to recover your file (assuming its enabled)

How can owncloud not notify the user?

Probably because there is a bug, either in the client or the data as reported by the server. We need more information to track this down, as this is most certainly not normal behavior.

Please provide details as per https://raw.githubusercontent.com/owncloud/client/master/issue_template.md.

[byroncoetsee]

Thanks for the response (pretty damn quick too!). I checked the trash but unfortunately seeing as the files never even got that far (server side), they aren't there.

I have since used a feature I was not aware of in OSX. Opening things in TextEdit and using File > Revert To, I was able to revert most of the changes back to their previous version... Maybe keep that in mind in case someone else falls victim to this rather potent godzilla-bug.

Expected behaviour

I would expect my local files to be pushed up to the server...

Actual behaviour

Server files overwrote local ones with no prompt. Set me back 20 days worth of development.

Steps to reproduce

Unfortunately, I don't know how to repro. The UI in general is already borderline useless in terms of being able to modify settings so nothing out of the ordinary was changed or set.

Server configuration

Operating system: linux

Web server: Apache 2.2.25

Database: MySQL 5.5.32-cll-lve

PHP version: 5.3.26

ownCloud version: 8.0.2 (Just upgrade to 8.0.3 now)

Storage backend:

Client configuration

Client version: Latest

Operating system: OSX Yosemite

OS language:

Installation path of client:

Logs

1. Logs are empty after restart.
2. Web server error log:
3. ownCloud log (data/owncloud.log):

Error PHP Only variables should be passed by reference at /home/switched/public_html/owncloud/lib/private/cache/fileglobalgc.php#50 2015-05-25T10:51:43+00:00
Error PHP Only variables should be passed by reference at /home/switched/public_html/owncloud/lib/private/cache/fileglobalgc.php#50 2015-05-25T08:27:24+00:00
Warning core Login failed: 'byroncoetsee' (Remote IP: '41.164.21.70', X-Forwarded-For: '') 2015-05-25T07:57:59+00:00
Error PHP Only variables should be passed by reference at /home/switched/public_html/owncloud/lib/private/cache/fileglobalgc.php#50 2015-05-06T13:16:00+00:00
Error PHP Only variables should be passed by reference at /home/switched/public_html/owncloud/lib/private/cache/fileglobalgc.php#50 2015-05-01T08:26:46+00:00
Error core unable to rename, file does not exists : files_versions/Swift.v 2015-04-30T11:28:19+00:00
Error PHP Undefined index: extension at /home/switched/public_html/owncloud/apps/files_versions/lib/storage.php#320 2015-04-30T11:28:18+00:00
Error PHP Only variables should be passed by reference at /home/switched/public_html/owncloud/lib/private/cache/fileglobalgc.php#50 2015-04-30T11:25:38+00:00
Error PHP Class 'Zend_Search_Lucene_Document' not found at /home/switched/public_html/owncloud/apps/search_lucene/lib/indexer.php#71 2015-04-30T11:24:35+00:00
Fatal index Exception: {"Message":"","Code":0,"Trace":"#0 \/home\/switched\/public_html\/owncloud\/lib\/private\/app.php(77): OC_App::loadApp('activity')\n#1 \/home\/switched\/public_html\/owncloud\/lib\/private\/util.php(75): OC_App::loadApps(Array)\n#2 \/home\/switched\/public_html\/owncloud\/lib\/base.php(766): OC_Util::setupFS()\n#3 \/home\/switched\/public_html\/owncloud\/index.php(36): OC::handleRequest()\n#4 {main}","File":"\/home\/switched\/public_html\/owncloud\/lib\/private\/app.php","Line":96} 2015-04-30T11:23:12+00:00
Error PHP Call to undefined method OC_Helper::findBinaryPath() at /home/switched/public_html/owncloud/lib/private/preview.php#813 2015-04-30T09:09:01+00:00
Error PHP Call to undefined method OC_Helper::findBinaryPath() at /home/switched/public_html/owncloud/lib/private/preview.php#813 2015-04-30T09:08:56+00:00
Error PHP exec() [function.exec]: Unable to fork [command -v 'ffmpeg' 2> /dev/null] at /home/switched/public_html/owncloud/lib/private/preview/movies.php#12 2015-04-30T08:57:22+00:00

[byroncoetsee]

As a sidenote, I got duplicate files showing up, but with "conflict" appended to the name along with some numbers... eg:

Route.swift
Route_conflict-20150506-115010.swift

[byroncoetsee]

...got own'd by owncloud

[ogoffart]

This looks similar to https://github.com/owncloud/client/issues/2325 in which backup are restored on the server, and then the client believe that the restored files are the newer because the server claim it has changed. (We have no way to know that a backup was restored)

[byroncoetsee]

Even if the files on the server show their "Last Edited" time as 20 days ago?
Surely the comparison that happens to determine which is newer is done using the same date/time that the "Last Edited" is using for its calculation...? If so, restoring backups should not cause this issue as the edit date is carried over.

[ogoffart]

Surely the comparison that happens to determine which is newer is done using the same date/time that the "Last Edited" is using for its calculation...

No, we rely on the server to tell us that there is a new file. We can't rely on last edited for a lot of reasons (timezone problems, people uploading old files on purpose, clock not synchronized, ...)

[guruz]

Closing this for now until it's clear that this is an oC bug

[mortee]

This issue seems to be left quiet for some time. I'd like to indicate that I think the current behavior is faulty, too. Maybe I should open a new issue?

My setup is one windows desktop client, and one server which mostly serves as an automated backup (and primitive versioning) for my work files. This also lets me easily migrate my work stuff to another desktop if necessary - but this happens very rarely, and most of the time the _only_ client that accesses the server is my desktop.

Actual behaviour

Given this setup, I'd expect to _never_ experience conflicts. What happens instead is that in a _lot_ of cases, a conflict arises from reasons involving mostly "Not allowed to upload this file because it is read-only on the server, restoring". This occurs mostly in Eclipse project folders, especially with .classpath files (dunno why), which revert my developments in unpredictable ways.

While I understand that OC is primarily a collaborative tool, I think the below guidelines should be seriously considered, to prevent loss of data/work:

Expected behaviour

When the local computer is the _only_ source of updates on the server, then the client should never, ever overwrite any local file from the server, in any case whatsoever. How this circumstance can be verified is a separate topic - but I think this outcome should be mandatory.

When there's a conflict, then the local file should be left unchanged, the server version downloaded into a tagged filename, and the user _notified_.

Especially if the conflict is caused by a failure to sync, like when the client thinks the file is read-only on the server.

(How can a file become read-only on the server, when it's uploaded from a local, non-read-only file, and there are no other sources of updates on the server?)

The local file should never be overwritten/reverted if it's been changed locally since the last sync. Even in a collaborative environment this implies "I know you've been working on this file, but the server version is obviously more important than your work, so I set yours aside".

Actually, it should _only_ be overwritten from the server when the local version has already been successfully synced/uploaded, it has been changed on the server since then, _and_ the local file does _not_ have local changes. If it does, it should be treated as a conflict, and handled as written above.

Steps to reproduce

I _think_ the only-client-and-still-conflicting behavior might be triggered by a large shared folder which takes long to even discover, and local changes which happen on a much faster pace than the client can keep up.

So I'd reproduce it by setting up a server with only one client accessing it, sharing a big folder with lots of files, putting an Eclipse workspace inside, starting to do work on it, and then watching the project settings and possibly source files being reverted.

Server configuration

Operating system: NetBSD
Web server: Apache 2.2.31
Database: MySQL 5.5.45
PHP version: 5.4.45
ownCloud version: 8.1.3 (stable)

Client configuration

Client version: 2.0.2
Operating system: Windows 8.1

[floledermann]

Experiencing this regularly with Windows client Version 2.1.1 (most recent one) when only editing office files on one single computer. Absolutely ridiculous to have a dataloss bug like this just casually closed "until it's clear that this is an oC bug", all development should be halted until this is researched properly.

[mortee]

totally agreed. the issue is reopened here BTW: #4222

[dragotin]

I am not yet convinced that this issue and #4222 have the same root cause.

@floledermann what of the above in the reproduction description said can you confirm? Do you also have a huge directory in which that happens? Is the probability proportional to the number of files in the directory? Does it only happen with MS Office, or also other programs operating on the files?

If you can reproduce the problem, please create a client log file and pass it by so that we can see what is happening.

[floledermann]

On further investigation I am not sure that I am affected by the same behavior than the OP.

Here's what happens in my case: I sometimes have a file open the whole day while working on it (say in Word or PowerPoint). During that time, I save the file multiple times (obviously, to avoid a data loss in case of a crash). I believe what happens is that the OwnCloud client notices a change of the file, and syncs to the server, but it can't overwrite the local file because it is still opened in Office. Or I may even have saved another time in the meantime.

So at the end of the day, I save my file and close the program, at which point the new version is renamed to "*_conflict-...", and the old version from the start of the day is restored at the original filename. Very confusing and not very trust-inducing, but at least I haven't experienced anything non-recoverable yet. (This is all a theory, but consistent with my experiences)

As I said, I am not sure if this is what the OP experienced or if this is a separate bug.

[guruz]

@floledermann For the word file: Is that doc or docx?
There should not be a reason for ownCloud to write to your local file, it just needs to read it and upload it to the server.
There were no concurrent changes by you (or someone else) to the file that are synced to the server?
Which server version are you on?

[floledermann]

In this case it is a .docx but it also happens with .pptx or even with text-based fieles (.js etc.). As I said this happens practically every time I have a file open for a longer time and do multiple saves, but never close the file.

I can guarantee that no modifications were made on the server or other machines. I have owncloud configured on several computers, but not of these were even running at the time.

Server version is 8.1.6 (as provided by my university, I wouldn't be able to upgrade that myself)

[hodyroff]

Is this still happening with the newest server version and newest desktop client?

[floledermann]

I have largely stopped using OwnCloud because of this bug, so I cannot contribute any recent experience. Also the server is controlled by central IT services at my university, so I would't be able to update that.

If I notice this behavior again, I will notify here.

[dragotin]

Between April and today I remember we fixed a couple of issues around MS Office files on Windows. So it would be absolutely worth to check with a recent client. Windows has several tricky ways to lock files when editing ;-)

Closing this one for now, please comment in case and we reopen.

[floledermann]

Again, this approach of just closing a critical dataloss bug "for now" without verification doesn't increase my trust in the product. I pointed out clearly that this is not only affecting MS office files, but also text files open in a text editor.

[dragotin]

Sure, I am happy to reopen again, but than it would be very beneficial if you would be able to provide debug information. It is not the case that these things do not work for everybody, it seems the problem you're seeing is specific to your installation somehow, which is certainly possible.

[mortee]

I'm baffled by this approach to just close a critical bug like that. Unfortunately I can't verify if it's gone away currently either, because my server broke down for unrelated reasons. But anyway, no, the problem is NOT specific to floledermann's circumstances, as several people (including me) have reported the issue, and even several duplicate tickets have been opened just because of this reason.

And, god forbid, no, it definitely shouldn't be dismissed by "fixing a couple of issues around MS Office files". This is a freaking data loss bug, and I think it should have a substantial and specific fix, which guarantees that files are not overwritten by "server versions" (actually older versions), even when there are no other clients accessing the same files. If this can happen because of some tricky locking mechanisms (apparently, with very simple text editor software, too), then apparently OC's way to determine what's newer is flawed.

Pretty bad for a data backup and collaboration tool.

[guruz]

We need more data, e.g. logs on client and server.

There is hundreds of thousands of oC users that did not report an issue yet, so it's hard to find out what is wrong in your situation or installation.

[guaguanco]

@guruz I'm plagued by this bug since my institution adoption of ownCloud, with OS X bundle files is just a nightmare. I'm trying to convince our sysadmins to get in touch with you to provide you more data.

To give you a feeling of what we are dealing with below there's the result of a typical working session on ownCloud share, and I'm using just a text editor (it's dated because it's from an open ticket with our sysadmins):

./Teaching/ILS/Exams/2016-09-15/ex_abstract_nop_conflict-20160912-131449.tex
./Teaching/ILS/Exams/2016-09-15/ex_abstract_nop_conflict-20160912-134542.tex
./Teaching/ILS/Exams/2016-09-15/ex_pbo_conflict-20160912-175654.tex
./Teaching/ILS/Exams/2016-09-15/ex_pbo_conflict-20160912-180216.tex
./Teaching/ILS/Exams/2016-09-15/ex_pbo_conflict-20160912-180338.tex
./Teaching/ILS/Exams/2016-09-15/ex_pbo_conflict-20160912-180808.tex
./Teaching/ILS/Exams/2016-09-15/ex_pbo_conflict-20160912-181031.tex
./Teaching/ILS/Exams/2016-09-15/ILS_exam_2016-09-15_full_conflict-20160912-125552.pdf
./Teaching/ILS/Exams/2016-09-15/ILS_exam_2016-09-15_full_conflict-20160912-173712.pdf
./Teaching/ILS/Exams/2016-09-15/ILS_exam_2016-09-15_full_conflict-20160912-173849.pdf
./Teaching/ILS/Exams/2016-09-15/ILS_exam_2016-09-15_full_conflict-20160912-174527.pdf
./Teaching/ILS/Exams/2016-09-15/ILS_exam_2016-09-15_full_conflict-20160912-175237.pdf
./Teaching/ILS/Exams/2016-09-15/ILS_exam_2016-09-15_full_conflict-20160912-175425.pdf
./Teaching/ILS/Exams/2016-09-15/ILS_exam_2016-09-15_full_conflict-20160912-175734.pdf
./Teaching/ILS/Exams/2016-09-15/ILS_exam_2016-09-15_full_conflict-20160912-175952.pdf

If this can help, I noticed that the faster is the network the worse it gets.

[floledermann]

If this can help, I noticed that the faster is the network the worse it gets.

Interesting observation. This could explain why I am experiencing this also more often at work (>1Gbps connection to server) than when working from home. Is something inside the ownCloud logic working on timestamps on a second or even millisecond granularity?

[dragotin]

@guaguanco do you see that only happening on a share, meaning a shared directory that is owned by somebody else, or also on a directory that you own? That might be worse a check.

Maybe there is a race in how the server handles sharing info somehow?

[floledermann]

In my setup none of the folders this is occurring in is shared (local folders on my disk owned by me).

[guaguanco]

sorry @dragotin, I used "share" inappropriately. Mostly, I get conflicts on folders that are not shared with other users.

--sergio

[scorretti]

Hi there. I have exactly the same issue working with Linux and Windows, and with different software. For an unknown reason, OwnCloud client overwrites some files while I'm editing them. With some software I had a warning about this, with other ones (among others, Matlab) I can observe the file I'm editing suddenly change, and my last modifications are lost (or in the conflict file).

I think that this is definitely a severe bug. At present time, my only work-around is to pause synchronization and start it again when I make a pause in my work. I suggest to introduce a minimal delay condition for synchronising the files, for instance \Delta T > 5 minutes, or something like that.

If it can help, I can provide more information (which one ?), conflict files, or even install and make run a debug-version of the client on my PC (Ubuntu 16.04 LTS).

Riccardo

[SamuAlfageme]

@scorretti sure! Thanks for your help trying to find the cause of the issue here, there's some stuff you can provide to improve the chances of finding the issue:

• Client logs
• Examples of what programs cause the issues (and if they create temporary/intermediate files while you work)
• Examples on some dummy file causing issues (maybe the software locks it to avoid others editing/reading and the user running the client's process does not have permissions on it...)

Thanks again 😊

[scorretti]

I'm
      sorry for eventual copies of this mail : my previous mail come
      back (perheaps due to the attached files), so I send it again.
---------
Thank
      you for your fast answer !

      I'm experiencing this problem mostly with two text editors : matlab's
      text editor and textstudio. In both cases I need to
      modify very often (and save) my work.

      In order to reproduce the problem, I wrote  a small program
      with matlab to simulate this frequent activity (let's say,
      every 10 seconds) :

    for k = 1 : 30
       % open the file, add a new line, and close it again
       fid = fopen('test.txt', 'a+');
       fprintf(fid, '%s : %i\n', char(datetime('now')), k);
       fclose(fid);

     % wait 10 seconds
     pause(10);                     
    end

      I got 4 conflicts in 5 minutes. The present file clearly shows
      that there is a problem, because some lines are missing :

    12-Apr-2017 14:50:03 : 1
    12-Apr-2017 14:50:33 : 4
    12-Apr-2017 14:50:43 : 5
    12-Apr-2017 14:50:53 : 6
    12-Apr-2017 14:51:03 : 7
    12-Apr-2017 14:51:13 : 8
    12-Apr-2017 14:51:23 : 9
    12-Apr-2017 14:51:33 : 10
    12-Apr-2017 14:51:43 : 11
    12-Apr-2017 14:51:53 : 12
    12-Apr-2017 14:52:03 : 13
    12-Apr-2017 14:52:33 : 16
    12-Apr-2017 14:52:43 : 17
    12-Apr-2017 14:52:53 : 18
    12-Apr-2017 14:53:03 : 19
    12-Apr-2017 14:53:13 : 20
    12-Apr-2017 14:53:23 : 21
    12-Apr-2017 14:53:33 : 22
    12-Apr-2017 14:53:43 : 23
    12-Apr-2017 14:53:53 : 24

      I join you the file in its present version, the conflict files
      and the log which I could save. I hope this will help.
      Thanks a lot.

      Riccardo




  J'ai lié 6 fichiers à ce message
    :

    log-owncloud.txt(3,9
        Mo)Boxhttps://app.box.com/s/zf9skechuh1pp0u47au3t91y9dv4thco
    test_conflict-20170412-144709.txt(8,0
        Ko)Boxhttps://app.box.com/s/w7scokd8zl2rchi8mchb658i8x4bil2q
    test_conflict-20170412-145012.txt(75
        octets)Boxhttps://app.box.com/s/y9i4rxsrbix2vq8lsbutwbmyvto3p14c
    test_conflict-20170412-145211.txt(331
        octets)Boxhttps://app.box.com/s/prk0er7d0u7wxzquyy3pxsxqpkqrrtq3
    test_conflict-20170412-145402.txt(669
        octets)Boxhttps://app.box.com/s/t7uulml6phb9q2aaibe375qcteh9eted
    test.txt(513
        octets)Boxhttps://app.box.com/s/vnvqu0p3omg4jp74exsmp0kw8j3qttjm

  Mozilla Thunderbird
    permet de partager facilement des fichiers volumineux.

Le 12/04/2017 à 12:19, Samuel Alfageme
  a écrit :


  @scorretti sure! Thanks for your help
    trying to find the cause of the issue here, there's some stuff
    you can provide to improve the chances of finding the issue:

    Client
        logs
    Examples of what programs cause the issues (and if they
      create temporary/intermediate files while you work)
    Examples on some dummy file causing issues (maybe the
      software locks it to avoid others editing/reading and the user
      running the client's process does not have permissions on
      it...)

  Thanks again 😊
  —
    You are receiving this because you were mentioned.
    Reply to this email directly, view
      it on GitHub, or mute
      the thread.







  {"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/owncloud/client","title":"owncloud/client","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/owncloud/client"}},"updates":{"snippets":[{"icon":"PERSON","message":"@SamuAlfageme in #3276: @scorretti sure! Thanks for your help trying to find the cause of the issue here, there's some stuff you can provide to improve the chances of finding the issue:\r\n\r\n- [Client logs](https://doc.owncloud.org/desktop/2.3/troubleshooting.html#log-files)\r\n- Examples of what programs cause the issues (and if they create temporary/intermediate files while you work) \r\n- Examples on some dummy file causing issues (maybe the software locks it to avoid others editing/reading and the user running the client's process does not have permissions on it...)\r\n\r\nThanks again 😊\r\n"}],"action":{"name":"View Issue","url":"https://github.com/owncloud/client/issues/3276#issuecomment-293535230"}}}


--

Riccardo Scorretti, PhD

CNRS Researcher - Bioengineering department.
University of Lyon, Lab. Ampère - UMR 5005 CNRS,
phone: [phone] - ([phone]
fax: [phone]
http://ampere-lyon.fr

[ogoffart]

@scorretti : Your links to the log does not seem to be valid.

I'm closing this issue because since a lot of time passed and a lot of changes were made since the bug was open. The client now detect backup restoration, checks were added so that modified files does not get overwritten.
Several people added comment which may or may not be the same bug. But to us developer this is not useful to keep this "bug" open.

Should there still be a problem with the latest version, please open a new issue with enough details.
You can also ask help through support or on the forum. See https://owncloud.org/support/

[exander77]

I don't want to swear, but how the fuck is not this issue fixed? My KeePass file just got reverted at least 10 days. This cannot happen.

[mortee]

Totally second that. Users (myself included) are tirelessly complaining about this issue for YEARS, and still no solution.

[exander77]

I am really baffled by this because I am not even sharing the storage with other users, I am just using it for backups from a single computer, so there should be no conflicts whatsoever and server should have either the older version or current version of the file. There should be no case when the local file is overwritten from the server, yet it happens.

Please at least introduce option: Don't fuck with my local files. Ever.
This option would prohibit Owncloud Client to overwrite local files and would consider local files either the current or newer version for all cases.

[mortee]

Yes, this used to be my situation, too. For quite a long time, I'm not using OC exactly because of this reason.

[hodyroff]

Sorry to hear this @exander77 can you please provide some information about your setup. Newest client 2.3.3? Can it be reproduced? ownCloud is used by millions of users worldwide and we happily look after this one when we get more information. I would suggest to open a new issue with the details but we will need logfiles from both server and client side from the time this occurs.
@mortee A lot of comments from you were on the conflict files, this is somewhat different from a pure overwrite and the currently designed behaviour, I understand that you like to have it different. Most recent consideration is to also upload all conflict files into the versioning system server side which would allow to recover any version.
I understand that there can be different use cases, but server is master is an important design choice and while we protect your data client side, this choice is pretty important for the use cases we mostly have.

[mortee]

I think we all get this "server is master" thing. But please understand that our problem with the behaviour described so many times is that in the case where there ever was one single client, the source and sole consumer of any data on the server, then regardless any theoretical and conceptional design choices about "server is master", there should be no situation _ever_ which in any way results in the server assuming to be smarter than that client, and overwriting any changes on it. That simply shouldn't happen, still respecting the "server is master" idea to its fullest. If that can happen in any case, then something's wrong with the synchronisation algorithm.

Anyway, sure, I haven't experimented with this recently. All I know is that this issue is rolling along for several years now, with several people complaining (and one can be certain that there are orders of magnitudes more users with the same problem who don't reach out here), and we never received a clear and unambiguous statement from any developer acknowledging the problem, and pledging to fix it, not to mention an explicit announcement informing us that it has been doubtlessly identified and fixed.

[jturcotte]

The unneeded conflict files part could be related to #5106.