Android: Sync conflict

[Queeq]

Steps to reproduce

Change nothing in existing auto-upload settings. Make sure new files are created in the folder configured for auto-upload.
or
Attempt manual file upload (files do not exist on the server)

I suspect this is started after the latest (3.11.0) update.

Expected behaviour

Files are uploaded

Actual behaviour

Files are not uploaded.

Notification: "File upload conflict. Pick which version to keep of \

In the app, in the "Uploads" screen, files are in the "Failed/pending restart" category with the message: "Sync conflict, please resolve manually". Pressing "Resolve conflict" does nothing. Pressing "Delete" does not delete anything, but attempts to upload are stopped.

Server configuration detail

Operating system: Linux 4.4.110 #62 SMP Fri Jan 5 20:38:04 EET 2018 x86_64

Webserver: nginx/1.10.3 (fpm-fcgi)

Database: mysql 10.0.38

PHP version:

7.0.33-0ubuntu0.16.04.12
Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, cgi-fcgi, mysqlnd, PDO, xml, apcu, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, iconv, igbinary, imagick, imap, intl, json, exif, mcrypt, mysqli, pdo_mysql, pdo_pgsql, pdo_sqlite, pgsql, Phar, posix, readline, redis, shmop, SimpleXML, sockets, sqlite3, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Zend OPcache

Nextcloud version: 15.0.14 - 15.0.14.1

Updated from an older Nextcloud/ownCloud or fresh install: Updated

Where did you install Nextcloud from: I don't remember

List of activated apps

Enabled:
 - accessibility: 1.1.0
 - activity: 2.8.2
 - cloud_federation_api: 0.1.0
 - comments: 1.5.0
 - dav: 1.8.2
 - federatedfilesharing: 1.5.0
 - federation: 1.5.0
 - files: 1.10.0
 - files_pdfviewer: 1.4.0
 - files_sharing: 1.7.0
 - files_texteditor: 2.7.0
 - files_trashbin: 1.5.0
 - files_versions: 1.8.0
 - files_videoplayer: 1.4.0
 - firstrunwizard: 2.4.0
 - gallery: 18.2.0
 - issuetemplate: 0.6.0
 - logreader: 2.0.0
 - lookup_server_connector: 1.3.0
 - nextcloud_announcements: 1.4.0
 - notifications: 2.3.0
 - oauth2: 1.3.0
 - password_policy: 1.5.0
 - provisioning_api: 1.5.0
 - serverinfo: 1.5.0
 - sharebymail: 1.5.0
 - support: 1.0.0
 - survey_client: 1.3.0
 - systemtags: 1.5.0
 - theming: 1.6.0
 - twofactor_backupcodes: 1.4.1
 - updatenotification: 1.5.0
 - workflowengine: 1.5.0
Disabled:
 - admin_audit
 - bruteforcesettings
 - encryption
 - files_external
 - user_ldap

Configuration (config/config.php)

{
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "dbtype": "mysql",
    "version": "15.0.14.1",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbtableprefix": "oc_",
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "forcessl": true,
    "loglevel": 1,
    "theme": "",
    "maintenance": false,
    "memcache.local": "\\OC\\Memcache\\APCu",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 6379
    },
    "trusted_domains": [
        "***REMOVED SENSITIVE VALUE***"
    ],
    "secret": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpmode": "smtp",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "25",
    "trashbin_retention_obligation": "auto",
    "updatechecker": false,
    "htaccess.RewriteBase": "\/",
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "updater.release.channel": "stable",
    "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***",
    "mysql.utf8mb4": true
}

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Android version: 9
Device model: SM-G930F
Stock or customized system: LineageOS
Nextcloud app version: 3.11.0

Logs

Web server error log

2020/03/27 22:20:48 [info] 64263#64263: *1162408 client <IP> closed keepalive connection
2020/03/27 22:20:48 [info] 64263#64263: *1162406 client <IP> closed keepalive connection
2020/03/27 22:21:27 [info] 64263#64263: *1162427 client <IP> closed keepalive connection

Nextcloud log

Empty

Web server access log

<IP> - phone [27/Mar/2020:22:43:25 +0200] "HEAD /remote.php/webdav/IMG_20200327_163254.jpg HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.11.0"
<IP> - phone [27/Mar/2020:22:43:25 +0200] "HEAD /apps/files/ HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.11.0"
<IP> - phone [27/Mar/2020:22:43:25 +0200] "HEAD /remote.php/webdav/IMG_20200327_163714.jpg HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.11.0"
<IP> - phone [27/Mar/2020:22:43:25 +0200] "HEAD /apps/files/ HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.11.0"

[Queeq]

Following advice in #3635, pinging @tobiasKaminsky and @ArisuOngaku

[zero-24]

It happen to me as well it might be a good idea to add a switch to turn this check off per auto upload rule? Or having a global option to turn that check off.

[Queeq]

Also I tried to roll back to Nextcloud Android client 3.10.1 and it started to work, so the problem is definitely introduced in 3.11.0.

[tobiasKaminsky]

What kind of files are those?
Auto upload is expected to work fine with files that not have the same file name, e.g. IMG-2020-03-30-090910.jpg, but not with e.g. backup1.sql

[zero-24]

Yes for me it is the WhatsApp Backup. WhatsApp creates two files the backup with date and time attaced (that uploads fine) and the latest version msgstore.db.crypt12 as well as some setting files that always have the same name and should be overwritten with the latest version but now create this conflict message.

I agree that in the most cases the message is fine but therefore i suggested to make it possible to switch the check off on an per auto upload rule basis.

[tobiasKaminsky]

Thanks for info.

@ArisuOngaku seems, we really need a default policy for auto upload folders.
Can you do this? ❤️
If not, I will schedule some time for it.

[GitOkkie]

I've got this problem with the log file of an app I use. Until now, the new one just overwrites the old one. Since this week, I get this message every day after using this app. Unfortunately the message keeps popping up, even after switching on "Also upload existing files".

The file name is "aCar.log" and this is my config for that directory:

[dbrand666]

We don't need an option. We just need to fix the conflict resolution. If the file has only changed on one end, it's fine to overwrite the one that hasn't changed. The only time it should report a conflict is if the file changed on both sides since the last sync.

[ArisuOngaku]

@tobiasKaminsky I'll do it! :)

@dbrand666 what if the local version is not supposed to change and it changes for a user-unexpected reason? Overwriting the remote file means loss of data.
This feature is called "auto upload", not "file/folder sync", so this would be misleading.
If there is no existing issue about this, you may want to open one as this is, imho, a design suggestion.

[ArisuOngaku]

@tobiasKaminsky My apologies, I widely underestimated the amount of work it'll take to develop this properly; I unfortunately can't do this in the near future. Sorry for the inconvenience.

[GitOkkie]

I don't know what the status of this bug is now. What I do know is that I get 10 to 20 alerts every single day for manual conflict resolution of this logfile. This was never a problem, but it is now.

I still want to upload this directory since it also conains timestamped backup files. I don't really need any smart sync capabilitiy, though I understand the new feature. It's just that it doesn't work for me. As you can see above, I've got it set to "Also upload existing files", yet it still wants me to decide if I want to overwrite the old one. 20 times per day.

Should I file a seperate bug for this problem?

[Queeq]

I agree with @dbrand666, even if it's "Auto upload", it's reasonable to expect that if a file is modified locally, it would be automatically uploaded again. The user should be fully aware what he uploads.

Overall, although slightly offtopic, why was it decided to do it like this? I guess the majority of people would expect mobile client to behave in the same way as desktop one, and this difference is somewhat confusing.

[tuedelue]

I can confirm these issues - I get lots auf sync conflicts for my auto uploaded pictures. Probably google photos is changing timestamps or whatever.
Regardless, I would expect the client to resolve these issues automatically - since the pics on the phone have newer timestamps, it should simply upload them again/overwrite the old file.
The workaround resolving all conflicts manually helps a couple of days, yet the same pictures are producing conflicts again in future.
Edit: I will check out the new upload existing option. Does it help?!

[jo-me]

Same issue here. I rolled back to 3.10.1 and autoupload is working again...

[ArisuOngaku]

--> #5810 should help with (if not fix) the issue

btw related/duplicate issue #5599

[piknew]

This issue is closed, however... Sync conflict which is discussed in #5810 seems to be for existing remote file. Strange, as I am getting sync conflict when auto upload tries to push file to server when remote file does not exist yet. Conflict with non-existing file?

[piknew]

Not fixed with #5810.

[snowtr]

This is not fixed. I can confirm that it's still an issue in 3.11.1

[Queeq]

Could this issue be reopened? I also still experience problems with 3.11.1.

[zero-24]

@snowtr @Queeq did you checked the settings from the auto upload? Per default the new behavior is enabled and you have to explizit configure the 'overwrite' behavior. That works great for me since the update.

[Queeq]

@zero-24 Yes, I did check and tried different setting combinations. It refuses to upload even newly created files, stating that there's a conflict and I should resolve it manually. On the Uploads screen I can't resolve it because it constantly retries the upload.

Could it happen because the files are on the external SD card? I know that from Android 9 (or maybe 8) there're intricate access policies to the external SD. I definitely gave all necessary access to the app, but still...

[farverio]

Had this issue but was fixed for me after upgrade -- thanks for all the help!

[snowtr]

This issue is not fixed. I am still having this issue.

Would it be helpful if I provided logs from the Android app?
If so, partial or full?
If partial, what section(s) do you want?

[ezaquarii]

On Sat, May 09, 2020 at 01:50:06PM -0700, snowtr wrote:

This issue is not fixed. I am still having this issue.

Would it be helpful if I provided logs from the Android app?

Possibly. Also, your app and server versions can play a role.

If so, partial or full?

If in doubt, please send a full log to @tobiasKaminsky. Do not post
it here.

[snowtr]

I've gotten back around to reproducing it and sent the logs to @tobiasKaminsky

[machinasdei]

yuup, its now May 20, 2020 and I'm on 3.11.1 and I have the same issue still. glad to see its not just me.

[WzL]

I have the same issue. Continuous popups within android 10 NC client with "File upload conflict" "Pick which version to keep". And if i click the popup, I get a blank screen which eventually turns into an "Nextclud isn't responding" timeout popup. If I select "Wait" the NC android client crashes.
The files in question are JPG pictures restored from a google cloud backup after a Phone restore.

Because there are continuously "File upload conflicts" detected by the NC android client, battery drain is enormous and I am forced to keep the phone connected to the charger whenever possible. Looking into rollback to previous version as mentioned above, the current situation is not workable batterydrain wise.

[JonnyHaystack]

Yeah I have the exact same problems. This feature seems to be completely unusable right now.

[chris246]

After specifying in the settings, what should be done on conflicts, I don't have the conflict issue any more.

[jo-me]

There should not be a conflict in the first place if you take a picture
that's auto-uploaded afterwards. And even if the phone manipulates it
afterwards due to geotagging or so, there should not be a conflict because
the phone has the newer version and that is enough for a re-upload without
conflicts.

Conflicts should only arise if the file was manipulated by two parties. If
nextcloud is changing pictures after upload then this needs to be addressed.

Until this is fixed I continue using the old app version that can be found
in GitHub releases.

Christoph Schwarzenberg [e-mail] schrieb am Di., 2. Juni
2020, 14:17:

After specifying in the settings, what should be done on conflicts, I
don't have the conflict issue any more.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/5755#issuecomment-637503399,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACO7FKDUJQT732ZFPZFSKLLRUTUWXANCNFSM4LVJLMJQ
.

[snowtr]

There should not be a conflict in the first place if you take a picture that's auto-uploaded afterwards. And even if the phone manipulates it afterwards due to geotagging or so, there should not be a conflict because the phone has the newer version and that is enough for a re-upload without conflicts. Conflicts should only arise if the file was manipulated by two parties. If nextcloud is changing pictures after upload then this needs to be addressed. Until this is fixed I continue using the old app version that can be found in GitHub releases. Christoph Schwarzenberg [e-mail] schrieb am Di., 2. Juni 2020, 14:17:
…
After specifying in the settings, what should be done on conflicts, I don't have the conflict issue any more. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5755 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACO7FKDUJQT732ZFPZFSKLLRUTUWXANCNFSM4LVJLMJQ .

That's why I took logs and uploaded them, because it is not doing that.

[e-alfred]

I log into files (GPSlogger) that are auto uploaded to Nextcloud. In the past, uploads happened after the file was finished and GPSlogger was closed, so the full file would be uploaded to Nextcloud. Currently, the Nextcloud app tries to upload the file while it is still opened by GPSlogger and getting written to and because it tries to upload the same filename multiple times, it asks for manual conflict resolution by the user. This results in files that are broken and not complete. This problem didn't happen on older releases.

[Queeq]

I agree there shouldn't be any conflict unless the file is modified on both sides. Just like the desktop client does: it uploads the file if it's modified locally without any issues, conflicts or prompts.

For me it refuses to upload any newly taken photos even when there are no traces of them on the server, no matter how I change the settings. Manual resolution doesn't work either. I imagine it's even worse if I record a video and the file is being written into.

[Oclair]

Please provide an automated keep both if sources are different

[GitOkkie]

This problem returned in 3.13.0 RC1

Since this update i keep getting File Upload Conflicts on the same logfile every time it update agian.

Configuration for this directory:

• Only upload on unmetered Wi-Fi: ON
• Only upload when charging: ON
• Also upload existing files: ON
• Use subfolders: OFF
• Original file will be... [ kept in original folder ]
• What to do if the file already exists? [ Overwrite remote version ]

Despite these settings I get bombarded with "File upload conflicts" on this logfile. Again...

[Maverobot]

Every single new video from the camera requires to "resolve the conflict", I wonder what kind of conflict does a newly taken video have?

[tobiasKaminsky]

This will be fixed via #6771
In this PR you can find a APK which you can install in parallel and test it.

[Queeq]

Just to remind, I have a camera folder set for auto upload. The problem has not gone away with 3.13.1 unfortunately. All newly made photos are considered a conflict by the client. I tried to change auto upload folder settings:

• set "Overwrite remote version"
• set "Also upload existing files"

First one had no effect. Second one triggered re-upload of everything. Once all older files were successfully uploaded, it still couldn't upload not synced "conflicting files". When trying to resolve the conflict, it shows a floating system message "Error creating conflict dialog!".

While roaming around, trying to restart the app, upload manually, etc., at some point it was able to show the dialog from "Uploads" screen. Checking one or the other file did not clear the conflict though, only deleting them seems to have helped. After that, for a file that was created later, I see that it attempts to fetch server version to show the comparison, which is obviously not there...

[snowtr]

Just to remind, I have a camera folder set for auto upload. The problem has not gone away with 3.13.1 unfortunately. All newly made photos are considered a conflict by the client. I tried to change auto upload folder settings:

• set "Overwrite remote version"
• set "Also upload existing files"

First one had no effect. Second one triggered re-upload of everything. Once all older files were successfully uploaded, it still couldn't upload not synced "conflicting files". When trying to resolve the conflict, it shows a floating system message "Error creating conflict dialog!".

While roaming around, trying to restart the app, upload manually, etc., at some point it was able to show the dialog from "Uploads" screen. Checking one or the other file did not clear the conflict though, only deleting them seems to have helped. After that, for a file that was created later, I see that it attempts to fetch server version to show the comparison, which is obviously not there...

Same for me.

[ltGuillaume]

Same here on v3.13.1, at least with every file that's _modified_ locally.

What happens for you guys if you choose to keep the file from the server (by checking "Already existing file")? For me it downloads the file from the server (all the way to 100%), then fails _every single time_ with the following message (and the upload conflict reappears):

Nextcloud v19.0.3
Android 9 (LineageOS 16)

[Queeq]

Strangely enough, newly created files by TitaniumBackup seem to be uploaded (though I'm not sure if all of them), but with Camera it doesn't work. Both folders are on external SD card.

[ltGuillaume]

I've also seen this issue for a subset of files created by Titanium Backup (on a sidenote, somehow the Nextcloud app data couldn't be restored by Titanium Backup, otherwise I would have gone back to v3.12, which did _not_ exhibit this sync problem for me).

[Queeq]

@ltGuillaume Rollback via TitaniumBackup to 3.10 worked fine for me, but only if I have old version and data restored, because 3.11 seems to have changed some data structures and 3.10 would crash if newer data version is there. Also the problem appeared in 3.11 for me, that's why I have to rollback to 3.10.

[tobiasKaminsky]

Fixed via #7102
There is an APK file in this PR, which you can install in parallel to your existing Nextcloud app.

[Queeq]

~Sorry, I must be missing something, but I don't see the attached APK in that MR. Should it be in the OP or in some other tab/message?~ OK, the bot has published it.

[Queeq]

With latest APK the problem is still present for me. The options for the folder are set as following:
Only upload on unmetered Wi-Fi: no
Only upload when charging: no
Also upload existing files: no
Use subfolders: no
What to do if the file already exists? Ask me every time

P.S. It's also quite inconsistent. Sometimes it does upload files, but then shows conflict for the file it has uploaded.

[ikke-t]

For me it's is stuck in certain file. It just keeps announcing once in the while, that there is sync conflict. I should select what to do. Which ever version I select, it won't be happy, but complains about sync problem soon again. There are thousands of pics in camera folder, but it always picks the same one today.

[tobiasKaminsky]

@ikke-t this should be a parallel installation, so there should be no uploads in queue yet?

@Queeq @ikke-t
Can you both test this

• take a screenshot
• let it be uploaded by auto upload
• clear all uploads in uploads view
• modify this screenshot locally
• this triggers a new upload
  -> choose any option on conflict dialog
• verify if new upload is vanished from upload queue

[Queeq]

@tobiasKaminsky Do you want us to configure separate screenshot folder for auto upload or it's fine to copy a newly made screenshot to another folder that's already configured to auto upload (Camera in my case)? Is it important for the screenshot to be created in an auto upload folder right away or it could be copied there?

[ikke-t]

My android creates a new image if I modify any image in gallery, so it won't reproduce edited image. There is some error state sometimes, which happens likely due some network problem, e.g. starting upload on 4G, but in middle switching to WLAN. Once that happens, it's common to get such forever error causing sync messages.

Somehow the android app fails to take action on what ever get's selected in the selection dialog for conflicts.

[ikke-t]

I now renamed the image at the server to get rid of this constant nagging of the nextcloud app. I'll investigate more once it happens again. It's not often when it happens, it's rather rare luckily.

[tobiasKaminsky]

@Queeq I did not try it with copy, so you can test it,but I am not sure if this is then a meaningful test.
Best is to do it exactly in the steps I wrote

[Queeq]

@tobiasKaminsky So I tried it with Screenshots folder (and the recent dev .apk). It fails at the very beginning: once I take the screenshot, Nextcloud tries to upload it and instantly fails claiming upload conflict. If I set to overwrite remote version in Auto Upload preferences for the folder, the result is the same.

[tigernero79]

in version 20.0.0.1 rc1 this bug has come back.

[tobiasKaminsky]

Is this really related to server?

[Queeq]

Most likely not. I'm still on my old 15th version.

[Queeq]

Tested it with 16338.apk. It's definitely better now and newly created files are uploaded. But I had a bunch of unsynced files due to this bug and it doesn't upload them. Manual upload results in conflict again, and it's indefinitely looping trying to upload them and failing unless I cancel it.

P.S. Tried to wipe the app settings and reconfigure, the conflicts are still there. But now it doesn't loop over them. Pressing "Resolve conflict" in Uploads screen does nothing.

[e-alfred]

Does this error cause the sync conflicts?

{"reqId":"X56-ZtKjspcMRQSgaRIspwAAAUU","level":0,"time":"2020-11-01T15:00:07+01:00","remoteAddr":"1.1.1.1","user":"user","app":"no app in context","method":"HEAD","url":"/owncloud/remote.php/webdav/SofortUpload/test/4546754456/CSVLogs/CSVLog_20201031_172528.csv","message":"Deprecated event type for {\"[object] (OCP\\SabrePluginEvent)\":{\"*statusCode\":200,\"*message\":\"\",\"*server\":{\"[object] (OCA\\DAV\\Connector\\Sabre\\Server)\":{\"tree\":\"[object] (OCA\\DAV\\Connector\\Sabre\\ObjectTree)\",\"*baseUri\":\"/owncloud/remote.php/webdav/\",\"httpResponse\":\"[object] (Sabre\\HTTP\\Response)\",\"httpRequest\":\"[object] (Sabre\\HTTP\\Request)\",\"sapi\":\"[object] (Sabre\\HTTP\\Sapi)\",\"*plugins\":{\"...\":\"Over 20 items, aborting normalization\"},\"transactionType\":\"head\",\"protectedProperties\":{\"...\":\"Over 20 items, aborting normalization\"},\"debugExceptions\":false,\"resourceTypeMapping\":[],\"enablePropfindDepthInfinity\":true,\"xml\":\"[object] (Sabre\\DAV\\Xml\\Service)\",\"*listeners\":{\"...\":\"Over 20 items, aborting normalization\"},\"*wildcardListeners\":[],\"*listenerIndex\":[],\"*logger\":null}},\"Symfony\\Contracts\\EventDispatcher\\EventpropagationStopped\":false}}: null","userAgent":"Mozilla/5.0 (Android) Nextcloud-android/3.13.1","version":"19.0.4.2"}

[Queeq]

I'm still experiencing sync conflict issues even with new files in Camera folder (same 16388 dev version). The difference now is that it doesn't attempt to upload automatically anymore (basically auto upload doesn't work with any combination of settings). If I try to upload manually it still claims there's a conflict...