Desktop: Avoid redownload by client of locally existing files (eg copied by rsync)

[rainer042]

Expected behaviour

Tell us what should happen
Files that have been copied via rsync into a clients local nextcloud folder are redownloaded by the nextcloud client altough this should be avoided.

Actual behaviour

I copied a testfile into a nexclouds client subfolder that I disabled for syncing in the nextcloud client before. Then I reenabled syncing again for this subfolder and the file was downloaded again from the server instead of using the local identical copy.

Steps to reproduce

1. Take a file from your nextcloud server and copy it outside from nextcloud to a client into the nextcloud local folder into the correct directory
2. Watch what nextcloud client will do when you enable syncing for this directory
3. You will see it will redownload the file overwriting the identical existing one (the one that has been copied "manually" eg via rsync.

Client configuration

Client version: 2.5.1

Operating system:
Linux, OpenSuE Leap 15.1
OS language: German

Qt version used by client package (Linux only, see also Settings dialog):
Built from Git revision b37cbe using Qt 5.9.7, OpenSSL 1.1.0i-fips 14 Aug 2018
Client package (From Nextcloud or distro) (Linux only):
From distro
Installation path of client:
/usr/bin/nextcloud

Server configuration

Nextcloud version: 16.0.4

Storage backend (external storage):
Disk

Logs

I found issue #3422 from 2017, where this problem was already discussed. However I did not find a solution up to now if there is any.

Personally what I want to do is what I described here:

avoid-complete-nextcloud-resync-to-client-if-data-are-already-on-the-client-via-rsync

Thanks in advance
Rainer

[stale[bot]]

This request did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[Germano0]

This request did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

This is definitely still a problem

[realkot]

I have the same issue..... Client REdownload already existing files...

[rainer042]

I still would gladly welcome this new feature to add existing data into the nextcloud sync process. For my use case I eventually chose the indirect path, by removing my nextcloud account, add it again telling nextcloud to keep existing files. This works but its definitively a little cumbersome.

[FriendFX]

For my use case I eventually chose the indirect path, by removing my nextcloud account, add it again telling nextcloud to keep existing files. This works but its definitively a little cumbersome.

@rainer042 this is interesting. Do you know whether those existing files will be synced again if they're changed later on?
I have a slightly different use case: I am installing a different OS on my client machine and plan to manually copy the (previously up-to date) nextcloud folder from a backup once done to avoid having to slowly re-download everything over the internet.
Another use case would be to take a USB flash drive ("USB stick") with all the nextcloud files from one client machine to another before installing the Nextcloud client there to avoid having to transfer everything from scratch.

[rainer042]

@FriendFX I think if you have installed your new os and then create a new nextcloud account and tell nextcloud that its data directory is exactly the directory with your copy from your nextcloud root folder backup and also tell nextcloud to keep existing data this should work without any problems. In this case ideally no file should be synced initially and if you change any of the files on the client or the server side afterwards a sync would be triggered by your nextcloud client. For me at least this works perfectly.
The USB copy method would probably also work, because it does not matter how the files are copied to the new client if the copy is "one to one" and you create a new nextcloud account afterwards as described above. In this USB scenario you only have to take care that eg the filenames are kept intact exactly as they are. If you copy your files directly on the USB stick not using a tar or zip archive and the USB stick uses FAT as its filesystem this might result in a change of the filenames regarding case like "ReAdMe" and "readme" which could then lead to trouble when you start your first sync.

[haraldkoch]

I recently updated the desktop client to 3.0.1, and it disabled sync of three of my folders due to their size. They were being synced before I upgraded the client (which is a separate bug). When I enabled sync of those three folders, the nextcloud client re-downloaded all 8Gb of data that was already on my local disk, logging "File has changed since discovery" messages. (Bandwidth isn't free, btw). So this is a sync defect that isn't only triggered by adding files with external software ...

[6pac]

+1, I have 2TB of local data to sync up to the server folder which is 99% identical. Re-downloading it all is exactly what I'm trying to avoid. I think I'm going to switch to SyncThing, which has a number of other aadvantages.

[PVince81]

checksum support ticket here: https://github.com/nextcloud/server/issues/11138

that would make it possible for the client to detect that the files are identical and skip syncing

now, without this, maybe an expert option somewhere to tell the sync client to assume that all files in a folder are identical and just retrieve the etag/metadata and update it locally

[PVince81]

I'm also struggling with this. Rsync tells me everything is identical, even timestamps, but the client still tried to redownload everything.
Am wondering if it would be possible to manually insert all entries in the sync db with etags retrieved from the backend (with an external script, as a one shot operation).

[6pac]

Yes, it look like it's not the file dates/checksums, it's whether they are in the local db. There doesn't appear to be any officially supported way of getting them there. You could possibly do that insert, but I suspect that by the time you do that, you could almost have created a PR ;-)

Anyway. I don't have time for that, so I am uysing Syncthing which seems much better suited for a one way sync and appears to be much more stable and mature.

[PVince81]

I actually managed to make it recognize existing files by deleting the local sync folder config but with keeping the files. Then I added back the sync folder and pointed the config at the existing data. It seems to have recognized existing files by mtime/size as it didn't resync all of them. So at least that one is a possible workaround.

[er-vin]

I'm also struggling with this. Rsync tells me everything is identical, even timestamps, but the client still tried to redownload everything.

For the record, it checks mtime, etag and inode. So if the inode changes it'll assume something changes locally.