Client: Virtual File Sync no longer working properly

[efrenbg1]

Since the beginning of this week all clients updated automatically to client version 2.7.2. This made Virtual File Sync to stop working properly. I have observed two behaviors:

1. With previous files synced -> New files are immediately deleted by clients with Virtual File Sync.
2. On a clean sync -> Folders are perfectly created, but files are not, causing the client to delete every single file on the server.

Expected behaviour

Working with Virtual File Sync as expected.

Actual behaviour

All files are missing from the client folder, causing a complete wipe of data on the server.

Steps to reproduce

1. Share folder with user x
2. Add Virtual File Sync with user x
3. Wait for second sync caused by missing files

Server configuration

Operating system: debian 10 (Linux 4.19.0 - amd64)

Web server: nginx/1.14.2

Database: 10.3.25-MariaDB

PHP version: 7.2.34

ownCloud version: 10.5.0

Storage backend (external storage): none

Client configuration

Client version: 2.7.2

Operating system: Windows 10 Pro 20H2

OS language: Spanish

Qt version used by client package (Linux only, see also Settings dialog): none

Client package (From ownCloud or distro) (Linux only): none

Installation path of client: C:\Program Files (x86)\ownCloud

Logs

1. Client logfile: (Nothing on the log file)

2. Web server error log: (Nothing on the log file)

3. Server logfile: (Nothing on the log file)

What I've tried

1. Full clean install of the client removing all %appdata% files

2. Dowgrading to previous client versions (only 2.5.4 seems to work as of right now)

3. Clean install of ownCloud server

4. Update Windows 10 to latest version

~ Thanks in advanced for your hard work ~

[voerg]

I can confirm the behavior of file deletions on the server if the virtual file sync is used.
Please provide an update.

[efrenbg1]

I have done more testing. The problem seems to be related to a Windows 10 update.

1. All clients (including version 2.7.2) work perfectly under compilation version:
   19042.662
2. Not working under compilation version:
   19042.685

Both Windows are on the 20H2 version. As a result of Windows update policy, all computers at my office updated at the beginning of this week causing also the update of the client (due to the reboot of the OS).

[Tenivar]

I have done more testing. The problem seems to be related to a Windows 10 update.

1. All clients (including version 2.7.2) work perfectly under compilation version:
   19042.662

2. Not working under compilation version:
   19042.685

Both Windows are on the 20H2 version. As a result of Windows update policy, all computers at my office updated at the beginning of this week causing also the update of the client (due to the reboot of the OS).

I can confirm that, we are trying to find what changed in the "cloud files" windows API. But if there was a major change in how windows 10 handle files, they would have announced it...
I wonder if that's just a parameter/function name that changed between versions

[hodyroff]

Thanks! We have identified the issue and are working on a fix.
Indeed the Windows 10 API changed and the error was not handled correctly resulting in the above described issue.
2.7.3 client is upcoming ...

[TheOneRing]

Fixed in 2.7.3 https://github.com/owncloud/client/releases/tag/v2.7.3

[butonic]

Is there a script to restore all files since a given date? Should be doable using bash & curl...

An moving files on the server CLI and then running occ file:scan would assign new file ids to the deleted files and all shares and metadata would be lost...

[welkineins]

@hodyroff Sorry for comment on a closed issue.

Can you share which API is changed in Windows 2020H2? It might affect other software use the same API.

Thanks.

[TheOneRing]

@hodyroff Sorry for comment on a closed issue.

Can you share which API is changed in Windows 2020H2? It might affect other software use the same API.

Thanks.

https://github.com/microsoft/Windows-classic-samples/issues/164

[JanAckermann]

@efrenbg1 @welkineins @voerg @Tenivar @butonic

Restore script is here
https://github.com/JanAckermann/owncloud-restore-trash

[stylefieber]

I'm a bit scared now to use ownCloud in production at my company, where I am responsible for the infrastructure. Could that happen again?

[dragotin]

Well. You might want to refer to https://github.com/microsoft/Windows-classic-samples/issues/164 to understand how that could happen at all and what levels are involved.

As much as I can't assure you that nifty things will never happy again, I can assure you that we do everything for our part to avoid that. This has been the most severe bug in the desktop syncing since years. I am grateful that we have mechanisms in place to help (trashbin).

[efrenbg1]

Well. You might want to refer to microsoft/Windows-classic-samples#164 to understand how that could happen at all and what levels are involved.

@dragotin Just out of curiosity. Don’t changes to the cloud file API show first in the Insiders program?

All changes made to the cloud file API (or others) are first tested and published to the Insiders channel, but considering they are making constant changes to that API I'm not really surprised that has passed under the radar.

[TheOneRing]

Well. You might want to refer to microsoft/Windows-classic-samples#164 to understand how that could happen at all and what levels are involved.

@dragotin Just out of curiosity. Don’t changes to the cloud file API show first in the Insiders program?

All changes made to the cloud file API (or others) are first tested and published to the Insiders channel, but considering they are making constant changes to that API I'm not really surprised that has passed under the radar.

I'm running the beta insiders preview and the problematic change 19041.685 was released at the same time as for the other Windows users.

[EdouardRt]

@efrenbg1 @welkineins @voerg @Tenivar @butonic

Could you please download this zip file, unpack in a folder,
read the read.me file

Please try to restore the files and provide feedback.
(comment also shared in issue #38208.)

Thanks, i did try and run the restore.php,

Having a warning on composer install, and an error on php restore.php execution.
Tryied both from the owncloud directory and from /home, with admin and user rights. won't work.
Got an idea ? Am i doing something wrong ? is it related to the https ?

Warning on installation :
Cannot create cache directory /home/desktop/.composer/cache/repo/https---packagist.org/, or directory is not writable. Proceeding without cache
Cannot create cache directory /home/desktop/.composer/cache/files/, or directory is not writable. Proceeding without cache
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Package operations: 2 installs, 0 updates, 0 removals

Installing sabre/uri (2.2.1): Downloading (100%)
Installing sabre/xml (2.2.3): Downloading (100%)
Generating autoload files
Error on execution :

php restore.php
Collection files to restore
PHP Fatal error: Uncaught Sabre\Xml\ParseException: The input element to parse is empty. Do not attempt to parse in /home/desktop/restore/vendor/sabre/xml/lib/Service.php:122
Stack trace:

0 /home/desktop/restore/lib/RestoreTrash.php(59): Sabre\Xml\Service->parse(false)

1 /home/desktop/restore/lib/RestoreTrash.php(25): RestoreTrash->collectTrashbinData()

2 /home/desktop/restore/restore.php(11): RestoreTrash->run()

3 {main}

thrown in /home/desktop/restore/vendor/sabre/xml/lib/Service.php on line 122

[JanAckermann]

@EdouardRt some permission errors, please try to follow the steps here:
https://askubuntu.com/questions/1077879/cannot-create-cache-directory-home-user-composer-cache-repo-https-packagi

[EdouardRt]

thank you for your help @JanAckermann ! it does go through the cache warning, but the error while restoring remains :

sudo -u www-data php restore.php
Collection files to restore
PHP Fatal error:  Uncaught Sabre\Xml\ParseException: The input element to parse is empty. Do not attempt to parse in /home/desktop/restore/vendor/sabre/xml/lib/Service.php:122
Stack trace:
#0 /home/desktop/restore/lib/RestoreTrash.php(59): Sabre\Xml\Service->parse(false)
#1 /home/desktop/restore/lib/RestoreTrash.php(25): RestoreTrash->collectTrashbinData()
#2 /home/desktop/restore/restore.php(10): RestoreTrash->run()
#3 {main}
  thrown in /home/desktop/restore/vendor/sabre/xml/lib/Service.php on line 122

My detailed configuration is available here.
Meanwhile, i wander if the naming of elements in my trashbin is "normal" ie. IMG_0550.JPG.d1607618465 ?

[JanAckermann]

@EdouardRt
Did you changed the variables in the restore.php as written in documentation?
https://github.com/JanAckermann/owncloud-restore-trash

This error usually occures if the script can't connect to the server

Meanwhile, i wander if the naming of elements in my trashbin is "normal" ie. IMG_0550.JPG.d1607618465 ?

Totaly "normal", please don't try to rename them

[EdouardRt]

@JanAckermann Yes, variables are defined, all 4 between single quotes : 'name' ; 'https://website.xt' ; 'p@sword' ; 'yyyy-mm-dd' .
I know i have a problem with the https certificate which needs browsing through chrome to hit the "proceed anyway" button.
the windows client connects though, but maybe a php command could be weaker ?

[JanAckermann]

@EdouardRt please try the following:

1. Get the new version of the restore script here
2. In restore.php set ignore SSL to true (without quotes)
3. Run the script again

[EdouardRt]

Thanks for the updated script !
Feeling sorry @JanAckermann : getting an error (404 !?) :-(
and can't find any log that would help /var/log/syslog or /php7.0-fpm.log or /apache2/log ...

sudo -u www-data php restore.php
Collection files to restore
ERROR: The requested URL returned error: 404 Not FoundPHP Fatal error:  Uncaught Sabre\Xml\ParseException: The input element to parse is empty. Do not attempt to parse in /home/desktop/restore/owncloud-restore-trash-master/vendor/sabre/xml/lib/Service.php:122
Stack trace:
#0 /home/desktop/restore/owncloud-restore-trash-master/lib/RestoreTrash.php(74): Sabre\Xml\Service->parse(false)
#1 /home/desktop/restore/owncloud-restore-trash-master/lib/RestoreTrash.php(27): RestoreTrash->collectTrashbinData()
#2 /home/desktop/restore/owncloud-restore-trash-master/restore.php(11): RestoreTrash->run()
#3 {main}
  thrown in /home/desktop/restore/owncloud-restore-trash-master/vendor/sabre/xml/lib/Service.php on line 122

[JanAckermann]

@EdouardRt you are welcome!

404 usually means that the requested resource does not exist.
<your-server>/remote.php/dav/trash-bin
should be callable via browser with the output:

This is the WebDAV interface. It can only be accessed by WebDAV clients such as the ownCloud desktop sync client.

Is the server reachable from the client you want to execute the script?

There is a new version of the script, please download the new version, do a composer install
and try to run the command like this:

➜ php restore.php --url="https://owncloud.local" --username="admin" --password="admin" --date="2020-12-08" 

An additional / at the end of the url can lead to problems

[EdouardRt]

@JanAckermann
Weird :
https://server.xt/remote.php/dav/trash-bin
Sends an owncloud error page with "Not Found File not found: trash-bin in 'root'"

While :
https://server.xt/remote.php/dav/
Sends the error you mention : "This is the WebDAV interface. It can only be accessed by WebDAV clients such as the ownCloud desktop sync client."

Trying the updated script right away.

[JanAckermann]

@EdouardRt

Sends an owncloud error page with "Not Found File not found: trash-bin in 'root'"
This must be some server configuration problem,

in order to get the script run this must be fixed

Edit

https://server.xt/remote.php/dav/trash-bin/
should work

[EdouardRt]

@JanAckermann
I do not get it :-(
/dav/files/user >> works fine
but
/dav/trash-bin/user >> does not (owncloud error "file not found")

I tried an occ repair > no change
An upgrade > already latest version

Browsing through the files, i see them in data/user/files_trashbin/files
But /dav/trash-bin won't grant access to it.

Any hint on where to look ? i did not find any relevant case elsewhere...

[EdouardRt]

@JanAckermann
I was able to force an upgrade (sounds like my repo was outdated) and had to also force a php upgrade.
Now am able to run the script. : 37k files found... The restore process is ongoing.
Thank you very much for your help !
Edouard

[JanAckermann]

@EdouardRt
Good to hear man!

[audouts]

The 2.7.2 client deleted hundreds of thousands of files from our server.

We could not access the "Trash" via the web interface:
This directory is unavailable, please check the logs or contact the administrator

The recovery script was unusable because it would take hundreds of days for it to recover the files.
(https://github.com/JanAckermann/owncloud-restore-trash)

I could not find a way to use propfind to get the file ID, which the ownCloud docs describe in the webDAV, TrashBin API. I made a minor change that allowed me to get the file IDs and then dumped a list with propfind. I then used the TrashBin API "restore" call and tried to restore the files.

Approximately 92% of the files succeeded. We now face other problems:

First, when I try the TrashBin API restore on one of the remaining files, it says that the file already exists. It does not. I cannot find it anywhere in the database (except the trash and the history, showing it was deleted). It's not in the filesystem.

Second, the web interface works now (presumably due to the lower number of files) but has no way of selecting files before/after a certain date. I don't want to restore files that were intentionally deleted before the client bug started deleting everything.

I decided to try using the TrashBin API to delete files from the trash, before a certain date. That brings me to the third problem. Apparently, I can only delete one file at a time via the API. I get an error about it being "locked" if I try to run concurrent calls. It's taking 70-80 seconds per delete, which means it will take days to complete this relatively small operation.

What can we do to solve this?

[dannyrobertson]

This is still an issue even with Windows client v2.7.6:
When setting up a new Windows machine, I downloaded and installed the latest client and connected to my existing ownCloud instance.
Afterwards I spent all day recovering files.

[TheOneRing]

Could you open a new issue and fill in the details?
Ideally I'd also need some logs.