Desktop: Client stuck in sync

[EchoDev]

I'm running Nextcloud 13 with E2E enabled. I first had this issue with the techpreview Windows client. I grabbed the source from the E2E encryption branche and built the windows client and I still have the same issue.

I can't see what the client is trying to sync and I don't see anything on the server logs. Anyone knows what it could be or how I could find out what is causing it.

[EchoDev]

Ok so I had some time to test the issue and it seems to be an issues with edited files being synced.

To reproduce this issue:

1. Have E2EE enabled on your server and desktop client
2. Place a (text) file in one of your encrypted folders and let the desktop client sync
3. Confirm via mobile client or via an other pc that the file has been synced correctly
4. Edit the (text) file on PC
5. The desktop client will be stuck in sync mode

[tcanabrava]

On Sat, Mar 10, 2018 at 1:12 AM, EchoDev [e-mail] wrote:

Ok so I had some time to test the issue and it seems to be an issues with
edited files being synced.

To reproduce this issue:

1. Have E2EE enabled on your server and desktop client
2. Place a (text) file in one of your encrypted folders and let the
   desktop client sync
3. Confirm via mobile client or via an other pc that the file has been
   synced correctly
4. Edit the (text) file on PC
5. The desktop client will be stuck in sync mode

Thanks. This really helps.
I'm debugging and trying to understand where it's failing.

>
>

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/client/issues/199#issuecomment-371981334,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AD1zUJQ2ajGNTZjj7bND74VeL-5HhdTEks5tcxpfgaJpZM4SWAi2
.

[camilasan]

is this fixed @tcanabrava? I didn't have this problem.
cc @dantti

[dantti]

I believe so, updating the file wasn't implemented till my last PR, @EchoDev can you pleas test again (using the v4 branch)?

[EchoDev]

I built the client from master but it just seems to crash on login/sync. (I couldn't build from the v4 branch since it is deleted)

[camilasan]

Could you share your logs? Where does it crash? Was it with a new user or an existing user?

[dantti]

Hmm I didn't know about the v4 branch merge, it is crashing on Windows?

[EchoDev]

There is nothing in the server logs regarding the crash. Does the client produce logs somewhere?

I did the following:

1. Try to log in with existing user A that had E2EE enabled and has passphrase entered previously -> client crashes
2. Create new user B, log in with it -> I get the encryption phrase -> I click Ok -> Crash
3. Try to log in with new user B -> Crash

I need to remove the cfg file in %appdata%/Nextcloud to make the client work at all since it immediately crashes upon start since it logs in/sync immediately.

If I go back to the tech preview client it doesn't crash but the issue of stuck in sync remains.

Tested on Windows 7 x64

[camilasan]

could you send the backtrace and which version of openssh you have @EchoDev?

[EchoDev]

@camilasan reverting the following commits fixed the crashes for me

https://github.com/nextcloud/client/commit/faf64958ec1f74c3842aa1f37716231057902077

https://github.com/nextcloud/client/commit/40b1f051a4d968e45556817272e132054c91aeba

I need some time to see if the syncing issue is fixed (Not being able to revoke and generate new E2EE passphrases is really a pita.)

[tabp0le]

So, I am also having issues with the latest merge of the E2E. (besides the issue with the logs, which is easily remedied by reverting one commit)

The sync will not complete, especially for large folders. It seems also to sync the root folder, and folder structure, but not the items contained in the folders. Nextcloud 13, master branch of client. Windows and Linux.

[EchoDev]

Same for me. The latest version has become worse than the techpreview.

I have the same problems as @tabp0le first time sync won't even work now. (It just stop randomly stop after syncing a few small files)

When I try to open the files from the Android client it gives me a "Locked" reply. I'm assuming the client isn't finishing the files it is uploading and locks up/stops?

Uploading, downloading, encrypting and decrypting of folders works fine from the Android client when the Desktop client isn't involved.

[tabp0le]

To follow up on what @EchoDev said, I've noticed the locking too. While it's trying to sync, an error in the server log shows up: "Cannot find file $filename" (Obviously I substituted $filename for the real list of files. It's really logspam, just spews for every single directory) I tried wiping redis, all caches, dropped the locking tables from the database. Nothing solves the issue.

Additionaly another weird error that shows up (for each file): "GOAWAY received, cannot start a request"

I have to say, I don't think the e2e is ready to be in master. IMHO it should be reverted.

[tcanabrava]

The biggest issue is the spec, it makes the end to end encryption single
threaded and because of a client limitation and I can only send one file
per time (no concurrent uploads, so it will be much slower). The locking
problem I need more info. I think I know what can be causing this but I’m
unsure if it’s the root cause.

On Sat, 28 Apr 2018 at 02:11 Tab Fitts [e-mail] wrote:

To follow up on what @EchoDev https://github.com/EchoDev said, I've
noticed the locking too. While it's trying to sync, an error in the server
log shows up: "Cannot find file $filename" (Obviously I substituted
$filename for the real list of files. It's really logspam, just spews for
every single directory) I tried wiping redis, all caches, dropped the
locking tables from the database. Nothing solves the issue.

Additionaly another weird error that shows up (for each file): "GOAWAY
received, cannot start a request"

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/client/issues/199#issuecomment-385123253,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AD1zUN_P7GSHpgU8fhXTBHQsnCXNEOiQks5ts7OhgaJpZM4SWAi2
.

[camilasan]

Nothing else we can improve @tcanabrava? Besides moving the metadata to the file?

[dantti]

@camilasan AFAIK webdav support attributes per file, which would make things a lot better than maintaining the single JSON file, while OTOH making more HTTP requests, still if we could discuss this new spec again we could release something more solid, when it was drafted there was no implementation to prove if it would work well.

[camilasan]

@dantti @tcanabrava the issues described here are related to the e2ee being slow or this could just be a different bug?

[jospoortvliet]

Guys, let's focus on making it stable, performance comes next. Obviously, locking the E2E folder, uploading a file, unlocking again, locking again, uploading, unlocking - that serializes server access and makes it slower. We can perhaps change the protocol later to allow uploading multiple files at once.

But for now, being slow is not a problem for the first release. Losing files is. Let's just make it stable, if we have to tell people:

1. it will be slower in syncing
2. it can handle only 50 files in a single E2E folder before performance becomes cripling

that is fine.

The attributes per file means a whole new design, the choice for a single JSON file was made for a reason so let's stick with that for now. We can discuss changes at the Nextcloud conference, provided we released this before then.

[dantti]

@jospoortvliet sure, but I have in mind a few important problems that aren't about performance, not sure discussing here is a good place.

[jospoortvliet]

Sure, if there are fundamental problems with the design/protocol that block implementation or make it insecure, they should be discussed here:
https://github.com/nextcloud/end_to_end_encryption_rfc/issues

[camilasan]

@EchoDev Could you try this build - please make sure to have a clean configuration - https://download.nextcloud.com/desktop/daily/Windows/Nextcloud-2.5.0.61352-daily-20180904.exe?
It solved a lot of known connecting and syncing issues.

[EchoDev]

@camilasan I will do a complete reinstall of Nextcloud server side and will try the new client in a few days.

[EchoDev]

@camilasan I had some time so I did a quick reinstall of Nextcloud and installed the new client.

TL;DR -> The client is still stuck

Full story:
I removed everything Nextcloud related from the server. I removed the data files, the nextcloud server software and emptied the SQL database.
I reinstalled the server software, enabled server side encryption and enabled TOTP. After this was all working correctly I enabled e2ee. After installing the client it properly gave me a new passphrase. After this I started the initial sync. I created a new folder, set the folder to encrypted in the client and moved some files to this folder but it doesn't seem to properly sync.
The desktop client is stuck on sync randomly on initial sync. I tried syncing 1 folder with ~250 files around 200MB but the sync just stops with no errors in logs (Server and client). My firewall shows pretty much no bandwidth usage so it halted for some reason without any errors. If I pause the sync, unpause and force a sync a few new files get synced again and randomly stops again. (I'm assuming at this point the files get uploaded properly, I didn't test it from a second device yet, if the files are readable on a second device everything is okay on that front I guess)

Question: If I first sync the files without e2ee encryption and I enable the encryption afterwards, will the files and folders be properly encrypted? If yes, I might just try that and see if editing files and syncing it back to the server works.

[EchoDev]

Did some more tests, all tests done on encrypted folders ofcourse

Test 1: Sync with the desktop client, only a few files get synced before the client locks up

E2EE Files synced from desktop are not accessible on mobile (Android). Only some of the files that are supposed to be synced show up in the Android client and when you try to download those files the client gives the error "Download failed - The file is no longer available on the server"

Test 2: Sync with the mobile Android client, the upload completed partially.
E2EE files and folders created and uploaded from the mobile client also have issues with the desktop client when the desktop client is active? I uploaded 20+ files at the same time but the desktop client seems to try to grab the files early? The desktop client says

Server replied "423 Locked" to "GET https://server/remote.php/dav/files/username/folder/-random-numbers-and-letters"

Random numbers and letters are probably the encrypted filenames on server. After a while this causes the mobile client uploads to be corrupted. I had to set the folder to unencrypted to be able to delete the files.

Test 3: Upload a single file from Android client and sync it with the desktop client
Creating and uploading 1 file from the mobile client seems to not cause any issues in the desktop client. Upload from mobile

Test 4: Upload multiple files from the Android client while the desktop client is turned off and sync the files after upload on mobile is done.
Again the uploads from Android got borked, after 10-15 files the client says it failed uploading files to the server, tho this time the original filenames were kept as they were. The files were again corrupted After a while the file that were actually synced are working fine in the mobile and desktop client. But another issue seems to pop up, the desktop client tries to resync the files it just downloaded from the server....It seems to think the files that were just downloaded into the encrypted folder are actually new files... Ofcourse it locks up after a few files and is stuck in sync yet again.

At this point I'm not sure if this is an desktop client issue or just an e2ee/nextcloud server software issue. Syncing single files seems to work okay but throwing 20+ files at either of the clients cause the files to become corrupted (on the server) or just stops syncing.

[mcronce]

I'm getting a hang from the desktop client during sync after upgrading from 2.3.3 to 2.5.0. I haven't made any other changes (e.g. server upgrade, server app upgrades, etc) since it was last working. This is on OSX High Sierra. I have the End-to-End Encryption add-on enabled on the server, so when the new desktop client first came up it did give me the passphrase. Anything I can do to gather information that might be helpful?

The hang seems to happen at some point during (or maybe right after?) scanning for changes. That's all I've been able to tell so far.

Thus far I've tried tailing .owncloudsync.log and watching md5s of the ._sync SQLite files, with no changes to either. I've also tried dtrussing the process, but unless I interact with a file in Finder or something all it seems to do is print bsdthread_ctl() calls over and over with these two sets of arguments, alternating:

    PID/THRD  SYSCALL(args)          = return
31131/0x41027c6:  bsdthread_ctl(0x100, 0x0, 0x133FB)         = 0 0
31131/0x41027c6:  bsdthread_ctl(0x100, 0x0, 0x0)         = 0 0

[mcronce]

Uhh...interesting. Maybe it was just spending a long time trying to process the list of changes. (It was pegging one CPU core when it appeared to be hanging.) Immediately after I posted that it stopped hanging and started downloading files. Maybe I posted prematurely.

[mlaverdiere]

I think I'm also experiencing this problem (or a variant) with the ND Linux/Fedora 29 client version 2.5.0git, with ND server 14.0.4 and E2E enabled (E2E seems to work fine with my Android client).

Steps to reproduce:

1. On Linux/Fedora 29, create a folder "Test" and encrypt it using the ND client. So far, so good, everything is in sync and I can see on ND web and ND Android the folder Test with the lock icon.

2. Still on Linux/Fedora29, put whatever file in the Test folder. ND client starts to sync but never completes the process.

[mlaverdiere]

Same problem on Windows with ND client version 2.5.1final (build 20181204), with NS server 14.0.4, i.e. client stuck on sync when adding a file in a E2EE folder.

[Alphakilo]

I get this behavior across:

• iOS (2.22.8.20),
• Android,
• Fedora 29 (2.5.1-1.fc29 from updates-testing),
• Ubuntu / Mint (Flatpak),
• Windows 10 (2.5.1)

when trying to sync E2E folders with my Nextcloud server (Nextcloud 15.0.2.0, Ubuntu 18.04, PHP 7.3, apache2 compiled with h2 and tls-1.3). The issue is consistent across multiple users.

Funny enough, the client that initially uploaded the E2EE folder can access it's contents afterwards.
All other sync clients fail. The Windows version crashes, leaving behind only an error in the Windows Eventlog.

The Apache Access log shows this:
192.168.0.50 - - [28/Jan/2019:10:06:08 +0100] "GET /remote.php/dav/files/user/E2EE-folder/subfolder/file.pdf HTTP/2.0" 423 258 "-" "Mozilla/5.0 (Linux) mirall/2.5.1git (Nextcloud)"

The sync client on Fedora shows this in the log window:

[OCC::WebFlowCredentials::stillValid    "Error transferring https://cloud.my.domain.tld/remote.php/dav/files/user/E2EE-folder/subfolder/file.txt - server replied: "
[OCC::SyncJournalDb::setErrorBlacklistEntry     Setting blacklist entry for  "E2EE-folder/subfolder/file.txt" 11 "Error transferring https://cloud.my.domain.tld/remote.php/dav/files/user/E2EE-folder/subfolder/file.txt - server replied: " 1548667448 0 1514381889 "78bb2317ea76d49ab6c3c12f706b5af6" "" 0
[OCC::blacklistUpdate   escalating soft error on  "E2EE-folder/subfolder/file.txt"  to normal error,  423
[OCC::PropagateItemJob::done    Could not complete propagation of "E2EE-folder/subfolder/file.txt" by OCC::PropagateDownloadFile(0x62bf55922840) with status 2 and error: "Error transferring https://cloud.my.domain.tld/remote.php/dav/files/user/E2EE-folder/subfolder/file.txt - server replied: "
[OCC::ActivityWidget::slotItemCompleted         Item  "E2EE-folder/subfolder/file.txt"  retrieved resulted in  "Error transferring https://cloud.my.domain.tld/remote.php/dav/files/user/E2EE-folder/subfolder/file.txt - server replied: "
[OCC::ActivityWidget::slotItemCompleted         Item  "E2EE-folder/subfolder/file.txt"  retrieved resulted in error  "Error transferring https://cloud.my.domain.tld/remote.php/dav/files/user/E2EE-folder/subfolder/file.txt - server replied: "

Both the nextcloud.log and the Apache error.log contain no further information.

@camilasan May I ask you to reconsider the closed status on this issue?
Never mind, it appeared closed but is open. Need more coffee. Sorry for the noise!

[DC7IA]

Problem still seems to persist in the current version.

[er-vin]

I sounds a lot like this one and #1862 are related.

[camilasan]

I am still experiencing it with the current desktop client from master (build 11.07), end to end encryption app master too and Nextcloud 19. (I created 4 encrypted folder, one in Windows, 2 in Linux, one in Android)

[camilasan]

• I have created a fresh user, set up everything on Linux, created new folder, encrypted it, added a file to it.
• On Windows, it got stuck syncing everything.
• Restarted the client in Windows and got this at first: 2020-07-12 16:36:07:821 [ critical nextcloud.sync.propagator.download.encrypted ]: Failed to find encrypted metadata information of remote file "bank files"
• Added another folder under the encrypted folder from Linux and added file in it - client on windows got stuck, Linux doesn't see the new files from Windows yet.
• Eventually synced but the file under the subfolder of an encrypted folder did not sync and Linux and Windows are reporting that everything got synced (green check icon)

Windows (see file is there)

Linux (file is not there)

• Edited this file under the subfolder of an encrypted folder to try to trigger sync again. Also the files under the encrypted folder are not marked as encrypted in the database (expected?)

• Windows client it is stuck syncing again.

• Forcing sync again gives me the green check icon but file it is still missing on Linux... and a few seconds later it gets stuck again.
• Restarting the client also doesn't help.

linux_debug.log
linux_sync.log
windows_debug.log
windows_sync.log

[er-vin]

From your logs it looks like there is something wrong on the server side. For some reason it rejects any folder locking attempt with a 403 (on both the Linux and the Windows sides). I think we need to find out why before you can try to reproduce the issue reported here, it's something different getting in your way.

[camilasan]

I am still able to reproduce this issue. This is now in my localhost, server 19, end to end encryption from master: I still have a bunch of 403 in the logs of Windows and Linux.

[DominiqueFuchs]

Testing report:

Nextcloud daily master 20200714; everything default + SSL,HTTP2
E2EE master d3471e1
Desktop Client master 5c442af

Client1, Client2; Both Windows

1. Add testuser (Web Admin)
2. Connect Client1 w/ testuser
3. Enable E2EE / mnemonic
4. Create folder e2ee-sync-test
5. Check encrypt folder in client
6. Create file test.txt in e2ee-sync-test
7. All fine, synced
8. Opened file, typed text, saved
9. All fine, synced
10. Added dummy files in e2ee-sync.test (50x2MB, 40x8MB, filled with 0)
11. All fine, synced, log is clean/error-free
12. Connect Client2 w/ testuser (Client1 still running on other machine)
13. All fine, synced, can see and edit encrypted files, logs clean
14. Open test.txt in Client2, edit text, save but leaving file opened in editor
15. All fine, client syncs, changes appear in Client1
16. Open test.txt in Client1, edit text, save but leaving file opened in editor
17. All fine, client syncs, sync attempt in Client2 but changes not appearing b/c still opened, no error (this is by design)
18. Closing file on Client 2, check back few mins later, changes from 16 were synced down

No problems that way. I can throw in another client tomorrow on a different platform. No android devices here but spinning up a Linux distro with Client3 should be possible

// @camilasan

nextcloud-server.log.zip

[NewRedsquare]

I think that E2E itself can work but it's not trustworthy :/
I mean high risks of losing data

[camilasan]

I am closing this issue because I managed to get the syncing working by not using https. It was an SSL issue on my side. it works like a charm now.