Since enabling end2end encryption Desktop Sync Agents are no longer syncing (on several Desktop clients Windows 10)
Everytime when the Desktop Agent is trying to sync
Restart Agent agent shortly will bring sync to "green" .. one/two minutes later agent tries to resync and this resync will hung completely.
Only one Folder for testing purposes, was encrypted since.
Syncing should work
Sync does not work anymore.
Also restarting the Agent, does not help, nor force restart.
Operating system: Linux 4.18.0-193.19.1.el8_2.x86_64 nextcloud/end_to_end_encryption#1 SMP Mon Sep 14 14:37:00 UTC 2020 x86_64
Webserver: Apache (fpm-fcgi)
Database: mysql 10.3.17
PHP version:
7.4.11
Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, cgi-fcgi, bcmath, bz2, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, gmp, iconv, intl, json, ldap, exif, mysqlnd, PDO, Phar, posix, shmop, SimpleXML, sockets, sodium, sqlite3, sysvmsg, sysvsem, sysvshm, tokenizer, xml, xmlwriter, xsl, mysqli, pdo_mysql, pdo_sqlite, xmlreader, apcu, igbinary, imagick, msgpack, smbclient, zip, memcached, redis, libsmbclient, Zend OPcache
Nextcloud version: 19.0.4 - 19.0.4.2
Updated from an older Nextcloud/ownCloud or fresh install:
Update from 19.0.3
Where did you install Nextcloud from: Nextcloud source
Signing status
Array
(
)
List of activated apps
Enabled:
- accessibility: 1.5.0
- activity: 2.12.1
- analytics: 2.5.0
- announcementcenter: 3.8.1
- apporder: 0.11.0
- audioplayer: 2.12.0
- audioplayer_editor: 0.3.0
- audioplayer_sonos: 1.2.0
- bookmarks: 3.4.4
- bruteforcesettings: 2.0.1
- calendar: 2.0.4
- circles: 0.19.7
- cloud_federation_api: 1.2.0
- comments: 1.9.0
- contacts: 3.4.1
- contactsinteraction: 1.0.0
- cookbook: 0.7.6
- cospend: 1.0.5
- data_request: 1.6.0
- dav: 1.15.0
- deck: 1.1.2
- dicomviewer: 1.2.2
- documentserver_community: 0.1.8
- drawio: 0.9.7
- end_to_end_encryption: 1.5.2
- event_update_notification: 1.0.2
- extract: 1.2.4
- federatedfilesharing: 1.9.0
- federation: 1.9.0
- files: 1.14.0
- files_3d: 0.3.2
- files_antivirus: 3.0.0
- files_automatedtagging: 1.9.0
- files_downloadactivity: 1.8.0
- files_markdown: 2.3.1
- files_mindmap: 0.0.23
- files_pdfviewer: 1.8.0
- files_photospheres: 1.19.1
- files_rightclick: 0.16.0
- files_sharing: 1.11.0
- files_trashbin: 1.9.0
- files_versions: 1.12.0
- files_videoplayer: 1.8.0
- firstrunwizard: 2.8.0
- flowupload: 1.0.0
- forms: 2.0.4
- gpxedit: 0.0.13
- gpxmotion: 0.0.11
- gpxpod: 4.2.2
- groupfolders: 7.1.0
- impersonate: 1.6.1
- issuetemplate: 0.7.0
- logreader: 2.4.0
- lookup_server_connector: 1.7.0
- maps: 0.1.6
- nextcloud_announcements: 1.8.0
- notes: 3.6.4
- notifications: 2.7.0
- oauth2: 1.7.0
- onlyoffice: 6.1.0
- passman: 2.3.6
- password_policy: 1.9.1
- photos: 1.1.0
- polls: 1.5.4
- privacy: 1.3.0
- provisioning_api: 1.9.0
- quicknotes: 0.6.1
- quota_warning: 1.8.0
- rainloop: 7.0.3
- ransomware_protection: 1.7.0
- recommendations: 0.7.0
- serverinfo: 1.9.0
- settings: 1.1.0
- sharebymail: 1.9.0
- socialsharing_diaspora: 2.1.0
- socialsharing_email: 2.1.0
- socialsharing_facebook: 2.1.0
- socialsharing_twitter: 2.1.0
- spreed: 9.0.4
- support: 1.2.1
- suspicious_login: 3.2.1
- systemtags: 1.9.0
- tasks: 0.13.5
- terms_of_service: 1.5.2
- text: 3.0.1
- theming: 1.10.0
- timemanager: 0.1.4
- twofactor_backupcodes: 1.8.0
- twofactor_totp: 5.0.0
- updatenotification: 1.9.0
- video_converter: 0.1.5
- viewer: 1.3.0
- workflow_ocr: 1.19.1
- workflow_pdf_converter: 1.4.0
- workflow_script: 1.4.0
- workflowengine: 2.1.0
Disabled:
- admin_audit
- breezedark
- dashboard
- drop_account
- encryption
- external
- files_accesscontrol
- files_external
- files_fulltextsearch
- fulltextsearch
- fulltextsearch_elasticsearch
- joplin
- jsloader
- passwords
- registration
- richdocuments
- socialsharing_googleplus
- survey_client
- user_ldap
- weather
Configuration (config/config.php)
{
"memcache.local": "\\OC\\Memcache\\APCu",
"filelocking.enabled": true,
"redis": {
"host": "***REMOVED SENSITIVE VALUE***",
"port": EMOVED SENSITIVE VALUE*,
"dbindex": EMOVED SENSITIVE VALUE*,
"timeout": EMOVED SENSITIVE VALUE*,
"password": "***REMOVED SENSITIVE VALUE***"
},
"instanceid": "***REMOVED SENSITIVE VALUE***",
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": [
".REMOVED SENSITIVE VALUE*.",
"REMOVED SENSITIVE VALUE*."
],
"datadirectory": "***REMOVED SENSITIVE VALUE***",
"overwrite.cli.url": "https:\/\/www.swiss7cloud.ch",
"htaccess.RewriteBase": "\/",
"overwriteprotocol": "REMOVED SENSITIVE VALUE",
"dbtype": "mysql",
"version": "19.0.4.2",
"dbname": "***REMOVED SENSITIVE VALUE***",
"dbhost": "***REMOVED SENSITIVE VALUE***",
"dbport": "",
"dbtableprefix": "oc_",
"mysql.utf8mb4": true,
"dbuser": "***REMOVED SENSITIVE VALUE***",
"dbpassword": "***REMOVED SENSITIVE VALUE***",
"installed": true,
"maintenance": false,
"theme": "",
"loglevel": 0,
"updater.release.channel": "stable",
"auth.bruteforce.protection.enabled": true,
"check_for_working_htaccess": true,
"data-fingerprint": "REMOVED SENSITIVE VALUE",
"mail_from_address": "***REMOVED SENSITIVE VALUE***",
"mail_smtpmode": "smtp",
"mail_smtpauthtype": "LOGIN",
"mail_domain": "***REMOVED SENSITIVE VALUE***",
"mail_smtpsecure": "REMOVED SENSITIVE VALUE",
"mail_smtpauth": 1,
"mail_smtpname": "***REMOVED SENSITIVE VALUE***",
"mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
"mail_smtphost": "***REMOVED SENSITIVE VALUE***",
"mail_smtpport": "REMOVED SENSITIVE VALUE",
"session_lifetime": 1200,
"session_keepalive": false,
"logtimezone": "REMOVED SENSITIVE VALUE\/REMOVED SENSITIVE VALUE",
"logfile": "\/media\/log\/nextcloud.log",
"knowledgebaseenabled": false,
"log_rotate_size": REMOVED SENSITIVE VALUE,
"mail_sendmailmode": "smtp",
"app_install_overwrite": [
"passman",
"dicomviewer",
"radio"
]
}
Are you using external storage, if yes which one: no
Are you using encryption: no
Are you using an external user-backend, if yes which one: no
Sync Agent version: 3.0.2
Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36 Edg/86.0.622.43
Operating system: W10
Web server error log
Insert your web server log here
Nextcloud log
Insert your Nextcloud log here
Browser log
Insert your browser log here, this could for example include:
a) The javascript console log
b) The network log
c) ...
This repo is about e2e server app.
I will transfer it to Desktop repo.
Do you experience spikes in the server consumption? There's a problem about the e2e server app that gets the server consumption to go crazy when there's a large amount of files, which leads to timeouts sent back to the client hence why the client stops syncing.
If that's the case we got a tentative workaround for this shipped with 3.0.3 later this week.
If that's not the case it might be something else in which case I'd like to get debug logs of the client during a sync to see where it gets stuck.
Hey Kevin .. i do not think so, i tried the e2ee with my User .. (not so much files) .. and with only 258 Files (*.jpg Files).
After copying/syncing/encrypting 13 Files .. the whole get stuck ..
i'm using a Intel Xeon E-2136 @ 3.30GHz | average CPU mark : 13,506
For the VM i'm using four vCPUS, Total memory: 8GB .. the VM itself gots never a spike - average load between 0 and 20%.
You can better decide, if this issue is resolved by the workaround (3.0.3) or there is another issue.
I did saved all the NC Logs .. if you would to check them .. or we could be repeat the case (but this is a productive system .. hm .. better test it on a test VM) ..
No that looks like something else. Note that I'm more interested in the client logs than the server logs though. For such a case server logs might not help much.
ok Kevin .. which client logs you're looking for ?
I'm referring to client debug logs, this can be achieved by adding the following in the nextcloud.cfg General group:
logDebug=true
logExpire=24
logDir=C:/nextcloud-logs
Also make sure C:\nextcloud-logs exists before launching the client. After launching, log files should appear in that folder, that's the files I'm interested in.
I didn't want to open another issue for a similar problem, but in my case the e2ee downloading task works very slowly (it takes minutes to check for changes in remote folders).
It works for downloading data, but very slowly.
I tried to upload around 10 MB of data from my computer to the encrypted folder, and it's completely stuck

I like this e2ee feature, i really want it to be working 馃憤
Here are the client desktop logs :
20201030_2137_owncloud.log.0.txt
20201030_2137_owncloud.log.1.txt
20201030_2138_owncloud.log.0.txt
20201031_1146_owncloud.log.0.txt
20201031_1147_owncloud.log.0.txt
20201031_1148_owncloud.log.0.txt
20201031_1152_owncloud.log.0.txt
Thanks NewRedsquare for delivering the logs .. i would need more time for this, because - i do not want to repeat to enable e2ee on my productive system - not that users will be affected probably.
I need to build this up from scratch on my virtual environment (like VirtualBox).
On the other hand Kevin, was this not also deeply tested in one of your NC environments (performance .. sql behave and so on) ? or is this client specific ?
I didn't want to open another issue for a similar problem, but in my case the e2ee downloading task works very slowly (it takes minutes to check for changes in remote folders).
It works for downloading data, but very slowly.
I tried to upload around 10 MB of data from my computer to the encrypted folder, and it's completely stuck
I like this e2ee feature, i really want it to be working +1
Here are the client desktop logs :
20201030_2137_owncloud.log.0.txt
20201030_2137_owncloud.log.1.txt
20201030_2138_owncloud.log.0.txt
20201031_1146_owncloud.log.0.txt
20201031_1147_owncloud.log.0.txt
20201031_1148_owncloud.log.0.txt
20201031_1152_owncloud.log.0.txt
Thanks for that, such logs are often what we miss to make progress on those. In your case it's blocked due to this:
2020-10-31 11:48:45:843 [ warning nextcloud.sync.networkjob ]: QNetworkReply::ContentAccessDenied "Le serveur a r茅pondu \"403 Forbidden\" 脿 \"POST https://***HIDDEN VALUE***/ocs/v2.php/apps/end_to_end_encryption/api/v1/lock/146663?format=json\"" QVariant(int, 403)
Then it tries again at 2020-10-31 11:53:03:013 with the same result.
The server is replying that the folder is already locked (there's a lock mechanism on the encrypted folders, it's part of the protocol, we have to lock/modify/unlock). That said in the logs you provided I don't see any other lock attempt so it doesn't feel like the client deadlocked itself.
Do you see anything in the server logs around the same time which would tell us why there is a stale lock? Would you happen to have another client holding the lock somehow? (shouldn't hold it forever of course but I'm thinking aloud here)
On the other hand Kevin, was this not also deeply tested in one of your NC environments (performance .. sql behave and so on) ? or is this client specific ?
Well, we knew that the performance on the client side could be better. There are a couple of bottlenecks coming from the existing architecture (this code base is fairly old compared to other clients) which we can't solve yet (mainly because some other pieces are not in the right place yet + manpower). So it's expected to be slower than it could be while uploading encrypted files.
As for the SQL going mad on the server side (I guess this is what you were referring to) which should be fixed by 3.0.3, this didn't show up in our tests. I suspect there was a threshold effect somewhere at an amount of file larger than what we got in our tests.
I didn't want to open another issue for a similar problem, but in my case the e2ee downloading task works very slowly (it takes minutes to check for changes in remote folders).
It works for downloading data, but very slowly.
I tried to upload around 10 MB of data from my computer to the encrypted folder, and it's completely stuck
I like this e2ee feature, i really want it to be working +1
Here are the client desktop logs :
20201030_2137_owncloud.log.0.txt
20201030_2137_owncloud.log.1.txt
20201030_2138_owncloud.log.0.txt
20201031_1146_owncloud.log.0.txt
20201031_1147_owncloud.log.0.txt
20201031_1148_owncloud.log.0.txt
20201031_1152_owncloud.log.0.txtThanks for that, such logs are often what we miss to make progress on those. In your case it's blocked due to this:
2020-10-31 11:48:45:843 [ warning nextcloud.sync.networkjob ]: QNetworkReply::ContentAccessDenied "Le serveur a r茅pondu \"403 Forbidden\" 脿 \"POST https://***HIDDEN VALUE***/ocs/v2.php/apps/end_to_end_encryption/api/v1/lock/146663?format=json\"" QVariant(int, 403)Then it tries again at 2020-10-31 11:53:03:013 with the same result.
The server is replying that the folder is already locked (there's a lock mechanism on the encrypted folders, it's part of the protocol, we have to lock/modify/unlock). That said in the logs you provided I don't see any other lock attempt so it doesn't feel like the client deadlocked itself.
Do you see anything in the server logs around the same time which would tell us why there is a stale lock? Would you happen to have another client holding the lock somehow? (shouldn't hold it forever of course but I'm thinking aloud here)
I don't have access to server logs, my nextcloud server is managed by a provider ( still have admin access to the nextcloud UI ) sadly
Ah too bad... Well I guess one thing we could try is to wait 24 hours while your client would be completely shutdown. Locks from the E2EE server app are supposed to be released after 24 hours.
Then you could try again and see if the same lock shows up in the client logs, or maybe we figure through those logs how a stale lock happened in the first place.
I have exactly the same problem as NewRedsquare, and in my case as well it seems to be related to the E2EE app not allowing the client to lock a folder.
Uploading and syncing to a just created end-to-end encrypted folder worked fine, but after restarting my machine sync is stuck displaying the "0 seconds remaining" message posted above when trying to add new files.
Absolutely nothing on server log at /var/www/nextcloud/data/nextcloud.log. I wonder if the E2EE-app has its own log-file somewhere?
Server version: 20.0.1.1
Client version: 3.0.3 (Windows)
Solved the issue by manually deleting E2EE-apps locks from the database with DELETE FROM oc_e2e_encryption_lock.
Main cause of the problem seems to be that somehow the E2EE-apps locks aren't removed/released properly. I think that a cron-task which would remove all of the locks periodically might work as a temporary fix.
Other things I tried:
DELETE FROM oc_file_locks WHERE 1 -> no effect, since E2EE-apps locks are stored in a different place.Main cause of the problem seems to be that somehow the E2EE-apps locks aren't removed/released properly. I think that a cron-task which would remove all of the locks periodically might work as a temporary fix.
Note that they're supposed to expire after 24h though.
nx-client-3.0.3.log
All right I'll stick to this issue. My sync stops as well, but so far I didn't see 403 Forbidden. I'm running into timeout issues on the other hand, if E2E is enabled (see attached log).
Gathered up some more information related to the locking-related sync-stuck-permanently issue.
This part of the E2EE-app (Sabre/LockPlugin.php) throws a HTTP 403 error with the log entry Write access to end-to-end encrypted folder requires token -no token sent if no token is sent by the client.
I have certainly seen the Write access to end-to-end encrypted folder requires token - no token sent error in the server logs many times before, but for some reason there is nothing in the logs from the time I took the log I linked above...
Write access to end-to-end encrypted folder requires token -no token sent on server logWrite access to end-to-end encrypted folder requires token -no token sent on server log.Write access to end-to-end encrypted folder requires token -no token sent on server log.Write access to end-to-end encrypted folder requires token -no token sent on server log.Write access to end-to-end encrypted folder requires token -no token sent on server log.Write access to end-to-end encrypted folder requires token -no token sent on server log.Seems to be an issue related to tokens not being sent properly by the desktop client, as there doesn't seem to be any related issues on android and iOS repositories.
Updated the summary above with some additional related issues. Hopefully it could be useful.
After manually deleting the E2EE-apps locks I haven't had any issues with syncing to encrypted folders, but I increased the servers logging level to DEBUG in hopes of possibly catching something in the future.
I don't have any C++ experience, but might dig in to the desktop clients source code to see if I could find anything strange with the token handling. Doing side-to-side comparison with the mobile clients token handling might be useful, as they don't seem to have any related issues.
Thanks for your efforts, you might want to enable logs on the client as well then. In the General group of your nextcloud.cfg I'd advise adding something like the following:
logDebug=true
logExpire=24
logDir=C:/nextcloud-desktop-logs
I'll let you adjust the logExpire, it's in hours so obviously the bigger the better since it looks like there's some time between those stale locks being created and the client actually getting stuck on one of those. But the bigger the most disk space consumed by the logs (in my tests we're around 100MB/day).
Note that what I'm proposing here will be the new defaults for 3.1.0 (+ a button to harvest it all in a zip file) to help everyone chase this kind of stuff.
I've had similar issues. NC desktop 3.0.3 showed "Waiting ..." forever on my clients (Linux, macOS). The problem appeared when upgrading frmo 19.0.4 to 20.0.1.
Disabling end2end encryption server-side mitigated the problem for me, but this is obviously suboptimal...
Note: running DELETE FROM oc_e2e_encryption_lock; did not help, the table was already empty for me. I really had to remove the e2ee plugin.
Note: running
DELETE FROM oc_e2e_encryption_lock;did not help, the table was already empty for me. I really had to remove the e2ee plugin.
@mpranj Did you see 'Connection timed out' messages in you client log? I'm just curious.
It appears once in the logs, yes.
But this appears >3000 times:
2020-11-03 13:34:29:274 [ warning nextcloud.sync.networkjob ]: Network job timeout QUrl("remote.php/webdav")
and this appears >6000 times:
[ info nextcloud.sync.networkjob ]: OCC::GetFolderEncryptStatusJob created for "https://example.com" + "remote.php/webdav" "OCC::ClientSideEncryption"
Server-side there was no high load or anything suspicious to my knowledge.
Someone has another logs to provide ? I had to disable e2ee, didn't want to lose work's files :/
it is the same for me, logs provided in https://github.com/nextcloud/desktop/issues/2314. Nothing changed. NC server now 20.0.2, client 3.1.x (trying daily as well)
Yes, trying to get some help from the server side and they've been busy so far. We'll need some adjustments to get this to rest and I can't solve this until this happens.
I have the same issue.
As soon as I try to sync the encrypted folder on the desktop client, the sync process stuck.
When I untick the encrypted folder, the sync completed without problem and icon turned green.
I am on the stable Nextcloud 19 and windows client 3.x
Same issue here, have 3,1.0 clients and NC 20.0.4 running on Fedora 32 and all clients just sit there waiting.
Very frustrating. Is someone working on it? There were logs provided. What else can we do?
Well yes I do. I'd expect this fixed with 3.2 not before. For those on Linux I'm interested in someone testing the AppImage from #2700 and letting us know how this goes (obviously warning ensue: it might kill puppies). This should hopefully fix the "server meltdown" case with E2EE. I don't expect it fixes the lock problem on some of the E2EE folder deletion we've been seeing (reported in another ticket) but that part is likely to be tackled during the 3.2 cycle as well.
Well yes I do. I'd expect this fixed with 3.2 not before. For those on Linux I'm interested in someone testing the AppImage from #2700 and letting us know how this goes (obviously warning ensue: it might kill puppies). This should hopefully fix the "server meltdown" case with E2EE. I don't expect it fixes the lock problem on some of the E2EE folder deletion we've been seeing (reported in another ticket) but that part is likely to be tackled during the 3.2 cycle as well.
Do we know when 3.2 is going to be officially released?
Well yes I do. I'd expect this fixed with 3.2 not before. For those on Linux I'm interested in someone testing the AppImage from #2700 and letting us know how this goes (obviously warning ensue: it might kill puppies). This should hopefully fix the "server meltdown" case with E2EE. I don't expect it fixes the lock problem on some of the E2EE folder deletion we've been seeing (reported in another ticket) but that part is likely to be tackled during the 3.2 cycle as well.
Do we know when 3.2 is going to be officially released?
It is aimed for end of March. This contains major changes to the engine so it'll need quite some QA time before we're confident releasing it in the wild.
Thanks for the timetable.
Even that means 3 more months of server pain OR not being able to use E2EE at all.
That would be 7 months after promoting it as "production ready" https://nextcloud.com/press/pr20200818/ ... anyway, at least there's light at the end of the tunnel.
Thanks for knowing that it till come in March, but still a long time.
What is the best way to disable e2ee encryption in the meantime?
I have one account that encrypts only one folder what is the best way to not loose data. What would happen to this folder if I deactivate e2ee? Would it get synced in plaintext again? Should I put this folder outside of nextcloud? I want to make Nextcloud usable for all other accounts until this is fixed (would downloading a client version before 3.x help if I don't need e2ee on these clients? Like 2.3.0.3, I believe this version was very stable)
Hope you can help us to have some manual/plan what to do in the meantime what is best to use the rest of the clients without e2ee.
Blessed Christmas by the way馃巹
A shame that this feature was published without testing
Most helpful comment
Gathered up some more information related to the locking-related sync-stuck-permanently issue.
Error source
This part of the E2EE-app (Sabre/LockPlugin.php) throws a HTTP
403error with the log entryWrite access to end-to-end encrypted folder requires token -no token sentif no token is sent by the client.I have certainly seen the
Write access to end-to-end encrypted folder requires token - no token senterror in the server logs many times before, but for some reason there is nothing in the logs from the time I took the log I linked above...Related issues from the E2EE-app repository:
Write access to end-to-end encrypted folder requires token -no token senton server logRelated issues from the desktop repository:
Write access to end-to-end encrypted folder requires token -no token senton server log.Write access to end-to-end encrypted folder requires token -no token senton server log.Write access to end-to-end encrypted folder requires token -no token senton server log.Write access to end-to-end encrypted folder requires token -no token senton server log.Related issue from the server repository:
Write access to end-to-end encrypted folder requires token -no token senton server log.Summary
Seems to be an issue related to tokens not being sent properly by the desktop client, as there doesn't seem to be any related issues on android and iOS repositories.