Server: 503 Encryption not ready: multikeydecrypt with share key failed

[CamZie]

Steps to reproduce

1. enable encryption
2. upload and download files

Expected behaviour

Nextcloud should allow downloading of files without any errors.

Actual behaviour

Cannot download some files. User is receiving errors that the server is temporarily unavailable (503) or that the server is in maintenance.

Server configuration

Operating system: Debian 8.10

Web server: NGINX 1.12

Database: MariaDB 10.0

PHP version: PHP 5.6

Nextcloud version: 12.0.2

Updated from an older Nextcloud/ownCloud or fresh install: Updated from an older Nextcloud version.

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - activity: 2.5.2
  - admin_audit: 1.2.0
  - bookmarks: 0.10.1
  - bruteforcesettings: 1.0.3
  - calendar: 1.5.7
  - comments: 1.2.0
  - contacts: 2.0.1
  - dav: 1.3.0
  - encryption: 1.6.0
  - federatedfilesharing: 1.2.0
  - files: 1.7.2
  - files_pdfviewer: 1.1.1
  - files_sharing: 1.4.0
  - files_texteditor: 2.4.1
  - files_trashbin: 1.2.0
  - files_versions: 1.5.0
  - files_videoplayer: 1.1.0
  - firstrunwizard: 2.1
  - gallery: 17.0.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.0.0
  - mail: 0.7.9
  - nextcloud_announcements: 1.1
  - notes: 2.3.2
  - notifications: 2.0.0
  - oauth2: 1.0.5
  - password_policy: 1.2.2
  - provisioning_api: 1.2.0
  - qownnotesapi: 17.5.0
  - serverinfo: 1.2.0
  - sharebymail: 1.2.0
  - systemtags: 1.2.0
  - tasks: 0.9.5
  - theming: 1.3.0
  - twofactor_backupcodes: 1.1.1
  - updatenotification: 1.2.0
  - workflowengine: 1.2.0
Disabled:
  - federation
  - files_external
  - survey_client
  - user_external
  - user_ldap

Nextcloud configuration:

Config report

    "system": {
        "instanceid": "ocpom4ncgfhghkwru",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "\/mnt\/***REMOVED SENSITIVE VALUE***\/data",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "12.0.2.0",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "Europe\/Zurich",
        "installed": true,
        "theme": "***REMOVED SENSITIVE VALUE***",
        "enable_previews": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "enable_avatars": false,
        "logdateformat": "Y-m-d_H:i:s",
        "updatechecker": false,
        "log_type": "errorlog",
        "logfile": "",
        "loglevel": 2,
        "customclient_desktop": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "trashbin_retention_obligation": "auto,90",
        "activity_expire_days": 90,
        "preview_max_scale_factor": 1,
        "preview_max_filesize_image": 10,
        "skeletondir": "***REMOVED SENSITIVE VALUE***",
        "mail_from_address": "no-reply",
        "mail_smtpmode": "php",
        "mail_smtpauthtype": "LOGIN",
        "mail_domain": "***REMOVED SENSITIVE VALUE***"}

Are you using encryption: yes

Client configuration

Browser:
Operating system: Nextcloud-iOS/2.19.2

Logs

Nextcloud log (data/nextcloud.log)

Nextcloud log

2018/02/10 04:14:07 [error] 32243#32243: *2115256 FastCGI sent in stderr: "PHP message: [owncloud]
[webdav][4] Exception: {"Exception":"Sabre\\DAV\\Exception\\ServiceUnavailable","Message":"Encryption
 not ready: multikeydecrypt with share key failed:error:0906D06C:PEM routines:PEM_read_bio:no start 
line","Code":0,"Trace":"#0 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php(85): 
OCA\\DAV\\Connector\\Sabre\\File->get()\n#1 [internal function]: Sabre\\DAV
\\CorePlugin->httpGet(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#2 \/var\
/www\/nextcloud\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, 
Array)\n#3 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(479): Sabre\\Event
\\EventEmitter->emit('method:GET', Array)\n#4 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV
\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), 
Object(Sabre\\HTTP\\Response))\n#5 \/var\/www\/nextcloud\/apps\/dav\/appinfo\/v1\/webdav.php(71): 
Sabre\\DAV\\Server->exec()\n#6 \/var\/www\/nextclo" while reading response header from upstream, client: 
***REMOVED SENSITIVE VALUE***, server: ***REMOVED SENSITIVE VALUE***, request: "GET 
/remote.php/webdav/Photos/2018/01/18-01-19%2018-37-42%200433.jpg HTTP/2.0", upstream: 
"fastcgi://unix:/var/run/php5-fpm.sock:", host: "***REMOVED SENSITIVE VALUE***"

[RandieM]

Same problem here with two different Nextcloud 12.0.2 installations. One installation is running on Debian 8 and the other is running on Debian 9, for what it's worth. The rest of my set-up is pretty much the same as @CamZie's. Any ideas?

[tflidd]

@schiessle

[albertogscotti]

Same problem on 13.0.2.
Happens on sharing encrypted directories / files.
Also: php occ encryption:migrate throws a lot of errors "An unhandled exception has been thrown:
ArgumentCountError: Too few arguments to function OCA\EncryptionMigration::__construct()"

[hostingnuggets]

Same problem here too with 13.0.1.

@schiessle what is the status or progress regarding this encryption related bug?

[berho]

I have the same problem on 13.0.2!
A lot of files can not be syncronized over dav. This Version of NextCloud is not stable to use in a productive environment!!

How can I get back my files??

[m33m33]

Same problem here on 13.0.4 stable release.
Server side encryption activated = impossible to share files.
This encryption feature should be disabled on the stable/production releases

[CamZie]

Is there any updates or news for this issue? This is starting to be a big problem since it is impossible to access the files anymore...

[Escubaer]

Is this reproducable with the newest versions eg. 12.0.9? https://github.com/nextcloud/server/wiki/Maintenance-and-Release-Schedule if not it seems rather something for a subscription service - if yes it should get immediate attention, indeed.

[CamZie]

We have upgraded our installation to 13.0.1 and this issue still persists. We haven't been able to identify the cause of this problem.

[RandieM]

@Escubaer, I am having trouble understanding your comment, as several other users (refer to comments above) have already reported that they experience this issue in versions 13.0.1, 13.0.2 and 13.0.4. People keep losing, possibly irreversibly, their data; how does such a major issue qualify as a case for a subscription service?

[Escubaer]

@RandieM I am not working for the vendor, just to make sure. I think I am trying to say and ask if this can be reproduced with a brand new setup eg. with 13.0.4 and with which steps or if this is random and rather happened suddenly in people's running environment. IMHO this will make debugging difficult and therefor it seems maybe more for the subscription/support service. Besides that no developer seems to come up with any idea or solution till now here ...

[Escubaer]

It is also maked as a feature whereas for you guys it sounds like a strong bug ...

[RandieM]

@Escubaer, when it comes to programming, I tend not to believe in "random" events. The described problem is triggered by something, which I am currently unable to identify. This also seems to be the case for @CamZie, according to his/her latest comment.

Besides, you do have a point when you say:

It is also maked as a feature whereas for you guys it sounds like a strong bug ...

I believe that this issue has been assigned the wrong label, as it is certainly not a feature, but a bug /cc @tflidd

[tflidd]

I believe that this issue has been assigned the wrong label, as it is certainly not a feature, but a bug /cc @tflidd

It just says that this topic is related to the server-side-encryption. There are different tags for feature requests ;-)

But regarding the number of users reporting this problem, it is probably more than just a single coincidence. I will put a bug-label to it.

[RandieM]

Thanks @tflidd for the explanation and for the assignment of the new label.

[berho]

I had this wired error once more today and I tested around but can't get any clue why that happens:

Upload from: ------------> Server Thumbnail Creation --------> Download to Windows Client
----------------------------(View and download with Browser)----------------------------------

iOS App ----------------------------> OK --------------------------------> Fail
iOS send To NextCloud--------------> OK --------------------------------> Fail
Browser (FF on W7) ------------------> OK -------------------------------> Fail
Windows Client ----------------------> OK -------------------------------> (uploaded)

Error in Logfile always:
SabreDAV\Exception\ServiceUnavailable: Encryption not ready: multikeydecrypt with share key failed:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error
/htdocs/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 88: OCADAVConnector\Sabre\File->get()
[internal function] SabreDAVCorePlugin->httpGet(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))

[CamZie]

Yet another user has this problem and they keep receiving this error when trying to access their files. multikeydecrypt with share key failed:error:0906D06C:PEM routines:PEM_read_bio:no start line

Any news on this as it is getting more and more critical?

[m33m33]

Issue still present on 13.0.5.

As a workaround, is it safe to follow https://docs.nextcloud.com/server/13/admin_manual/configuration_files/encryption_configuration.html and decrypt files with occ ?

No, or I may do something wrong...

After using : php occ encryption:decrypt-all user1

The files are still encrypted on the storage, and users get a "bad signature" on all files. Better have a good backup.

In nextcloud.log :
"Exception: {\"Exception\":\"OCP\\Encryption\\Exceptions\\GenericEncryptionException\",\"Message\":\"Bad Signature\",\"Code\":0,\"Trace\":\"#0 \/mnt\/sd0d\/usr\/pkg\/share\/nextcloud\/apps\/encryption\/lib\/Crypto\/Crypt.php(465)

A decrypted file "About.txt":
file data/user1/files/Documents/About.txt
data/user1/files/Documents/About.txt: data ===> should be "text"

First few lines of About.txt:
"
HBEGIN:oc_encryption_module:OC_DEFAULT_MODULE:cipher:AES-256-CTR:signed:true:HEND----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
"
Still encrypted...

[RandieM]

It seems that his behaviour is sometimes triggered by a password change, although I do have users in the same installation that have never changed their password, yet they experience this problem.

Any help would be greatly appreciated, as an increasing number of my users are permanently losing access to their files!

[m33m33]

A clue about this issue: it seems related to public link shared files only:

A. I share a file with a user of my nextcloud instance: the user can open the file.
B. I share a file with a public link (url): the link is unusable and throws the multikeydecrypt error message.

[RandieM]

@m33m33, thanks for posting. Initially, I also thought that this was the case, but, in my experience, it does not only happen with shared files.

[CamZie]

Are there any updates or news for this issue?

Just as @RandieM and @m33m33 mentioned, I have also noticed that these are mostly triggered by a password change or shared files, but some of my users also do not have either of them but are still experiencing this problem. Any help would be greatly appreciated.

[m33m33]

Bug still lives in v14. Exterminate. Exterminate. Exterminate.

Considering total NC removal under users grunts.

"
Can't read file multikeydecrypt with share key failed:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error
"

[albertogscotti]

Dropping NC too

[m33m33]

Another clue about this issue: it seems image format files are not affected.

A. I share a picture (.jpg) with a public link : the destination user can open the link and the image shows in NC viewer.
B. I share a document (.pdf, .odt...) with a public link : the link is unusable and throws the multikeydecrypt error message.

[hostingnuggets]

@m33m33 The behavior you describe in your point A might be the effect of the cache: my assumption here is that image files get cached unencrypted and this picture file you shared with a public link is then accessed directly from the cache, that's why it works.

Have a look at my comment here and the answers below on the nextcloud forum: https://help.nextcloud.com/t/nextcloud-14-focus-on-security-and-compliance/36116/2

In my comment I have asked the nextcloud core team why they don't seem to care about fixing and even replying to all the server-side encryption issues...

[m33m33]

@m33m33 The behavior you describe in your point A might be the effect of the cache: my assumption here is that image files get cached unencrypted and this picture file you shared with a public link is then accessed directly from the cache, that's why it works.

You are right. I am fooled by the preview from cache, if I click on "download" the picture don't show and the multikey failure message appears :(

[schiessle]

I have the feeling that this issue mixes many potential different problems together. E.g. the original issue says that the user gets a "503 Nextcloud unavailable or in maintenance mode" which I never saw and I don't know how this could be triggered by the server side encryption. The other error messages posted here make more sense but I still struggle to find the necessary information and what all this reports have in common in order to try to reproduce it.

So my request to everyone in this issue. Can someone of you describe a step by step scenario with the latest Nextcloud version (13.0.6 or 14, because they contain some changes to make the file cache updates more robust) where they can reliable reproduce the issue?

If I have something like this I'm happy to give it another try and see if I can reproduce it.

[m33m33]

NC 14, current user with server side encryption enabled.

1. Upload a new file : picture.jpg

2. Create a public link

3. Give public link to an external anonymous user : the preview is OK.
   Click on "Download" shows "Can't read file multikeydecrypt with share key failed:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error"

[schiessle]

@m33m33 I just tried it and couldn't reproduce it. Does the error happens for you reliable with all files? Is it a fresh installation or a update from a older version. On which version did you enabled encryption? Do you still use per-user keys or the new master key?

[m33m33]

@schiessle it happens on 100% files.
It's an update, I've been updating to all new versions since 13.whatever.
I enabled server side encryption with per-user key on v13.whatever, sorry I don't remember the exact release, but it should be the current release as of 16 Jun (when I joined this ticket).

[schiessle]

@m33m33 thanks. I tested it with the master key. I will try it with per user keys as well.

[schiessle]

Sadly I also couldn't reproduce it with per user keys. Can you check this folder data/<owner>/files_encryption/keys/files/<filename>/OC_DEFAULT_MODULE and tell me what's in it?

After you created a public link there should be three files in it:

• <owner>.shareKey
• fileKey
• pubShare_<some-random-chars>.shareKey

Also please post the error message you see in the nextcloud.log directly after you tried to download the file from the public link.

[m33m33]

Hi,
issue still present in 14.0.1.

Yes, a public link does create three files in OC_DEFAULT_MODULE:
fileKey {user}.shareKey pubShare_{8 caracters}.shareKey

Log extract directly after a click on "download:"
https://paste.ee/p/NG6R2

[m33m33]

Another clue about this issue to help you narrow the search :

1. Internal user create a folder and share it with public link.
2. Foreign user upload files to the folder
3. Both internal and foreign user can preview and download files without encryption error.

[chrsch]

This is still an issue in 14.0.8, is this solved in NC15/16 already?

[m33m33]

Still in 15.x

[wandoliver]

Just ran into the same problem on NC 15. No previews are generated anymore, always running into this error message. For both, shared and non-shared images.

[chrsch]

@schiessle I would like to nail this down in current NC15. Do you have some hints where to start digging in the encryption app? Thanks.

[m33m33]

@schiessle I would like to nail this down in current NC15. Do you have some hints where to start digging in the encryption app? Thanks.

Sorry, had to take the NC service down because of data loss with this issue. Can't provide much more help now :/

It's that simple: setup a NC, use it a little, then enable server side encryption, use it a little more and it may happen.

[moonmeister]

Issue still happening in v16...in fact it only started happening for me after i upgraded to v16.

[luisBSBDATA]

Also happening here on 13.0.2.. It was running just fine for more than a year now and suddenly today it started to happen. I'm extremely scared as it's sensitive and important data and can't seem to recover it

[hurrrrricaaaaane]

Same here with nextcloud 16 after enabling 2 Factor Auth and E2E encryption

[Zmmfly]

Same problem after decrypt all files, but still encrypted.

[Zmmfly]

Version:16.3

[bjo81]

Same sh*t, different instance on 16.0.3.

[tylon85]

The same here: https://github.com/nextcloud/notes/issues/328

Nextcloud: 16.0.4

Only that in this case it is not about shared files, but about notes of the Notes android app.

The whole thing started with the activation of 2FA. Since then the files of the notes (via android app and web browser) have been duplicated when trying to open them and the name has been extended by "(2)".
So "filename/note.txt" to "filename/note (2).txt".

This worked for a while and now (probably after an update) it doesn't let any of the files/notes open anymore, but it always comes the already known error message.

Can it be that Nextcloud has stored or finds two different keys and does not know which one to take? Can I influence this somehow?

I have deactivated 2FA again, but the problem remains.
Currently "Encrypt the home storage" is still activated. Would it change anything if I deactivated it or is all data useless in any case?

[tylon85]

Even after deactivating the server-side encryption, it is not possible to access the data. The data remains encrypted with an (now) unknown key.

Maybe already encrypted data was encrypted again with another key. The first key does not exist anymore and therefore decryption is impossible.

I think that an update has destroyed everything and there is no possibility to access the data anymore.

This is a real worst-case scenario for a software that is widely used by the ITZBund (Informationstechnikzentrum Bund in Germany), as well as by authorities in France, Sweden and the Netherlands, and really shameful.

[Fjodor42]

Also bitten by this, and will be happy to provide any logs deemed necessary to nail this.

@schiessle: Are you still involved in this?

[Fjodor42]

To clarify:

I am not entirely unwilling to actually grant access to my instance, or to the actual server itself, provided that the developer to be such such access can be vouched for by other (lead) developers.

What I am seeing specifically, is the following, on a file in a folder that is both shared with me from a user, and has a share link, with the file opening fine for my own user, but throwing an error when accessed through the share link to the folder (the listing of the folder seems fine in both cases):

Can't read file

multikeydecrypt with share key failed:error:04065084:rsa routines:rsa_ossl_private_decrypt:data too large for modulus

@schiessle , if you're there, or anyone else if not: Let me know if there is anything I can do to help debug

Googling "data too large for modulus" doesn't really get me anywhere, except that it could be related to padding, could be related to the RSA implementation as it pertains to async features, and could be something else entirely, but my Google-fu might be weak here...

[espressoelf]

I had a similar problem when I tried to sync from the server (17.0.1) to desktop (2.6.1stable-Win64 (build 20191105). The serverlogs said multikeydecrypt with share key failed:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error and Encryption not ready: multikeydecrypt with share key failed:error:04065072:rsa routines:rsa_ossl_private_decrypt:padding check failed but also something about {"reqId":"IliU9KMskc6JQB3MLhMV","level":3,"time":"2019-12-18T14:37:15+00:00","remoteAddr":"...","user":"zh","app":"no app in context","method":"GET","url":"\/remote.php\/dav\/files\/zh\/SofortUpload\/Camera\/2019\/12\/20191217_095312.jpg","message":"Couldn't re-calculate unencrypted size for files\/SofortUpload\/Camera\/2019\/12\/20191217_095312.jpg","userAgent":"Mozilla\/5.0 (Windows) mirall\/2.6.1stable-Win64 (build 20191105) (Nextcloud)","version":"17.0.1.1"}

I was able to sync again by logging out and back in in the desktop app.

[acheronstyix]

I have the same problem with NC 18.0.3.
I've enabled server side encryption and sometimes, some files can't be sync from NC desktop to NC server. It happens randomly.

One workaround is to rename the file to another name and rename it to its first name to unblock the synchronization.

None of these files are share from NC.

[FlorentCoppint]

No news about this issue since February 2018 ?

I got the problem with Nextcloud 18 and up-to-date client, only with files intensively modified client-side.

I use S3 as primary storage.

[lectergta]

Same problem here.
This is crazy, why does this error still occur for over 2 years and there is no solution yet.
I hope s.o. could help us here...

[mnelemans]

I just encountered this problem on all my files uploaded before today, which came in as quite the shock to say the least. After much digging I've been able to discover the issue that caused the error for me. Apparently, my master private key had been replaced during (or after?) a minor upgrade process (according to the timestamps), which made it that all new files were able to work just fine but the encrypted files that were created using the "old" master private key were now corrupted.

I have no clue as to why my master private key got replaced, but I'm happy I had a backup that was able to restore my old version. I imagine the same issue could happen on any of the private keys, not just the master one which could explain why this may also happen on a select number of files.

All I can really say is that this error seems to pop up when decrypting the key files, and not the files themselves. If able, try finding older versions of the encryption keys for the corrupted files and hopefully that is enough to at least restore them. A good overview of all the file paths can be found on the wiki. I would be interested in hearing what caused the key replacement, but looking at the history of this issue that is unlikely.

[lectergta]

Thank you so much mnelemans! You saved my Life. I replaced the current key files with the ones in my backup. Now I can access all files again! Thx a lot.

[juliushaertl]

The documentation states pretty clearly that without a backup of your keys all data might be lost, so proper backups are really mandatory if you consider enabling encryption. However there might of course also be other occurrences where the private key might get corrupted, so a backup is really worth as the docs state :wink:

You should regularly backup all encryption keys to prevent permanent data loss.
The encryption keys are stored in the following directories:

data/<user>/files_encryption
Users' private keys and all other keys necessary to decrypt the users' files
data/files_encryption
private keys and all other keys necessary to decrypt the files stored on a
system wide external storage

Of course this should not happen though Nextcloud, so any help tracing down the issue is welcome. If anyone is seeing this issue, please try to check the modification time of your master key and see if there is maybe some relation to an update or request from your webserver access log.

[FlorentCoppint]

In the case of S3 as primary storage, what are the path of keys to backup ? I don't think there's user's key but only global keys in files_encryption path

[juliushaertl]

If this happened immediately initial setup of encryption https://github.com/nextcloud/server/pull/22018 should help with this.

[juliushaertl]

In the case of S3 as primary storage, what are the path of keys to backup ? I don't think there's user's key but only global keys in files_encryption path

For object storage files are stored by file id which you would need to fetch from the filecache table first.

[tusharsharma27]

Is this issue resolved? I got the same issue in a fine running Owncloud 10 server. None of my users are able to access the default owncloud_manual.pdf file which is shared once the account is created.
I have got the older keys also but not sure when did the actual key changed as there are lot of files which were uploaded after the key change. It is a random error for which there does not seem to be an explanation made by anyone. Even the original uploader of the files is not able the see the files.

[tusharsharma27]

Any update guys? I have made a replica of the server and can give access to anyone over remote. I want people to understand this issue. I have master key enabled. A newly created user was able to check all the file after first login but after a random time, while the user did not make much activity, now he is not able to see the files which he uploaded and never shared. This is quite a serious issue. Please help. Inbox me at tushar.sharma.[email protected]