Server: Decryption (symmetric) of content fails using s3

[TecJon]

Steps to reproduce

1. Update to 14 beta 4 using update
2. enable external storage
3. enabled default encryption without home encryption
4. add s3 external storage (for everyone) with previews, ssl and encryption
5. create a folder within the external storage using the nc web interface
6. upload a file (larger than 50 KB) using the web interface
7. Download the file

Expected behaviour

It should download

Actual behaviour

File not found, status 500. And nextcloud errors.

Server configuration

Operating system:
ubuntu
Web server:
nginx
Database:
mysql 5.7.23
PHP version:
7.0.30
Nextcloud version: (see Nextcloud admin page)
14 beta 4
Updated from an older Nextcloud/ownCloud or fresh install:
fresh 13 install, switch channel to beta, update
Where did you install Nextcloud from:
one click hoster and private virtual server
Signing status:

no errors

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "example.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/example.com",
        "dbtype": "mysql",
        "version": "14.0.0.16",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "updater.release.channel": "beta",
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

Are you using external storage, if yes which one: local/smb/sftp/...
s3
Are you using encryption: yes/no
yes
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no

Client configuration

Browser:
Firefox 61, Chrome
Operating system:
Win 10

Logs

Web server error log

2018/08/20 20:47:20 [error] 5484#5484: *155594 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Error: Access to undeclared static property: OC\Files\Filesystem::$normalizedPathCache in /var/www/nextcloud_test/nextcloud/lib/private/Files/Filesystem.php:797
Stack trace:
#0 /var/www/nextcloud_test/nextcloud/lib/private/Files/View.php(2056): OC\Files\Filesystem::normalizePath('/User/files/s3...')
#1" while reading upstream, client: xx.xxx.x.xx, server: example.com, request: "GET /remote.php/webdav/s3-test/testfolder/test.zip?downloadStartSecret=gjce5rdhn9f HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.0-fpm.sock:", host: "example.com"
2018/08/20 20:47:20 [error] 5484#5484: *155594 FastCGI sent in stderr: " /var/www/nextcloud_test/nextcloud/lib/private/Files/View.php(1180): OC\Files\View->unlockFile('/s3-test/testfo...', 1)
#2 [internal function]: OC\Files\View->OC\Files\{closure}()
#3 /var/www/nextcloud_test/nextcloud/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php(109): call_user_func(Object(Closure))
#4 [internal function]: Icewind\Streams\CallbackWrapper->stream_close()
#5 {main}
  thrown in /var/www/nextcloud_test/nextcloud/lib/private/Files/Filesystem.php on line 797" while reading upstream, client: xx.xxx.x.xx, server: example.com, request: "GET /remote.php/webdav/s3-test/testfolder/test.zip?downloadStartSecret=gjce5rdhn9f HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.0-fpm.sock:", host: "example.com"

Nextcloud log (data/nextcloud.log)

{"reqId":"LAYjbdNvkZWxDTgY9Y61","level":4,"time":"2018-08-15T19:01:35+00:00","remoteAddr":"xx.xxx.x.xx","user":"User","app":"webdav","method":"GET","url":"\/remote.php\/webdav\/s3-test\/testfolder\/test.zip?downloadStartSecret=urxlbk0g3t","message":{"Exception":"OC\\Encryption\\Exceptions\\DecryptionFailedException","Message":"Encryption library: Decryption (symmetric) of content failed: ","Code":0,"Trace":[{"file":"\/var\/www\/nextcloud_test\/nextcloud\/apps\/encryption\/lib\/Crypto\/Crypt.php","line":469,"function":"decrypt","class":"OCA\\Encryption\\Crypto\\Crypt","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/apps\/encryption\/lib\/Crypto\/Encryption.php","line":380,"function":"symmetricDecryptFileContent","class":"OCA\\Encryption\\Crypto\\Crypt","type":"->","args":["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***",1,"*** sensitive parameter replaced ***"]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/lib\/private\/Files\/Stream\/Encryption.php","line":422,"function":"decrypt","class":"OCA\\Encryption\\Crypto\\Encryption","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/lib\/private\/Files\/Stream\/Encryption.php","line":262,"function":"readCache","class":"OC\\Files\\Stream\\Encryption","type":"->","args":[]},{"function":"stream_read","class":"OC\\Files\\Stream\\Encryption","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/Wrapper.php","line":83,"function":"fread","args":[null,8192]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php","line":91,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"Icewind\\Streams\\CallbackWrapper","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/3rdparty\/sabre\/http\/lib\/Sapi.php","line":80,"function":"stream_copy_to_stream","args":[null,null,"2091586"]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":498,"function":"sendResponse","class":"Sabre\\HTTP\\Sapi","type":"::","args":[{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https:\/\/example.com\/remote.php\/webdav\/s3-test\/testfolder\/test.zip?downloadStartSecret=urxlbk0g3t","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/apps\/dav\/appinfo\/v1\/webdav.php","line":80,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/remote.php","line":163,"args":["\/var\/www\/nextcloud_test\/nextcloud\/apps\/dav\/appinfo\/v1\/webdav.php"],"function":"require_once"}],"File":"\/var\/www\/nextcloud_test\/nextcloud\/apps\/encryption\/lib\/Crypto\/Crypt.php","Line":586,"Hint":"Encryption library: Decryption (symmetric) of content failed: ","CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko\/20100101 Firefox\/61.0","version":"14.0.0.14"}
{"reqId":"W3R7FAJsXrNgOTjW@Tg4","level":4,"time":"2018-08-15T19:12:21+00:00","remoteAddr":"xx.xxx.x.xx","user":"User","app":"webdav","method":"GET","url":"\/cloud\/remote.php\/webdav\/AmazonS3\/testfolder\/test.zip?downloadStartSecret=xyyhi8z50o","message":{"Exception":"OC\\Encryption\\Exceptions\\DecryptionFailedException","Message":"Encryption library: Decryption (symmetric) of content failed: ","Code":0,"Trace":[{"file":"\/www\/htdocs\/w017847f\/example.com\/apps\/encryption\/lib\/Crypto\/Crypt.php","line":469,"function":"decrypt","class":"OCA\\Encryption\\Crypto\\Crypt","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/www\/htdocs\/w017847f\/example.com\/apps\/encryption\/lib\/Crypto\/Encryption.php","line":379,"function":"symmetricDecryptFileContent","class":"OCA\\Encryption\\Crypto\\Crypt","type":"->","args":["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***",1,"*** sensitive parameter replaced ***"]},{"file":"\/www\/htdocs\/w017847f\/example.com\/lib\/private\/Files\/Stream\/Encryption.php","line":479,"function":"decrypt","class":"OCA\\Encryption\\Crypto\\Encryption","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/www\/htdocs\/w017847f\/example.com\/lib\/private\/Files\/Stream\/Encryption.php","line":299,"function":"readCache","class":"OC\\Files\\Stream\\Encryption","type":"->","args":[]},{"function":"stream_read","class":"OC\\Files\\Stream\\Encryption","type":"->","args":[8192]},{"file":"\/www\/htdocs\/w017847f\/example.com\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/Wrapper.php","line":91,"function":"fread","args":[null,8192]},{"file":"\/www\/htdocs\/w017847f\/example.com\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php","line":91,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"Icewind\\Streams\\CallbackWrapper","type":"->","args":[8192]},{"file":"\/www\/htdocs\/w017847f\/example.com\/3rdparty\/sabre\/http\/lib\/Sapi.php","line":80,"function":"stream_copy_to_stream","args":[null,null,"2091586"]},{"file":"\/www\/htdocs\/w017847f\/example.com\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":498,"function":"sendResponse","class":"Sabre\\HTTP\\Sapi","type":"::","args":[{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/www\/htdocs\/w017847f\/example.com\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https:\/\/example.com\/remote.php\/webdav\/AmazonS3\/testfolder\/test.zip?downloadStartSecret=xyyhi8z50o","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/www\/htdocs\/w017847f\/example.com\/apps\/dav\/appinfo\/v1\/webdav.php","line":80,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"\/www\/htdocs\/w017847f\/example.com\/remote.php","line":163,"args":["\/www\/htdocs\/w017847f\/example.com\/apps\/dav\/appinfo\/v1\/webdav.php"],"function":"require_once"}],"File":"\/www\/htdocs\/w017847f\/example.com\/apps\/encryption\/lib\/Crypto\/Crypt.php","Line":586,"Hint":"Encryption library: Decryption (symmetric) of content failed: ","CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko\/20100101 Firefox\/61.0","version":"14.0.0.15"}

Preview errors
{"reqId":"9ZvzPD5diCrqtAL6ySQs","level":3,"time":"2018-08-20T18:39:38+00:00","remoteAddr":"xx.xxx.x.xx","user":"User","app":"PHP","method":"GET","url":"\/core\/preview?fileId=389&c=5b7479ee3e228&x=32&y=32&forceIcon=0","message":"fread() expects parameter 1 to be resource, boolean given at \/var\/www\/nextcloud_test\/nextcloud\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#858","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko\/20100101 Firefox\/61.0","version":"14.0.0.16"}
{"reqId":"9ZvzPD5diCrqtAL6ySQs","level":3,"time":"2018-08-20T18:39:38+00:00","remoteAddr":"xx.xxx.x.xx","user":"User","app":"PHP","method":"GET","url":"\/core\/preview?fileId=389&c=5b7479ee3e228&x=32&y=32&forceIcon=0","message":"fclose() expects parameter 1 to be resource, boolean given at \/var\/www\/nextcloud_test\/nextcloud\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#859","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko\/20100101 Firefox\/61.0","version":"14.0.0.16"}
{"reqId":"9ZvzPD5diCrqtAL6ySQs","level":3,"time":"2018-08-20T18:39:38+00:00","remoteAddr":"xx.xxx.x.xx","user":"User","app":"PHP","method":"GET","url":"\/core\/preview?fileId=389&c=5b7479ee3e228&x=32&y=32&forceIcon=0","message":"fread(): Length parameter must be greater than 0 at \/var\/www\/nextcloud_test\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/Wrapper.php#91","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko\/20100101 Firefox\/61.0","version":"14.0.0.16"}

Browser log

nothing

@schiessle I tried it on two independent test installations. One is a one-click install on a shared web space. The other is a installation on a virtual server with the zip from nextcloud.com. For both installations I installed NC 13 and switched to the beta channel. Both installations produce the same error when trying to download a file:

Some notes that apply for both test installations:

• configured external storage (s3) system-wide
• for all users, no limitations
• enabled default encryption without home encryption
• no errors using external s3 storage without encryption
• SSL is enabled.
• Style path and legacy authentication is disabled
• preview is enabled
• Compatibility with Mac NFD encoding is disabled
• uploading a file on the web interface works without problems
• Tiny files of 3 KB are no problem to download.
• larger files (160 KB) fail when trying to download (or open) see error above.
• the files uploaded in nextcloud also appear on the bucket and seem to be encrypted

What S3 storage do you use?

Not sure which information you could need. I configured two s3 buckets, one for each installations. One has default AES-256 encryption enabled, the other one hasn't. One is located in us east, the other in europe central. Let me know if any other info can be of help.

This was originally posted here https://github.com/nextcloud/server/issues/8299#issuecomment-413361312
nginx error log is new.

[nextcloud-bot]

GitMate.io thinks possibly related issues are https://github.com/nextcloud/server/issues/8299 (Decryption failed with s3), https://github.com/nextcloud/server/issues/5516 (nextcloud does not use s3 external storrage), https://github.com/nextcloud/server/issues/6796 (Nextcloud fails to handle large files when using S3 object store (with fix)), and https://github.com/nextcloud/server/issues/10455 (File upload fails with encryption on s3 storage "ObjectUpload not found").

[MorrisJobke]

cc @schiessle

[thorsten19]

Thanks @TecJon for reporting and @schiessle for looking into this.
Unfortunately I ran into this exact issue not knowing that S3 as external storage and used together with server side encryption leads to this issue.

As it seems that it is taking time to resolve this I suggest that the team updates the documentation to avoid other people run into this
Ideally this should be mentioned in the documentation about configuring external storage

https://docs.nextcloud.com/server/15/admin_manual/configuration_files/external_storage_configuration_gui.html#enabling-external-storage-support

Spent several hours uploading files to a fresh Nextcloud (v.15.0.5 with the latest apps for external storage and encryption) just to find out that downloading the same files won't work.

[PabloCastellano]

Is anyone working on this issue? This is an important feature for me and I would like to give it a try, but I need some guidance. Do we know where the bug is?

[chriswayg]

Its been almost a year, and it is still not working. I thought the main purpose of encryption was to be able to encrypt files located on outside storage. S3 should really be a priority.

Where can we see the status of the bug? Has there been any progress?
Small text-files sometimes work, while everything else fails.

[paulcalabro]

Wow, I'm glad I had backups. This almost rendered a bunch of important files useless to me. It would be nice to see this fixed or at least a warning about potential issues you might experience when using encryption with s3 for external storage.

[TecJon]

Hello @schiessle,
Can you provide prioritization of this ticket? When will this be considered?

cc @MorrisJobke @skjnldsv

[skjnldsv]

@Tecjon, please don't mention people randomly. I have nothing to do with encryption! :)

[qguv]

Hi, is this issue triaged at all? It's hard to justify using an external storage provider if it's not possible to encrypt the data on the third party servers.

[kdamianakis]

Apologies, just to clarify; would it be S3 enabled encryption from Nextcloud that is not working or the S3 default AES-256 encryption that is the problem? If it is the latter, i.e. the one that is handled by S3 and returns back the files unencrypted to the NextCloud service could someone please let me know?

[benbouillet]

I'm experiencing the issue too.
This would be great for it to be corrected, as S3-based storage is an awesome feature.

[koreywithak]

If I understand this correctly, the only issue is trying to use an _encrypted S3 bucket_ in addition to NC encryption? I don't see why an encrypted S3 bucket is that beneficial if the data NC puts on it is encrypted using its encryption module. The content of the file that leaves the NC server is encrypted before leaving the server and going to the bucket. Does the OP and others want to encrypt metadata or the file/folder structure? However, it seems like the metadata is not encrypted by AWS anyways. I am not an expert with this stuff, so I assume I am misunderstanding something.

To clarify what I have done: I set up a test folder with basically the same stuff as the OP and I have no issues, but _my S3 bucket is not encrypted_. Only the stuff NC loads onto it is encrypted (which is all I think I really need?). I created some folders and files in NC after turning on encryption. I can confirm that if I download the files from S3 directly and try to open them they are not understood (encrypted). When I download files from within NC after uploading them via NC (I tried a 6.2 MB pdf file and a 2.6 MB jpeg file), I can still view them with no issue. I am on NC 17.0.1.

[koreywithak]

@TecJon Am I correct that this issue is if you have S3 encryption enabled _and_ Nextcloud encryption? If so, why do you need the S3 encryption?

[kdamianakis]

In my case S3 encryption does not work as the REST or CLI or Code command does not seem to include the correct header for the encryption to be enabled on the S3 file even if the S3 bucket is set to enforce encryption. Using S3 as nextcloud primary storage in AWS.

Kon

On 13 Dec 19, at 21:33, USER19464729 <[email protected][e-mail]> wrote:

@TecJonhttps://github.com/TecJon Am I correct that this issue is if you have S3 encryption enabled and Nextcloud encryption? If so, why do you need the S3 encryption?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/nextcloud/server/issues/10767?email_source=notifications&email_token=ACWUET6ONTCZR36CAD3RFF3QYPPQRA5CNFSM4FQSEALKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG3AVVQ#issuecomment-565578454, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACWUET46PRAXZGT3JKA7VWDQYPPQRANCNFSM4FQSEALA.

[koreywithak]

I still don't understand why anyone is using S3 encryption... I am using Nextcloud's default encryption module and everything that is loaded onto the S3 bucket is encrypted by Nextcloud. Why encrypt the S3 bucket?

[kdamianakis]

Hi,

Thank you for this!

Is this confirmed to be working? I thought it could not be enable for a primary S3 storage.If I enable it now will it encrypt all new files/updates?

Kind Regards
Kon

On 14 Dec 19, at 22:55, USER19464729 <[email protected][e-mail]> wrote:

I still don't understand why anyone is using S3 encryption... I am using Nextcloud's default encryption module and everything that is loaded onto the S3 bucket is encrypted by Nextcloud. Why encrypt the S3 bucket?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/nextcloud/server/issues/10767?email_source=notifications&email_token=ACWUETYLGN7ALBN6SNGJ3XLQYVB5VA5CNFSM4FQSEALKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG4LFJY#issuecomment-565752487, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACWUET3PNVRVNGG6NSRUFOTQYVB5VANCNFSM4FQSEALA.

[koreywithak]

Oh. I am using S3 as an attached external storage--not my primary storage. That was the piece I was misunderstanding.

[kdamianakis]

I believe that it may be that the header will need to be set correctly, if
using CLI command line structure for moving the file to S3 it should be
done with this command
https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-encryption.html.
Unfortunately I do not know enough about the internals of NextCloud to see
where the change would need to be made. It it is an API PUT request, the
header can be manually fixed to "To encrypt an object at the time of
upload, you need to add a header called x-amz-server-side-encryption to the
request to tell S3 to encrypt the object using SSE-C, SSE-S3, or SSE-KMS" (
https://aws.amazon.com/blogs/security/how-to-prevent-uploads-of-unencrypted-objects-to-amazon-s3/
)

On Thu, Nov 28, 2019 at 8:43 PM Benjamin Bouillet [e-mail]
wrote:

I'm experiencing the issue too.
This would be great for it to be corrected, as S3-based storage is an
awesome feature.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/server/issues/10767?email_source=notifications&email_token=ACWUET6TR43DRPIFP2K6H5DQWAGNZA5CNFSM4FQSEALKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFNIWEI#issuecomment-559581969,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACWUET4TPWBILZHTUJK4TNDQWAGNZANCNFSM4FQSEALA
.

--
Regards,
Konstantinos Damianakis
Vouchcom Limited
m [phone]
e Vouchcom [[email protected]]

[kellybyrd]

I'm having these same symptoms with a fresh NC 18.0.4 install:

• S3-compatible service (Wasabi) as primary storage
• NC server-side encryption turned on. With "Encrypt the home storage" checked.
• No separate encryption on the S3 service.

I can't download or preview any files uploaded after "Encrypt the home storage" is checked. The errors in the log say:
"Exception": "OCP\\Encryption\\Exceptions\\GenericEncryptionException","Message": "Bad Signature",