Server: Encryption not working with S3 object storage as primary storage

Created on 14 Oct 2018 · 12Comments · Source: nextcloud/server

Steps to reproduce

install Nextcloud 14.0.3 with S3 object storage (DigitalOcean Spaces) as primary storage
enable encryption (including the home storage)
upload files larger than about 100 kb
access the files via web interface

Expected behaviour

The file should be listed in the files app, thumbnails should be created for the appropriate file types, it should be possible to preview the files and download them.

Actual behaviour

The files appear in the files app, but thumbnail generation and attempts to open them and download them fail leaving "Couldn't re-calculate unencrypted size", "OCP\Encryption\Exceptions\GenericEncryptionException: Bad Signature" and "OCP\Encryption\Exceptions\GenericEncryptionException: Missing Signature" errors in the logs.

Occasionally the thumbnails get created and the files become available, but even then it can require two or more attempts before they can be downloaded because the first attempts fails with a browser error (file unavailable, or 0-size file downloaded) or because the https:///remote.php/webdav/?downloadStartSecret= link times out. Even then the file can later become inaccessible again.

Small files seem more likely to work, but I could be mistaken. The default files created by Nextcloud, who aren't encrypted, work regularly, but as soon as I launch occ encryption:encrypt-all, they become unavailable (no thumbnail, preview and download stop working).

On the same server with the same settings but encryption disabled everything works correctly.

Server configuration

Operating system: Ubuntu 18.04.1

Web server: nginx 1.14.0-0ubuntu1.1

Database: mariadb 10.1.34-0ubuntu0.18.04.1

PHP version: 7.2.10-0ubuntu0.18.04.1

Nextcloud version: 14.0.3

Updated from an older Nextcloud/ownCloud or fresh install: fresh install

Where did you install Nextcloud from: tar.bz2 file downloaded from the website

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - accessibility: 1.0.1
  - activity: 2.7.0
  - cloud_federation_api: 0.0.1
  - comments: 1.4.0
  - dav: 1.6.0
  - encryption: 2.2.0
  - federatedfilesharing: 1.4.0
  - federation: 1.4.0
  - files: 1.9.0
  - files_pdfviewer: 1.3.2
  - files_sharing: 1.6.2
  - files_texteditor: 2.6.0
  - files_trashbin: 1.4.1
  - files_versions: 1.7.1
  - files_videoplayer: 1.3.0
  - firstrunwizard: 2.3.0
  - gallery: 18.1.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.2.0
  - nextcloud_announcements: 1.3.0
  - notifications: 2.2.1
  - oauth2: 1.2.1
  - password_policy: 1.4.0
  - provisioning_api: 1.4.0
  - serverinfo: 1.4.0
  - sharebymail: 1.4.0
  - support: 1.0.0
  - survey_client: 1.2.0
  - systemtags: 1.4.0
  - theming: 1.5.0
  - twofactor_backupcodes: 1.3.1
  - updatenotification: 1.4.1
  - workflowengine: 1.4.0
Disabled:
  - admin_audit
  - files_external
  - user_external
  - user_ldap

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "objectstore": {
            "class": "OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "***REMOVED SENSITIVE VALUE***",
                "autocreate": true,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "hostname": "ams3.digitaloceanspaces.com",
                "use_ssl": true
            }
        },
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "14.0.3.0",
        "overwrite.cli.url": "http:\/\/localhost",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "updater.release.channel": "production",
    }
}

Are you using external storage, if yes which one: no (DigitalOcean Spaces as primary object storage)

Are you using encryption: yes

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Firefox 60 ESR, Safari on iOS 12.01

Operating system: Ubuntu 18.04, iOS 12.0.1 on iPad

Logs

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"wdmxC19wMAqHfNQsAHBE","level":3,"time":"2018-10-14T12:18:09+00:00","remoteAddr":"XX.XX.XX.XX","user":"admin","app":"no app in context","method":"GET","url":"\/core\/preview?fileId=242&c=5bc3290044318&x=32&y=32&forceIcon=0","message":"Couldn't re-calculate unencrypted size for files\/3B71DDB6-86FD-45ED-9DF2-E816580BA98F.jpeg","userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:60.0) Gecko\/20100101 Firefox\/60.0","version":"14.0.3.0"}
{"reqId":"wdmxC19wMAqHfNQsAHBE","level":3,"time":"2018-10-14T12:18:09+00:00","remoteAddr":"XX.XX.XX.XX","user":"admin","app":"no app in context","method":"GET","url":"\/core\/preview?fileId=242&c=5bc3290044318&x=32&y=32&forceIcon=0","message":{"Exception":"OCP\\Encryption\\Exceptions\\GenericEncryptionException","Message":"Bad Signature","Code":0,"Trace":[{"file":"\/var\/www\/html\/nextcloud\/apps\/encryption\/lib\/Crypto\/Crypt.php","line":463,"function":"checkSignature","class":"OCA\\Encryption\\Crypto\\Crypt","type":"->","args":["<encoded string>",null,"c6577216a2141526aaee15e81281833fbbab823afdab5c6760118fd4de06fdb4"]},{"file":"\/var\/www\/html\/nextcloud\/apps\/encryption\/lib\/Crypto\/Encryption.php","line":379,"function":"symmetricDecryptFileContent","class":"OCA\\Encryption\\Crypto\\Crypt","type":"->","args":["*** sensitive parameter replaced ***",null,"AES-256-CTR",1,"*** sensitive parameter replaced ***"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php","line":581,"function":"decrypt","class":"OCA\\Encryption\\Crypto\\Encryption","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php","line":500,"function":"fixUnencryptedSize","class":"OC\\Files\\Storage\\Wrapper\\Encryption","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php","line":164,"function":"verifyUnencryptedSize","class":"OC\\Files\\Storage\\Wrapper\\Encryption","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php","line":401,"function":"filesize","class":"OC\\Files\\Storage\\Wrapper\\Encryption","type":"->","args":["*** sensitive parameter replaced ***"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Files\/Storage\/Wrapper\/Wrapper.php","line":298,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Encryption","type":"->","args":["*** sensitive parameter replaced ***","r"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Files\/View.php","line":1144,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["*** sensitive parameter replaced ***","r"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Files\/View.php","line":986,"function":"basicOperation","class":"OC\\Files\\View","type":"->","args":["fopen","\/3B71DDB6-86FD-45ED-9DF2-E816580BA98F.jpeg",["read"],"r"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Files\/View.php","line":997,"function":"fopen","class":"OC\\Files\\View","type":"->","args":["3B71DDB6-86FD-45ED-9DF2-E816580BA98F.jpeg","r"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Preview\/Image.php","line":53,"function":"toTmpFile","class":"OC\\Files\\View","type":"->","args":["3B71DDB6-86FD-45ED-9DF2-E816580BA98F.jpeg"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Preview\/GeneratorHelper.php","line":59,"function":"getThumbnail","class":"OC\\Preview\\Image","type":"->","args":["3B71DDB6-86FD-45ED-9DF2-E816580BA98F.jpeg",4096,4096,false,{"__class__":"OC\\Files\\View"}]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Preview\/Generator.php","line":194,"function":"getThumbnail","class":"OC\\Preview\\GeneratorHelper","type":"->","args":[{"__class__":"OC\\Preview\\JPEG"},{"__class__":"OC\\Files\\Node\\File"},4096,4096]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Preview\/Generator.php","line":118,"function":"getMaxPreview","class":"OC\\Preview\\Generator","type":"->","args":[{"__class__":"OC\\Files\\SimpleFS\\SimpleFolder"},{"__class__":"OC\\Files\\Node\\File"},"image\/jpeg"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/PreviewManager.php","line":205,"function":"getPreview","class":"OC\\Preview\\Generator","type":"->","args":[{"__class__":"OC\\Files\\Node\\File"},32,32,true,"fill","image\/jpeg"]},{"file":"\/var\/www\/html\/nextcloud\/core\/Controller\/PreviewController.php","line":175,"function":"getPreview","class":"OC\\PreviewManager","type":"->","args":[{"__class__":"OC\\Files\\Node\\File"},32,32,true,"fill"]},{"file":"\/var\/www\/html\/nextcloud\/core\/Controller\/PreviewController.php","line":147,"function":"fetchPreview","class":"OC\\Core\\Controller\\PreviewController","type":"->","args":[{"__class__":"OC\\Files\\Node\\File"},32,32,false,false,"fill"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/AppFramework\/Http\/Dispatcher.php","line":166,"function":"getPreviewByFileId","class":"OC\\Core\\Controller\\PreviewController","type":"->","args":[242,32,32,false,false,"fill"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/AppFramework\/Http\/Dispatcher.php","line":99,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OC\\Core\\Controller\\PreviewController"},"getPreviewByFileId"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/AppFramework\/App.php","line":118,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OC\\Core\\Controller\\PreviewController"},"getPreviewByFileId"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/AppFramework\/Routing\/RouteActionHandler.php","line":47,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OC\\Core\\Controller\\PreviewController","getPreviewByFileId",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"_route":"core.Preview.getPreviewByFileId"}]},{"function":"__invoke","class":"OC\\AppFramework\\Routing\\RouteActionHandler","type":"->","args":[{"_route":"core.Preview.getPreviewByFileId"}]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Route\/Router.php","line":297,"function":"call_user_func","args":[{"__class__":"OC\\AppFramework\\Routing\\RouteActionHandler"},{"_route":"core.Preview.getPreviewByFileId"}]},{"file":"\/var\/www\/html\/nextcloud\/lib\/base.php","line":987,"function":"match","class":"OC\\Route\\Router","type":"->","args":["\/core\/preview"]},{"file":"\/var\/www\/html\/nextcloud\/index.php","line":42,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"\/var\/www\/html\/nextcloud\/apps\/encryption\/lib\/Crypto\/Crypt.php","Line":483,"Hint":"Bad Signature","CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:60.0) Gecko\/20100101 Firefox\/60.0","version":"14.0.3.0"}

1. to develop bug encryption (server-side)

Source

plumbeo

👍13

Most helpful comment

Wish there was more traction on this issue since it seems pretty severe. I think if anything, enabling encryption on an instance that has an objectstore with a class of '\OC\Files\ObjectStore\S3' should fail or give a warning stating the unstable nature of encryption on such an objectstore. I'd like to look more into this issue as well, will have to brush up on my php skills.

Sorry if my tone is a little negative, just learned of this issue after uploading 100GB of data, only to find it's now essentially garbage.

craysiii on 11 Apr 2019

👍9

All 12 comments

GitMate.io thinks possibly related issues are https://github.com/nextcloud/server/issues/10455 (File upload fails with encryption on s3 storage "ObjectUpload not found"), https://github.com/nextcloud/server/issues/3294 (No Previews/Thumbnails with Primary Storage S3 + Encryption), https://github.com/nextcloud/server/issues/3748 (Primary S3 Storage Backend + Encryption problem with file size, mimetype and first download failed), https://github.com/nextcloud/server/issues/6113 (S3 won't connect as external storage), and https://github.com/nextcloud/server/issues/8617 (Previews not working when using external AWS S3 storage and encryption).

nextcloud-bot on 14 Oct 2018

Looks similar to #10767

MorrisJobke on 15 Oct 2018

cc @nextcloud/encryption

MorrisJobke on 15 Oct 2018

I'm having the same issue with same log entries. Ubuntu 16.04, Nextcloud 14.0.3, apache 2.4.34. I tried to upload from android app and from web ui, same results. Upload is ok but the file can not be previewed or opened after downloading. I'm using wasabi s3 as external storage.

daylightsports on 17 Oct 2018

👍1

Now that NC 15.0 is out:
I can replicate the issue consistently with Nextcloud 15.0 on Ubuntu 18.04 either with apache2 or nginx.

My workflow is I set up Nextcloud to the point where the admin user is created and the database connection is specified. I then edit config.php with my S3 storage array. Back-end is minio.
I can see it connect and drop files into the bucket no problems. I then enable the 'Default encryption module 2.3.0" and turn on server-side encryption including home directories. Now the log is spammed with errors for any new objects created.

{"reqId":"3fijiDnpFGZCJqrcM2mx","level":3,"time":"2018-12-16T02:12:15+01:00","remoteAddr":"1.2.3.148","user":"admin","app":"no app in context","method":"GET","url":"/core/preview?fileId=243&c=5c15a66e54ac8&x=625&y=625&forceIcon=0","message":"Couldn't re-calculate unencrypted size for files/cantdisclose.pdf","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0","version":"15.0.0.10","id":"5c15a937e6081"}

drphr4ud on 16 Dec 2018

Hey guys, I decided to look into this problem because I really wanted to use encryption on S3 object storage. I wrote down my notes and a possible fix in #14027. Maybe someone wants to give it a try? I'd love to get this fixed in a future version of NC. Also more input is appreciated, maybe someone can come up with a better solution.