Server: S3 as primary storage : empty files are impossible to upload

[ArnY]

Steps to reproduce

1. set up a nextcloud instance with a s3 bucket as primary storage
2. create an empty file on your hard drive
3. try to upload the empty file or drop the file in your synch'ed folder

Expected behaviour

the file should be uploaded to the nextcloud instance, on the s3 bucket.

Actual behaviour

in both case (upload from web or using the synchronization client), the upload will fail with the following errors

cloudprod3.cprv.univ-nantes.prive/syslog:Dec 20 14:28:10 cloudprod3.cprv.univ-nantes.prive ownCloud[2272]: {objectstore} Could not create object urn:oid:5857935 for files/truc.txt.ocTransferId1292085902.part: {"Exception":"Aws\\S3\\Exception\\S3MultipartUploadException","Message":"An exception occurred while completing a multipart upload: Error executing \"CompleteMultipartUpload\" on \"http:\/\/cloudrgw.ha.univ-nantes.prive\/cloudprod\/urn%3Aoid%3A5857935?uploadId=2~8qS66aBMfjoVHA2bLLWdZstr9wQGnVx\"; AWS HTTP error: Client error response [url] http:\/\/cloudrgw.ha.univ-nantes.prive\/cloudprod\/urn%3Aoid%3A5857935?uploadId=2~8qS66aBMfjoVHA2bLLWdZstr9wQGnVx [status code] 400 [reason phrase] Bad Request MalformedXML (client):  - <?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Code>MalformedXML<\/Code><BucketName>cloudprod<\/BucketName><RequestId>tx000000000000000120f00-005a3a656a-3727e-default<\/RequestId><HostId>3727e-default-default<\/HostId><\/Error>","Code":0,"Trace":"#0 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(203): Aws\\Multipart\\AbstractUploadManager->Aws\\Multipart\\{closure}(Object(Aws\\S3\\Exception\\S3Exception))\n#1 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(156): GuzzleHttp\\Promise\\Promise::callHandler(2, Object(Aws\\S3\\Exception\\S3Exception), Array)\n#2 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/TaskQueue.php(47): GuzzleHttp\\Promise\\Promise::GuzzleHttp\\Promise\\{closure}()\n#3 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(234): GuzzleHttp\\Promise\\TaskQueue->run()\n#4 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(267): GuzzleHttp\\Promise\\Promise->waitIfPending()\n#5 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(225): GuzzleHttp\\Promise\\Promise->invokeWaitList()\n#6 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(267): GuzzleHttp\\Promise\\Promise->waitIfPending()\n#7 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(225): GuzzleHttp\\Promise\\Promise->invokeWaitList()\n#8 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(62): GuzzleHttp\\Promise\\Promise->waitIfPending()\n#9 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Coroutine.php(65): GuzzleHttp\\Promise\\Promise->wait()\n#10 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(246): GuzzleHttp\\Promise\\Coroutine->GuzzleHttp\\Promise\\{closure}(true)\n#11 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(223): GuzzleHttp\\Promise\\Promise->invokeWaitFn()\n#12 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(267): GuzzleHttp\\Promise\\Promise->waitIfPending()\n#13 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(225): GuzzleHttp\\Promise\\Promise->invokeWaitList()\n#14 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(62): GuzzleHttp\\Promise\\Promise->waitIfPending()\n#15 \/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/Multipart\/AbstractUploadManager.php(83): GuzzleHttp\\Promise\\Promise->wait()\n#16 \/var\/www\/nextcloud\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php(105): Aws\\Multipart\\AbstractUploadManager->upload()\n#17 \/var\/www\/nextcloud\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php(81): OC\\Files\\ObjectStore\\S3->multiPartUpload('urn:oid:5857935', Resource id #19)\n#18 \/var\/www\/nextcloud\/lib\/private\/Files\/ObjectStore\/ObjectStoreStorage.php(406): OC\\Files\\ObjectStore\\S3->writeObject('urn:oid:5857935', Resource id #19)\n#19 \/var\/www\/nextcloud\/lib\/private\/Files\/ObjectStore\/ObjectStoreStorage.php(306): OC\\Files\\ObjectStore\\ObjectStoreStorage->writeBack('\/uncloud-tmp\/ph...', 'files\/truc.txt....')\n#20 [internal function]: OC\\Files\\ObjectStore\\ObjectStoreStorage->OC\\Files\\ObjectStore\\{closure}()\n#21 \/var\/www\/nextcloud\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php(109): call_user_func(Object(Closure))\n#22 [internal function]: Icewind\\Streams\\CallbackWrapper->stream_close()\n#23 \/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/File.php(135): fclose(Resource id #17)\n#24 \/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/Directory.php(151): OCA\\DAV\\Connector\\Sabre\\File->put(Resource id #13)\n#25 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(1096): OCA\\DAV\\Connector\\Sabre\\Directory->createFile('truc.txt', Resource id #13)\n#26 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php(525): Sabre\\DAV\\Server->createFile('files\/abelard-a...', Resource id #13, NULL)\n#27 [internal function]: Sabre\\DAV\\CorePlugin->httpPut(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#28 \/var\/www\/nextcloud\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#29 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(479): Sabre\\Event\\EventEmitter->emit('method:PUT', Array)\n#30 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#31 \/var\/www\/nextcloud\/apps\/dav\/lib\/Server.php(258): Sabre\\DAV\\Server->exec()\n#32 \/var\/www\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php(33): OCA\\DAV\\Server->exec()\n#33 \/var\/www\/nextcloud\/remote.php(162): require_once('\/var\/www\/nextcl...')\n#34 {main}","File":"\/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/Multipart\/AbstractUploadManager.php","Line":136}
cloudprod3.cprv.univ-nantes.prive/syslog:Dec 20 14:28:10 cloudprod3.cprv.univ-nantes.prive ownCloud[2272]: {webdav} Exception: {"Exception":"Sabre\\DAV\\Exception","Message":"An exception occurred while completing a multipart upload: Error executing \"CompleteMultipartUpload\" on \"http:\/\/cloudrgw.ha.univ-nantes.prive\/cloudprod\/urn%3Aoid%3A5857935?uploadId=2~8qS66aBMfjoVHA2bLLWdZstr9wQGnVx\"; AWS HTTP error: Client error response [url] http:\/\/cloudrgw.ha.univ-nantes.prive\/cloudprod\/urn%3Aoid%3A5857935?uploadId=2~8qS66aBMfjoVHA2bLLWdZstr9wQGnVx [status code] 400 [reason phrase] Bad Request MalformedXML (client):  - <?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Code>MalformedXML<\/Code><BucketName>cloudprod<\/BucketName><RequestId>tx000000000000000120f00-005a3a656a-3727e-default<\/RequestId><HostId>3727e-default-default<\/HostId><\/Error>","Code":0,"Trace":"#0 \/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/File.php(159): OCA\\DAV\\Connector\\Sabre\\File->convertToSabreException(Object(Aws\\S3\\Exception\\S3MultipartUploadException))\n#1 \/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/Directory.php(151): OCA\\DAV\\Connector\\Sabre\\File->put(Resource id #13)\n#2 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(1096): OCA\\DAV\\Connector\\Sabre\\Directory->createFile('truc.txt', Resource id #13)\n#3 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php(525): Sabre\\DAV\\Server->createFile('files\/abelard-a...', Resource id #13, NULL)\n#4 [internal function]: Sabre\\DAV\\CorePlugin->httpPut(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#5 \/var\/www\/nextcloud\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#6 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(479): Sabre\\Event\\EventEmitter->emit('method:PUT', Array)\n#7 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#8 \/var\/www\/nextcloud\/apps\/dav\/lib\/Server.php(258): Sabre\\DAV\\Server->exec()\n#9 \/var\/www\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php(33): OCA\\DAV\\Server->exec()\n#10 \/var\/www\/nextcloud\/remote.php(162): require_once('\/var\/www\/nextcl...')\n#11 {main}","File":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/File.php","Line":567}

Adding just one letter in the file fixes the problem.

I first suspected our radosgw cluster, but it doesn't seem to be at fault since uploading the same empty file with s3cmd succeeds:

~$ s3cmd put test.txt s3://cloudprod
upload: 'toto' -> 's3://cloudprod/test.txt'  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done
~$ s3cmd ls s3://cloudprod/test.txt 
2017-12-20 14:02         0   s3://cloudprod/test.txt
~$ s3cmd del s3://cloudprod/test.txt
delete: 's3://cloudprod/test.txt'

Server configuration

Operating system: Debian 9.1

Web server: apache 2.4

Database: mariadb 10.1 + galera 3

PHP version: 7.0

Nextcloud version: 12.0.4 and 13beta1

Updated from an older Nextcloud/ownCloud or fresh install: updated from

Where did you install Nextcloud from: nextcloud.com's download servers

Signing status:

Signing status

Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

"No errors have been found."

List of activated apps:

App list

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your Nextcloud installation folder

Enabled:

• activity: 2.5.2
• apporder: 0.4.1
• bruteforcesettings: 1.0.2
• comments: 1.2.0
• dav: 1.3.0
• external: 2.0.3
• federatedfilesharing: 1.2.0
• federation: 1.2.0
• files: 1.7.2
• files_pdfviewer: 1.1.1
• files_sharing: 1.4.0
• files_texteditor: 2.4.1
• files_trashbin: 1.2.0
• files_versions: 1.5.0
• files_videoplayer: 1.1.0
• firstrunwizard: 2.1
• gallery: 17.0.0
• impersonate: 1.0.1
• logreader: 2.0.0
• lookup_server_connector: 1.0.0
• nextcloud_announcements: 1.1
• notifications: 2.0.0
• oauth2: 1.0.5
• password_policy: 1.2.2
• provisioning_api: 1.2.0
• quota_warning: 1.1.1
• serverinfo: 1.2.0
• sharebymail: 1.2.0
• survey_client: 1.0.0
• twofactor_backupcodes: 1.1.1
• twofactor_totp: 1.3.1
• twofactor_u2f: 1.4.0
• updatenotification: 1.2.0
• user_ldap: 1.2.1
• workflowengine: 1.2.0
  Disabled:
• admin_audit
• encryption
• files_external
• systemtags
• theming
• user_external
  

Nextcloud configuration:

Config report

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your Nextcloud installation folder

or 

Insert your config.php content here. 
Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …)

...
"objectstore": {
"class": "OC\Files\ObjectStore\S3",
"arguments": {
"bucket": "cloudprod",
"autocreate": true,
"key": "XXXXXXXXXXXXXXXXXX",
"secret": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"hostname": "cloudrgw.ha.univ-nantes.prive",
"port": 80,
"use_ssl": false,
"region": "optional",
"use_path_style": true
}
},
...

Are you using external storage, if yes which one: S3 as primary storage

Are you using encryption: no

Are you using an external user-backend, if yes which one: LDAP

[MorrisJobke]

cc @icewind1991

[icewind1991]

This is a limitation of amazon and not something we can do much about

[ArnY]

Actually, i'm not using amazon's s3 but radosgw, not sure this makes a difference. But I also showed in the ticket that s3cmd does upload the empty files without any trouble so it doesn't looks the problem is on the storage side but indeed in nextcloud.

[Marlinc]

It appears I've ran into the same issue with radosgw. It only happened after upgrading to Nextcloud 12.0.4

[KroniK907]

I have also run into this issue with S3 as my primary storage. Appears to be new since my upgrade to 12.0.4

[simie7]

I confirm that we have same issue after upgrade to 12.0.4. We use CEPH as primary S3 storage.

[ibr123]

Hi,
I have the same issue, I have and my server is linked to Amazon S3, I downloaded the latest of NextCloud server which is after the date of this issue, yet the problem still occurs
Please refer to this issue
Thanks

[icewind1991]

Please see if https://github.com/nextcloud/server/pull/8033 fixes the problem for you

[ArnY]

It does indeed fix the problem! Thanks. Can we expect that commit to be backported into the next 12.0 version?

[icewind1991]

Fix should be available in 12.0.6 and 13.0.1

[ibr123]

Hi, from where I can download these versions?

[MorrisJobke]

They are not released yet. Planned release is mid march.

[MorrisJobke]

Fixed via #8033 (master) and there are also the backport to 12.0.6 and 13.0.1 linked.