Hi
Bacbklaze just launched B2 v1.0.
Backblaze is a company that hosts data in the cloud at a REALLY low price and with B2 you can use it like AWS storage.
I'd be great if you could use B2 to sync your Nextcloud server to Backblaze. (The usage of B2 isn't the same as AWS but it would be nice to just login and have it work in the web interface)(I know I can setup the command line tool but I feel like it'be great if I just had to enter username and password (and two step auth) like on Synology NAS's)
https://www.backblaze.com/b2/partner-synology.html
Thanks
thibaultmol
All 51 comments
MariusBluem
on 2 Jul 2016
I second this request. I am considering a switch from ownCloud to Nextcloud and would love for this to be a viable option. My existing server does not have expandable storage and the pricing of S3 is too high to be practical for personal use.
How difficult would this be? I'm slightly considering looking into the code for the S3 external storage, I imagine it would be very similar, if somebody would be willing to give some pointers I could try my hand at this.
GrahamJenkins
on 15 Jul 2016
+1, I too would be ecstatic to have this as an option. Thanks for linking those docs.
EnMod
on 21 Jul 2016
Does anybody know how much work this would involve or how long it might take a competent developer?
GrahamJenkins
on 1 Sep 2016
First, I am brand new to Nexctloud and have no idea how it implements external storage, so I am a long way from jumping in and starting work on this.
Some useful background however:
As far as I know, BackBlaze doesn't provide an official B2 API client in any language.
There are a few PHP Wrappers/Classes/SDKs/Clients in development for B2 independently though. None of those seem fully mature or widely used though. Most are labeled alpha/beta.
- The most recent/active/popular is: https://github.com/cwhite92/b2-sdk-php
- This one is even more popular but appears stalled: https://github.com/kamilz/b2backblaze
Other notes:
There is also a FUSE written in Python that seems mature: https://github.com/sondree/b2_fuse
If someone wanted they could probably use that to mount B2 storage directly to the server running Nextcloud and I'm guessing use it directly (again, I'm too new to NextCloud to even know for sure that's viable, but it seems like it might be).
Finally for other interested in building integrations and the requirments for being listed on the integrations page: https://www.backblaze.com/b2/docs/integrations.html
jb510
on 19 Sep 2016
I'd definitely like to see this as well. I've been waiting since they first announced the service last year but as of yet neither OwnCloud, Pydio, nor NextCloud seem to be attempting it. (I've taken a look into implementing it myself but my lack of knowledge in PHP and owncloud's plugin system seemed insurmountable).
Some additional points though:
- No, I don't believe there are any official clients, but the API is a rather simple REST-based one with similarities to S3.
- The FUSE implementation is less than ideal as the developer notes it is not production-ready and includes numerous limitations (concurrency issues, slow performance with > 1000 files)
I've been surprised there hasn't been more interest in adding this to one of the open source cloud solutions. Is there perhaps some guidance for how someone like myself could implement this? I imagine it would largely be the same as the S3 implementation.
Edit: I'm taking a look and I think I can take a stab at this.
8BitAce
on 22 Oct 2016
:+1: this would be yet another compelling reason for me to finally make that switch from oc to nc
WebSpider
on 25 Oct 2016
馃憤
tommyent
on 30 Nov 2016
I'd like to add my +1 to this, and request that two factor auth be supported.
jakimfett
on 15 Dec 2016
Please stop 馃憤 such feature requests ... instead of this - you may want to develop or find somebody who wants to do make this real 馃槣 ... you may also want to consider about posting a bug bounty on it:
http://bountysource.com/teams/nextcloud/issues
In this issue, we should only discuss this feature technically and should not tell each other how much we want it 馃槄 THX. However: Thanks for your interest in this topic. If you want to react to specific issues or pull requests on GitHub, you can also use reactions (https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comments) 馃槈
MariusBluem
on 15 Dec 2016
despens
on 2 Jan 2017
I just ran across this: https://github.com/ncw/rclone/ It has a mount feature I plan on trying with my nextcloud
jasonehines
on 8 Jan 2017
It would be nice to see full support for b2 as a regular backend (external storages & primary storage). I've tried using s3proxy, but nextcloud fails to create files in the bucket. I assume it's probably related to some of the limitations listed below. b2 works a bit differently if you take a look at the protocol.
- bucket names must be between 6 and 50 characters and consistent of only letters, number, and hyphens
- does not return ETag as an MD5 hash, instead returning SHA1 hash
- does not support blob-level access control
- does not support conditional GET
- does not support copying objects
- does not support Content-Disposition, Content-Encoding, Content-Language, Content-MD5, or Expiry headers
- multipart upload requires at least 2 parts
For those that just need a basic implementation this fork of s3ql does seem to work well. It can be a bit of a chore to get it compiled, and it makes it almost impossible to scale horizontally. For deployments that don't need to load balance a handful of front-ends s3ql will get the job done. It is better than most fuse based s3 file systems as it will cache to disk, or in my case an SSD array.
https://github.com/s3ql/s3ql/pull/8
https://github.com/sylvainlehmann/s3ql
IamDH4
on 4 Jul 2017
I'm really excited about this s3ql backblaze fork!
bitshark
on 14 Jul 2017
Would be a nice feature in nc13. ;)
benginoe
on 27 Aug 2017
Minio just added experimental B2 support. It looks like minio can be used as a gateway to native B2 buckets. Providing a way to transparently use an s3 compatible API.
I'll be investigating this further. But I thought I would share the link. I would still like to see official support for B2 in nextcloud, but this might be a good workaround.
IamDH4
on 4 Oct 2017
B2 external storage will not be implemented in the server code itself, but it can be implemented as an app that brings this into files_external, like the files_external_dropbox app does this for Dropbox for example: https://github.com/icewind1991/files_external_dropbox
Nevertheless I will not close this ticket here, because there is a bounty on it.
MorrisJobke
on 11 Jan 2018
also would be interested in b2 support but cannot help for lack of php fu there is this though:
https://github.com/cwhite92/b2-sdk-php
https://github.com/gliterd/backblaze-b2
tilllt
on 7 Apr 2018
For those looking, I tried using Minio's experimental B2 support in the edge channel and mounting it to NextCloud as an external S3 storage source. There are unfortunately quite a few issues, due to the unimplemented CopyObject API and some odd behavior with creating folders. I didn't have time to dig into it, but it's definitely not going to work out of the box.
codyhazelwood
on 1 May 2018
hi @codyhazelwood, how about minio b2 gateway as primary storage? i have same issue with google cloud storage as external s3 storage, but work perfect with primary storage.
najcardboyz
on 3 May 2018
Good call @najcardboyz. I tried playing with that for a little while, and it works with no errors, but it is unusably slow. It takes 20 - 40 seconds to change pages in the Nextcloud UI. Uploads and downloads are still speedy though, it's just slow navigating through the UI.
codyhazelwood
on 4 May 2018
yes, i've tried too and slow, maybe because latency from data center vps to backblaze b2, or anything else. I will try wasabi for later, have you tried it?
najcardboyz
on 7 May 2018
Can I ask why "B2 external storage will not be implemented in the server code itself"? If I were to develop it, and try to get it merged, would it be rejected? Is it a thing where "We've accepted Swift, S3, Azure, and are just done with all these protocols"?
Vaelatern
on 5 Feb 2019
Hey @vaelatern ... Great to hear you want to implement this feature 馃檲 We recommend implementing it as a separate app expanding files_external-app instead of adding B2-specific code into this repository.
You can find an example storage provider as an app over here: https://github.com/icewind1991/files_external_dropbox
We are of course here to assist you if you have any questions 馃槆
MariusBluem
on 5 Feb 2019
Can I put primary storage on a files_external-app, like I can for S3 at present? And thank you, I'm hoping my employer wants this feature enough to pay for it too.
Vaelatern
on 5 Feb 2019
file external can using s3, if you want use s3 as primary storage it can, but external storage for primary storage is different function
najcardboyz
on 6 Feb 2019
My reason for working on this would be a deploy in a cloud environment, where I do not want extra copies of my files. Right now, all my files are on S3, but it would be nice to have all those files on B2, and not cloned to some hard disk in the cloud where I'd be paying twice for the remote storage.
Vaelatern
on 6 Feb 2019
may be able to use rclone mount function, but this is still beta / testing, or s3fs or some similar
najcardboyz
on 7 Feb 2019
I think it makes sense to tackle the problem in two steps. First implement a external storage app and then try to make it available as primary storage. I don't know the implementation details but maybe @icewind1991 can give you some hints and can point you to the corresponding s3 code for inspiration.
schiessle
on 5 Mar 2019
hi @codyhazelwood, how about minio b2 gateway as primary storage? i have same issue with google cloud storage as external s3 storage, but work perfect with primary storage.
I am starting with nextCloud+GCP (VM it's runing with domain+SSL) but i can't find related information of how to setup a Google Cloud Storage as Internal/External storage. Can you tell me where did you found the documentation or any step-by-step guide?
Fenrihr
on 16 Jun 2019
Backblaze recently updated their API to include b2_copy_file and b2_copy_part. These two APIs have been in beta for a while but they were taken out of beta and released to the public recently.
These two previously absent APIs should make the conversion of S3 API calls to B2 API calls more straightforward, which means writing an external storage app for B2 should be easier and the final app should be more stable. Is the best way of writing this app to integrate with the files_external app and add a dropdown item for B2?
@codyhazelwood pointed out Minio, which can act as a translation layer and convert S3 API calls to B2 calls, and tested this software with Nextcloud as a workaround:
For those looking, I tried using Minio's experimental B2 support in the edge channel and mounting it to NextCloud as an external S3 storage source. There are unfortunately quite a few issues, due to the unimplemented CopyObject API and some odd behavior with creating folders. I didn't have time to dig into it, but it's definitely not going to work out of the box.
These two APIs have still not been integrated into Minio (however there is an open issue https://github.com/minio/minio/issues/7703), but when they are Minio will be a more stable workaround.
Anyone looking to implement this feature into Nextcloud directly (@icewind1991) might want to look at Minio's B2 Gateway and see how it is converting the API calls. They also list known limitations which may be helpful.
Additionally, rclone implemented these two APIs and released them in version 1.48. So the code associated with https://github.com/ncw/rclone/issues/3210 may be helpful in writing this app.
HHousen
on 9 Jul 2019
Currently connecting Nextcloud (17.0.0 rc2) to a Minio B2 gateway will not work in read-write mode. Read-only mode somehow works on my deployment, but no extensive testing has been done. A Minio maintainer considers this a bug from Nextcloud.
Jamesits
on 22 Sep 2019
Tricky. S3 does not complain about uploading small files with Multipart Upload.
I think we could try to replace the MultipartUploader here with the ObjectUploader because it uses Multipart only if required: https://github.com/nextcloud/3rdparty/blob/793150ff5c84584b3d2fba9ddbdf9069d65d0d19/aws/aws-sdk-php/src/S3/ObjectUploader.php#L106-L108
kesselb
on 22 Sep 2019
Someone of you had the chance to test the above patch?
kesselb
on 30 Oct 2019
@kesselb I gave your patch a try. It seems to fix small file uploads and I was able to get past installation this time using Minio/B2, but upload of large files fails with the follow error:
Sabre\DAV\Exception\BadRequest: Expected filesize of 10485760 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 15728640 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.
joelkennedy
on 3 Nov 2019
@kesselb
i tried your patch with local minio installation. before you patch i was unable to upload files larger than 512mb,
minio logs (before patch):
Nov 15 22:31:55 Khadas minio[8405]: time="2019-11-15T22:31:55+08:00" level=error msg="Unable to create object part." cause="read tcp 192.168.101.54:9000->192.168.101.42:62874: i/o timeout" source="[object-handlers.go:831:objectAPIHandlers.PutObjectPartHandler()]" stack="fs-v1-helpers.go:272:fsCreateFile fs-v1-multipart.go:547:fsObjects.PutObjectPart <autogenerated>:383:(*fsObjects).PutObjectPart object-handlers.go:828:objectAPIHandlers.PutObjectPartHandler api-router.go:46:(objectAPIHandlers).PutObjectPartHandler-fm"
nextcloud logs (before patch) : https://gist.github.com/Cinerar/dd97c81ba2a4c06c1ca0660430e300e6
now everything looks ok, tested for files up to 10gb
Cinerar
on 15 Nov 2019
@Cinerar hey, would you mind sharing your nextcloud config?
i tried the path with minio and b2 and couldn't get it to work, so I believe it may be a config issue.
thanks in advance!
c-reiter
on 13 Dec 2019
Nothing special some tweaks in php
upload_max_filesize = 16G
post_max_size = 16G
default_socket_timeout = 3600
max_input_time = 3600
max_execution_time = 3600
memory_limit = 1024M
Cinerar
on 13 Dec 2019
Thanks @Cinerar @joelkennedy :+1:
@autoize I've seen your blog post about the patch. Any issues found during the tests?
kesselb
on 4 Feb 2020
@kesselb Sometimes when loading the dashboard it seems to timeout or be missing some of the menu icons, with a 500 Internal Server error. Clearing the browser cache and logging in again seems to do the trick. It might be related to the latency between the Minio S3-to-B2 gateway and Backblaze's datacenter.
With your patch and the Default encryption module app enabled (not necessary to actually enable server side encryption), large files can be uploaded no problem - concurrent with @Cinerar's experience.
Tested it with a video file up to 200 MB - so the original issue seems to be fixed.
Here are the errors in the log when it times out:
TypeError: Argument 1 passed to OC\Files\Cache\CacheQueryBuilder::whereFileId() must be of the type int, null given, called in /var/www/nextcloud/lib/private/Files/Cache/Cache.php on line 655
/var/www/nextcloud/lib/private/Files/Cache/Cache.php - line 655:
OC\Files\Cache\CacheQueryBuilder->whereFileId(null)
/var/www/nextcloud/lib/private/Files/Cache/Cache.php - line 579:
OC\Files\Cache\Cache->moveFromCache(OC\Files\Cache\Cache {}, "appdata_oc7 ... t", "appdata_oc7 ... s")
/var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php - line 481:
OC\Files\Cache\Cache->move("appdata_oc7 ... t", "appdata_oc7 ... s")
/var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php - line 405:
OC\Files\ObjectStore\ObjectStoreStorage->writeStream("appdata_oc7 ... s", null, 1)
/var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php - line 313:
OC\Files\ObjectStore\ObjectStoreStorage->writeBack("/tmp/oc_tmp_EQyQjP-.css", "appdata_oc7 ... s")
<<closure>>
OC\Files\ObjectStore\ObjectStoreStorage->OC\Files\ObjectStore\{closure}("*** sensiti ... *")
/var/www/nextcloud/3rdparty/icewind/streams/src/CallbackWrapper.php - line 121:
call_user_func(Closure {})
<<closure>>
Icewind\Streams\CallbackWrapper->stream_close()
/var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php - line 426:
fclose(null)
/var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php - line 381:
OC\Files\ObjectStore\ObjectStoreStorage->file_put_contents("appdata_oc7 ... s", " ")
/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php - line 354:
OC\Files\ObjectStore\ObjectStoreStorage->touch("appdata_oc7 ... s", 1580857454)
/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php - line 364:
OC\Files\Storage\Wrapper\Wrapper->touch("appdata_oc7 ... s", null)
/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php - line 354:
OC\Files\Storage\Wrapper\Availability->touch("appdata_oc7 ... s", null)
/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php - line 354:
OC\Files\Storage\Wrapper\Wrapper->touch("appdata_oc7 ... s", null)
/var/www/nextcloud/lib/private/Files/View.php - line 1162:
OC\Files\Storage\Wrapper\Wrapper->touch("appdata_oc7 ... s")
/var/www/nextcloud/lib/private/Files/View.php - line 569:
OC\Files\View->basicOperation("touch", "/appdata_oc ... s", [ "touch","create","write"], null)
/var/www/nextcloud/lib/private/Files/Node/Folder.php - line 184:
OC\Files\View->touch("/appdata_oc ... s")
/var/www/nextcloud/lib/private/Files/SimpleFS/SimpleFolder.php - line 84:
OC\Files\Node\Folder->newFile("9160-ce11-results.css")
/var/www/nextcloud/lib/private/Template/SCSSCacher.php - line 312:
OC\Files\SimpleFS\SimpleFolder->newFile("9160-ce11-results.css")
/var/www/nextcloud/lib/private/Template/SCSSCacher.php - line 182:
OC\Template\SCSSCacher->cache("/var/www/ne ... s", "9160-ce11-results.css", "results.scss", OC\Files\Sim ... {}, "/core/search/css")
/var/www/nextcloud/lib/private/Template/CSSResourceLocator.php - line 110:
OC\Template\SCSSCacher->process("/var/www/nextcloud", "core/search/css/results.scss", "core")
/var/www/nextcloud/lib/private/Template/CSSResourceLocator.php - line 62:
OC\Template\CSSResourceLocator->cacheAndAppendScssIfExist("/var/www/nextcloud", "core/search/css/results.scss")
/var/www/nextcloud/lib/private/Template/ResourceLocator.php - line 78:
OC\Template\CSSResourceLocator->doFind("search/css/results")
/var/www/nextcloud/lib/private/TemplateLayout.php - line 310:
OC\Template\ResourceLocator->find([ "css/serve ... "])
/var/www/nextcloud/lib/private/TemplateLayout.php - line 215:
OC\TemplateLayout::findStylesheetFiles([ "css/serve ... "])
/var/www/nextcloud/lib/private/legacy/template.php - line 184:
OC\TemplateLayout->__construct("user", "files")
/var/www/nextcloud/lib/public/AppFramework/Http/TemplateResponse.php - line 167:
OC_Template->fetchPage({ usedSpaceP ... ]})
/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 123:
OCP\AppFramework\Http\TemplateResponse->render()
/var/www/nextcloud/lib/private/AppFramework/App.php - line 125:
OC\AppFramework\Http\Dispatcher->dispatch(OCA\Files\Co ... {}, "index")
/var/www/nextcloud/lib/private/AppFramework/Routing/RouteActionHandler.php - line 47:
OC\AppFramework\App::main("OCA\\Files\ ... r", "index", OC\AppFramew ... {}, { _route: "files.view.index"})
<<closure>>
OC\AppFramework\Routing\RouteActionHandler->__invoke({ _route: "files.view.index"})
/var/www/nextcloud/lib/private/Route/Router.php - line 299:
call_user_func(OC\AppFramew ... {}, { _route: "files.view.index"})
/var/www/nextcloud/lib/base.php - line 1008:
OC\Route\Router->match("/apps/files/")
/var/www/nextcloud/index.php - line 38:
OC::handleRequest()
Aws\S3\Exception\S3Exception: Error executing "PutObject" on "https://minio.example.com/cloudstoreb2/urn%3Aoid%3A486"; AWS HTTP error: Server error: `PUT https://minio.example.com/cloudstoreb2/urn%3Aoid%3A486` resulted in a `500 Internal Server Error` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>InternalError</Code><Message>We encountered an internal error, pleas (truncated...) InternalError (server): We encountered an internal error, please try again. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message><Key>urn:oid:486</Key><BucketName>cloudstoreb2</BucketName><Resource>/cloudstoreb2/urn:oid:486</Resource><RequestId>15F0558375D807DB</RequestId><HostId>e7482b37-0b14-4984-96cc-d3924ce7ebe8</HostId></Error>
/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/WrappedHttpHandler.php - line 97:
Aws\WrappedHttpHandler->parseError("*** sensiti ... *", GuzzleHttp\Psr7\Request {}, Aws\Command {}, [])
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 203:
Aws\WrappedHttpHandler->Aws\{closure}("*** sensiti ... *")
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 174:
GuzzleHttp\Promise\Promise::callHandler(2, "*** sensiti ... *", [ GuzzleHttp ... }])
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/RejectedPromise.php - line 40:
GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}("*** sensiti ... *")
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/TaskQueue.php - line 47:
GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}("*** sensiti ... *")
/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php - line 98:
GuzzleHttp\Promise\TaskQueue->run()
/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php - line 125:
GuzzleHttp\Handler\CurlMultiHandler->tick()
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 246:
GuzzleHttp\Handler\CurlMultiHandler->execute(true)
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 223:
GuzzleHttp\Promise\Promise->invokeWaitFn()
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 267:
GuzzleHttp\Promise\Promise->waitIfPending()
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 225:
GuzzleHttp\Promise\Promise->invokeWaitList()
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 267:
GuzzleHttp\Promise\Promise->waitIfPending()
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 225:
GuzzleHttp\Promise\Promise->invokeWaitList()
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 62:
GuzzleHttp\Promise\Promise->waitIfPending()
/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/ObjectUploader.php - line 90:
GuzzleHttp\Promise\Promise->wait()
/var/www/nextcloud/lib/private/Files/ObjectStore/S3ObjectTrait.php - line 95:
Aws\S3\ObjectUploader->upload()
/var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php - line 468:
OC\Files\ObjectStore\S3->writeObject("urn:oid:486", null)
/var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php - line 405:
OC\Files\ObjectStore\ObjectStoreStorage->writeStream("appdata_oc7 ... s", null, 152574)
/var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php - line 313:
OC\Files\ObjectStore\ObjectStoreStorage->writeBack("/tmp/oc_tmp_I6kjnj-.css", "appdata_oc7 ... s")
<<closure>>
OC\Files\ObjectStore\ObjectStoreStorage->OC\Files\ObjectStore\{closure}("*** sensiti ... *")
/var/www/nextcloud/3rdparty/icewind/streams/src/CallbackWrapper.php - line 121:
call_user_func(Closure {})
<<closure>>
Icewind\Streams\CallbackWrapper->stream_close()
/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Encryption.php - line 236:
fclose(null)
/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php - line 256:
OC\Files\Storage\Wrapper\Encryption->file_put_contents("appdata_oc7 ... s", ":root {\n-- ... }")
/var/www/nextcloud/lib/private/Files/View.php - line 1160:
OC\Files\Storage\Wrapper\Wrapper->file_put_contents("appdata_oc7 ... s", ":root {\n-- ... }")
/var/www/nextcloud/lib/private/Files/View.php - line 694:
OC\Files\View->basicOperation("file_put_contents", "/appdata_oc ... s", [ "update","write"], ":root {\n-- ... }")
/var/www/nextcloud/lib/private/Files/Node/File.php - line 71:
OC\Files\View->file_put_contents("/appdata_oc ... s", ":root {\n-- ... }")
/var/www/nextcloud/lib/private/Files/SimpleFS/SimpleFile.php - line 108:
OC\Files\Node\File->putContent(":root {\n-- ... }")
/var/www/nextcloud/lib/private/Template/IconsCacher.php - line 146:
OC\Files\SimpleFS\SimpleFile->putContent(":root {\n-- ... }")
/var/www/nextcloud/lib/private/Template/SCSSCacher.php - line 337:
OC\Template\IconsCacher->setIconsCss(".icon-room{ ... }")
/var/www/nextcloud/lib/private/Template/SCSSCacher.php - line 182:
OC\Template\SCSSCacher->cache("/var/www/ne ... s", "c5f2-ce11-icons.css", "icons.scss", OC\Files\Sim ... {}, "/apps/files_sharing/css")
/var/www/nextcloud/lib/private/Template/CSSResourceLocator.php - line 110:
OC\Template\SCSSCacher->process("/var/www/ne ... g", "css/icons.scss", "files_sharing")
/var/www/nextcloud/lib/private/Template/CSSResourceLocator.php - line 85:
OC\Template\CSSResourceLocator->cacheAndAppendScssIfExist("/var/www/ne ... g", "css/icons.scss", "files_sharing")
/var/www/nextcloud/lib/private/Template/ResourceLocator.php - line 78:
OC\Template\CSSResourceLocator->doFind("css/icons")
/var/www/nextcloud/lib/private/TemplateLayout.php - line 310:
OC\Template\ResourceLocator->find([ "css/serve ... "])
/var/www/nextcloud/lib/private/TemplateLayout.php - line 215:
OC\TemplateLayout::findStylesheetFiles([ "css/serve ... "])
/var/www/nextcloud/lib/private/legacy/template.php - line 184:
OC\TemplateLayout->__construct("user", "files")
/var/www/nextcloud/lib/public/AppFramework/Http/TemplateResponse.php - line 167:
OC_Template->fetchPage({ usedSpaceP ... ]})
/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 123:
OCP\AppFramework\Http\TemplateResponse->render()
/var/www/nextcloud/lib/private/AppFramework/App.php - line 125:
OC\AppFramework\Http\Dispatcher->dispatch(OCA\Files\Co ... {}, "index")
/var/www/nextcloud/lib/private/AppFramework/Routing/RouteActionHandler.php - line 47:
OC\AppFramework\App::main("OCA\\Files\ ... r", "index", OC\AppFramew ... {}, { _route: "files.view.index"})
<<closure>>
OC\AppFramework\Routing\RouteActionHandler->__invoke({ _route: "files.view.index"})
/var/www/nextcloud/lib/private/Route/Router.php - line 299:
call_user_func(OC\AppFramew ... {}, { _route: "files.view.index"})
/var/www/nextcloud/lib/base.php - line 1008:
OC\Route\Router->match("/apps/files/")
/var/www/nextcloud/index.php - line 38:
OC::handleRequest()
Caused by GuzzleHttp\Exception\ServerException: Server error: `PUT https://minio.example.com/cloudstoreb2/urn%3Aoid%3A486` resulted in a `500 Internal Server Error` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>InternalError</Code><Message>We encountered an internal error, pleas (truncated...)
/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php - line 66:
GuzzleHttp\Exception\RequestException::create(GuzzleHttp\Psr7\Request {}, "*** sensiti ... *")
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 203:
GuzzleHttp\Middleware::GuzzleHttp\{closure}("*** sensiti ... *")
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 156:
GuzzleHttp\Promise\Promise::callHandler(1, "*** sensiti ... *", [ GuzzleHttp ... l])
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/TaskQueue.php - line 47:
GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}("*** sensiti ... *")
/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php - line 98:
GuzzleHttp\Promise\TaskQueue->run()
/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php - line 125:
GuzzleHttp\Handler\CurlMultiHandler->tick()
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 246:
GuzzleHttp\Handler\CurlMultiHandler->execute(true)
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 223:
GuzzleHttp\Promise\Promise->invokeWaitFn()
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 267:
GuzzleHttp\Promise\Promise->waitIfPending()
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 225:
GuzzleHttp\Promise\Promise->invokeWaitList()
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 267:
GuzzleHttp\Promise\Promise->waitIfPending()
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 225:
GuzzleHttp\Promise\Promise->invokeWaitList()
/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 62:
GuzzleHttp\Promise\Promise->waitIfPending()
/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/ObjectUploader.php - line 90:
GuzzleHttp\Promise\Promise->wait()
/var/www/nextcloud/lib/private/Files/ObjectStore/S3ObjectTrait.php - line 95:
Aws\S3\ObjectUploader->upload()
/var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php - line 468:
OC\Files\ObjectStore\S3->writeObject("urn:oid:486", null)
/var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php - line 405:
OC\Files\ObjectStore\ObjectStoreStorage->writeStream("appdata_oc7 ... s", null, 152574)
/var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php - line 313:
OC\Files\ObjectStore\ObjectStoreStorage->writeBack("/tmp/oc_tmp_I6kjnj-.css", "appdata_oc7 ... s")
<<closure>>
OC\Files\ObjectStore\ObjectStoreStorage->OC\Files\ObjectStore\{closure}("*** sensiti ... *")
/var/www/nextcloud/3rdparty/icewind/streams/src/CallbackWrapper.php - line 121:
call_user_func(Closure {})
<<closure>>
Icewind\Streams\CallbackWrapper->stream_close()
/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Encryption.php - line 236:
fclose(null)
/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php - line 256:
OC\Files\Storage\Wrapper\Encryption->file_put_contents("appdata_oc7 ... s", ":root {\n-- ... }")
/var/www/nextcloud/lib/private/Files/View.php - line 1160:
OC\Files\Storage\Wrapper\Wrapper->file_put_contents("appdata_oc7 ... s", ":root {\n-- ... }")
/var/www/nextcloud/lib/private/Files/View.php - line 694:
OC\Files\View->basicOperation("file_put_contents", "/appdata_oc ... s", [ "update","write"], ":root {\n-- ... }")
/var/www/nextcloud/lib/private/Files/Node/File.php - line 71:
OC\Files\View->file_put_contents("/appdata_oc ... s", ":root {\n-- ... }")
/var/www/nextcloud/lib/private/Files/SimpleFS/SimpleFile.php - line 108:
OC\Files\Node\File->putContent(":root {\n-- ... }")
/var/www/nextcloud/lib/private/Template/IconsCacher.php - line 146:
OC\Files\SimpleFS\SimpleFile->putContent(":root {\n-- ... }")
/var/www/nextcloud/lib/private/Template/SCSSCacher.php - line 337:
OC\Template\IconsCacher->setIconsCss(".icon-room{ ... }")
/var/www/nextcloud/lib/private/Template/SCSSCacher.php - line 182:
OC\Template\SCSSCacher->cache("/var/www/ne ... s", "c5f2-ce11-icons.css", "icons.scss", OC\Files\Sim ... {}, "/apps/files_sharing/css")
/var/www/nextcloud/lib/private/Template/CSSResourceLocator.php - line 110:
OC\Template\SCSSCacher->process("/var/www/ne ... g", "css/icons.scss", "files_sharing")
/var/www/nextcloud/lib/private/Template/CSSResourceLocator.php - line 85:
OC\Template\CSSResourceLocator->cacheAndAppendScssIfExist("/var/www/ne ... g", "css/icons.scss", "files_sharing")
/var/www/nextcloud/lib/private/Template/ResourceLocator.php - line 78:
OC\Template\CSSResourceLocator->doFind("css/icons")
/var/www/nextcloud/lib/private/TemplateLayout.php - line 310:
OC\Template\ResourceLocator->find([ "css/serve ... "])
/var/www/nextcloud/lib/private/TemplateLayout.php - line 215:
OC\TemplateLayout::findStylesheetFiles([ "css/serve ... "])
/var/www/nextcloud/lib/private/legacy/template.php - line 184:
OC\TemplateLayout->__construct("user", "files")
/var/www/nextcloud/lib/public/AppFramework/Http/TemplateResponse.php - line 167:
OC_Template->fetchPage({ usedSpaceP ... ]})
/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 123:
OCP\AppFramework\Http\TemplateResponse->render()
/var/www/nextcloud/lib/private/AppFramework/App.php - line 125:
OC\AppFramework\Http\Dispatcher->dispatch(OCA\Files\Co ... {}, "index")
/var/www/nextcloud/lib/private/AppFramework/Routing/RouteActionHandler.php - line 47:
OC\AppFramework\App::main("OCA\\Files\ ... r", "index", OC\AppFramew ... {}, { _route: "files.view.index"})
<<closure>>
OC\AppFramework\Routing\RouteActionHandler->__invoke({ _route: "files.view.index"})
/var/www/nextcloud/lib/private/Route/Router.php - line 299:
call_user_func(OC\AppFramew ... {}, { _route: "files.view.index"})
/var/www/nextcloud/lib/base.php - line 1008:
OC\Route\Router->match("/apps/files/")
/var/www/nextcloud/index.php - line 38:
OC::handleRequest()
autoize
on 5 Feb 2020
@kesselb I gave your patch a try. It seems to fix small file uploads and I was able to get past installation this time using Minio/B2, but upload of large files fails with the follow error:
Sabre\DAV\Exception\BadRequest: Expected filesize of 10485760 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 15728640 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.
I have the same issue
maprambo
on 22 Feb 2020
I have the same issue
If you enable the Default Encryption Module app, this error message will go away, and the files > 10 MB will be successfully uploaded. You do not actually have to enable server side encryption - simply enable the app and leave the checkbox unchecked.
autoize
on 22 Feb 2020
https://www.backblaze.com/blog/backblaze-b2-s3-compatible-api/ I think this should help 馃檪
rogatty
on 4 May 2020
https://www.backblaze.com/blog/backblaze-b2-s3-compatible-api/ I think this should help slightly_smiling_face
It is indeed.. now I only need to migrate my data from the old bucket to the new s3 compatible bucket.
much-doge
on 5 May 2020
Class C transactions cap reached 75% after 1 day configured in nextcloud
Setting "Check for changes" to never fixed it
tsgoff
on 5 May 2020
I've noticed that Backblaze seems to be storing 2 identical versions of each file Nextcloud uploads when connected with the new S3 compatible API (one of which is immediately hidden but costs towards the data stored)- can anyone else observe this behaviour? It happens with the mobile app and desktop sync, but the gvfs connector on GNOME doesn't seem to trigger the issue, which is strange as they both send a single PUT command to the server. Server version is 18.0.4.
blueish4
on 7 May 2020
I've noticed that Backblaze seems to be storing 2 identical versions of each file Nextcloud uploads when connected with the new S3 compatible API (one of which is immediately hidden but costs towards the data stored)- can anyone else observe this behaviour? It happens with the mobile app and desktop sync, but the gvfs connector on GNOME doesn't seem to trigger the issue, which is strange as they both send a single PUT command to the server. Server version is 18.0.4.
I also experienced the same issue. Copying files form my local storage to external B2 storage from within web interface creates duplicate files in B2
much-doge
on 15 May 2020
I believe this is because Nextcloud creates a .part file and then renames the file once the upload is complete. Rename in S3 means PutObjectCopy+DeleteObject, which is how you end up with one live and one hidden object. My solution for now is to use the "Keep only the last version of the file" lifecycle setting, which means B2 will delete the hidden copy after a day.
I also had to disable file locking as it was causing problems with file deletion. Is probably not needed for S3 anyways (S3 objects always have atomic consistency):
'filelocking.enabled' => false,
This should already be disabled (by default):
'filesystem_check_changes' => 0,
joyov
on 26 May 2020
To anyone who applied the patch I posted: Revert it.
I spent some time today to polish it more and submit a pull request for Nextcloud but run into various issues. The storage implementation assumes that the number of bytes read from a stream are the number of bytes written to the remote storage. But the ObjectUploader always reads the first 5 MB of a stream to check if MultipartUpload or PutObject should be used thus the counter for read bytes is always (at least) 5 MB to high. This is somehow related to those "Argument 1 passed to OC\Files\Cache\CacheQueryBuilder::whereFileId() must be of the type int, null given" errors. Default encryption module workarounds this problem because the size of the encrypted file is always higher than the actual content.
I still strongly advice to use it.
kesselb
on 22 Jul 2020
Class C transactions cap reached 75% after 1 day configured in nextcloud
Setting "Check for changes" to never fixed it
I'm facing the same problem with a B2 external storage, but setting "Check for changes" to "never" doesn't help. Today, one single nextcloud desktop client generated about 250 class C transactions in about 30 minutes.
Is there anything else I could try?
kalatabe
on 19 Aug 2020
This does not affect only B2 but other S3 compatible filesystems dealing (badly) with multipart uploads for small files.
This leads to a lot of fclose, cacheQueryBuilder and other Expected filesize of errors while using multipart uploads.
acsfer
on 11 Nov 2020
Related issues
williambargent
路
3Comments
georgehrke
路
3Comments
ChristophWurst
路
3Comments
blackcrack
路
3Comments
jancborchardt
路
3Comments
Most helpful comment
Can I ask why "B2 external storage will not be implemented in the server code itself"? If I were to develop it, and try to get it merged, would it be rejected? Is it a thing where "We've accepted Swift, S3, Azure, and are just done with all these protocols"?