Server: Object storage as primary storage is broken in Nextcloud12

Created on 24 Jul 2017  ·  30Comments  ·  Source: nextcloud/server

Hi,

We use successfully Nextcloud11 together with swift object storage as primary storage. However, using Nextcloud12, this seems to be broken.

Cheers,

Ron

Steps to reproduce

1.
2.
3.

Expected behaviour

Tell us what should happen

Actual behaviour

Tell us what happens instead

Server configuration

Operating system:

Web server:

Database:

PHP version:

Nextcloud version: (see Nextcloud admin page)

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from:

Signing status:


Signing status

Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

List of activated apps:


App list

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your Nextcloud installation folder

Nextcloud configuration:


Config report

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your Nextcloud installation folder

or 

Insert your config.php content here. 
Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …)

Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption: yes/no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...

LDAP configuration (delete this part if not used)


LDAP config

With access to your command line run e.g.:
sudo -u www-data php occ ldap:show-config
from within your Nextcloud installation folder

Without access to your command line download the data/owncloud.db to your local
computer or access your SQL server remotely and run the select query:
SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap';


Eventually replace sensitive data as the name/IP-address of your LDAP server or groups.

Client configuration

Browser:

Operating system:

Logs

Web server error log


Web server error log

Insert your webserver log here

Nextcloud log (data/nextcloud.log)


Nextcloud log

Insert your Nextcloud log here

Browser log


Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

0. Needs triage bug filesystem needs info
Source
picture rontrompert
👍3

Most helpful comment

Update! I got S3 working as primary storage on nextcloud v12.

Currently it seems that the only way to make this work is to re-install nextcloud with the 'objectstore' parameters in config.php.

First I had to make sure that the objectstore was properly added to config/config.php and then In the same file I set 'installed' => true, to false and then reloaded my nextcloud home page.

Then I had to go create a NEW database on my DB server and give nextcloud the credentials for the new database. The old one will mess things up and I tried several ways of dealing with it but was unsuccessful. So I would suggest just making a new database.

Then I re-ran the installation process and everything came up correctly and I could see the proper files in my S3 bucket.

Basically the solution is that you have to know how to set up an object store as your primary storage before you ever install nextcloud and have your object store settings in config.php before you ever run the first installation.

Overall this is NOT clear on the primary storage page of the documentation.

That page even says:

Configuring a primary object store on an existing Nextcloud instance will make all existing files on the instance inaccessible.

Which implies that you should be able to just add the code shown below and it should work, but leave your old files inaccessible. This does not seem to be the case. It seems you must install completely fresh if you want to use an object store as your primary storage.

picture KroniK907 on 29 Aug 2017
👍5 🎉4

All 30 comments

We use successfully Nextcloud11 together with swift object storage as primary storage. However, using Nextcloud12, this seems to be broken.

We have multiple instances running with Swift as well as with S3 (they share a lot of code) and both work just fine. Are there more details what is not working?

MorrisJobke picture MorrisJobke on 25 Jul 2017

We have a nextcloud 11.0.3.2 install that we updated to 12, I unpacked the tarball on top on the old one and ran the updater. Now, the site gives a 500 server error, which goes away when I remove the multibucket-object store configuration from config.php

Here is the config:
``` $CONFIG = array (
'instanceid' => 'hidden',
'passwordsalt' => 'hidden',
'secret' => 'hidden',
'trusted_domains' =>
array (
0 => 'hidden',
1 => 'hidden',
),
'datadirectory' => '/var/www/test/data',
'overwrite.cli.url' => 'https://hidden',
'dbtype' => 'mysql',
'version' => '12.0.0.29',
'dbname' => 'hidden',
'dbhost' => 'hidden',
'dbport' => '',
'dbtableprefix' => 'oc_',
'dbuser' => 'hidden',
'dbpassword' => 'hidden',
'installed' => true,
'overwritehost' => 'hidden',
'overwriteprotocol' => 'https',
'forwarded_for_headers' =>
array (
0 => 'HTTP_X_FORWARDED_FOR',
1 => 'HTTP_FORWARDED_FOR',
),
'memcache.local' => '\OC\Memcache\APCu',
'memcache.distributed' => '\OC\Memcache\Redis',
'memcache.locking' => '\OC\Memcache\Redis',
'redis' =>
array (
'host' => 'hidden',
'port' => 6379,
'timeout' => 0.0,
'password' => '',
'dbindex' => 0,
),
'filelocking.enabled' => 'true',
'trusted_proxies' =>
array (
0 => 'hidden',
),
'mail_domain' => 'hidden',
'appcodechecker' => false,
'updatechecker' => false,
'enable_previews' => true,
'preview_max_x' => 1024,
'preview_max_y' => 1024,
'preview_max_filesize_image' => 100,
'enabledPreviewProviders' =>
array (
0 => 'OC\Preview\PNG',
1 => 'OC\Preview\JPEG',
2 => 'OC\Preview\GIF',
3 => 'OC\Preview\BMP',
4 => 'OC\Preview\XBitmap',
5 => 'OC\Preview\TXT',
6 => 'OC\Preview\MarkDown',
),
'appstoreenabled' => false,
'has_internet_connection' => false,
'auth.bruteforce.protection.enabled' => true,
'enable_avatars' => true,
'cron_log' => true,
'log_type' => 'file',
'logfile' => '/var/log/hidden.log',
'logtimezone' => 'Europe/Amsterdam',
'loglevel' => '3',
'logdateformat' => 'd/M/Y:H:i:s',
'objectstore_multibucket' =>
array (
'class' => 'OC\Files\ObjectStore\Swift',
'arguments' =>
array (
'autocreate' => true,
'username' => 'hidden',
'password' => 'hidden',
'region' => 'RegionOne',
'url' => 'https://hidden:5000/v2.0',
'tenantName' => 'hidden',
'serviceName' => 'Swift',
'urlType' => 'internal',
),
),
'ldapIgnoreNamingRules' => false,
'ldapProviderFactory' => '\OCA\User_LDAP\LDAPProviderFactory',
'updater.secret' => 'hidden',
'maintenance' => false,
'theme' => '',
);


Without the 'objectstore_multibucket' entry, I get no errors, but with that enabled I get a 500 server error.

Log:

{"reqId":"WXcaOHiatmRRY6wSrC9MFAAAABM","level":3,"time":"25\/Jul\/2017:12:15:20","remoteAddr":"xxx.xxx.xxx.xxx","user":"admin","app":"objectstore","method":"GET","url":"\/index.php\/apps\/files\/?fileNotFound=1","message":"Could not create object: Invalid url type","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko\/20100101 Firefox\/54.0","version":"12.0.0.29"}
{"reqId":"WXcaOHiatmRRY6wSrC9MFAAAABM","level":3,"time":"25\/Jul\/2017:12:15:20","remoteAddr":"xxx.xxx.xxx.xxx","user":"admin","app":"objectstore","method":"GET","url":"\/index.php\/apps\/files\/?fileNotFound=1","message":"Could not create object: Invalid url type","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko\/20100101 Firefox\/54.0","version":"12.0.0.29"}
{"reqId":"WXcaOHiatmRRY6wSrC9MFAAAABM","level":3,"time":"25\/Jul\/2017:12:15:20","remoteAddr":"xxx.xxx.xxx.xxx","user":"admin","app":"objectstore","method":"GET","url":"\/index.php\/apps\/files\/?fileNotFound=1","message":"Could not create object: Invalid url type","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko\/20100101 Firefox\/54.0","version":"12.0.0.29"}
{"reqId":"WXcaOHiatmRRY6wSrC9MFAAAABM","level":3,"time":"25\/Jul\/2017:12:15:20","remoteAddr":"xxx.xxx.xxx.xxx","user":"admin","app":"index","method":"GET","url":"\/index.php\/apps\/files\/?fileNotFound=1","message":"Exception: {\"Exception\":\"OCP\\Files\\NotFoundException\",\"Message\":\"\",\"Code\":0,\"Trace\":\"#0 \\/var\\/www\\/test\\/lib\\/private\\/Files\\/Node\\/Node.php(217): OC\\Files\\Node\\Node->getFileInfo()\n#1 \\/var\\/www\\/test\\/lib\\/private\\/Files\\/Node\\/Node.php(117): OC\\Files\\Node\\Node->getPermissions()\n#2 \\/var\\/www\\/test\\/lib\\/private\\/Files\\/Node\\/File.php(62): OC\\Files\\Node\\Node->checkPermissions(2)\n#3 \\/var\\/www\\/test\\/lib\\/private\\/Files\\/SimpleFS\\/SimpleFile.php(95): OC\\Files\\Node\\File->putContent('(function($) {\\n...')\n#4 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/JSCombiner.php(163): OC\\Files\\SimpleFS\\SimpleFile->putContent('(function($) {\\n...')\n#5 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/JSCombiner.php(89): OC\\Template\\JSCombiner->cache('\\/var\\/www\\/test\\/c...', 'merged-template...', Object(OC\\Files\\SimpleFS\\SimpleFolder))\n#6 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/JSResourceLocator.php(105): OC\\Template\\JSCombiner->process('\\/var\\/www\\/test', 'core\\/js\\/merged-...', 'core')\n#7 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/JSResourceLocator.php(68): OC\\Template\\JSResourceLocator->cacheAndAppendCombineJsonIfExist('\\/var\\/www\\/test', 'core\\/js\\/merged-...')\n#8 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/ResourceLocator.php(75): OC\\Template\\JSResourceLocator->doFind('js\\/merged-templ...')\n#9 \\/var\\/www\\/test\\/lib\\/private\\/TemplateLayout.php(249): OC\\Template\\ResourceLocator->find(Array)\n#10 \\/var\\/www\\/test\\/lib\\/private\\/TemplateLayout.php(132): OC\\TemplateLayout::findJavascriptFiles(Array)\n#11 \\/var\\/www\\/test\\/lib\\/private\\/legacy\\/template.php(205): OC\\TemplateLayout->__construct('user', 'files')\n#12 \\/var\\/www\\/test\\/lib\\/public\\/AppFramework\\/Http\\/TemplateResponse.php(157): OC_Template->fetchPage()\n#13 \\/var\\/www\\/test\\/lib\\/private\\/AppFramework\\/Http\\/Dispatcher.php(113): OCP\\AppFramework\\Http\\TemplateResponse->render()\n#14 \\/var\\/www\\/test\\/lib\\/private\\/AppFramework\\/App.php(114): OC\\AppFramework\\Http\\Dispatcher->dispatch(Object(OCA\\Files\\Controller\\ViewController), 'index')\n#15 \\/var\\/www\\/test\\/lib\\/private\\/AppFramework\\/Routing\\/RouteActionHandler.php(47): OC\\AppFramework\\App::main('ViewController', 'index', Object(OC\\AppFramework\\DependencyInjection\\DIContainer), Array)\n#16 [internal function]: OC\\AppFramework\\Routing\\RouteActionHandler->__invoke(Array)\n#17 \\/var\\/www\\/test\\/lib\\/private\\/Route\\/Router.php(299): call_user_func(Object(OC\\AppFramework\\Routing\\RouteActionHandler), Array)\n#18 \\/var\\/www\\/test\\/lib\\/base.php(1000): OC\\Route\\Router->match('\\/apps\\/files\\/')\n#19 \\/var\\/www\\/test\\/index.php(40): OC::handleRequest()\n#20 {main}\",\"File\":\"\\/var\\/www\\/test\\/lib\\/private\\/Files\\/Node\\/Node.php\",\"Line\":97}","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko\/20100101 Firefox\/54.0","version":"12.0.0.29"}
{"reqId":"WXcaOHiatmRRY6wSrC9MFAAAABM","level":3,"time":"25\/Jul\/2017:12:15:20","remoteAddr":"xxx.xxx.xxx.xxx","user":"admin","app":"objectstore","method":"GET","url":"\/index.php\/apps\/files\/?fileNotFound=1","message":"Could not create object: Invalid url type","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko\/20100101 Firefox\/54.0","version":"12.0.0.29"}
{"reqId":"WXcaOHiatmRRY6wSrC9MFAAAABM","level":3,"time":"25\/Jul\/2017:12:15:20","remoteAddr":"xxx.xxx.xxx.xxx","user":"admin","app":"objectstore","method":"GET","url":"\/index.php\/apps\/files\/?fileNotFound=1","message":"Could not create object: Invalid url type","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko\/20100101 Firefox\/54.0","version":"12.0.0.29"}
{"reqId":"WXcaOHiatmRRY6wSrC9MFAAAABM","level":3,"time":"25\/Jul\/2017:12:15:20","remoteAddr":"xxx.xxx.xxx.xxx","user":"admin","app":"objectstore","method":"GET","url":"\/index.php\/apps\/files\/?fileNotFound=1","message":"Could not create object: Invalid url type","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko\/20100101 Firefox\/54.0","version":"12.0.0.29"}
{"reqId":"WXcaOHiatmRRY6wSrC9MFAAAABM","level":3,"time":"25\/Jul\/2017:12:15:20","remoteAddr":"xxx.xxx.xxx.xxx","user":"admin","app":"core","method":"GET","url":"\/index.php\/apps\/files\/?fileNotFound=1","message":"Exception: {\"Exception\":\"OCP\\Files\\NotFoundException\",\"Message\":\"\",\"Code\":0,\"Trace\":\"#0 \\/var\\/www\\/test\\/lib\\/private\\/Files\\/Node\\/Node.php(217): OC\\Files\\Node\\Node->getFileInfo()\n#1 \\/var\\/www\\/test\\/lib\\/private\\/Files\\/Node\\/Node.php(117): OC\\Files\\Node\\Node->getPermissions()\n#2 \\/var\\/www\\/test\\/lib\\/private\\/Files\\/Node\\/File.php(62): OC\\Files\\Node\\Node->checkPermissions(2)\n#3 \\/var\\/www\\/test\\/lib\\/private\\/Files\\/SimpleFS\\/SimpleFile.php(95): OC\\Files\\Node\\File->putContent('(function($) {\\n...')\n#4 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/JSCombiner.php(163): OC\\Files\\SimpleFS\\SimpleFile->putContent('(function($) {\\n...')\n#5 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/JSCombiner.php(89): OC\\Template\\JSCombiner->cache('\\/var\\/www\\/test\\/c...', 'merged-template...', Object(OC\\Files\\SimpleFS\\SimpleFolder))\n#6 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/JSResourceLocator.php(105): OC\\Template\\JSCombiner->process('\\/var\\/www\\/test', 'core\\/js\\/merged-...', 'core')\n#7 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/JSResourceLocator.php(68): OC\\Template\\JSResourceLocator->cacheAndAppendCombineJsonIfExist('\\/var\\/www\\/test', 'core\\/js\\/merged-...')\n#8 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/ResourceLocator.php(75): OC\\Template\\JSResourceLocator->doFind('js\\/merged-templ...')\n#9 \\/var\\/www\\/test\\/lib\\/private\\/TemplateLayout.php(249): OC\\Template\\ResourceLocator->find(Array)\n#10 \\/var\\/www\\/test\\/lib\\/private\\/TemplateLayout.php(132): OC\\TemplateLayout::findJavascriptFiles(Array)\n#11 \\/var\\/www\\/test\\/lib\\/private\\/legacy\\/template.php(205): OC\\TemplateLayout->__construct('user', 'files')\n#12 \\/var\\/www\\/test\\/lib\\/public\\/AppFramework\\/Http\\/TemplateResponse.php(157): OC_Template->fetchPage()\n#13 \\/var\\/www\\/test\\/lib\\/private\\/AppFramework\\/Http\\/Dispatcher.php(113): OCP\\AppFramework\\Http\\TemplateResponse->render()\n#14 \\/var\\/www\\/test\\/lib\\/private\\/AppFramework\\/App.php(114): OC\\AppFramework\\Http\\Dispatcher->dispatch(Object(OCA\\Files\\Controller\\ViewController), 'index')\n#15 \\/var\\/www\\/test\\/lib\\/private\\/AppFramework\\/Routing\\/RouteActionHandler.php(47): OC\\AppFramework\\App::main('ViewController', 'index', Object(OC\\AppFramework\\DependencyInjection\\DIContainer), Array)\n#16 [internal function]: OC\\AppFramework\\Routing\\RouteActionHandler->__invoke(Array)\n#17 \\/var\\/www\\/test\\/lib\\/private\\/Route\\/Router.php(299): call_user_func(Object(OC\\AppFramework\\Routing\\RouteActionHandler), Array)\n#18 \\/var\\/www\\/test\\/lib\\/base.php(1000): OC\\Route\\Router->match('\\/apps\\/files\\/')\n#19 \\/var\\/www\\/test\\/index.php(40): OC::handleRequest()\n#20 {main}\",\"File\":\"\\/var\\/www\\/test\\/lib\\/private\\/Files\\/Node\\/Node.php\",\"Line\":97}","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko\/20100101 Firefox\/54.0","version":"12.0.0.29"}
{"reqId":"WXcaOHiatmRRY6wSrC9MFAAAABM","level":3,"time":"25\/Jul\/2017:12:15:20","remoteAddr":"xxx.xxx.xxx.xxx","user":"admin","app":"core","method":"GET","url":"\/index.php\/apps\/files\/?fileNotFound=1","message":"Exception: {\"Exception\":\"OCP\\Files\\NotFoundException\",\"Message\":\"\",\"Code\":0,\"Trace\":\"#0 \\/var\\/www\\/test\\/lib\\/private\\/Files\\/Node\\/Node.php(217): OC\\Files\\Node\\Node->getFileInfo()\n#1 \\/var\\/www\\/test\\/lib\\/private\\/Files\\/Node\\/Node.php(117): OC\\Files\\Node\\Node->getPermissions()\n#2 \\/var\\/www\\/test\\/lib\\/private\\/Files\\/Node\\/File.php(62): OC\\Files\\Node\\Node->checkPermissions(2)\n#3 \\/var\\/www\\/test\\/lib\\/private\\/Files\\/SimpleFS\\/SimpleFile.php(95): OC\\Files\\Node\\File->putContent('(function($) {\\n...')\n#4 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/JSCombiner.php(163): OC\\Files\\SimpleFS\\SimpleFile->putContent('(function($) {\\n...')\n#5 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/JSCombiner.php(89): OC\\Template\\JSCombiner->cache('\\/var\\/www\\/test\\/c...', 'merged-template...', Object(OC\\Files\\SimpleFS\\SimpleFolder))\n#6 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/JSResourceLocator.php(105): OC\\Template\\JSCombiner->process('\\/var\\/www\\/test', 'core\\/js\\/merged-...', 'core')\n#7 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/JSResourceLocator.php(68): OC\\Template\\JSResourceLocator->cacheAndAppendCombineJsonIfExist('\\/var\\/www\\/test', 'core\\/js\\/merged-...')\n#8 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/ResourceLocator.php(75): OC\\Template\\JSResourceLocator->doFind('js\\/merged-templ...')\n#9 \\/var\\/www\\/test\\/lib\\/private\\/TemplateLayout.php(249): OC\\Template\\ResourceLocator->find(Array)\n#10 \\/var\\/www\\/test\\/lib\\/private\\/TemplateLayout.php(132): OC\\TemplateLayout::findJavascriptFiles(Array)\n#11 \\/var\\/www\\/test\\/lib\\/private\\/legacy\\/template.php(205): OC\\TemplateLayout->__construct('error', '')\n#12 \\/var\\/www\\/test\\/lib\\/private\\/Template\\/Base.php(132): OC_Template->fetchPage()\n#13 \\/var\\/www\\/test\\/lib\\/private\\/legacy\\/template.php(342): OC\\Template\\Base->printPage()\n#14 \\/var\\/www\\/test\\/index.php(59): OC_Template::printExceptionErrorPage(Object(OCP\\Files\\NotFoundException))\n#15 {main}\",\"File\":\"\\/var\\/www\\/test\\/lib\\/private\\/Files\\/Node\\/Node.php\",\"Line\":97}","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko\/20100101 Firefox\/54.0","version":"12.0.0.29"}
```

jmdeboer-surfsara picture jmdeboer-surfsara on 25 Jul 2017

I also seem to run into trouble with a clean install of nextcloud 12 as well as the 13 master branch checked out.

Edit: Just to confirm, I unpacked the 12.0.0 tar in a clean directory, copied just the above config.php and data directory from the updated install, I get the same error. It's worth noting the Swift config has not changed from when it was working in Nextcloud 11.

jmdeboer-surfsara picture jmdeboer-surfsara on 25 Jul 2017

Edit: Just to confirm, I unpacked the 12.0.0 tar in a clean directory, copied just the above config.php and data directory from the updated install, I get the same error. It's worth noting the Swift config has not changed from when it was working in Nextcloud 11.

Weird. because we fixed the multi bucket config for 12.0.0: #4798

Could you try to use a single bucket option and check if it is broken there as well? And then maybe disable debug log? For 12.0.1 we also improved a bit the debug log for the primary object storage. (you need to install the daily-stable12 or the latest RC of 12.0.1 to get those changes - 12.0.1 is about to release soon).

MorrisJobke picture MorrisJobke on 25 Jul 2017

'urlType' => 'internal',

And the urlType needs to be internalUrl or publicUrl: https://github.com/nextcloud/server/blob/c54a59d51eb924e77df6f6d2bb79ea466d169221/lib/private/Files/ObjectStore/Swift.php#L126-L126

This is the reason for the log messages.

MorrisJobke picture MorrisJobke on 25 Jul 2017

I will change the urlType but the docs appear to be incorrect:
https://docs.nextcloud.com/server/12/admin_manual/configuration_files/primary_storage.html

That is, 'urlType' => 'internal' is mentioned in the single bucket config snippet, no mention in the multi-bucket section.

jmdeboer-surfsara picture jmdeboer-surfsara on 25 Jul 2017

I installed a fresh copy of N C12.0.0.29, with a fresh database. Initially I set it up via the web interface, so it has 'normal' storage. This works, including redis, memcache etc. So it's basically the config as listed above.

When adding the config sections for single or multi bucket swift (and performing a filescan), the web interface only partially renders.

I see the top bar and the sidebar, but the main section of the page remains empty. The dropdown from the gear icon top-right no longer appears. My browser tells me:

"The resource from “https://hidden/index.php/js/activity/activity-sidebar.js?v=2b3ba31c3e5d41de6f94372ea5cff05a-0” was blocked due to MIME type mismatch (X-Content-Type-Options: nosniff)."

Log files say nothing.

Syncing with the desktop client and the android client works.

jmdeboer-surfsara picture jmdeboer-surfsara on 25 Jul 2017

Well, for some reason I don't quite understand the webinterface did not render correctly with object storage enabled, tested on linux with firefox and chromium, and windows 10 with IE and firefox. I then upgraded to 12.0.1 RC2 and then it renders across the board.

However... in 12.0.1 RC2 storing objects no longer works :(

{"reqId":"WXhaEZW5q6kSiPAA1q3TigAAAAI","level":3,"time":"26\/Jul\/2017:11:00:01","remoteAddr":"xxx.xxx.xxx.xxx","user":"admin","app":"objectstore","method":"PUT","url":"\/remote.php\/dav\/files\/admin\/Documents\/icinga-docs-1067.pdf","message":"Could not create object urn:oid:1423 for files\/Documents\/icinga-docs-1067.pdf.ocTransferId1201964647.part: {\"Exception\":\"Guzzle\\\\Http\\\\Exception\\\\ClientErrorResponseException\",\"Message\":\"Client error response\\n[status code] 401\\n[reason phrase] Unauthorized\\n[url] https:\\\/\\\/**hidden**\\\/v1\\\/KEY_**hidden**\\\/35\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/test\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Message\\\/Request.php(145): Guzzle\\\\Http\\\\Exception\\\\BadResponseException::factory(Object(Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest), Object(Guzzle\\\\Http\\\\Message\\\\Response))\\n#1 [internal function]: Guzzle\\\\Http\\\\Message\\\\Request::onRequestError(Object(Guzzle\\\\Common\\\\Event), 'request.error', Object(Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher))\\n#2 \\\/var\\\/www\\\/test\\\/3rdparty\\\/symfony\\\/event-dispatcher\\\/EventDispatcher.php(174): call_user_func(Array, Object(Guzzle\\\\Common\\\\Event), 'request.error', Object(Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher))\\n#3 \\\/var\\\/www\\\/test\\\/3rdparty\\\/symfony\\\/event-dispatcher\\\/EventDispatcher.php(43): Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher->doDispatch(Array, 'request.error', Object(Guzzle\\\\Common\\\\Event))\\n#4 \\\/var\\\/www\\\/test\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Message\\\/Request.php(589): Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher->dispatch('request.error', Object(Guzzle\\\\Common\\\\Event))\\n#5 \\\/var\\\/www\\\/test\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Message\\\/Request.php(378): Guzzle\\\\Http\\\\Message\\\\Request->processResponse(Array)\\n#6 \\\/var\\\/www\\\/test\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Message\\\/EntityEnclosingRequest.php(49): Guzzle\\\\Http\\\\Message\\\\Request->setState('complete', Array)\\n#7 \\\/var\\\/www\\\/test\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(286): Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest->setState('complete', Array)\\n#8 \\\/var\\\/www\\\/test\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(244): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->processResponse(Object(Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest), Object(Guzzle\\\\Http\\\\Curl\\\\CurlHandle), Array)\\n#9 \\\/var\\\/www\\\/test\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(227): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->processMessages()\\n#10 \\\/var\\\/www\\\/test\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(211): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->executeHandles()\\n#11 \\\/var\\\/www\\\/test\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(105): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->perform()\\n#12 \\\/var\\\/www\\\/test\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMultiProxy.php(91): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->send()\\n#13 \\\/var\\\/www\\\/test\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Client.php(282): Guzzle\\\\Http\\\\Curl\\\\CurlMultiProxy->send()\\n#14 \\\/var\\\/www\\\/test\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Message\\\/Request.php(198): Guzzle\\\\Http\\\\Client->send(Object(Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest))\\n#15 \\\/var\\\/www\\\/test\\\/3rdparty\\\/rackspace\\\/php-opencloud\\\/lib\\\/OpenCloud\\\/ObjectStore\\\/Service.php(117): Guzzle\\\\Http\\\\Message\\\\Request->send()\\n#16 \\\/var\\\/www\\\/test\\\/lib\\\/private\\\/Files\\\/ObjectStore\\\/Swift.php(150): OpenCloud\\\\ObjectStore\\\\Service->createContainer('35')\\n#17 \\\/var\\\/www\\\/test\\\/lib\\\/private\\\/Files\\\/ObjectStore\\\/Swift.php(243): OC\\\\Files\\\\ObjectStore\\\\Swift->init()\\n#18 \\\/var\\\/www\\\/test\\\/lib\\\/private\\\/Files\\\/ObjectStore\\\/ObjectStoreStorage.php(399): OC\\\\Files\\\\ObjectStore\\\\Swift->writeObject('urn:oid:1423', Resource id #25)\\n#19 \\\/var\\\/www\\\/test\\\/lib\\\/private\\\/Files\\\/ObjectStore\\\/ObjectStoreStorage.php(306): OC\\\\Files\\\\ObjectStore\\\\ObjectStoreStorage->writeBack('\\\/tmp\\\/oc_tmp_IVc...', 'files\\\/Documents...')\\n#20 [internal function]: OC\\\\Files\\\\ObjectStore\\\\ObjectStoreStorage->OC\\\\Files\\\\ObjectStore\\\\{closure}()\\n#21 \\\/var\\\/www\\\/test\\\/3rdparty\\\/icewind\\\/streams\\\/src\\\/CallbackWrapper.php(109): call_user_func(Object(Closure))\\n#22 [internal function]: Icewind\\\\Streams\\\\CallbackWrapper->stream_close()\\n#23 \\\/var\\\/www\\\/test\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php(135): fclose(Resource id #20)\\n#24 \\\/var\\\/www\\\/test\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/Directory.php(151): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->put(Resource id #10)\\n#25 \\\/var\\\/www\\\/test\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1096): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Directory->createFile('icinga-docs-106...', Resource id #10)\\n#26 \\\/var\\\/www\\\/test\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(525): Sabre\\\\DAV\\\\Server->createFile('files\\\/admin\\\/Doc...', Resource id #10, NULL)\\n#27 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPut(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#28 \\\/var\\\/www\\\/test\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#29 \\\/var\\\/www\\\/test\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(479): Sabre\\\\Event\\\\EventEmitter->emit('method:PUT', Array)\\n#30 \\\/var\\\/www\\\/test\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#31 \\\/var\\\/www\\\/test\\\/apps\\\/dav\\\/lib\\\/Server.php(253): Sabre\\\\DAV\\\\Server->exec()\\n#32 \\\/var\\\/www\\\/test\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(33): OCA\\\\DAV\\\\Server->exec()\\n#33 \\\/var\\\/www\\\/test\\\/remote.php(162): require_once('\\\/var\\\/www\\\/test\\\/a...')\\n#34 {main}\",\"File\":\"\\\/var\\\/www\\\/test\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Exception\\\/BadResponseException.php\",\"Line\":43}","userAgent":"Mozilla\/5.0 (Linux) mirall\/2.3.2 (Nextcloud)","version":"12.0.1.1"}
{"reqId":"WXhaEZW5q6kSiPAA1q3TigAAAAI","level":4,"time":"26\/Jul\/2017:11:00:01","remoteAddr":"xxx.xxx.xxx.xxx","user":"admin","app":"webdav","method":"PUT","url":"\/remote.php\/dav\/files\/admin\/Documents\/icinga-docs-1067.pdf","message":"Exception: {\"Exception\":\"Sabre\\\\DAV\\\\Exception\",\"Message\":\"Client error response\\n[status code] 401\\n[reason phrase] Unauthorized\\n[url] https:\\\/\\\/**hidden**\\\/v1\\\/KEY_**hidden**\\\/35\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/test\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php(159): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->convertToSabreException(Object(Guzzle\\\\Http\\\\Exception\\\\ClientErrorResponseException))\\n#1 \\\/var\\\/www\\\/test\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/Directory.php(151): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->put(Resource id #10)\\n#2 \\\/var\\\/www\\\/test\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1096): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Directory->createFile('icinga-docs-106...', Resource id #10)\\n#3 \\\/var\\\/www\\\/test\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(525): Sabre\\\\DAV\\\\Server->createFile('files\\\/admin\\\/Doc...', Resource id #10, NULL)\\n#4 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPut(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#5 \\\/var\\\/www\\\/test\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#6 \\\/var\\\/www\\\/test\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(479): Sabre\\\\Event\\\\EventEmitter->emit('method:PUT', Array)\\n#7 \\\/var\\\/www\\\/test\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#8 \\\/var\\\/www\\\/test\\\/apps\\\/dav\\\/lib\\\/Server.php(253): Sabre\\\\DAV\\\\Server->exec()\\n#9 \\\/var\\\/www\\\/test\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(33): OCA\\\\DAV\\\\Server->exec()\\n#10 \\\/var\\\/www\\\/test\\\/remote.php(162): require_once('\\\/var\\\/www\\\/test\\\/a...')\\n#11 {main}\",\"File\":\"\\\/var\\\/www\\\/test\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php\",\"Line\":567}","userAgent":"Mozilla\/5.0 (Linux) mirall\/2.3.2 (Nextcloud)","version":"12.0.1.1"}

Reverting to 12.0.0.29 makes it work again (android and desktop, browser still no-go), using the exact same config save the version number. (I 'downgrade' by switching apache's document root, which might not be recommended practice but I am pragmatic)

Edit: Clearing the browser cache seems to have fixed the rendering problem, at least after downgrading. Which is still a bit weird, since I tested with clean browser install on a VM before. And strangely, 12.0.1 now works too.

I suppose the only 'real' issue that remains in the incorrect value for urlType in the documentation, maybe the updater should also check for it so as not to break the upgrade from version 11.

jmdeboer-surfsara picture jmdeboer-surfsara on 26 Jul 2017

I just did a clean install of nextcloud 12 on Ubuntu 16.04, copy pasted the s3 configuration details from https://docs.nextcloud.com/server/12/admin_manual/configuration_files/primary_storage.html changed the details to my details, server breaks, comment those details out and it comes up fine.

<?php
$CONFIG = array (
  'instanceid' => 'example',
  'passwordsalt' => 'example',
  'secret' => 'example',
  'trusted_domains' =>
  array (
    0 => 'localhost',
    1 => '212.x.x.x',
  ),
  'datadirectory' => '/var/www/html/nextcloud/data',
  'overwrite.cli.url' => 'http://localhost',
  'dbtype' => 'mysql',
  'version' => '12.0.0.29',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_kode',
  'dbpassword' => 'example',
  'installed' => true,
  'objectstore' => array(
        'class' => 'OC\\Files\\ObjectStore\\S3',
        'arguments' => array(
                'bucket' => 'nextcloud-s3',
                'autocreate' => true,
                'key'    => 'mykey',
                'secret' => 'mysecret',
                'hostname' => 'example.com',
                'port' => 1234,
                'use_ssl' => true,
                'region' => 'optional',
                // required for some non amazon s3 implementations
                'use_path_style'=>true
        ),
  ),
);

With that config the tab just whirls and whirls, and nothing get set in the apache2 error.log

I also tried commenting out hostname, port and region as these are supposedly optional, at which point it gets an error 500 instead

I was getting Could not create object: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint: \"nextcloud-s3.s3.amazonaws.com\ so I changed the hostname to nextcloud-s3.s3.amazonaws.com, then I started getting Could not create object: The request signature we calculated does not match the signature you provided. Check your key and signing method.
I've tried creating a new IAM user and creating new keys and secret and the same thing

KodeStar picture KodeStar on 28 Jul 2017

Made some progress, set use_path_style to false now at least the login comes up, it now error 500's on index.php/apps/files/ after logging in I'd also updated to 12.0.1 RC3

I've tried the same settings with Nextcloud 11, and it seems to work with that ok.

KodeStar picture KodeStar on 29 Jul 2017

For me, a fresh docker pull from the latest offical Nextcloud docker breaks with 500 error when setting Object storage as primary. I did some experimentation with Linuxserver's builds because they had less problems getting it running, and the release versions were easier for me to parse.

Cmd (v11.0.3 works with S3 backend):
docker run -d --name="nextcl" --net="bridge" -e TZ="America/New_York" -e "PUID"="99" -e "PGID"="100" -p 8086:443/tcp -v "/data":"/data":rw -v "/appdata/nextcloud":"/config":rw linuxserver/nextcloud:59

Cmd (v12 Broken with S3 backend)
docker run -d --name="nextcl" --net="bridge" -e TZ="America/New_York" -e "PUID"="99" -e "PGID"="100" -p 8086:443/tcp -v "/data":"/data":rw -v "/appdata/nextcloud":"/config":rw linuxserver/nextcloud:63

Darkknight33 picture Darkknight33 on 5 Aug 2017

Can confirm that v12 with S3 backend is broken and does not work.

my config.php:

<?php
$CONFIG = array (
  'instanceid' => 'id',
  'passwordsalt' => 'salt',
  'secret' => 'secret',
  'trusted_domains' => 
  array (
    0 => 'drive.example.com',
  ),
  'datadirectory' => '/var/www/drive.example.com/public/data/',
  'overwrite.cli.url' => 'https://drive.example.com',
  'dbtype' => 'mysql',
  'version' => '12.0.2.0',
  'dbname' => 'nextcloud',
  'dbhost' => '10.138.0.4:3306',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'password',
  'installed' => true,
  'objectstore' => array(
      'class' => 'OC\\Files\\ObjectStore\\S3',
      'arguments' => array(
          'bucket' => 'bucketname',
          'key' => 'key',
          'secret' => 'secret',
          'hostname' => 's3-us-west-2.amazonaws.com',
          'use_ssl' => true,
      ),
  ),
);

When I use the above config I get a nextcloud 500 error screen. If I comment out the objectstore section everything works fine.

Here is the log I got:

OCP\Files\NotFoundException: 
0.  /var/www/drive.example.com/public/apps/files/lib/Controller/ViewController.php - line 131: OC_Helper getStorageInfo('/', false)
1.  /var/www/drive.example.com/public/apps/files/lib/Controller/ViewController.php - line 160: OCA\Files\Controller\ViewController->getStorageInfo()
2.  [internal function] OCA\Files\Controller\ViewController->index('', '', NULL, false)
3.  /var/www/drive.example.com/public/lib/private/AppFramework/Http/Dispatcher.php - line 160: call_user_func_array(Array, Array)
4.  /var/www/drive.example.com/public/lib/private/AppFramework/Http/Dispatcher.php - line 90: OC\AppFramework\Http\Dispatcher->executeController(Object(OCA\Files\Controller\ViewController), 'index')
5.  /var/www/drive.example.com/public/lib/private/AppFramework/App.php - line 114: OC\AppFramework\Http\Dispatcher->dispatch(Object(OCA\Files\Controller\ViewController), 'index')
6.  /var/www/drive.example.com/public/lib/private/AppFramework/Routing/RouteActionHandler.php - line 47: OC\AppFramework\App main('ViewController', 'index', Object(OC\AppFramework\DependencyInjection\DIContainer), Array)
7.  [internal function] OC\AppFramework\Routing\RouteActionHandler->__invoke(Array)
8.  /var/www/drive.example.com/public/lib/private/Route/Router.php - line 299: call_user_func(Object(OC\AppFramework\Routing\RouteActionHandler), Array)
9.  /var/www/drive.example.com/public/lib/base.php - line 1004: OC\Route\Router->match('/apps/files/')
10.  /var/www/drive.example.com/public/index.php - line 48: OC handleRequest()
11.  {main}

If I use the exact same S3 info using the external files plugin it mounts it just fine, so I know that my credentials and everything works correctly.

KroniK907 picture KroniK907 on 29 Aug 2017

Update! I got S3 working as primary storage on nextcloud v12.

Currently it seems that the only way to make this work is to re-install nextcloud with the 'objectstore' parameters in config.php.

First I had to make sure that the objectstore was properly added to config/config.php and then In the same file I set 'installed' => true, to false and then reloaded my nextcloud home page.

Then I had to go create a NEW database on my DB server and give nextcloud the credentials for the new database. The old one will mess things up and I tried several ways of dealing with it but was unsuccessful. So I would suggest just making a new database.

Then I re-ran the installation process and everything came up correctly and I could see the proper files in my S3 bucket.

Basically the solution is that you have to know how to set up an object store as your primary storage before you ever install nextcloud and have your object store settings in config.php before you ever run the first installation.

Overall this is NOT clear on the primary storage page of the documentation.

That page even says:

Configuring a primary object store on an existing Nextcloud instance will make all existing files on the instance inaccessible.

Which implies that you should be able to just add the code shown below and it should work, but leave your old files inaccessible. This does not seem to be the case. It seems you must install completely fresh if you want to use an object store as your primary storage.

KroniK907 picture KroniK907 on 29 Aug 2017
👍5 🎉4

The solution that @KroniK907 provided didn't work for me. Either the autocreate option for S3 is broken in v12.0.3.3 or the IAM user that I created specifically for this has not sufficient privileges. I can't really tell. I attached the AmazonS3FullAccess policy to said user. Not sure if this is sufficient or not. I simply kept getting errors that the bucket doesn't exist.

Which is why I simply created the bucket in the S3 console upfront and removed the autocreate flag. This worked for me and it seems to have switched to the bucket as PrimaryStorage.

codemusings picture codemusings on 28 Sep 2017
👍1

A quick guide how to install Nextcloud with object storage as primary storage:

  • extract the tar file
  • create a file storage.config.php (part before .config.php does not matter. NC will read the config.php first - if it exists - and then read and merge all *.config.php files in order of the file name into it. That means that the entries of the last file will overwrite previous ones)
<?php

$CONFIG = [
    'objectstore_multibucket' => [
        'class' => 'OC\\Files\\ObjectStore\\S3',
        'arguments' => [
            'bucket' => 'abc',
            'num_buckets' => 64,
            'key' => '123',
            'secret' => 'abc',
            'hostname' => 'localhost',
            'port' => '4567',
            'use_ssl' => false,
            'use_path_style' => true,
        ],
    ],
];
  • install Nextcloud via the web or CLI based installer: i.e. occ maintenance:install --admin-user=admin --admin-pass=admin --database=mysql --database-name=nextcloud --database-user=nextcloud --database-pass=password
  • that instance now uses the object storage as primary storage without a broken file cache or zombie files in the data dir
MorrisJobke picture MorrisJobke on 28 Sep 2017
🎉5 👍1

I followed the steps that @KroniK907 listed and was able to get a DigitalOcean Space used as my primary storage. That was quite a pain, though luckily I was already working with a blank slate.

I am not sure if this is working as intended, but since moving to objectstorage, all the files on my DigitalOcean Space are "urn:oid:##" instead of their real file names. I am pretty sure I have some zombie files in there already but am unable to identify them.

jknigga picture jknigga on 26 Oct 2017

Hi Morris,

Unfortunately, there is still a bug in the latest 12.0.3.3 release, like @jmdeboer-surfsara mentioned on July 25th.
When the installation of the environment is already done with local storage and later on you change the primary storage from normale local POSIX filesystem to an external Swift or S3 storage by changing the config file, you will get an "Internal Server" error.

{"reqId":"<my-reqid>","level":3,"time":"03\/Nov\/2017:12:31:26","remoteAddr":"<my-ip>","user":"<my-user>","app":"index","method":"GET","url":"\/index.php\/apps\/files\/","message":"Exception: {\"Exception\":\"OCP\\\\Files\\\\NotFoundException\",\"Message\":\"\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/stash\\\/apps\\\/files\\\/lib\\\/Controller\\\/ViewController.php(131): OC_Helper::getStorageInfo('\\\/', false)\\n#1 \\\/var\\\/www\\\/stash\\\/apps\\\/files\\\/lib\\\/Controller\\\/ViewController.php(160): OCA\\\\Files\\\\Controller\\\\ViewController->getStorageInfo()\\n#2 [internal function]: OCA\\\\Files\\\\Controller\\\\ViewController->index('', '', NULL, false)\\n#3 \\\/var\\\/www\\\/stash\\\/lib\\\/private\\\/AppFramework\\\/Http\\\/Dispatcher.php(160): call_user_func_array(Array, Array)\\n#4 \\\/var\\\/www\\\/stash\\\/lib\\\/private\\\/AppFramework\\\/Http\\\/Dispatcher.php(90): OC\\\\AppFramework\\\\Http\\\\Dispatcher->executeController(Object(OCA\\\\Files\\\\Controller\\\\ViewController), 'index')\\n#5 \\\/var\\\/www\\\/stash\\\/lib\\\/private\\\/AppFramework\\\/App.php(114): OC\\\\AppFramework\\\\Http\\\\Dispatcher->dispatch(Object(OCA\\\\Files\\\\Controller\\\\ViewController), 'index')\\n#6 \\\/var\\\/www\\\/stash\\\/lib\\\/private\\\/AppFramework\\\/Routing\\\/RouteActionHandler.php(47): OC\\\\AppFramework\\\\App::main('ViewController', 'index', Object(OC\\\\AppFramework\\\\DependencyInjection\\\\DIContainer), Array)\\n#7 [internal function]: OC\\\\AppFramework\\\\Routing\\\\RouteActionHandler->__invoke(Array)\\n#8 \\\/var\\\/www\\\/stash\\\/lib\\\/private\\\/Route\\\/Router.php(299): call_user_func(Object(OC\\\\AppFramework\\\\Routing\\\\RouteActionHandler), Array)\\n#9 \\\/var\\\/www\\\/stash\\\/lib\\\/base.php(1004): OC\\\\Route\\\\Router->match('\\\/apps\\\/files\\\/')\\n#10 \\\/var\\\/www\\\/stash\\\/index.php(48): OC::handleRequest()\\n#11 {main}\",\"File\":\"\\\/var\\\/www\\\/stash\\\/lib\\\/private\\\/legacy\\\/helper.php\",\"Line\":543}","userAgent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10.12; rv:56.0) Gecko\/20100101 Firefox\/56.0","version":"12.0.3.3"}

The bucket is created in the background, but unfortunately the storage link in the database is not updated. Should it be possible to check of this record is up-to-date, so you will be able to migrate later on from local storage to an alternative storage platform?

Thanks!

T0mWz picture T0mWz on 3 Nov 2017

When the installation of the environment is already done with local storage and later on you change the primary storage from normale local POSIX filesystem to an external Swift or S3 storage by changing the config file, you will get an "Internal Server" error.

This is not supported. You either install with a POSIX filesystem from the beginning or with S3/Swift from the beginning. Switching after using one is not supported.

MorrisJobke picture MorrisJobke on 3 Nov 2017

If that is so, then why does the note in the documentation talk about switching the storage and that it makes the files inaccessible (not that it breaks the whole installation)?

Note
Configuring a primary object store on an existing Nextcloud instance will make all existing files on the instance inaccessible.

CherryDT picture CherryDT on 4 Dec 2017
👍7

I cannot get S3 support no matter whose example code I try, even with wiping the database and reinstalling clean. My creds have been tested with other apps, so it is definitely an issue with the S3 primaryStorage.

Does anyone have working, tested config+steps that I can duplicate for a running setup?

spacemonkey picture spacemonkey on 30 Dec 2017
👍1

I was able to get this to work after following the instructions that @MorrisJobke wrote. I did not re-extract the archive. I did the following:

1. dropped the nextcloud database (I am using MySQL. This may be excessive behavior but it's a new install for me so I didn't care if I lost the DB)
# mysql -uroot -p
mysql> drop database nextcloud;
mysql> quit

2. created the alternate config file for objectstore (Region must be set correctly)

File config/objectstore.config.php:

<?php
$CONFIG = array (
'objectstore' => array(
'class' => 'OC\\Files\\ObjectStore\\S3',
'arguments' => array(
'bucket' => 'bucketname',
'autocreate' => true,
'key' => 'keystring',
'secret' => 'secretstring',
'use_ssl' => true,
'region' => 'us-east-1'
),
),
);

3. ran the occ maintenance:install command.
sudo -u www-data php occ maintenance:install --database "mysql" --database-name "nextcloud" --database-user "root" --database-pass "dbpassword" --admin-user "admin" --admin-pass "newpass"

martinsah picture martinsah on 12 Jan 2018
👍1

Hi,

i tried to make Amazon S3 as my primary storage as well following the link
keep in mind i have NextCloud 12.0.4.3, is it also broken for this version?

this is what i have added to the config.php

 'objectstore' =>
  array(
    'class' => 'OC\\Files\\ObjectStore\\S3',
    'arguments' => array(
            'bucket' => 'nextcloud',
        'autocreate' => true,
        'key'    => 'XXXXXX',
        'secret' => 'xxxxxxxxxxxxxxxxx',
        'hostname' => 's3-us-east-1.amazonaws.com',
        'use_ssl' => false,
        'region' => 'us-east-1',
      ),
  ),

any idea?

ibr123 picture ibr123 on 15 Jan 2018

@ibr123 Did you do this on a fresh install? You cant switch from a standard POSIX setup to an S3 setup. You must re-install nextcloud using one of the methods mentioned in the comments above to get the S3 setup working. Please read through the comments in this thread for a few different methods of doing that.

KroniK907 picture KroniK907 on 16 Jan 2018

@KroniK907 actually the first time I tried it, the part of configuring (shared in my previous comment) wasn't in the config file, maybe I didn't save or when I restarted apache it wasn't saved, but I tired it again on a fresh install, and it worked fine, for knowing how to configure the S3 the following link is useful
Thanks

ibr123 picture ibr123 on 17 Jan 2018

I've been tweaking a nextcloudPI installation at the beginning but done the php installer fresh install as well. Different config options tried.
RPI3 Debian + S3 CEPH Rados, so maybe this has any relevance. Thank you.
nextcloud.log

Found this: https://github.com/nextcloud/server/pull/7570

maxgrass picture maxgrass on 17 Jan 2018

With 13+ we heavily improved the functionality of object storage as primary storage. Please test again with this version and report back if you have problems. We have multiple big instances that work fine with S3 and Swift on 13 or 14. Thanks for the feedback anyway.

MorrisJobke picture MorrisJobke on 29 Nov 2018

Just because we battled with this for some time and found the solution in this thread:

With version 17 and working owith an already functioning site using local storage we switched to Swift Object Storage as primary storage with no problem using MorrisJoke's seperate config.php suggestion above.

No database commands needed; it just worked. We did go in and out of maintenance mode but not sure even that is necessary.

$CONFIG = array (
'objectstore' => array(
'class' => '\OC\Files\ObjectStore\Swift',
'arguments' => array(
'username' => 'xxx',
'password' => 'xxx',
'bucket' => 'xxx',
'autocreate' => true,
'region' => 'xxx',
'url' => 'xxx',
'tenantName' => 'xxx',
'serviceName' => 'swift',
),
),
);

Warning - all existing files and folders are deleted from NextCloud however they remain in the local storage folder.

Files in the (fresh) bucket are renamed and look like urn:oid:xxx

klaatuic picture klaatuic on 6 Nov 2019

maintenance:install

Did not work for me I am on Nextcloud 18.0.3 and seems it still has this same issue
I did a new installation (didn't know we have to create an additional file for bucket) and landed on this page followed everything but seems maintenance: install is no longer available

cinghaman picture cinghaman on 1 Apr 2020

Bumped into the same issue (18.0.3). Nothing from the workarounds above helped me to get it working with S3 whether on fresh install or old if using k8s. Looks like the bug still persists and keeps traveling from one version to another without any improvements...

gondaz picture gondaz on 14 Apr 2020

Bumped into the same issue (18.0.3). Nothing from the workarounds above helped me to get it working with S3 whether on fresh install or old if using k8s. Looks like the bug still persists and keeps traveling from one version to another without any improvements...

Hi I finally got it working
1: Install a fresh copy
2: Once installed logged into the admin dashboard > apps > enable external storage
3: Go back to your command line
4: Edit the config.php file change install to false
5: create a storage.config.php file add objectstorage details (bucket/api keys)
6: Run Maintenance occ command

PS: Make sure the bucket name you are adding is not already existing
This now works for me but it created another problem When I delete anything in nextcloud it does not get deleted in the bucket so it keeps on storing everything even if i permanently remove it.

cinghaman picture cinghaman on 14 Apr 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

juliushaertl picture juliushaertl  ·  3Comments
williambargent picture williambargent  ·  3Comments
rullzer picture rullzer  ·  3Comments
j-ed picture j-ed  ·  3Comments
brylie picture brylie  ·  3Comments