Server: OpenStack Swift external storage broke after upgrade to 14.0.0

[mlanner]

Steps to reproduce

1. Upgrade from 13.0.0 to 14.0.0
2. Try to load external storage folders with Swift as backend
   3.

Expected behaviour

Should be able to access files

Actual behaviour

Complains that the setup is incorrect. This worked fine in 13.0.0. It appears that the new OpenStack v3 integration has broken previous v2 integration.

Server configuration

Operating system: Ubuntu 16.04.5

Web server: Apache2

Database: MySQL 5.7

PHP version: 7.0

Nextcloud version: (see Nextcloud admin page)

Updated from an older Nextcloud/ownCloud or fresh install: Upgrade from NC

Where did you install Nextcloud from: Zip

Signing status:

Signing status

Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

List of activated apps:

App list

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your Nextcloud installation folder

Nextcloud configuration:

Config report

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your Nextcloud installation folder

or 

Insert your config.php content here. 
Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …)

Are you using external storage, if yes which one: Swift

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

LDAP configuration (delete this part if not used)

LDAP config

With access to your command line run e.g.:
sudo -u www-data php occ ldap:show-config
from within your Nextcloud installation folder

Without access to your command line download the data/owncloud.db to your local
computer or access your SQL server remotely and run the select query:
SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap';


Eventually replace sensitive data as the name/IP-address of your LDAP server or groups.

Client configuration

Browser:

Operating system:

Logs

Web server error log

Web server error log

Insert your webserver log here

Nextcloud log (data/nextcloud.log)

Nextcloud log

Insert your Nextcloud log here

Browser log

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

[mlanner]

Also, when clicking on the folder (in pink) the following message appears:

There was an error with message: Empty response from the server. Do you want to review mount point config in personal settings page?

If one clicks on the "OK" button, it simply sends me back to the folder list and not to the "mount point config in personal settings page" as expected.

I have also confirmed that the Nextcloud server can reach the Swift cluster just fine.

[nextcloud-bot]

GitMate.io thinks possibly related issues are https://github.com/nextcloud/server/issues/4730 (Storage), https://github.com/nextcloud/server/issues/3789 (External storage), https://github.com/nextcloud/server/issues/11225 (PHP Fatal error after upgrade to 14.0.0), https://github.com/nextcloud/server/issues/8041 ([primary storage on Openstack swift] Update to 12.0.5 / 13RC breaks Nextcloud), and https://github.com/nextcloud/server/issues/1282 ( Authentication problem after upgrade to 10.0.0).

[mlanner]

None of the above bot-suggested tickets appear to be relevant.

[teleassist]

Same issue for me, i've got this message from command line updater :

`Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Set log level to debug
Updating database schema
Updated database
Checking for update of app activity in appstore
An unhandled exception has been thrown:
Error: Call to undefined method GuzzleHttpClient::request() in /var/www/nextcloud/3rdparty/php-opencloud/openstack/src/Common/Api/OperatorTrait.php:118
Stack trace:

0 /var/www/nextcloud/3rdparty/php-opencloud/openstack/src/Common/Api/OperatorTrait.php(126): OpenStackCommonServiceAbstractService->sendRequest(Object(OpenStackCommonApiOperation), Array)

1 /var/www/nextcloud/3rdparty/php-opencloud/openstack/src/Identity/v2/Service.php(52): OpenStackCommonServiceAbstractService->execute(Array, Array)

2 /var/www/nextcloud/lib/private/Files/ObjectStore/SwiftFactory.php(147): OpenStackIdentityv2Service->generateToken(Array)

3 /var/www/nextcloud/lib/private/Files/ObjectStore/SwiftFactory.php(117): OCFilesObjectStoreSwiftFactory->auth(Object(OpenStackIdentityv2Service), '8WcCwwSpbKpm@ht...')

4 /var/www/nextcloud/lib/private/Files/ObjectStore/SwiftFactory.php(189): OCFilesObjectStoreSwiftFactory->getClient()

5 /var/www/nextcloud/lib/private/Files/ObjectStore/SwiftFactory.php(177): OCFilesObjectStoreSwiftFactory->createContainer()

6 /var/www/nextcloud/lib/private/Files/ObjectStore/Swift.php(57): OCFilesObjectStoreSwiftFactory->getContainer()

7 /var/www/nextcloud/lib/private/Files/ObjectStore/Swift.php(89): OCFilesObjectStoreSwift->getContainer()

8 /var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php(276): OCFilesObjectStoreSwift->readObject('urn:oid:336')

9 /var/www/nextcloud/lib/private/Files/Storage/Common.php(191): OCFilesObjectStoreObjectStoreStorage->fopen('appdata_oc0lh55...', 'r')

10 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(244): OCFilesStorageCommon->file_get_contents('appdata_oc0lh55...')

11 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php(266): OCFilesStorageWrapperWrapper->file_get_contents('appdata_oc0lh55...')

12 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(244): OCFilesStorageWrapperAvailability->file_get_contents('appdata_oc0lh55...')

13 /var/www/nextcloud/lib/private/Files/View.php(1146): OCFilesStorageWrapperWrapper->file_get_contents('appdata_oc0lh55...')

14 /var/www/nextcloud/lib/private/Files/View.php(588): OCFilesView->basicOperation('file_get_conten...', '/appdata_oc0lh5...', Array)

15 /var/www/nextcloud/lib/private/Files/Node/File.php(52): OCFilesView->file_get_contents('/appdata_oc0lh5...')

16 /var/www/nextcloud/lib/private/Files/SimpleFS/SimpleFile.php(88): OCFilesNodeFile->getContent()

17 /var/www/nextcloud/lib/private/App/AppStore/Fetcher/Fetcher.php(146): OCFilesSimpleFSSimpleFile->getContent()

18 /var/www/nextcloud/lib/private/Installer.php(385): OCAppAppStoreFetcherFetcher->get()

19 /var/www/nextcloud/lib/private/Updater.php(445): OCInstaller->isUpdateAvailable('activity')

20 /var/www/nextcloud/lib/private/Updater.php(256): OCUpdater->upgradeAppStoreApps(Array)

21 /var/www/nextcloud/lib/private/Updater.php(126): OCUpdater->doUpgrade('14.0.3.0', '13.0.6.1')

22 /var/www/nextcloud/core/Command/Upgrade.php(262): OCUpdater->upgrade()

23 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(251): OCCoreCommandUpgrade->execute(Object(SymfonyComponentConsoleInputArgvInput), Object(SymfonyComponentConsoleOutputConsoleOutput))

24 /var/www/nextcloud/3rdparty/symfony/console/Application.php(946): SymfonyComponentConsoleCommandCommand->run(Object(SymfonyComponentConsoleInputArgvInput), Object(SymfonyComponentConsoleOutputConsoleOutput))

25 /var/www/nextcloud/3rdparty/symfony/console/Application.php(248): SymfonyComponentConsoleApplication->doRunCommand(Object(OCCoreCommandUpgrade), Object(SymfonyComponentConsoleInputArgvInput), Object(SymfonyComponentConsoleOutputConsoleOutput))

26 /var/www/nextcloud/3rdparty/symfony/console/Application.php(148): SymfonyComponentConsoleApplication->doRun(Object(SymfonyComponentConsoleInputArgvInput), Object(SymfonyComponentConsoleOutputConsoleOutput))

27 /var/www/nextcloud/lib/private/Console/Application.php(213): SymfonyComponentConsoleApplication->run(Object(SymfonyComponentConsoleInputArgvInput), Object(SymfonyComponentConsoleOutputConsoleOutput))

28 /var/www/nextcloud/console.php(96): OCConsoleApplication->run()

29 /var/www/nextcloud/occ(11): require_once('/var/www/nextcl...')

`

[teleassist]

Any news for this ?

[felix-walter]

I also encountered this issue today and got the following log message:

{"reqId":"<redacted>","level":1,"time":"November 21, 2018 21:38:20","remoteAddr":"<redacted>","user":"felix","app":"no app in context","method":"PUT","url":"\/apps\/files_external\/globalstorages\/1","message":"External storage not available: Storage unauthorized. Scope has to be defined for V3 requests","userAgent":"<redacted>","version":"14.0.3.0"}

Searching the last part of the message I found the following commit which is in v14, but not in v13: https://github.com/nextcloud/server/pull/9280/commits/ba71918ad2f6fb336852fd492e5e819eb38201c9

Thus, the issue seems that we cannot set the required scope in the external storage configuration UI. Strangely, it also does not work when using OpenStack v2 authentication.

[pecuna]

NC 14.0.6 still can not integrate with Openstack Object storage (Mitaka + Ceph Jewel)
Nothing is showing in logs (NC and Keystone). Identity endpoint is v3.

[vpecinka]

Hi @teleassist

this is maybe off topic, but you can run into it later: I noticed you use urn:oid: object prefix. Similar to my case (I have id: prefix for objects). I have other problem and found that there is problem in Guzzlehttp. Maybe this patch (tuned for you) will help you, please try and give me feedback. The problem I found is that the library handles wrongly : in relative urls.

--- nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php 2019-01-01 03:35:39.796954000 +0100
+++ nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php 2019-01-01 03:35:39.796954000 +0100
@@ -142,6 +142,9 @@

     private function buildUri($uri, array $config)
     {
+         // fix by VP
+         $uri = str_replace("/urn:oid:", "/urn%3Aoid%3A", $uri);
+
         // for BC we accept null which would otherwise fail in uri_for
         $uri = Psr7\uri_for($uri === null ? '' : $uri);

[skydiablo]

NC 15, same here!

I also encountered this issue today and got the following log message:

{"reqId":"<redacted>","level":1,"time":"November 21, 2018 21:38:20","remoteAddr":"<redacted>","user":"felix","app":"no app in context","method":"PUT","url":"\/apps\/files_external\/globalstorages\/1","message":"External storage not available: Storage unauthorized. Scope has to be defined for V3 requests","userAgent":"<redacted>","version":"14.0.3.0"}

Searching the last part of the message I found the following commit which is in v14, but not in v13: ba71918

Thus, the issue seems that we cannot set the required scope in the external storage configuration UI. Strangely, it also does not work when using OpenStack v2 authentication.

[kesselb]

Yes. There is no option to set the scope.

https://github.com/nextcloud/server/blob/2e36069e24406455ad3f3998aa25e2a949d1402a/apps/files_external/lib/Lib/Auth/OpenStack/OpenStackV3.php#L44

If you change the file like below it should be possible to define a scope. Unfortunately I don't have a working test setup for OpenStack :disappointed:

new DefinitionParameter('url', $l->t('Identity endpoint URL')),
new DefinitionParameter('scope', $l->t('Scope'))

[kesselb]

Ref https://github.com/nextcloud/server/pull/9280 https://github.com/nextcloud/server/issues/9286

[kesselb]

Yes. There is no option to set the scope.

server/apps/files_external/lib/Lib/Auth/OpenStack/OpenStackV3.php

Line 44 in 2e36069
new DefinitionParameter('url', $l->t('Identity endpoint URL'))

If you change the file like below it should be possible to define a scope. Unfortunately I don't have a working test setup for OpenStack disappointed

new DefinitionParameter('url', $l->t('Identity endpoint URL')),
new DefinitionParameter('scope', $l->t('Scope'))

I think this is not gonna work :rofl: (https://developer.openstack.org/api-ref/identity/v3/index.html#password-authentication-with-unscoped-authorization looks like one input is not enough)

[mlanner]

@danielkesselberg What do you need to test against? I can definitely get you a Swift endpoint to test against if that's what you need.

[kesselb]

Thank you! I wrote something on the pull request and hope that either @rullzer or @icewind1991 could pick this up.

[skydiablo]

this is an working setup for my current test-env (NC 15):

  'objectstore' => array(
        'class' => 'OC\\Files\\ObjectStore\\Swift',
        'arguments' => array(
                'user' => array(
                      'name' => 'XXXXXXXXXX',
                      'password' => 'XXXXXXXXXXXXXXXXXXXXX',
                      'domain'   => [
                          'name' => 'Default'
                      ],
                ),
                'scope' => [
                        'project' => [
                                'name' => 'XXXXXXXXXXXX',
                                'domain' => [
                                        'name' => 'Default',
                                ],
                        ],
                ],
                'bucket' => 'nextcloud',
                'region' => 'RegionOne',
                'url' => 'http://XXX.XXX.XXX.XXX:5000/v3',
        ),
  ),

[kesselb]

this is an working setup for my current test-env (NC 15):

If you want to use Swift as objectstorage. For external storage it's not possible to set a scope (right now).

[alejandrocq]

Hi! Any update on this? I had to comment these lines to be able to configure an openstack v3 object storage (provided by OVH) as external storage:

https://github.com/nextcloud/server/blob/master/lib/private/Files/ObjectStore/SwiftFactory.php#L126-L129

I think this restriction should be removed for the moment until we can configure scope, or try to find a quick solution.

[asve23fq]

Needed to comment lines suggested by @alejandrocq in 17.0.1 for OVH object storage to work again.

[asve23fq]

After updating to 17.0.2 you still need to comment lines 126-129 for OVH object storage to work, but on the page posted by @alejandrocq I see that wrong lines are getting highlighted. The file on github has 259 lines, but my local SwiftFactory.php file has only 251 lines. I'm not sure why, but the bottom line is that these lines need to be commented:

126 # if (!isset($this->params['scope'])) {
127 # throw new StorageAuthException('Scope has to be defined for V3 requests');
128 # }
129 #

[xThrodx]

OVH will drop the support for the v2 API on March 24, 2020, as explained there.

It looks like it will be necessary to use the v3 API and manually modify the SwiftFactory.php to continue using Openstack from OVH as a backend.

[mirtouf]

Thanks for the quick & dirty fix for OVH !
I am following this issue to see the outcomes.

[frank-dspeed]

this needs asap documentation so swift ovh users are aware of this result.

[kesselb]

cc @icewind1991 @rullzer :ping_pong:

[Jbertan42]

I can confirm that after commenting the scope line in lib/private/Files/ObjectStore/SwiftFactory.php
it works with v3 openstack!

[remidebette]

Indeed it works with OVH hosted swift, as stated in https://github.com/nextcloud/server/issues/5882#issuecomment-586758620

Can this be fixed on the nextcloud side so that we do not have to temper with the code?
Thank you

[frank-dspeed]

@remidebette as your the one who verifyed its working can you make a conclusion did you comment out the
https://github.com/nextcloud/server/blob/master/lib/private/Files/ObjectStore/SwiftFactory.php#L134-L136

# if (!isset($this->params['scope'])) {
# throw new StorageAuthException('Scope has to be defined for V3 requests');
# }
#

?
is it still needed? or do you simply supply a fake scope?

[dinacel]

@remidebette as your the one who verifyed its working can you make a conclusion did you comment out the
https://github.com/nextcloud/server/blob/master/lib/private/Files/ObjectStore/SwiftFactory.php#L134-L136

# if (!isset($this->params['scope'])) {
# throw new StorageAuthException('Scope has to be defined for V3 requests');
# }
#

?
is it still needed? or do you simply supply a fake scope?

I can confirm it is still required to comment out these lines because there's no way to set the scope (at least in the external storage GUI).

[zizzfizzix]

is it still needed? or do you simply supply a fake scope?

I'm using swift as primary storage and setting scope made it work, no need to edit any nextcloud code.

[fxeon]

is it still needed? or do you simply supply a fake scope?

I'm using swift as primary storage and setting scope made it work, no need to edit any nextcloud code.

Hi, can you please provide your config.php, i'm stuck with V2 and can't get V3 to work. Thanks

Edit: I think I got it

  ' objectstore' => array(
         'class' => 'OC\\Files\\ObjectStore\\Swift',
         'arguments' => array(
                 'autocreate' => true,
                 'user' => [
                         'name' => 'MyUserName',
                         'password' => 'SupaStrongPasswd',
                         'domain' => [
                                 'name' => 'Default'
                         ]
                 ],
                 'scope' => [
                         'project' => [
                                 'name' => 'OS_PROJECT_NAME',
                                 'domain' => [
                                         'name' => 'Default',
                                 ],
                         ],
                 ],
                 'serviceName' => 'swift',
                 'region' => 'GRA',
                 'url' => "https://auth.cloud.ovh.net/v3",
                 'bucket' => 'nextcloud'
         )
   ),

I had to clean files in CLI to get it working
sudo -u www-data php occ files:cleanup

[alviss7]

Thanks a lot, I managed to make it work with Keystone v3, the mail documentation is confusing, I plan to fix it today as soon as I have time:
https://docs.nextcloud.com/server/18/admin_manual/configuration_files/primary_storage.html#openstack-swift.

[zizzfizzix]

Edit: I think I got it

  ' objectstore' => array(
         'class' => 'OC\\Files\\ObjectStore\\Swift',
         'arguments' => array(
                 'autocreate' => true,
                 'user' => [
                         'name' => 'MyUserName',
                         'password' => 'SupaStrongPasswd',
                         'domain' => [
                                 'name' => 'Default'
                         ]
                 ],
                 'scope' => [
                         'project' => [
                                 'name' => 'OS_PROJECT_NAME',
                                 'domain' => [
                                         'name' => 'Default',
                                 ],
                         ],
                 ],
                 'serviceName' => 'swift',
                 'region' => 'GRA',
                 'url' => "https://auth.cloud.ovh.net/v3",
                 'bucket' => 'nextcloud'
         )
   ),

Yeah, that's it.

I had to clean files in CLI to get it working
sudo -u www-data php occ files:cleanup

Mine was throwing an error when I left it (late at night), but started working apparently a couple minutes later (as I checked the next day).

[fxeon]

The cleanup part might indeed not be needed. It happened that I could gain access to my nextcloud page right after I did it. Beforehand it kept throwing me "Storage temporarily not available". As it was late as well I didn't investigate any further.

[humpfhumpf]

Please, do not forget that External Storage to OVH is also impacted (the ones configured in database in oc_external_mounts table), and not only Primary Storage.