Server: [primary storage on Openstack swift] Update to 12.0.5 / 13RC breaks Nextcloud

[harryhb]

Issue logged after discussion on support: https://help.nextcloud.com/t/broken-after-update-to-12-0-5/26300

Steps to reproduce

1. Upgrade from 12.0.4 tot 12.0.5 / 13RC through admin interface
   2.
   3.

Expected behaviour

Upgrade script runs; through web interface the upgrade can be finalized and system is back up and running.

Actual behaviour

• The "button" on the web interface cannot be activated => Finish upgrade through CLI
• The site login screen can be reached, but login fails (buttons cannot be activated)

Server configuration

Operating system: Debian stable

Web server: Apache/2.4.25 (Debian)

Database: mariadb Ver 15.1 Distrib 10.1.26-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

PHP version: PHP 7.0.27-0+deb9u1

Nextcloud version: 12.0.5 / 13RC

Updated from an older Nextcloud/ownCloud or fresh install: Updated from 12.0.4

Where did you install Nextcloud from: Through admin interface

Signing status: Not possible to login

List of activated apps:

App list

Enabled:

• activity: 2.5.2
• admin_audit: 1.2.0
• bruteforcesettings: 1.0.3
• calendar: 1.5.7
• caniupdate: 0.1.1
• comments: 1.2.0
• contacts: 2.0.1
• dav: 1.3.1
• federatedfilesharing: 1.2.0
• federation: 1.2.0
• files: 1.7.2
• files_downloadactivity: 1.1.1
• files_pdfviewer: 1.1.1
• files_sharing: 1.4.0
• files_texteditor: 2.4.1
• files_trashbin: 1.2.0
• files_versions: 1.5.0
• files_videoplayer: 1.1.0
• firstrunwizard: 2.1
• gallery: 17.0.0
• logreader: 2.0.0
• lookup_server_connector: 1.0.0
• mail: 0.7.8
• metadata: 0.6.0
• nextcloud_announcements: 1.1
• notifications: 2.0.0
• oauth2: 1.0.5
• ocsms: 1.12.2
• passman: 2.1.4
• password_policy: 1.2.2
• provisioning_api: 1.2.0
• qownnotesapi: 17.5.0
• serverinfo: 1.2.0
• sharebymail: 1.2.0
• spreed: 2.0.2
• survey_client: 1.0.0
• systemtags: 1.2.0
• tasks: 0.9.5
• twofactor_backupcodes: 1.1.1
• updatenotification: 1.2.0
• workflowengine: 1.2.0
  

Nextcloud configuration:

Config report

{
"system": {
"objectstore": {
"class": "OC\Files\ObjectStore\Swift",
"arguments": {
"username": "xx",
"password": "REMOVED SENSITIVE VALUE",
"bucket": "nextcloud_swift",
"autocreate": true,
"region": "xx",
"url": "https:\/\/identity.stack.cloudvps.com\/v2.0",
"tenantName": "xx",
"serviceName": "swift"
}
},
"passwordsalt": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"trusted_domains": [
"DOMAIN"
],
"datadirectory": "xx",
"overwrite.cli.url": "xx",
"dbtype": "mysql",
"version": "xx",
"dbname": "xx",
"dbhost": "localhost",
"dbport": "",
"dbtableprefix": "oc_",
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"installed": true,
"forcessl": true,
"logtimezone": "UTC",
"trashbin_retention_obligation": "auto",
"mail_from_address": "info",
"mail_smtpmode": "smtp",
"mail_domain": "xx",
"mail_smtpsecure": "tls",
"mail_smtpauthtype": "xx",
"mail_smtpauth": 1,
"mail_smtphost": "xx",
"mail_smtpport": "xx",
"mail_smtpname": "REMOVED SENSITIVE VALUE",
"mail_smtppassword": "REMOVED SENSITIVE VALUE",
"app.mail.smtp.timeout": 10,
"memcache.local": "\OC\Memcache\Redis",
"filelocking.enabled": "true",
"memcache.distributed": "\OC\Memcache\Redis",
"memcache.locking": "\OC\Memcache\Redis",
"redis": {
"host": "localhost",
"port": 6379,
"timeout": 0,
"dbindex": 0
},
"appstore.experimental.enabled": true,
"maintenance": false,
"theme": "",
"loglevel": 0,
"updater.secret": "REMOVED SENSITIVE VALUE",
"instanceid": "xx"
}
}

Are you using external storage, if yes which one: Primary storage on Openstack/Swift

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Firefox & Chromium

Operating system: Debian testing

Logs

Web server error log

Web server error log

No errors

Nextcloud log (data/nextcloud.log)

Nextcloud log

Browser log

Browser log

ReferenceError: OC is not defined

[Arvi89]

Any news? This is kind of a big deal.

[harryhb]

I reverted back to 12.0.4 as this bug is a deal breaker for nextcloud usage.

[Arvi89]

I find it amazing to not have an answer from anyone (even just to ask for more information). It's a serious issue, and it's kind of ignored. A minor update just completely broke the nextcloud installation, but hey, no big deal, this affects only few people, no worry.

[mduller]

I am actually holding off performing the upgrade because of this issue that "affects only few people" -- I currently don't have the luxury of spending time on rolling back if something goes wrong (I am in charge of 4 Nextcloud systems). Would like to see some insights in this ticket to inspire confidence to perform the update. That confidence is currently missing.

[Schmuuu]

I find it amazing to not have an answer from anyone (even just to ask for more information). It's a serious issue, and it's kind of ignored. A minor update just completely broke the nextcloud installation, but hey, no big deal, this affects only few people, no worry.

Hi Arvi,

I totally understand your frustration about your broken NC server due to the minor update to 12.0.5.
Let's put some focus on this issue.

@ChristophWurst @MorrisJobke @jospoortvliet @rullzer Who can help? Quite a few users broke their installation by updating to 12.0.5
The community couldn't help so far. Some insights of developers is needed.

Please advise.

[hitam4450]

Updating the server instance is always a total mess! Minor or major update is always trouble !
Plan enough time for manual update, that is the only way to (more - or less ) safe to update!

[rullzer]

I can't find the bug I think this is...

Anyway.

Could you try:

1. Remove the css and js folder from your appdata_* folder
2. Run: ./occ files:scan-app-data
3. Do a force refresh in your browser ctrl + F5

@skjnldsv @juliushaertl I think https://github.com/nextcloud/server/pull/8078 would help here as well.

[Arvi89]

mm I don't have any appdata folder, where is it supposed to be?

[rullzer]

@Arvi89 in your data folder. It is data/appdata_<instance id>.

[Arvi89]

Yeah, I don't have that folder.
(I use openstack as my main storage)

[rullzer]

Ah right then your data folder is indeed empty I guess.

I can try to come up with a SQL query... because if you just purge it from the DB on object store it will just create new items :)

[rullzer]

DELETE
FROM oc_filecache
WHERE path LIKE 'appdata%/js/%'
OR path LIKE 'appdata%/css/%';

Try that. Of course make sure you have a backup and all that ;)

[Arvi89]

No luck. The query was executed properly (203 rows affected), but it doesn't work better.

[rullzer]

That is strange... and nothing in your log files?

[Arvi89]

{"reqId":"IvGS68ERn3bybTiy7Ohm","level":3,"time":"2018-01-28T21:31:12+00:00","remoteAddr":"90.91.101.134","user":"arvi89","app":"objectstore","method":"PROPFIND","url":"\/remote.php\/dav\/files\/arvi89\/","message":"Could not delete object urn:oid:30816 for cache\/IMG_20180104_164836.jpg-chunking-1917752040-4: {\"Exception\":\"Guzzle\\\\Http\\\\Exception\\\\ClientErrorResponseException\",\"Message\":\"Client error response\\n[status code] 404\\n[reason phrase] Not Found\\n[url] https:\\\/\\\/storage.gra1.cloud.ovh.net\\\/v1\\\/AUTH_xxxxxxxxxxxxxxxxxxxxxxAUTH_xxxxxxxxxxxxxxxxxxxxxxxx\\\/nextcloud\\\/urn:oid:30816\",\"Code\":0,\"Trace\":\"#0 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Message\\\/Request.php(145): Guzzle\\\\Http\\\\Exception\\\\BadResponseException::factory(Object(Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest), Object(Guzzle\\\\Http\\\\Message\\\\Response))\\n#1 [internal function]: Guzzle\\\\Http\\\\Message\\\\Request::onRequestError(Object(Guzzle\\\\Common\\\\Event), 'request.error', Object(Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher))\\n#2 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/symfony\\\/event-dispatcher\\\/EventDispatcher.php(174): call_user_func(Array, Object(Guzzle\\\\Common\\\\Event), 'request.error', Object(Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher))\\n#3 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/symfony\\\/event-dispatcher\\\/EventDispatcher.php(43): Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher->doDispatch(Array, 'request.error', Object(Guzzle\\\\Common\\\\Event))\\n#4 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Message\\\/Request.php(589): Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher->dispatch('request.error', Object(Guzzle\\\\Common\\\\Event))\\n#5 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Message\\\/Request.php(378): Guzzle\\\\Http\\\\Message\\\\Request->processResponse(Array)\\n#6 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Message\\\/EntityEnclosingRequest.php(49): Guzzle\\\\Http\\\\Message\\\\Request->setState('complete', Array)\\n#7 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(286): Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest->setState('complete', Array)\\n#8 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(244): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->processResponse(Object(Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest), Object(Guzzle\\\\Http\\\\Curl\\\\CurlHandle), Array)\\n#9 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(227): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->processMessages()\\n#10 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(211): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->executeHandles()\\n#11 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(105): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->perform()\\n#12 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMultiProxy.php(91): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->send()\\n#13 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Client.php(282): Guzzle\\\\Http\\\\Curl\\\\CurlMultiProxy->send()\\n#14 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Message\\\/Request.php(198): Guzzle\\\\Http\\\\Client->send(Object(Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest))\\n#15 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/rackspace\\\/php-opencloud\\\/lib\\\/OpenCloud\\\/ObjectStore\\\/Resource\\\/DataObject.php(384): Guzzle\\\\Http\\\\Message\\\\Request->send()\\n#16 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/ObjectStore\\\/Swift.php(278): OpenCloud\\\\ObjectStore\\\\Resource\\\\DataObject->delete()\\n#17 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/ObjectStore\\\/ObjectStoreStorage.php(189): OC\\\\Files\\\\ObjectStore\\\\Swift->deleteObject('urn:oid:30816')\\n#18 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(261): OC\\\\Files\\\\ObjectStore\\\\ObjectStoreStorage->unlink('cache\\\/IMG_20180...')\\n#19 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Availability.php(285): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->unlink('cache\\\/IMG_20180...')\\n#20 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(261): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Availability->unlink('cache\\\/IMG_20180...')\\n#21 [internal function]: OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->unlink('cache\\\/IMG_20180...')\\n#22 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/apps\\\/files_trashbin\\\/lib\\\/Storage.php(231): call_user_func_array(Array, Array)\\n#23 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/apps\\\/files_trashbin\\\/lib\\\/Storage.php(163): OCA\\\\Files_Trashbin\\\\Storage->doDelete('cache\\\/IMG_20180...', 'unlink')\\n#24 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(1159): OCA\\\\Files_Trashbin\\\\Storage->unlink('cache\\\/IMG_20180...')\\n#25 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(721): OC\\\\Files\\\\View->basicOperation('unlink', '\\\/IMG_20180104_1...', Array)\\n#26 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/Cache\\\/File.php(188): OC\\\\Files\\\\View->unlink('\\\/IMG_20180104_1...')\\n#27 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/base.php(833): OC\\\\Cache\\\\File->gc()\\n#28 [internal function]: OC::{closure}(*** sensitive parameters replaced ***)\\n#29 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/Hooks\\\/EmitterTrait.php(99): call_user_func_array(Object(Closure), Array)\\n#30 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/Hooks\\\/PublicEmitter.php(33): OC\\\\Hooks\\\\BasicEmitter->emit('\\\\\\\\OC\\\\\\\\User', 'postLogin', Array)\\n#31 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/User\\\/Session.php(359): OC\\\\Hooks\\\\PublicEmitter->emit('\\\\\\\\OC\\\\\\\\User', 'postLogin', Array)\\n#32 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/User\\\/Session.php(550): OC\\\\User\\\\Session->completeLogin(*** sensitive parameters replaced ***)\\n#33 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/User\\\/Session.php(326): OC\\\\User\\\\Session->loginWithPassword(*** sensitive parameters replaced ***)\\n#34 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/lib\\\/private\\\/User\\\/Session.php(400): OC\\\\User\\\\Session->login(*** sensitive parameters replaced ***)\\n#35 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/Auth.php(129): OC\\\\User\\\\Session->logClientIn(*** sensitive parameters replaced ***)\\n#36 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Auth\\\/Backend\\\/AbstractBasic.php(105): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Auth->validateUserPass(*** sensitive parameters replaced ***)\\n#37 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/Auth.php(252): Sabre\\\\DAV\\\\Auth\\\\Backend\\\\AbstractBasic->check(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#38 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/Auth.php(154): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Auth->auth(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#39 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Auth\\\/Plugin.php(201): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Auth->check(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#40 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Auth\\\/Plugin.php(150): Sabre\\\\DAV\\\\Auth\\\\Plugin->check(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#41 [internal function]: Sabre\\\\DAV\\\\Auth\\\\Plugin->beforeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#42 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#43 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(466): Sabre\\\\Event\\\\EventEmitter->emit('beforeMethod', Array)\\n#44 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#45 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Server.php(258): Sabre\\\\DAV\\\\Server->exec()\\n#46 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(33): OCA\\\\DAV\\\\Server->exec()\\n#47 \\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/remote.php(162): require_once('\\\/home\\\/cloud\\\/pub...')\\n#48 {main}\",\"File\":\"\\\/home\\\/cloud\\\/public_html\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Exception\\\/BadResponseException.php\",\"Line\":43}","userAgent":"Mozilla\/5.0 (Windows) mirall\/2.3.2 (build 1) (Nextcloud)","version":"12.0.5.3"}

[avmediadesign]

Any news about this (serious) issue ?

[rullzer]

So, long story short I have no idea. As I tried several upgrades and they all didn't work.

If somebody could get me an account on a system with this bug maybe I could debug better.
Just ping me on IRC (Freenode, #nextcloud-dev)

[avmediadesign]

So what is the actual recommendation?
I reverted back to 12.0.4 but I obviously have an error message about code integrity.

[harryhb]

@rullzer Did you try a server with primary storage on swift? That seems the mutual item for people suffering from the bug?

[avmediadesign]

I confirm using swift as primary storage.

[rullzer]

I don't have a test system with swift..... I'll see if I can come up with something. But it won't be this week I'm afraid.

[Arvi89]

Well, the sooner the better, I still can't use my nextcloud storage, and I don't want to revert to 12.0.4 as I'm worried to lose data.
It takes 10 mins to install, I you don't do it I'll install you something on a vps later...

[tdttester]

I'm experiencing the same problem with the 12.0.5 update.
I'm using OpenStack Swift as primary storage too.

Also, I tried on a test environment to upgrade a 12.0.4 Nextcloud instance using OpenStack Swift as primary storage to the latest RC of NextCloud 13, and it has the same problem.

[harryhb]

I guess we should be looking towards the Openstack swift implementation as the root cause? I changed the title of this issue to reflect.

Additionally added 13RC as impacted version

[rullzer]

I just wrote a small app to help here:

Could you try:

1. cd apps
   2.wget https://github.com/rullzer/appdatacleaner/releases/download/v1.0.0/appdatacleaner.tar.gz
2. tar -xzvf appdatacleaner.tar.gz
3. cd ..
4. ./occ app:enable appdatacleaner:
5. ./occ appdatacleaner:css
6. ./occ appdatacleaner:js
7. Go to your cloud in your browser and reload

[Arvi89]

Did all that, it didnt fix the issue.

[leoverdura]

I'm experiencing the same issue. Nextcloud doesn't show any graphics in the web-ui but seems to work in the background. I'm still receiving sync messages for example.

See the screens below. All links to the graphics/theme disappeared somehow.

[Arvi89]

Yep, that's the issue. But it doesn't seem to worry the team more than that, which is weird as it makes the instance unusable.

[dnwk]

I have a similar experience. It looks like SWIFT backend is the culprit. I switch to use S3 backend and it went on well. I think NextCloud need to look at swift storage backend and work on it. Even mounting swift storage as external storage is unstable. Very frequently, it failed to upload a file.

[deejay104]

I can confirm that this issue is linked to swift as primary storage.
I did a fresh install of version 13. It is working fine, then I have just change the configuration file to point a swift storage and the issue appear. Removing the objectstore configuration from config.php and everything is working fine again.

[rullzer]

Does somebody have a working swift setup for me to test against? Because the docker images I could find did not work.

[deejay104]

My setup was working with 12.0.4. Is it what you are asking ?

[deejay104]

And I can also confirm that this test instance is working fine with S3.

[deejay104]

For information, I have reverted the change done in this commit :
https://github.com/nextcloud/server/commit/ac29608db6816312b7d558fc3ac8b86ad34ad282
(removed the RetryWrapper::wrap function in the 2 files)
After that my instance is working again. I hope that it can help to resolve this issue.

[Arvi89]

Does somebody have a working swift setup for me to test against? Because the docker images I could find did not work.

What do you mean, you guys seriously still haven't installed a nextcloud instance with swift, really???

[tdttester]

They are releasing a product with a feature that they can't test whether it works or not. It's amazing!

[tessus]

@Schmuuu @skjnldsv why the thumbs down? it's the truth, otherwise this would have been a showstopper for a release, or at least I hope so.
The only dev in this discussion said that he can't test/reproduce, because the Docker images don't work.
If the CI pipeline had sufficient tests, the commit that broke nextcloud would've never been merged.

So nothing what @tdttester said is wrong.

[rullzer]

Lets keep it civil people. We are constantly updating and extending our test cases. This test matrix is already big and getting bigger every day.

We are in the process of setting up automated tests against swift right now. But this takes time and resources which are limited.

Hopefully we have a fix for you to test soon. But comments like that don't make us work any faster. The opposite is more likely.

[tessus]

Please note that I did not attack anybody personally. I also did not use any swear words, nor harsh language. Don't deflect by calling the truth uncivil.

I know that it is hard to have test cases for every single edge case, but this doesn't change the fact that my comment above was correct.

[skjnldsv]

@tessus it's called diplomacy. Saying things sarcastically has never improved the dialogue between people.

E.g: three sentences, one same outcome

1. Maybe if you had done your work correctly and implemented the proper testing, you wouldn't be there
2. Maybe we should consider adding some more tests for the next time so things like that doesn't occur again
3. Releasing a product with a feature that they can't test whether it works or not. It's amazing!

Which one do you prefer? I made my choice :)

[tessus]

@skjnldsv ok, my bad. I'm blunt and I expect others to be blunt with me as well. for me, all 3 sentences have the same meaning: I screwed up and thus have to fix it.
I would never take any of it personally or think "oops, that sounded harsh and hurt my feelings".

But I agree with you, the second one may "sound" nicer to a lot of people. Personally I don't put value in how something sounds, but the context.

Disclaimer: I have a genuine distaste for sugar-coating and making things sound nice for reasons of diplomacy and political correctness. I'm working on it, but it may take a while... ;-)

Anyway, I don't want to derail this ticket any further. I just wanted to make my position clear.
(I won't add any more comments to this ticket.)

P.S.: I also love sarcasm.

[skjnldsv]

@tessus no worries. But yes, it's not about sugar coating things, it's about telling the other person you're with him to fix the issue (which is productive) and not focusing on the issue itself (which is counter-productive) :)

[rullzer]

I think this is the same issue as https://github.com/nextcloud/server/pull/8233

The fix should be trivial to apply on 13 and 12. Please test. I'll make sure it is backported if it fixes it.

[Arvi89]

Yep, this is working. Adding the return statement fixed my problem! Thank you!

[rullzer]

Merged and will be shipped with 12.0.6, 13.0.1 and 14

Master: https://github.com/nextcloud/server/pull/8233
Stable 13: https://github.com/nextcloud/server/pull/8250
Stable 12: https://github.com/nextcloud/server/pull/8251

Grab the patch from your version. It is trivial to apply.

Also see https://github.com/nextcloud/server/pull/8235, this should get more swift test coverage to the server so this happening again will be a lot less likely.

[leoverdura]

I just manually updated to Nextcloud 13. Procedure like described in https://docs.nextcloud.com/server/13/admin_manual/maintenance/manual_upgrade.html
That helped me out. It works all fine again as far as I can see.

[BloodMotion]

Same issue, for upgrade N12 -> 13, fixed with https://github.com/nextcloud/server/pull/8250
Thanks @rullzer !

[linuxipho]

8250 works perfectly for 12 to 13 upgrade with Swift ObjectStore.

Thanks again @rullzer