Server: [WORKAROUND] NC14 RC2: occ upgrade hangs on running cronjob it states

[eggithub]

I cant give to much info right now due to lack of time but this is the command line output of the upgrade process:
sudo -u www-data php /var/www/nextcloud/occ upgrade
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Set log level to debug
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …

A cronjob was running at that time, killed the cron process but occ upgrade keeps giving this output.

When im home I provide more info (mysql/php7/etc..).
Tell me what else you need apart from the template. I use my own upgrade script. perhaps that might help.

to be continued...

Cheers

[nextcloud-bot]

GitMate.io thinks possibly related issues are https://github.com/nextcloud/server/issues/6960 (occ upgrade is outdated), https://github.com/nextcloud/server/issues/10918 (Occ Upgrade forces to run once a day), https://github.com/nextcloud/server/issues/9562 (Check before running "upgrade" if cronjob is still running and wait for it to finish), https://github.com/nextcloud/server/issues/8092 (Upgrade notice should not be displayed when occ is run with --output=json), and https://github.com/nextcloud/server/issues/4726 (Upgrading Nextcloud sometimes causes Cronjob Deadlocks).

[snopy234]

Try to change your current version in _[nextcloud path]/config/config.php_ to a lower version. In my case i've changed 14.0.0.17 to 14.0.0.0
Then use _./occ upgrade_ to upgrade your nextcloud. I hope this helps for you and sorry for my language problems. Have a nice day!
PS: Please let me know if it has worked.

[kesselb]

See #9992 for more information about this issue

[awrede]

@snopy234 I ran into the same issue. Changing version in config.php from 14.0.0.17 to 14.0.0.0 and then running ./occ upgrade made the upgrade run to completion.

[eggithub]

disabled the nextcloud cronjob
waited for >12 hours
then I was able to update

[snopy234]

@awrede nice to hear that it's working :)

[hchristo]

Try to change your current version in _[nextcloud path]/config/config.php_ to a lower version. In my case i've changed 14.0.0.17 to 14.0.0.0
Then use _./occ upgrade_ to upgrade your nextcloud. I hope this helps for you and sorry for my language problems. Have a nice day!
PS: Please let me know if it has worked.

Thanks, this ben Worked, Working, yes Working... THX!

[jgrete]

This issue is still present in 14.0.1 and the workaround is the same.
Very strange...

[snopy234]

Is my workaround still working? I havn't any problems since i changed the version

[jgrete]

@snopy234 :+1:
Yepp! THanks a lot!

[Larx]

Workaround does not work for me::
`Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Set log level to debug
Updating database schema
Updated database
Checking for update of app bookmarks in appstore
An unhandled exception has been thrown:
Error: Call to undefined method GuzzleHttpClient::request() in /srv/www/htdocs/nextcloud/lib/private/Http/Client/Client.php:161
Stack trace:

0 /srv/www/htdocs/nextcloud/lib/private/App/AppStore/Fetcher/Fetcher.php(106): OCHttpClientClient->get('https://apps.ne...', Array)

1 /srv/www/htdocs/nextcloud/lib/private/App/AppStore/Fetcher/AppFetcher.php(80): OCAppAppStoreFetcherFetcher->fetch('', '')

2 /srv/www/htdocs/nextcloud/lib/private/App/AppStore/Fetcher/Fetcher.php(170): OCAppAppStoreFetcherAppFetcher->fetch('', '')

3 /srv/www/htdocs/nextcloud/lib/private/Installer.php(385): OCAppAppStoreFetcherFetcher->get()

4 /srv/www/htdocs/nextcloud/lib/private/Updater.php(445): OCInstaller->isUpdateAvailable('bookmarks')

5 /srv/www/htdocs/nextcloud/lib/private/Updater.php(256): OCUpdater->upgradeAppStoreApps(Array)

6 /srv/www/htdocs/nextcloud/lib/private/Updater.php(126): OCUpdater->doUpgrade('14.0.1.1', '14.0.0.1')

7 /srv/www/htdocs/nextcloud/core/Command/Upgrade.php(262): OCUpdater->upgrade()

8 /srv/www/htdocs/nextcloud/3rdparty/symfony/console/Command/Command.php(251): OCCoreCommandUpgrade->execute(Object(SymfonyComponentConsoleInputArgvInput), Object(SymfonyComponentConsoleOutputConsoleOutput))

9 /srv/www/htdocs/nextcloud/3rdparty/symfony/console/Application.php(946): SymfonyComponentConsoleCommandCommand->run(Object(SymfonyComponentConsoleInputArgvInput), Object(SymfonyComponentConsoleOutputConsoleOutput))

10 /srv/www/htdocs/nextcloud/3rdparty/symfony/console/Application.php(248): SymfonyComponentConsoleApplication->doRunCommand(Object(OCCoreCommandUpgrade), Object(SymfonyComponentConsoleInputArgvInput), Object(SymfonyComponentConsoleOutputConsoleOutput))

11 /srv/www/htdocs/nextcloud/3rdparty/symfony/console/Application.php(148): SymfonyComponentConsoleApplication->doRun(Object(SymfonyComponentConsoleInputArgvInput), Object(SymfonyComponentConsoleOutputConsoleOutput))

12 /srv/www/htdocs/nextcloud/lib/private/Console/Application.php(213): SymfonyComponentConsoleApplication->run(Object(SymfonyComponentConsoleInputArgvInput), Object(SymfonyComponentConsoleOutputConsoleOutput))

13 /srv/www/htdocs/nextcloud/console.php(96): OCConsoleApplication->run()

14 /srv/www/htdocs/nextcloud/occ(11): require_once('/srv/www/htdocs...')

`

Here the bookmarks app causes the trouble, but before the activity and accessibility app caused the problem. I disabled them.

How to go on?

PS: Nextcloud causes trouble at almost every update. As much as I like it, it would be great if the update process would me much more robust.

[Larx]

Disabling files_external_gdrive did the job... (wasn't there a time when nextcloud disabled all but the core apps by itself before updating?)

[jgrete]

I hadn't activated that plugin, so the cause of your trouble was different.
Yes, I'm a little bit fed up also with these ever crahing updates. On most updates cycles I wasn't able to do a smooth update on my home installation. I could recover from all chrashes, but on my companies installation I'm much more conservative when it comes to nextcloud updates...

[cvandesande]

Also got this after the 14.0.1 upgrade. The workaround from @hchristo worked

[MorrisJobke]

A cronjob was running at that time, killed the cron process but occ upgrade keeps giving this output.

Killing the cron job caused the trouble actually here, because we store a "is something running" timestamp for each cron job and it only get updated by the job itself once it finished. The cronjob also just finishes the currently running one and then updates the DB table.

Best would be the SQL query from https://github.com/nextcloud/server/issues/9992#issue-335459638, but be aware that if the cron.php is still running somewhere that this could cause some issues.

[MorrisJobke]

Also this would be helpful to fix the cause of this issue:

SELECT * FROM oc_jobs WHERE reserved_at <> 0;

[snopy234]

@cvandesande which workaround from @hchristo do you mean?

[cvandesande]

@snopy234 where you downgrade the version in config.php to 14.0.0.0

[michidk]

Same issue. Query (@MorrisJobke):

SELECT * FROM oc_jobs WHERE reserved_at <> 0;

oc_jobs

id	class	argument	last_run	last_checked	reserved_at	execution_duration
748	OC\Preview\BackgroundCleanupJob	null	1538766002	1538766001	1538766001	0

Also tried updating Nextcloud manually, following this guide: https://docs.nextcloud.com/server/13/admin_manual/maintenance/manual_upgrade.html - with the same result: stuck, waiting for cron to finish.

Is there an hotfix to get Nextcloud working again?

[michidk]

Fixed it by setting the results of the query SELECT * FROM oc_jobs WHERE reserved_at <> 0; at field reserved_at to 0.

[MorrisJobke]

cc @rullzer for the background job in previews :/

[GLLM]

Hi all, I also have this issue.

• I got it while upgrading from 14.0.0 to 14.0.1. Solved it by changing version in config.php from 14.0.0.17 to 14.0.0.0 and then running ./occ upgrade

today, upgraded from 14.0.1 to 14.0.2 and also got it.
I'll try @michidk workaround ...

_Feedback/Edit_ : solved by doing SELECT * FROM oc_jobs WHERE reserved_at <> 0;
and setting it to 0 on the one line that I got.

Thanks

[moretocome]

Saw this error for the first time when upgrading from 14.0.2 to 14.0.3 with commandline updater.phar. Solved by changing version to 14.0.0.0 in config.php

[pkmkrishnakumar]

@GLLM

How to do this please give in detail here (For novice user)

This error occured during 14 ---> 14.1 & now 14.1 -----> 14.2

[MorrisJobke]

How to do this please give in detail here (For novice user)

This error occured during 14 ---> 14.1 & now 14.1 -----> 14.2

See my comment in https://github.com/nextcloud/server/issues/10949#issuecomment-426033160

[GLLM]

Hi all
Got the same issue once again from 14.0.2 > 14.0.3

Solved (again) it by looking in the _oc_jobs_ table, found the line with field reserved_at <> 0, and set it up back to 0.

Thanks

[benneti]

Hi,
Got the same issue (for the first time) from 14.0.2 > 14.0.3,
The fix suggested by @snopy234 worked, so thank you for this.

[MorrisJobke]

The big question is, why there are those entries. What type of cron job do you have? Do you run them in ajax mode?

[eggithub]

I just use the Cron option for background jobs.

Could it be related to the News App? I reinstalled that before I updated the nextcloud instance. I know from experience that News takes a long time to update after installing a new version because it happens dat my feeds aren't accessible for hours. And since this app also uses cron for updates, could it be that it is holding up something. Just guessing... But can I check this Cron update behaviour somewhere in the logs?

Running this nextcloud instance on an RPI3...

[MorrisJobke]

Could it be related to the News App? I reinstalled that before I updated the nextcloud instance. I know from experience that News takes a long time to update after installing a new version because it happens dat my feeds aren't accessible for hours. And since this app also uses cron for updates, could it be that it is holding up something. Just guessing... But can I check this Cron update behaviour somewhere in the logs?

Not really - except it is a news background job, but it seems to be many different ones in here.

[benneti]

I also use cron (provided by debian stable package) and have the news app installed

[enoch85]

Sorry for bumping, just got this on one of the servers I maintain going from 14.0.1 -- 14.0.3.

root@cloud:/var/scripts# sudo -u www-data php /var/www/nextcloud/occ upgrade
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Set log level to debug
Turned on maintenance mode
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
^C
root@cloud:/var/scripts# sudo -u postgr^C
root@cloud:/var/scripts# mysql -u root 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 922369
Server version: 5.7.23-0ubuntu0.16.04.1 (Ubuntu)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SELECT * FROM oc_jobs WHERE reserved_at <> 0;
ERROR 1046 (3D000): No database selected
mysql> use nextcloud_db
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> SELECT * FROM oc_jobs WHERE reserved_at <> 0;
+----+------------------------------------------------+----------+------------+--------------+-------------+--------------------+
| id | class                                          | argument | last_run   | last_checked | reserved_at | execution_duration |
+----+------------------------------------------------+----------+------------+--------------+-------------+--------------------+
| 20 | OCA\Survey_Client\BackgroundJobs\MonthlyReport | null     | 1539676802 |   1539676802 |  1539676802 |                 13 |
+----+------------------------------------------------+----------+------------+--------------+-------------+--------------------+
1 row in set (0,00 sec)

mysql> SELECT * FROM oc_jobs WHERE reserved_at <> 0;
+----+------------------------------------------------+----------+------------+--------------+-------------+--------------------+
| id | class                                          | argument | last_run   | last_checked | reserved_at | execution_duration |
+----+------------------------------------------------+----------+------------+--------------+-------------+--------------------+
| 20 | OCA\Survey_Client\BackgroundJobs\MonthlyReport | null     | 1539676802 |   1539676802 |  1539676802 |                 13 |
+----+------------------------------------------------+----------+------------+--------------+-------------+--------------------+
1 row in set (0,00 sec)

mysql> UPDATE oc_jobs SET reserved_at=0
    -> ;
Query OK, 1 row affected (0,00 sec)
Rows matched: 24  Changed: 1  Warnings: 0

mysql> SELECT * FROM oc_jobs WHERE reserved_at <> 0;
Empty set (0,00 sec)

mysql> exit
Bye
root@cloud:/var/scripts# nano nextcloud_update.sh 
root@cloud:/var/scripts# nano nextcloud_update.sh 
root@cloud:/var/scripts# bash nextcloud_update.sh 
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Set log level to debug
Updating database schema
Updated database
Disabled incompatible app: bruteforcesettings
Disabled incompatible app: calendar
Disabled incompatible app: contacts
Disabled incompatible app: impersonate
Checking for update of app accessibility in appstore
Checked for update of app "accessibility" in appstore 
Checking for update of app activity in appstore
Checked for update of app "activity" in appstore 
Checking for update of app cloud_federation_api in appstore
Checked for update of app "cloud_federation_api" in appstore 
Checking for update of app comments in appstore
Checked for update of app "comments" in appstore 
Checking for update of app dav in appstore
Checked for update of app "dav" in appstore 
Checking for update of app federatedfilesharing in appstore
Checked for update of app "federatedfilesharing" in appstore 
Checking for update of app federation in appstore
Checked for update of app "federation" in appstore 
Checking for update of app files in appstore
Checked for update of app "files" in appstore 
Checking for update of app files_external in appstore
Checked for update of app "files_external" in appstore 

All good now.

[Larx]

Any idea how to get rid of this? I'm kind of sick of going into manual mode to repair NC after every update. Please please make the updates more robust!!

[GAS85]

Meet the same issue during update to 14.0.3

[MorrisJobke]

We removed this via #12188

Will be shipped in 14.0.4

[sscholl]

@snopy234 Worked indeed!

[snopy234]

Thank you to share this with us @sscholl and @benneti :)

btw i found the solution for the update error from 15 beta 2 to 15 rc 2 ... what are you doing with the updater at each update @nextcloud-bot @karlitschek :(