Core: When the DB server goes away, HTTP Status code stays 200 and oC shows exception in browser

[enbrnz]

Steps to reproduce

1. Install ownCloud
2. Stop MySQL or MariaDB server
3. Visit ownCloud URL

Expected behaviour

This should show a 500 Server Error

Actual behaviour

Webserver replies with HTTP Status code 200 and the following exception is shown to end users:

Exception occurred while logging exception: Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] No such file or directory
#0 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /var/www/owncloud/lib/private/DB/Connection.php(144): Doctrine\DBAL\Connection->setTransactionIsolation(2)
#5 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\DB\Connection->__construct(Array, Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventManager))
#6 /var/www/owncloud/lib/private/DB/ConnectionFactory.php(145): Doctrine\DBAL\DriverManager::getConnection(Array, Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventManager))
#7 /var/www/owncloud/lib/private/Server.php(529): OC\DB\ConnectionFactory->getConnection('mysql', Array)
#8 /var/www/owncloud/lib/composer/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#9 /var/www/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(108): Pimple\Container->offsetGet('DatabaseConnect...')
#10 /var/www/owncloud/lib/private/ServerContainer.php(86): OC\AppFramework\Utility\SimpleContainer->query('DatabaseConnect...')
#11 /var/www/owncloud/lib/private/Server.php(1213): OC\ServerContainer->query('DatabaseConnect...')
#12 /var/www/owncloud/lib/private/Server.php(404): OC\Server->getDatabaseConnection()
#13 /var/www/owncloud/lib/composer/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#14 /var/www/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(108): Pimple\Container->offsetGet('AppConfig')
#15 /var/www/owncloud/lib/private/ServerContainer.php(86): OC\AppFramework\Utility\SimpleContainer->query('AppConfig')
#16 /var/www/owncloud/lib/private/Server.php(1143): OC\ServerContainer->query('AppConfig')
#17 /var/www/owncloud/lib/private/Server.php(583): OC\Server->getAppConfig()
#18 /var/www/owncloud/lib/composer/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#19 /var/www/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(108): Pimple\Container->offsetGet('AppManager')
#20 /var/www/owncloud/lib/private/ServerContainer.php(86): OC\AppFramework\Utility\SimpleContainer->query('AppManager')
#21 /var/www/owncloud/lib/private/Server.php(1411): OC\ServerContainer->query('AppManager')
#22 /var/www/owncloud/lib/private/legacy/app.php(346): OC\Server->getAppManager()
#23 /var/www/owncloud/lib/private/legacy/app.php(110): OC_App::getEnabledApps()
#24 /var/www/owncloud/lib/base.php(577): OC_App::loadApps(Array)
#25 /var/www/owncloud/lib/base.php(994): OC::init()
#26 /var/www/owncloud/index.php(53): require_once('/var/www/ownclo...')
#27 {main}

Server configuration

ownCloud version: 10.2.1

Logs

Web server access log

ubuntu.local:443 192.168.122.1 - - [21/Aug/2019:14:42:32 +0000] "GET / HTTP/1.1" 200 3224 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"
ubuntu.local:443 192.168.122.1 - - [21/Aug/2019:14:42:32 +0000] "GET /favicon.ico HTTP/1.1" 404 606 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"

ownCloud log (data/owncloud.log)

nothing

[DeepDiver1975]

What is displayed in the browser?

[cdamken]

What is displayed in the browser?

This is what is displayed in the browser:

Exception occurred while logging exception: Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] No such file or directory
#0 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /var/www/owncloud/lib/private/DB/Connection.php(144): Doctrine\DBAL\Connection->setTransactionIsolation(2)
#5 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\DB\Connection->__construct(Array, Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventManager))
#6 /var/www/owncloud/lib/private/DB/ConnectionFactory.php(145): Doctrine\DBAL\DriverManager::getConnection(Array, Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventManager))
#7 /var/www/owncloud/lib/private/Server.php(529): OC\DB\ConnectionFactory->getConnection('mysql', Array)
#8 /var/www/owncloud/lib/composer/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#9 /var/www/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(108): Pimple\Container->offsetGet('DatabaseConnect...')
#10 /var/www/owncloud/lib/private/ServerContainer.php(86): OC\AppFramework\Utility\SimpleContainer->query('DatabaseConnect...')
#11 /var/www/owncloud/lib/private/Server.php(1213): OC\ServerContainer->query('DatabaseConnect...')
#12 /var/www/owncloud/lib/private/Server.php(404): OC\Server->getDatabaseConnection()
#13 /var/www/owncloud/lib/composer/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#14 /var/www/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(108): Pimple\Container->offsetGet('AppConfig')
#15 /var/www/owncloud/lib/private/ServerContainer.php(86): OC\AppFramework\Utility\SimpleContainer->query('AppConfig')
#16 /var/www/owncloud/lib/private/Server.php(1143): OC\ServerContainer->query('AppConfig')
#17 /var/www/owncloud/lib/private/Server.php(583): OC\Server->getAppConfig()
#18 /var/www/owncloud/lib/composer/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#19 /var/www/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(108): Pimple\Container->offsetGet('AppManager')
#20 /var/www/owncloud/lib/private/ServerContainer.php(86): OC\AppFramework\Utility\SimpleContainer->query('AppManager')
#21 /var/www/owncloud/lib/private/Server.php(1411): OC\ServerContainer->query('AppManager')
#22 /var/www/owncloud/lib/private/legacy/app.php(346): OC\Server->getAppManager()
#23 /var/www/owncloud/lib/private/legacy/app.php(110): OC_App::getEnabledApps()
#24 /var/www/owncloud/lib/base.php(577): OC_App::loadApps(Array)
#25 /var/www/owncloud/lib/base.php(994): OC::init()
#26 /var/www/owncloud/index.php(53): require_once('/var/www/ownclo...')
#27 {main}

The whole Stacktrace

[ho4ho]

Related by https://github.com/nextcloud/server/pull/9995 ?

[jvillafanez]

@DeepDiver1975 @PVince81 It seems we need a "broken beyond repair" kind of log to be used in several endpoints, such as index.php, remote.php, public.php and others.

The main usage is to report broken installations / states such as the above, when the DB connection is down. The problem is that the ownCloud's logger (and maybe other services such as the request) access to the DB because it needs to fetch the user. This makes ownCloud crash trying to handle the previous exception.

Expected code should be something like:

try {
  normalRequestHandling(.....)
} catch (\Exception $e) {
  try {
    normalExceptionHandling($e);
  } catch (\Exception $e) {
    abnormalExceptionHandling($e);
  }
}

The challenge is that this "abnormalExceptionHandling" can't rely on ownCloud.
We could send a 599 HTTP status with a plain text body, and log a message in a different file (we can't log into the normal owncloud.log file because the logger could be broken, so we can't format properly the log message)

At the moment, there are several unhandled exceptions in apache because we're trying to log when the logger is broken inside the "normal" catch block.

Note that this should be as homogeneous as possible among all the endpoints

[micbar]

PR got merged.

Closing this ticket, will be released with 10.4.0