Core: MariaDB 10.2.7: Install broken on master due to database column options

[tomneedham]

Summary

Symptom:

An exception occurred while executing 
'ALTER TABLE oc_calendarsubscriptions CHANGE `lastmodified` lastmodified INT UNSIGNED DEFAULT NULL NOT NULL'

Reason: MariaDB v10.2.7+ is not supported by doctrine/dbal prior to v2.7.0
OC 10.0.4 - Fix on top of older doctrine/dbal: https://github.com/owncloud/core/pull/29240
OC 10.4.0 - doctrine/dbal updated to v2.8.0: https://github.com/owncloud/core/pull/36290
Quick older doctrine/dbal patch: https://github.com/owncloud/core/issues/28695#issuecomment-332174798

Initial report:

Master as of today 15/8/17

PHP 7.1.8-2+ubuntu16.04.1+deb.sury.org+4
10.2.7-MariaDB-10.2.7+maria~xenial

1. I setup a new laptop, with apache, git, node, mariadb, php7.1.
2. Git cloned core, ran make
3. Setup db, user
4. ran install via occ

tomneedham@tomneedham-X453MA:/var/www/html$ sudo -u www-data ./occ maintenance:install -vv --database=mysql --database-name=owncloud --database-host=localhost --database-user=owncloud --database-pass=owncloud --admin-user=Tom --admin-pass=Tom
ownCloud is not installed - only a limited number of commands are available
Error while trying to create admin user: An exception occurred while executing 'ALTER TABLE oc_calendarsubscriptions CHANGE `lastmodified` lastmodified INT UNSIGNED DEFAULT NULL NOT NULL':

SQLSTATE[42000]: Syntax error or access violation: 1067 Invalid default value for 'lastmodified'

Obviously we cant have default=null and NOT NULL....!

Traced this to line https://github.com/owncloud/core/blob/master/core/Migrations/Version20170101215145.php#L213

Dav install migration is here: https://github.com/owncloud/core/blob/master/apps/dav/appinfo/Migrations/Version20170116150538.php#L387 and only specifies unsigned => true.

I think this should have been in the dav app migrations.... https://github.com/owncloud/core/commit/de295caa4b2d7b7192ff95170b11b31690e079e4

Weirdly I can't reproduce on another machine.

Could this be related to now having the 8/9 => 10 migrations in core, which include the setup for the dav app. So, during install, we are 'installing' dav before dav has had a change to do its setup?? Maybe....? Brain is still thinking this through.

@PVince81 @DeepDiver1975 ?

[tomneedham]

I added in logging to show the migrations being performed. You can see it is running the 8/9 => 10 migrations, despite this being a fresh 10 install. This means the 8/9 migratuions are 'installing' the dav app despite it not being installed yet, which is tricking future core migrations into thinking we are just doing an update here and hence modifying the dav tables, which are not actually up to date yet.

[tomneedham]

Confirmed that removing https://github.com/owncloud/core/blob/master/core/Migrations/Version20170101010100.php fixes the issue for a clean install

[tomneedham]

Should https://github.com/owncloud/core/blob/master/core/Migrations/Version20170101010100.php not check if we are updating, not a fresh install?

[PVince81]

we shouldn't detect install vs update in a migration, but we can detect whether a table/index already exists or not or a column is of a specific type.

[hanwentao]

I also encountered this issue when upgrading from 9.1 to 10.0. The generated SQL statement has DEFAULT NULL NOT NULL, which is contradictory. Look at the end of file Version20170101215145.php. Should we specify a default value for migration of lastmodified column?

[PVince81]

moving to "planned".

@tomneedham did you have any news regarding this ?

[TheFiZi]

Just upgraded from owncloud-files-9.1.6-1.1.noarch to owncloud-files-10.0.3-1.1.noarch on a CentOS 6 64-bit box.

PHP 7.0.22
MariaDB 10.2.8
Apache 2.2

Upgrading via the WebUI returns this:

Preparing update
Set log level to debug
Turned on maintenance mode
Updating database schema
Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'ALTER TABLE oc_calendarsubscriptions CHANGE `lastmodified` lastmodified INT UNSIGNED DEFAULT NULL NOT NULL': SQLSTATE[42000]: Syntax error or access violation: 1067 Invalid default value for 'lastmodified'

Upgrading via the CLI

-bash-4.1$ ./occ upgrade
ownCloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
2017-09-19T17:02:02+00:00 Set log level to debug
2017-09-19T17:02:02+00:00 Turned on maintenance mode
2017-09-19T17:02:02+00:00 Repair step: Repair MySQL database engine
2017-09-19T17:02:02+00:00 Repair step: Repair MySQL collation
2017-09-19T17:02:02+00:00 Repair info: Change collation for oc_migrations ...
2017-09-19T17:02:02+00:00 Repair step: Repair SQLite autoincrement
2017-09-19T17:02:02+00:00 Repair step: Repair duplicate entries in oc_lucene_status
2017-09-19T17:02:02+00:00 Repair info: lucene_status table does not exist -> nothing to do
2017-09-19T17:02:02+00:00 Repair step: Upgrade app code from the marketplace
2017-09-19T17:02:02+00:00 Repair info: Enabling market app to assist with update
2017-09-19T17:02:02+00:00 Repair info: Using market to update existing apps
2017-09-19T17:02:02+00:00 Repair info: Attempting to update the following existing compatible apps from market:
2017-09-19T17:02:02+00:00 Repair info: Fetching app from market: activity
2017-09-19T17:02:03+00:00 Repair info:
2017-09-19T17:02:03+00:00 Repair info: Fetching app from market: comments
2017-09-19T17:02:05+00:00 Repair info: App (comments) is not known at the marketplace.
2017-09-19T17:02:05+00:00 Repair info: Fetching app from market: configreport
2017-09-19T17:02:06+00:00 Repair info: App (configreport) is not known at the marketplace.
2017-09-19T17:02:06+00:00 Repair info: Fetching app from market: dav
2017-09-19T17:02:07+00:00 Repair info: App (dav) is not known at the marketplace.
2017-09-19T17:02:07+00:00 Repair info: Fetching app from market: federatedfilesharing
2017-09-19T17:02:09+00:00 Repair info: App (federatedfilesharing) is not known at the marketplace.
2017-09-19T17:02:09+00:00 Repair info: Fetching app from market: federation
2017-09-19T17:02:10+00:00 Repair info: App (federation) is not known at the marketplace.
2017-09-19T17:02:10+00:00 Repair info: Fetching app from market: files
2017-09-19T17:02:11+00:00 Repair info: App (files) is not known at the marketplace.
2017-09-19T17:02:11+00:00 Repair info: Fetching app from market: files_external
2017-09-19T17:02:11+00:00 Repair info: App (files_external) is not installed
2017-09-19T17:02:11+00:00 Repair info: Fetching app from market: files_pdfviewer
2017-09-19T17:02:13+00:00 Repair info:
2017-09-19T17:02:13+00:00 Repair info: Fetching app from market: files_sharing
2017-09-19T17:02:14+00:00 Repair info: App (files_sharing) is not known at the marketplace.
2017-09-19T17:02:14+00:00 Repair info: Fetching app from market: files_texteditor
2017-09-19T17:02:15+00:00 Repair info:
2017-09-19T17:02:15+00:00 Repair info: Fetching app from market: files_trashbin
2017-09-19T17:02:16+00:00 Repair info: App (files_trashbin) is not known at the marketplace.
2017-09-19T17:02:16+00:00 Repair info: Fetching app from market: files_versions
2017-09-19T17:02:18+00:00 Repair info: App (files_versions) is not known at the marketplace.
2017-09-19T17:02:18+00:00 Repair info: Fetching app from market: files_videoplayer
2017-09-19T17:02:19+00:00 Repair info: App (files_videoplayer) is not known at the marketplace.
2017-09-19T17:02:19+00:00 Repair info: Fetching app from market: firstrunwizard
2017-09-19T17:02:20+00:00 Repair info: App (firstrunwizard) is not known at the marketplace.
2017-09-19T17:02:20+00:00 Repair info: Fetching app from market: market
2017-09-19T17:02:22+00:00 Repair info:
2017-09-19T17:02:22+00:00 Repair info: Fetching app from market: notifications
2017-09-19T17:02:23+00:00 Repair info: App (notifications) is not known at the marketplace.
2017-09-19T17:02:23+00:00 Repair info: Fetching app from market: provisioning_api
2017-09-19T17:02:24+00:00 Repair info: App (provisioning_api) is not known at the marketplace.
2017-09-19T17:02:24+00:00 Repair info: Fetching app from market: systemtags
2017-09-19T17:02:25+00:00 Repair info: App (systemtags) is not known at the marketplace.
2017-09-19T17:02:25+00:00 Repair info: Fetching app from market: templateeditor
2017-09-19T17:02:27+00:00 Repair info: App (templateeditor) is not known at the marketplace.
2017-09-19T17:02:27+00:00 Repair info: Fetching app from market: updatenotification
2017-09-19T17:02:28+00:00 Repair info: App (updatenotification) is not known at the marketplace.
2017-09-19T17:02:28+00:00 Repair info: App was not updated: activity
2017-09-19T17:02:28+00:00 Repair info: App was not updated: comments
2017-09-19T17:02:28+00:00 Repair info: App was not updated: configreport
2017-09-19T17:02:28+00:00 Repair info: App was not updated: dav
2017-09-19T17:02:28+00:00 Repair info: App was not updated: federatedfilesharing
2017-09-19T17:02:28+00:00 Repair info: App was not updated: federation
2017-09-19T17:02:28+00:00 Repair info: App was not updated: files
2017-09-19T17:02:28+00:00 Repair info: App was not updated: files_external
2017-09-19T17:02:28+00:00 Repair info: App was not updated: files_pdfviewer
2017-09-19T17:02:28+00:00 Repair info: App was not updated: files_sharing
2017-09-19T17:02:28+00:00 Repair info: App was not updated: files_texteditor
2017-09-19T17:02:28+00:00 Repair info: App was not updated: files_trashbin
2017-09-19T17:02:28+00:00 Repair info: App was not updated: files_versions
2017-09-19T17:02:28+00:00 Repair info: App was not updated: files_videoplayer
2017-09-19T17:02:28+00:00 Repair info: App was not updated: firstrunwizard
2017-09-19T17:02:28+00:00 Repair info: App was not updated: market
2017-09-19T17:02:28+00:00 Repair info: App was not updated: notifications
2017-09-19T17:02:28+00:00 Repair info: App was not updated: provisioning_api
2017-09-19T17:02:28+00:00 Repair info: App was not updated: systemtags
2017-09-19T17:02:28+00:00 Repair info: App was not updated: templateeditor
2017-09-19T17:02:28+00:00 Repair info: App was not updated: updatenotification
2017-09-19T17:02:28+00:00 Updating database schema
2017-09-19T17:02:28+00:00 Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'ALTER TABLE oc_calendarsubscriptions CHANGE `lastmodified` lastmodified INT UNSIGNED DEFAULT NULL NOT NULL':

SQLSTATE[42000]: Syntax error or access violation: 1067 Invalid default value for 'lastmodified'
2017-09-19T17:02:28+00:00 Update failed
2017-09-19T17:02:28+00:00 Maintenance mode is kept active
2017-09-19T17:02:28+00:00 Reset log level

[fsantiago07044]

same issue i had as @TheFiZi ; just reported on the mailing list as well.

[TheFiZi]

@fsantiago07044 I've also posted here: https://central.owncloud.org/t/centos-6-upgrade-to-oc-10-0-3-failed/9544

[PVince81]

@tomneedham and I looked at various theories why this would happen.

Could everybody here post the result of oc_migrations from your broken instance ?

The theories:
1) Some migrations ran in the wrong order, or some migrations were skipped: so far this looks impossible as they are sorted by ASCII sort based on their timestamp of the file name, which should sort fine on any platform. Or Doctrine/DBAL bug ?

2) Potential weird migration conflict. Apparently some migrations are defined twice: https://github.com/owncloud/core/blob/master/core/Migrations/Version20170101010100.php#L424 and https://github.com/owncloud/core/blob/master/apps/dav/appinfo/Migrations/Version20170116150538.php#L338. Regardless, the migration run order should always be "core" first then apps, so the result should be deterministic.

3) Potential older calendar tables: if a calendar app from OC 8.2 was present during setup/upgrade, it's default_enable tag would enable it regardless of max-version. However it turns out that the old calendar app didn't have any "oc_calendarsubscriptions" table, so this is not this issue here.

4) A Doctrine/DBAL bug that would make the SQL query not be generated properly. This one could be explored further by looking at the DBAL code and find out what combination of attributes could generate the above invalid query.

5) Anything else that could mess up the migration order.

Seeing the contents of oc_migrations before and after a failed upgrade or install would be useful as we could find out whether migrations were actually skipped (which would confirm one theory) or they ran but skipped a table update.

@tomneedham does that summarize everything we talked about so far ?

[fsantiago07044]

On September 19, 2017 5:27:56 PM EDT, Vincent Petry [e-mail] wrote:

@tomneedham and I looked at various theories why this would happen.

Could everybody here post the result of oc_migrations from your
broken instance ?

The theories:
1) Some migrations ran in the wrong order, or some migrations were
skipped: so far this looks impossible as they are sorted by ASCII sort
based on their timestamp of the file name, which should sort fine on
any platform. Or Doctrine/DBAL bug ?

2) Potential weird migration conflict. Apparently some migrations are
defined twice:
https://github.com/owncloud/core/blob/master/core/Migrations/Version20170101010100.php#L424
and
https://github.com/owncloud/core/blob/master/apps/dav/appinfo/Migrations/Version20170116150538.php#L338.
Regardless, the migration run order should always be "core" first then
apps, so the result should be deterministic.

3) Potential older calendar tables: if a calendar app from OC 8.2 was
present during setup/upgrade, it's default_enable tag would enable it
regardless of max-version. However it turns out that the old calendar
app didn't have any "oc_calendarsubscriptions" table, so this is not
this issue here.

4) A Doctrine/DBAL bug that would make the SQL query not be generated
properly. This one could be explored further by looking at the DBAL
code and find out what combination of attributes could generate the
above invalid query.

5) Anything else that could mess up the migration order.

Seeing the contents of oc_migrations before and after a failed
upgrade or install would be useful as we could find out whether
migrations were actually skipped (which would confirm one theory) or
they ran but skipped a table update.

@tomneedham does that summarize everything we talked about so far ?

Excuse my naivety; how do you obtain the contents of oc_migrations?

Thanks,

Fabian S.

OpenPGP: 3C3FA072ACCB7AC5DB0F723455502B0EEB9070FC

[TheFiZi]

@fsantiago07044

To get the contents of oc_migraitons:

1. SSH into your server
2. Run mysql -u <OWNCLOUD MYSQL USERNAME> -p
3. Enter the password for the \
4. USE <OWNCLOUD DB NAME>;
5. SELECT * FROM oc_migrations;

Here is mine:

| app      | version        |
+----------+----------------+
| activity | 20161122085340 |
| activity | 20161122092159 |
| activity | 20170131134507 |
| activity | 20170724182159 |
| core     | 20170101010100 |
+----------+----------------+
5 rows in set (0.00 sec)

Fun side note. I don't even use the calendar function in OwnCloud. Just file syncing.

[fsantiago07044]

On September 19, 2017 6:15:35 PM EDT, Eric [e-mail] wrote:

@fsantiago07044

To get the contents of oc_migraitons:

1. SSH into your server
2. Run mysql -u <OWNCLOUD MYSQL USERNAME> -p
3. Enter the password for the \
4. USE <OWNCLOUD DB NAME>;
5. SELECT * FROM oc_migrations;

Here is mine:

| app      | version        |
+----------+----------------+
| activity | 20161122085340 |
| activity | 20161122092159 |
| activity | 20170131134507 |
| activity | 20170724182159 |
| core     | 20170101010100 |
+----------+----------------+
5 rows in set (0.00 sec)

Oh got it. Didn't realize that meant a db table. Here's mine:

+----------+----------------+
| app | version |
+----------+----------------+
| activity | 20161122085340 |
| activity | 20161122092159 |
| activity | 20170131134507 |
| activity | 20170724182159 |
| core | 20170101010100 |
+----------+----------------+
5 rows in set (0.01 sec)

Thanks.

--

Thanks,

Fabian S.

OpenPGP: 3C3FA072ACCB7AC5DB0F723455502B0EEB9070FC

[fri-K]

I have exact the same problem, here is content of my oc_migrations:
+----------+----------------+
| app | version |
+----------+----------------+
| activity | 20161122085340 |
| activity | 20161122092159 |
| activity | 20170131134507 |
| activity | 20170724182159 |
| core | 20170101010100 |
| core | 20170111103310 |
| core | 20170213215145 |
| core | 20170214112458 |
| core | 20170221114437 |
| core | 20170221121536 |
| core | 20170315173825 |
| core | 20170320173955 |
| core | 20170418154659 |
| dav | 20170116150538 |
| dav | 20170116170538 |
| dav | 20170202213905 |
| dav | 20170202220512 |
+----------+----------------+

[PVince81]

Wow, thanks guys. For reference, here is how mine looks like on an instance upgraded to 10.0.3 over the years (went through almost all possible versions since OC 4.5)

This is for a working instance:

| app                  | version        |
+----------------------+----------------+
| activity             | 20161122085340 |
| activity             | 20161122092159 |
| activity             | 20170131134507 |
| activity             | 20170724182159 |
| core                 | 20170101010100 |
| core                 | 20170101215145 |
| core                 | 20170111103310 |
| core                 | 20170213215145 |
| core                 | 20170214112458 |
| core                 | 20170221114437 |
| core                 | 20170221121536 |
| core                 | 20170315173825 |
| core                 | 20170320173955 |
| core                 | 20170418154659 |
| core                 | 20170516100103 |
| core                 | 20170526104128 |
| core                 | 20170605143658 |
| core                 | 20170711191432 |
| core                 | 20170804201253 |
| dav                  | 20170116150538 |
| dav                  | 20170116170538 |
| dav                  | 20170202213905 |
| dav                  | 20170202220512 |
| dav                  | 20170427182800 |
| dav                  | 20170519091921 |
| dav                  | 20170526100342 |
| dav                  | 20170711193427 |
| federatedfilesharing | 20170804201125 |
| federatedfilesharing | 20170804201253 |
| files_external       | 20170814051424 |
| files_sharing        | 20170804201125 |
| files_sharing        | 20170804201253 |
| files_trashbin       | 20170804201125 |
| files_trashbin       | 20170804201253 |
+----------------------+----------------+

[PVince81]

For @TheFiZi and @fsantiago07044 this tells us that apart from the activity app migrations, nothing else than "20170101010100" has run before, and "20170101010100" is the one with the calendar subscriptions bug in the SQL queries.

This invalidates the theories about bugs in the migrations order where another migration would cause trouble for the previous one.

Now if you were upgrading from 9.1, the table in question already existed there. So what Doctrine/DBAL does here with the migrations is detect the existing schema from the database and compute it, then apply the existing things. So we should look at how the table looked like in 9.1.6 and find a way how the schema was processed before and after the migration in PHP in a way that could cause this weird message.

For those who did not upgrade and had a new install instead (with an empty database!), it still remains mysterious. @tomneedham is there a likely chance that this one time you setup OC from scratch you forgot to reset the database ?

[wcz-txp]

Setting default for lastmodified from NULL to current_timestamp helped me. After that sudo -u www-data ./occ upgrade ran through.

ownCloud 10.0.3-1.1 upgraded from 9.1.6-1.1
PHP 7.0.23-1~dotdeb+8.1
MariaDB-10.2.8
Debian GNU/Linux 8.9 (jessie)

[fsantiago07044]

September 20, 2017 5:57 AM, "Andrij" [e-mail] wrote:

Setting default for lastmodified from NULL to current_timestamp helped me. After that sudo -u
www-data ./occ upgrade ran through.

ownCloud 10.0.3-1.1 upgraded from 9.1.6-1.1
PHP 7.0.23-1dotdeb+8.1
MariaDB-10.2.8+mariajessie-log
Debian GNU/Linux 8.9 (jessie)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

ok,

i tried this; it technically "worked" but then in my admin panels i started receiving many errors such as "failed to request notifications. please try to refresh the page manually." this appears on various admin panels, etc.

I have everything backed up so as a test, i then ran "yum remove owncloud-files", and dropped the db.

i created a new empty db and reinstalled owncloud-files.

as a fresh install now, and during the initial web based setup, it failed with the same original error during the db setup. but the db had been populated with the schema.

i then changed the lastmodified default value as suggested above, and again, it "worked" but i still see the same errors throughout the interface regarding notifications. and this is now a fresh install with no users (other than the admin created during setup), extra apps, data, etc.

so what now? thanks.

--

Thanks,

Fabian S.

OpenPGP: 3C3FA072ACCB7AC5DB0F723455502B0EEB9070FC

[PVince81]

as a fresh install now, and during the initial web based setup, it failed with the same original error during the db setup. but the db had been populated with the schema.

Ok, so this tells us that it's not an upgrade issue at all and something is wrong either with the code that generates the SQL query or the code that reads the schema from the DB.

[VicDeo]

as a temporary workaround I can suggest to use MariaDB 10.1 as the issue is MariaDB 10.2.7+ specific
Related upstream issue https://github.com/doctrine/dbal/issues/2817

there is also untested unofficial patch for Doctrine

Updating the method _getPortableTableColumnDefinition of DBAL/Schema/MysqlSchemaManager

    $options = array(
        'length'        => $length,
        'unsigned'      => (bool) (strpos($tableColumn['type'], 'unsigned') !== false),
        'fixed'         => (bool) $fixed,
        // REPLACE THE LINE BELOW
        //    'default'       => isset($tableColumn['default']) ? $tableColumn['default'] : null,
        // BY THIS ONE
        'default'       => isset($tableColumn['default']) && !($tableColumn['default'] === 'NULL' && $tableColumn['null'] === 'YES') ? $tableColumn['default'] : null,
        'notnull'       => (bool) ($tableColumn['null'] != 'YES'),
        'scale'         => null,
        'precision'     => null,
        'autoincrement' => (bool) (strpos($tableColumn['extra'], 'auto_increment') !== false),
        'comment'       => isset($tableColumn['comment']) && $tableColumn['comment'] !== ''
            ? $tableColumn['comment']
            : null,
    );

[enoch85]

I'm using MariaDB 10.2 in the ownCloud VM and just got this:

+ mysql -u root -pJxBP#SOMjO493cdimgsamvNNmLzN# -e 'CREATE DATABASE IF NOT EXISTS owncloud_db;'
+ cd /var/www/owncloud
+ check_command sudo -u www-data php occ maintenance:install --data-dir /var/ocdata --database mysql --database-name owncloud_db --database-user root --database-pass JxBP#SOMjO493cdimgsamvNNmLzN# --admin-user ocadmin --admin-pass owncloud
+ eval 'sudo -u www-data php occ maintenance:install --data-dir /var/ocdata --database mysql --database-name owncloud_db --database-user root --database-pass JxBP#SOMjO493cdimgsamvNNmLzN# --admin-user ocadmin --admin-pass owncloud'
++ sudo -u www-data php occ maintenance:install --data-dir /var/ocdata --database mysql --database-name owncloud_db --database-user root --database-pass JxBP#SOMjO493cdimgsamvNNmLzN# --admin-user ocadmin --admin-pass owncloud
ownCloud is not installed - only a limited number of commands are available
Error while trying to create admin user: An exception occurred while executing 'ALTER TABLE oc_calendarsubscriptions CHANGE `lastmodified` lastmodified INT UNSIGNED DEFAULT NULL NOT NULL':

SQLSTATE[42000]: Syntax error or access violation: 1067 Invalid default value for 'lastmodified'

Maybe not so much help, but I really think you should fix this as soon as possible. I use the same code in Nextcloud and have no issues at all.

Thanks!

[TheFiZi]

@VicDeo That appears to have resolved my problem.

Here is the output of my upgrade log:

-bash-4.1$ ./occ upgrade
ownCloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
2017-09-24T20:51:51+00:00 Set log level to debug
2017-09-24T20:51:51+00:00 Repair step: Repair MySQL database engine
2017-09-24T20:51:51+00:00 Repair step: Repair MySQL collation
2017-09-24T20:51:51+00:00 Repair info: All tables already have the correct collation -> nothing to do
2017-09-24T20:51:51+00:00 Repair step: Repair SQLite autoincrement
2017-09-24T20:51:51+00:00 Repair step: Repair duplicate entries in oc_lucene_status
2017-09-24T20:51:51+00:00 Repair info: lucene_status table does not exist -> nothing to do
2017-09-24T20:51:51+00:00 Repair step: Upgrade app code from the marketplace
2017-09-24T20:51:51+00:00 Repair info: Enabling market app to assist with update
2017-09-24T20:51:51+00:00 Repair info: Using market to update existing apps
2017-09-24T20:51:51+00:00 Repair info: Attempting to update the following existing compatible apps from market:
2017-09-24T20:51:51+00:00 Repair info: Fetching app from market: activity
2017-09-24T20:51:53+00:00 Repair info:
2017-09-24T20:51:53+00:00 Repair info: Fetching app from market: comments
2017-09-24T20:51:55+00:00 Repair info: App (comments) is not known at the marketplace.
2017-09-24T20:51:55+00:00 Repair info: Fetching app from market: configreport
2017-09-24T20:51:56+00:00 Repair info: App (configreport) is not known at the marketplace.
2017-09-24T20:51:56+00:00 Repair info: Fetching app from market: dav
2017-09-24T20:51:58+00:00 Repair info: App (dav) is not known at the marketplace.
2017-09-24T20:51:58+00:00 Repair info: Fetching app from market: federatedfilesharing
2017-09-24T20:51:59+00:00 Repair info: App (federatedfilesharing) is not known at the marketplace.
2017-09-24T20:51:59+00:00 Repair info: Fetching app from market: federation
2017-09-24T20:52:00+00:00 Repair info: App (federation) is not known at the marketplace.
2017-09-24T20:52:00+00:00 Repair info: Fetching app from market: files
2017-09-24T20:52:02+00:00 Repair info: App (files) is not known at the marketplace.
2017-09-24T20:52:02+00:00 Repair info: Fetching app from market: files_external
2017-09-24T20:52:02+00:00 Repair info: App (files_external) is not installed
2017-09-24T20:52:02+00:00 Repair info: Fetching app from market: files_pdfviewer
2017-09-24T20:52:03+00:00 Repair info:
2017-09-24T20:52:03+00:00 Repair info: Fetching app from market: files_sharing
2017-09-24T20:52:04+00:00 Repair info: App (files_sharing) is not known at the marketplace.
2017-09-24T20:52:04+00:00 Repair info: Fetching app from market: files_texteditor
2017-09-24T20:52:05+00:00 Repair info:
2017-09-24T20:52:05+00:00 Repair info: Fetching app from market: files_trashbin
2017-09-24T20:52:07+00:00 Repair info: App (files_trashbin) is not known at the marketplace.
2017-09-24T20:52:07+00:00 Repair info: Fetching app from market: files_versions
2017-09-24T20:52:08+00:00 Repair info: App (files_versions) is not known at the marketplace.
2017-09-24T20:52:08+00:00 Repair info: Fetching app from market: files_videoplayer
2017-09-24T20:52:09+00:00 Repair info: App (files_videoplayer) is not known at the marketplace.
2017-09-24T20:52:09+00:00 Repair info: Fetching app from market: firstrunwizard
2017-09-24T20:52:11+00:00 Repair info: App (firstrunwizard) is not known at the marketplace.
2017-09-24T20:52:11+00:00 Repair info: Fetching app from market: market
2017-09-24T20:52:12+00:00 Repair info:
2017-09-24T20:52:12+00:00 Repair info: Fetching app from market: notifications
2017-09-24T20:52:13+00:00 Repair info: App (notifications) is not known at the marketplace.
2017-09-24T20:52:13+00:00 Repair info: Fetching app from market: provisioning_api
2017-09-24T20:52:15+00:00 Repair info: App (provisioning_api) is not known at the marketplace.
2017-09-24T20:52:15+00:00 Repair info: Fetching app from market: systemtags
2017-09-24T20:52:16+00:00 Repair info: App (systemtags) is not known at the marketplace.
2017-09-24T20:52:16+00:00 Repair info: Fetching app from market: templateeditor
2017-09-24T20:52:17+00:00 Repair info: App (templateeditor) is not known at the marketplace.
2017-09-24T20:52:17+00:00 Repair info: Fetching app from market: updatenotification
2017-09-24T20:52:18+00:00 Repair info: App (updatenotification) is not known at the marketplace.
2017-09-24T20:52:18+00:00 Repair info: App was not updated: activity
2017-09-24T20:52:18+00:00 Repair info: App was not updated: comments
2017-09-24T20:52:18+00:00 Repair info: App was not updated: configreport
2017-09-24T20:52:18+00:00 Repair info: App was not updated: dav
2017-09-24T20:52:18+00:00 Repair info: App was not updated: federatedfilesharing
2017-09-24T20:52:18+00:00 Repair info: App was not updated: federation
2017-09-24T20:52:18+00:00 Repair info: App was not updated: files
2017-09-24T20:52:18+00:00 Repair info: App was not updated: files_external
2017-09-24T20:52:18+00:00 Repair info: App was not updated: files_pdfviewer
2017-09-24T20:52:18+00:00 Repair info: App was not updated: files_sharing
2017-09-24T20:52:18+00:00 Repair info: App was not updated: files_texteditor
2017-09-24T20:52:18+00:00 Repair info: App was not updated: files_trashbin
2017-09-24T20:52:18+00:00 Repair info: App was not updated: files_versions
2017-09-24T20:52:18+00:00 Repair info: App was not updated: files_videoplayer
2017-09-24T20:52:18+00:00 Repair info: App was not updated: firstrunwizard
2017-09-24T20:52:18+00:00 Repair info: App was not updated: market
2017-09-24T20:52:18+00:00 Repair info: App was not updated: notifications
2017-09-24T20:52:18+00:00 Repair info: App was not updated: provisioning_api
2017-09-24T20:52:18+00:00 Repair info: App was not updated: systemtags
2017-09-24T20:52:18+00:00 Repair info: App was not updated: templateeditor
2017-09-24T20:52:18+00:00 Repair info: App was not updated: updatenotification
2017-09-24T20:52:18+00:00 Updating database schema
2017-09-24T20:52:22+00:00 Updated database
2017-09-24T20:52:22+00:00 Updating <federatedfilesharing> ...
2017-09-24T20:52:22+00:00 Updated <federatedfilesharing> to 0.3.1
2017-09-24T20:52:22+00:00 Updating <dav> ...
2017-09-24T20:52:23+00:00 Updated <dav> to 0.3.0
2017-09-24T20:52:23+00:00 Updating <files_sharing> ...
2017-09-24T20:52:24+00:00 Updated <files_sharing> to 0.10.1
2017-09-24T20:52:24+00:00 Updating <files_trashbin> ...
2017-09-24T20:52:24+00:00 Updated <files_trashbin> to 0.9.1
2017-09-24T20:52:24+00:00 Updating <notifications> ...
2017-09-24T20:52:24+00:00 Updated <notifications> to 0.3.1
2017-09-24T20:52:24+00:00 Repair step: Repair mime types
2017-09-24T20:52:24+00:00 Repair step: Generate ETags for file where no ETag is present.
2017-09-24T20:52:24+00:00 Repair info: ETags have been fixed for 0 files/folders.
2017-09-24T20:52:24+00:00 Repair step: Clean tags and favorites
2017-09-24T20:52:24+00:00 Repair info: 0 tags of deleted users have been removed.
2017-09-24T20:52:24+00:00 Repair info: 0 tags for delete files have been removed.
2017-09-24T20:52:24+00:00 Repair info: 0 tag entries for deleted tags have been removed.
2017-09-24T20:52:24+00:00 Repair info: 0 tags with no entries have been removed.
2017-09-24T20:52:24+00:00 Repair step: Drop old database tables
2017-09-24T20:52:24+00:00 Drop old database tables
2017-09-24T20:52:24+00:00
                                                    2017-09-24T20:52:24+00:00  Done
 28/28 [============================] 100%2017-09-24T20:52:24+00:00
2017-09-24T20:52:24+00:00 Repair step: Drop old background jobs
2017-09-24T20:52:24+00:00 Repair step: Remove getetag entries in properties table
2017-09-24T20:52:24+00:00 Repair info: Removed 0 unneeded "{DAV:}getetag" entries from properties table.
2017-09-24T20:52:24+00:00 Repair step: Repair outdated OCS IDs
2017-09-24T20:52:24+00:00 Repair step: Repair invalid shares
2017-09-24T20:52:24+00:00 Repair step: Remove old share propagation app entries
2017-09-24T20:52:24+00:00 Repair step: Fix permissions so avatars can be stored again
2017-09-24T20:52:24+00:00 Fix permissions so avatars can be stored again
                                                    2017-09-24T20:52:24+00:00  Done
 2/2 [============================] 100%2017-09-24T20:52:24+00:00
2017-09-24T20:52:24+00:00 Repair step: Move user avatars outside the homes to the new location
2017-09-24T20:52:24+00:00 Move user avatars outside the homes to the new location
                                                    2017-09-24T20:52:25+00:00  Done
 3/3 [============================] 100%2017-09-24T20:52:25+00:00
2017-09-24T20:52:25+00:00 Repair step: Remove shares of a users root folder
2017-09-24T20:52:25+00:00 Repair step: Repair unmerged shares
2017-09-24T20:52:25+00:00 Starting code integrity check...
2017-09-24T20:52:38+00:00 Finished code integrity check
2017-09-24T20:52:38+00:00 Update successful
2017-09-24T20:52:38+00:00 Maintenance mode is kept active
2017-09-24T20:52:38+00:00 Reset log level

I had to manually take OwnCloud out of maintenance mode. It is now reporting that I am running ownCloud 10.0.3 (stable)

For those of you who want to try here is a step by step:

1. Login to your server and navigate to the root of your OwnCloud installation
2. Make a backup of the file we're going to edit cp lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Schema/MySqlSchemaManager.php lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Schema/MySqlSchemaManager.php.bak
3. Edit lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Schema/MySqlSchemaManager.php
4. Search for getPortableTableColumnDefinition (around line 105)
5. Scroll down till you find the block @VicDeo describes (around line 181)
6. Locate (around line 185) 'default' => isset($tableColumn['default']) ? $tableColumn['default'] : null, and change it to 'default' => isset($tableColumn['default']) && !($tableColumn['default'] === 'NULL' && $tableColumn['null'] === 'YES') ? $tableColumn['default'] : null,
7. Save and close the file
8. Run ./occ upgrade
9. Profit $$$$

[enoch85]

@TheFiZi Thanks for that detailed post. Will have to try tomorrow.

[VicDeo]

Please do not post anything below before reading this summary.

Summary

1. Introduced by breaking changes in MariaDB 10.2.7 https://jira.mariadb.org/browse/MDEV-13132
2. Upstream issue https://github.com/doctrine/dbal/issues/2817
3. Upstream fix (is being discussed, not merged yet) https://github.com/doctrine/dbal/pull/2825

So, What should I do now?

Option 1. Not an option really. Do not update / wait until the fix is merged into Doctrine and adopted by ownCloud
Option 2. Not an option really. Do not use MariaDB 10.2 / switch to 10.1 or MySQL, Percona, etc
Option 3. Patch your lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Schema/MySqlSchemaManager.php:

cd /path/to/your/owncloud
wget https://gist.github.com/VicDeo/bb0689104baeb5ad2371d3fdb1a013ac/raw/04bb98e08719a04322ea883bcce7c3e778e3afe1/DoctrineMariaDB102.patch
patch -p0 < DoctrineMariaDB102.patch

Profit!

[fsantiago07044]

September 26, 2017 7:54 AM, "Viktar Dubiniuk" [e-mail] wrote:

Please do not post anything below before reading this summary.
Summary

• Introduced by breaking changes in MariaDB 10.2
• Upstream issue doctrine/dbal#2817
• Upstream fix (is being discussed, no merged yet) doctrine/dbal#2825
  So, What should I do now?
  Option 1. Not an option really. Do not update wait until the fix is merged into Doctrine and
  adopted by ownCloud
  Option 2. Not an option really. Do not use MariaDB 10.2 (switch to 10.1 or MySQL, Percona, etc)
  Option 3. Patch your lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Schema/MySqlSchemaManager.php:

cd /path/to/your/owncloud
wget
https://gist.github.com/VicDeo/bb0689104baeb5ad2371d3fdb1a013ac/raw/04bb98e08719a04322ea883bcce7c3e7
8e3afe1/DoctrineMariaDB102.patch
patch -p0 < DoctrineMariaDB102.patch
Profit!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

worked. thanks. i still see a notification error but i will submit a new issue for this. thanks.

--

Thanks,

Fabian S.

OpenPGP: 3C3FA072ACCB7AC5DB0F723455502B0EEB9070FC

[enoch85]

@VicDeo The patch did it's job, but still some log entry that you should know about:

+ patch -p0
patching file lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Schema/MySqlSchemaManager.php
+ cd /var/www/owncloud
+ check_command sudo -u www-data php occ maintenance:install --data-dir /var/ocdata --database mysql --database-name owncloud_db --database-user root --database-pass UVfT9IVpkRl7gTpsE@CrgYrKTsCe --admin-user ocadmin --admin-pass owncloud
+ eval 'sudo -u www-data php occ maintenance:install --data-dir /var/ocdata --database mysql --database-name owncloud_db --database-user root --database-pass UVfT9IVpkRl7gTpsE@CrgYrKTsCe --admin-user ocadmin --admin-pass owncloud'
++ sudo -u www-data php occ maintenance:install --data-dir /var/ocdata --database mysql --database-name owncloud_db --database-user root --database-pass UVfT9IVpkRl7gTpsE@CrgYrKTsCe --admin-user ocadmin --admin-pass owncloud
ownCloud is not installed - only a limited number of commands are available
{"reqId":"1qHmatQhy9i5D0Hqi4j3","level":3,"time":"2017-09-26T17:15:51+00:00","remoteAddr":"","user":"--","app":"PHP","method":"--","url":"--","message":"Undefined variable: user at \/var\/www\/owncloud\/lib\/public\/Util.php#193"}
ownCloud was successfully installed
{"reqId":"1qHmatQhy9i5D0Hqi4j3","level":3,"time":"2017-09-26T17:15:53+00:00","remoteAddr":"","user":"--","app":"PHP","method":"--","url":"--","message":"chmod(): No such file or directory at \/var\/www\/owncloud\/lib\/private\/Log\/Owncloud.php#122"}

[PVince81]

Assigning to @VicDeo to keep track of upstream fix.

[PVince81]

Moving to "planned" for the next release, 10.0.5.

Reason is that unit tests are failing, so it's not only about bringing in the updated DBAL stuff but also fixing additional issues in OC to make it properly compatible.

[VicDeo]

@PVince81 failing tests are not related to OC. It seems all default values are quoted in MariaDB now, not only null.
E.g. DEFAULT -1 is stored as '-1' . When diff is computed '-1' <> -1 and it produces alter table statement with DEFAULT -1 once again.
It makes no harm in comparison with NOT NULL DEFAULT NULL.
But should be fixed anyway...

[enoch85]

It makes no harm in comparison with NOT NULL DEFAULT NULL.
But should be fixed anyway...

So maybe apply the patch in 10.0.4 and fix the quoting in 10.0.5?

[PVince81]

That is a possibility yes, but we need to make sure our CI doesn't break then. @VicDeo will look for a solution.

[PVince81]

@enoch85 ok, we found a solution. This will be in 10.0.4 RC1 that will be released in the coming days. Would be cool if you could test it with your VM once it's there.

[enoch85]

@PVince81 If I find the time I will.

[clayrisser]

I'm still getting this error

[VicDeo]

@jamrizzi see above "This will be in 10.0.4 RC1"
Are you running ownCloud 10.0.4 RC1 where it is fixed?

[PVince81]

10.0.4 RC1 hasn't been released yet

[clayrisser]

I'm running it from the most recent docker container.

[PVince81]

@jamrizzi link to the docker container ? what version of OC does it contain ?

[clayrisser]

https://hub.docker.com/_/owncloud/

I'm using latest

[PVince81]

What version are you seeing in the settings page ? It is likely to be 10.0.3 from the release tarballs, not stable10 nor daily

[clayrisser]

What version should I use. Currently, I reverted back to 9, but that's not ideal.

[VicDeo]

@jamrizzi the fact is that none of PHP applications that use Doctrine library as a database abstraction layer are compatible with MariaDB 10.2.7 and above due to breaking changes introduced in this MariaDB version.

ownCloud 10.0.4 has a workaround to support MariaDB 10.2.7+ while Doctrine still has no support for it at the moment. It is being discussed for a while in Doctrine and has been not added to it's codebase yet.

See https://github.com/owncloud/core/issues/28695#issuecomment-332174798 for details (including an easy hotfix).

[clayrisser]

I tried postgres and sqlite, and they didn't work either. It might have been a different issue with them though. I'll try an older database.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.