Semanticmediawiki: runtime error when running refreshLinks.php or rebuildall.php

Created on 9 Oct 2019  路  4Comments  路  Source: SemanticMediaWiki/SemanticMediaWiki

We just updated SMW from 3.0.2 to 3.1, and MW from 1.33.0 to 1.33.1. When we now run refreshLinks.php it gets through the refreshing redirects part but part way through refreshing links table we get the following error:

RuntimeException from line 123 of ...xxxx.../extensions/SemanticMediaWiki/src/MediaWiki/Connection/TransactionHandler.php: Trying to begin a new section transaction while SMW\SQLStore\SQLStoreUpdater::doDataUpdate is still active!

refreshLinks is run as part of rebuildall.php, which we run as a nightly cron job. It did not cause any problems before this update. I did the updates on two wikis, and both now throw this exception.

Setup and configuration

  • SMW version: 3.1
  • MW version: 1.33.1
  • PHP version: 7.2.11
  • DB system (MySQL, Blazegraph, etc.) and version: MySQL 5.7.25-log

Stack trace

[ps481774]$ php refreshL*
Refreshing redirects table.
Starting from page_id 1 of 831.
100
200
300
400
500
600
700
800
Refreshing links tables.
Starting from page_id 1 of 831.
100
[56180a3aacef6f1acaf577a5] [no req]   RuntimeException from line 123 of /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/MediaWiki/Connection/TransactionHandler.php: Trying to begin a new section transaction while SMW\SQLStore\SQLStoreUpdater::doDataUpdate is still active!
Backtrace:
#0 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/MediaWiki/Connection/Database.php(677): SMW\MediaWiki\Connection\TransactionHandler->markSectionTransaction(string)
#1 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/SQLStore/SQLStoreUpdater.php(195): SMW\MediaWiki\Connection\Database->beginSectionTransaction(string)
#2 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/SQLStore/SQLStore.php(303): SMW\SQLStore\SQLStoreUpdater->doDataUpdate(SMW\SemanticData)
#3 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/Store.php(238): SMW\SQLStore\SQLStore->doDataUpdate(SMW\SemanticData)
#4 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/DataUpdater.php(403): SMW\Store->updateData(SMW\SemanticData)
#5 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/DataUpdater.php(289): SMW\DataUpdater->updateData()
#6 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/DataUpdater.php(193): SMW\DataUpdater->runUpdate()
#7 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/ParserData.php(453): SMW\DataUpdater->doUpdate()
#8 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(296): SMW\ParserData->updateStore()
#9 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(216): SMW\MediaWiki\Jobs\UpdateJob->updateStore(SMW\ParserData)
#10 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(137): SMW\MediaWiki\Jobs\UpdateJob->parse_content()
#11 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(94): SMW\MediaWiki\Jobs\UpdateJob->doUpdate()
#12 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/SQLStore/RedirectStore.php(201): SMW\MediaWiki\Jobs\UpdateJob->run()
#13 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/SQLStore/EntityStore/EntityIdManager.php(222): SMW\SQLStore\RedirectStore->updateRedirect(integer, string, integer)
#14 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/SQLStore/RedirectUpdater.php(293): SMW\SQLStore\EntityStore\EntityIdManager->updateRedirect(integer, string, integer)
#15 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/SQLStore/SQLStoreUpdater.php(299): SMW\SQLStore\RedirectUpdater->updateRedirects(SMW\DIWikiPage)
#16 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/SQLStore/SQLStoreUpdater.php(224): SMW\SQLStore\SQLStoreUpdater->doFlatDataUpdate(SMW\SemanticData)
#17 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/SQLStore/SQLStore.php(303): SMW\SQLStore\SQLStoreUpdater->doDataUpdate(SMW\SemanticData)
#18 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/Store.php(238): SMW\SQLStore\SQLStore->doDataUpdate(SMW\SemanticData)
#19 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/Store.php(275): SMW\Store->updateData(SMW\SemanticData)
#20 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/SQLStore/RedirectUpdater.php(498): SMW\Store->clearData(SMW\DIWikiPage)
#21 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/SQLStore/RedirectUpdater.php(160): SMW\SQLStore\RedirectUpdater->moveAsRedirect(SMW\DIWikiPage, SMW\DIWikiPage, integer, integer, array)
#22 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/SQLStore/SQLStoreUpdater.php(430): SMW\SQLStore\RedirectUpdater->doUpdate(SMW\DIWikiPage, SMW\DIWikiPage, array)
#23 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/SQLStore/SQLStore.php(321): SMW\SQLStore\SQLStoreUpdater->changeTitle(Title, Title, integer, integer)
#24 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/DataUpdater.php(457): SMW\SQLStore\SQLStore->changeTitle(Title, Title, integer, integer)
#25 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/DataUpdater.php(428): SMW\DataUpdater->doUpdateUnknownRedirectTarget(SMW\SemanticData, Title)
#26 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/DataUpdater.php(397): SMW\DataUpdater->checkOnRequiredRedirectUpdate(SMW\SemanticData)
#27 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/DataUpdater.php(289): SMW\DataUpdater->updateData()
#28 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/DataUpdater.php(193): SMW\DataUpdater->runUpdate()
#29 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/ParserData.php(453): SMW\DataUpdater->doUpdate()
#30 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/MediaWiki/Hooks/LinksUpdateConstructed.php(120): SMW\ParserData->updateStore(array)
#31 /home/...xxx.../ccmdbpriv.kuality.ca/extensions/SemanticMediaWiki/src/MediaWiki/Hooks.php(815): SMW\MediaWiki\Hooks\LinksUpdateConstructed->process(LinksUpdate)
#32 /home/...xxx.../ccmdbpriv.kuality.ca/includes/Hooks.php(174): SMW\MediaWiki\Hooks->onLinksUpdateConstructed(LinksUpdate)
#33 /home/...xxx.../ccmdbpriv.kuality.ca/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#34 /home/...xxx.../ccmdbpriv.kuality.ca/includes/deferred/LinksUpdate.php(168): Hooks::run(string, array)
#35 /home/...xxx.../ccmdbpriv.kuality.ca/includes/Storage/DerivedPageDataUpdater.php(1304): LinksUpdate->__construct(Title, ParserOutput, boolean)
#36 /home/...xxx.../ccmdbpriv.kuality.ca/includes/Storage/DerivedPageDataUpdater.php(1567): MediaWiki\Storage\DerivedPageDataUpdater->getSecondaryDataUpdates(boolean)
#37 /home/...xxx.../ccmdbpriv.kuality.ca/includes/page/WikiPage.php(2134): MediaWiki\Storage\DerivedPageDataUpdater->doSecondaryDataUpdates(array)
#38 /home/...xxx.../ccmdbpriv.kuality.ca/maintenance/refreshLinks.php(275): WikiPage->doSecondaryDataUpdates(array)
#39 /home/...xxx.../ccmdbpriv.kuality.ca/maintenance/refreshLinks.php(198): RefreshLinks::fixLinksFromArticle(integer, boolean)
#40 /home/...xxx.../ccmdbpriv.kuality.ca/maintenance/refreshLinks.php(83): RefreshLinks->doRefreshLinks(integer, boolean, string, boolean, boolean)
#41 /home/...xxx.../ccmdbpriv.kuality.ca/maintenance/doMaintenance.php(96): RefreshLinks->execute()
#42 /home/...xxx.../ccmdbpriv.kuality.ca/maintenance/refreshLinks.php(491): require_once(string)
#43 {main}

Steps to reproduce

run refreshLinks.php

bug store-sql
Source
picture kuality

Most helpful comment

This relates to #4082 (and #3780) and because of the refactoring in
the SQLStore, I missed a place and the fact that no test is in place
to check a particular condition this went unnoticed.

The issue itself is hard to test (even for integration tests) and only
manifest itself under certain conditions that aren't easily be
reproducible (see comments in #4082).

If time permits, I'll try to prepare a patch file during the weekend
or next week since I'm hard pressed for time.

On 10/10/19, H. C. Kruse notifications@github.com wrote:

I am seeing the same error with:

  • SMW version: 3.1.0
  • MW version: 1.33.0
  • PHP version: 7.2.23
  • DB system: MariaDB 10.4.8

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/4323#issuecomment-540579592

picture mwjames on 10 Oct 2019
❤2

All 4 comments

I am seeing the same error with:

  • SMW version: 3.1.0
  • MW version: 1.33.0
  • PHP version: 7.2.23
  • DB system: MariaDB 10.4.8
octfx picture octfx on 10 Oct 2019

This relates to #4082 (and #3780) and because of the refactoring in
the SQLStore, I missed a place and the fact that no test is in place
to check a particular condition this went unnoticed.

The issue itself is hard to test (even for integration tests) and only
manifest itself under certain conditions that aren't easily be
reproducible (see comments in #4082).

If time permits, I'll try to prepare a patch file during the weekend
or next week since I'm hard pressed for time.

On 10/10/19, H. C. Kruse notifications@github.com wrote:

I am seeing the same error with:

  • SMW version: 3.1.0
  • MW version: 1.33.0
  • PHP version: 7.2.23
  • DB system: MariaDB 10.4.8

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/4323#issuecomment-540579592

mwjames picture mwjames on 10 Oct 2019
❤2

If time permits, I'll try to prepare a patch file during the weekend
or next week since I'm hard pressed for time.

The attached patch file (hereby required changes) should fix the issue.

On 10/11/19, James HK jamesin.hongkong.1@gmail.com wrote:

This relates to #4082 (and #3780) and because of the refactoring in
the SQLStore, I missed a place and the fact that no test is in place
to check a particular condition this went unnoticed.

The issue itself is hard to test (even for integration tests) and only
manifest itself under certain conditions that aren't easily be
reproducible (see comments in #4082).

If time permits, I'll try to prepare a patch file during the weekend
or next week since I'm hard pressed for time.

On 10/10/19, H. C. Kruse notifications@github.com wrote:

I am seeing the same error with:

  • SMW version: 3.1.0
  • MW version: 1.33.0
  • PHP version: 7.2.23
  • DB system: MariaDB 10.4.8

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/4323#issuecomment-540579592

From ad0c40c48b67a276c5c18c4ef57dac223ad6fab5 Mon Sep 17 00:00:00 2001
From: mwjames jamesin.hongkong.1@gmail.com
Date: Sat, 12 Oct 2019 23:47:01 +0900
Subject: Use constant for update section transaction, refs 3780, 4082, 4323

diff --git a/src/SQLStore/RedirectStore.php b/src/SQLStore/RedirectStore.php
index a52c39f8d..c8c06478f 100644
--- a/src/SQLStore/RedirectStore.php
+++ b/src/SQLStore/RedirectStore.php
@@ -193,11 +193,21 @@ class RedirectStore {
}
}

-
- $canRun = $this->isCommandLineMode && !$connection->inSectionTransaction( 'SMWSQLStore3Writers::doDataUpdate' );
+ // Generally, redirect updates can be lazily run during the online processing
+ $immediateMode = false;
+
+ // #4082, #4323
+ // If possible allow an immediate execution but ensure that no section
+ // transaction is open and causes the redirect update to run before the
+ // initial transaction which otherwise could cause data inconsistencies
+ if (
+ $this->isCommandLineMode &&
+ !$connection->inSectionTransaction( SQLStore::UPDATE_TRANSACTION ) ) {
+ $immediateMode = true;
+ }

    foreach ( $jobs as $job ) {

- if ( $canRun ) {
+ if ( $immediateMode ) {
$job->run();
} else {
$job->lazyPush();
diff --git a/src/SQLStore/SQLStore.php b/src/SQLStore/SQLStore.php
index 6a79a7cf8..d5b498cf9 100644
--- a/src/SQLStore/SQLStore.php
+++ b/src/SQLStore/SQLStore.php
@@ -117,6 +117,11 @@ class SQLStore extends Store {
*/
const ID_AUXILIARY_TABLE = 'smw_object_aux';

  • /**

    • Identifies the UPDATE transaction

  • */

  • const UPDATE_TRANSACTION = 'sql/transaction/update';
    +
    /**

    • @var SQLStoreFactory
      */
      diff --git a/src/SQLStore/SQLStoreUpdater.php b/src/SQLStore/SQLStoreUpdater.php
      index 247f48ea4..4ef2e6d5c 100644
      --- a/src/SQLStore/SQLStoreUpdater.php
      +++ b/src/SQLStore/SQLStoreUpdater.php
      @@ -191,8 +191,8 @@ class SQLStoreUpdater {

      $connection = $this->store->getConnection( 'mw.db' );

  • // MW 1.33+

  • $connection->beginSectionTransaction( __METHOD__ );
  • // MW 1.33+, #3780

  • $connection->beginSectionTransaction( SQLStore::UPDATE_TRANSACTION );

    $subobjectListFinder = $this->factory->newSubobjectListFinder();

@@ -272,7 +272,7 @@ class SQLStoreUpdater {
$changeOp
] );

  • $connection->endSectionTransaction( __METHOD__ );

  • $connection->endSectionTransaction( SQLStore::UPDATE_TRANSACTION );
    }

    /**
    diff --git a/tests/phpunit/Unit/SQLStore/RedirectStoreTest.php b/tests/phpunit/Unit/SQLStore/RedirectStoreTest.php
    index 48e791ef0..f153c133c 100644
    --- a/tests/phpunit/Unit/SQLStore/RedirectStoreTest.php
    +++ b/tests/phpunit/Unit/SQLStore/RedirectStoreTest.php
    @@ -245,6 +245,68 @@ class RedirectStoreTest extends \PHPUnit_Framework_TestCase {
    $instance->updateRedirect( 42, 'Foo', NS_MAIN );
    }

  • public function testUpdateRedirect_OnCommandLine_ActiveSectionTransaction() {
    +

  • $this->connection->expects( $this->once() )
  • ->method( 'inSectionTransaction' )
  • ->with( $this->equalTo( \SMW\SQLStore\SQLStore::UPDATE_TRANSACTION ) )
  • ->will( $this->returnValue( true ) );
    +
  • $this->jobQueue->expects( $this->once() )
  • ->method( 'lazyPush' );
    +
  • $row = new \stdClass;
  • $row->ns = NS_MAIN;
  • $row->t = 'Bar';
    +
  • $this->connection->expects( $this->once() )
  • ->method( 'select' )
  • ->will( $this->returnValue( [ $row ] ) );
    +
  • $propertyTable = $this->getMockBuilder( '\SMW\SQLStore\PropertyTableDefinition' )
  • ->disableOriginalConstructor()
  • ->getMock();
    +
  • $propertyTable->expects( $this->once() )
  • ->method( 'getFields' )
  • ->will( $this->returnValue( [ 'Foo' => \SMW\SQLStore\TableBuilder\FieldType::FIELD_ID ] ) );
    +
  • $store = $this->getMockBuilder( '\SMW\SQLStore\SQLStore' )
  • ->disableOriginalConstructor()
  • ->setMethods( [ 'getPropertyTables' ] )
  • ->getMock();
    +
  • $store->setConnectionManager( $this->connectionManager );
    +
  • $store->expects( $this->once() )
  • ->method( 'getPropertyTables' )
  • ->will( $this->returnValue( [ $propertyTable ] ) );
    +
  • $store->setOption(
  • \SMW\Store::OPT_CREATE_UPDATE_JOB,
  • true
  • );
    +
  • $this->testEnvironment->addConfiguration(
  • 'smwgEnableUpdateJobs',
  • true
  • );
    +
  • $store->setOption(
  • 'smwgEnableUpdateJobs',
  • true
  • );
    +
  • $instance = new RedirectStore(
  • $store
  • );
    +
  • $instance->setCommandLineMode( true );
  • $instance->setEqualitySupportFlag( SMW_EQ_FULL );
    +
  • $instance->updateRedirect( 42, 'Foo', NS_MAIN );

  • }
    +
    public function testUpdateRedirectNotEnabled() {

    $store = $this->getMockBuilder( '\SMW\SQLStore\SQLStore' )

    diff --git a/tests/phpunit/Unit/SQLStore/SQLStoreUpdaterTest.php b/tests/phpunit/Unit/SQLStore/SQLStoreUpdaterTest.php
    index afe29d6bc..8079b0dbb 100644
    --- a/tests/phpunit/Unit/SQLStore/SQLStoreUpdaterTest.php
    +++ b/tests/phpunit/Unit/SQLStore/SQLStoreUpdaterTest.php
    @@ -4,6 +4,7 @@ namespace SMW\Tests\SQLStore;

    use SMW\DIWikiPage;
    use SMW\SQLStore\SQLStoreUpdater;
    +use SMW\SQLStore\SQLStore;
    use Title;

    /**
    @@ -185,6 +186,14 @@ class SQLStoreUpdaterTest extends \PHPUnit_Framework_TestCase {
    ->disableOriginalConstructor()
    ->getMock();

  • $database->expects( $this->once() )

  • ->method( 'beginSectionTransaction' )
  • ->with( $this->equalTo( SQLStore::UPDATE_TRANSACTION ) );
    +
  • $database->expects( $this->once() )
  • ->method( 'endSectionTransaction' )
  • ->with( $this->equalTo( SQLStore::UPDATE_TRANSACTION ) );
    +
    $propertyTableInfoFetcher = $this->getMockBuilder( '\SMW\SQLStore\PropertyTableInfoFetcher' )
    ->disableOriginalConstructor()
    ->getMock();
mwjames picture mwjames on 16 Oct 2019
🎉1

If you do not want to wait for the release of SMW 3.1.1 you can update to the latest revision of the legacy branch 3.1 [0] by specifying "3.1.x@dev" instead of "3.1.0" in your "composer.local.json" file.

[0] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/docs/INSTALL.md#installation-of-development-versions-and-release-candidates

kghbln picture kghbln on 5 Nov 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Larivact picture Larivact  路  4Comments
simontaurus picture simontaurus  路  3Comments
SteveRMann picture SteveRMann  路  4Comments
WolfgangFahl picture WolfgangFahl  路  3Comments
alex-mashin picture alex-mashin  路  4Comments