Semanticmediawiki: rebuildData.php gives error: BIGINT UNSIGNED value is out of range in smw_prop_stats.usage_count

Created on 7 Feb 2021  路  5Comments  路  Source: SemanticMediaWiki/SemanticMediaWiki

Setup

  • SMW version: 3.2.2
  • MW version: 1.35.1
  • PHP version: 7.3
  • DB system (MySQL, Blazegraph, etc.) and version: MariaDB 10.3.27

Issue

I am upgrading SMW from 2.5 to 3.2.2 and upon completion the wiki came up saying I was missing an upgrade key. I ran populateHashField.php which got the wiki going but the SMW elements were blank. I thought maybe I have to rebuild the data, which I did by running rebuildData.php however this fails half way through each time, with or without --auto-recovery.

I had already run the usual maintenance/update.php so I tried SMW's setupStore.php which seemed to output much the same as update.php did, all of which completed without any problem. However I still can't get rebuildData.php to complete and the copy of the wiki I am testing this on still appears to be missing data in the SMW properties.

The message is long and a bit difficult to decipher so I included it all:

Rebuilding semantic data ...
   ... selecting 1 to 13793 IDs ...
   ... updating ...                                     1492 / 13798 ( 11%)


Wikimedia\Rdbms\DBTransactionStateError from line 1501 of /../w/includes/libs/rdbms/database/Database.php: Cannot execute query from SMW\SQLStore\PropertyStatisticsStore::setUsageCount while transaction status is ERROR
#0 /../w/includes/libs/rdbms/database/Database.php(1220): Wikimedia\Rdbms\Database->assertQueryIsCurrentlyAllowed('UPDATE  `smw_pr...', 'SMW\\SQLStore\\Pr...')
#1 /../w/includes/libs/rdbms/database/Database.php(2457): Wikimedia\Rdbms\Database->query('UPDATE  `smw_pr...', 'SMW\\SQLStore\\Pr...', 128)
#2 /../w/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->update('`smw_prop_stats...', Array, 'p_id = 13209', 'SMW\\SQLStore\\Pr...', Array)
#3 /../w/includes/libs/rdbms/database/DBConnRef.php(375): Wikimedia\Rdbms\DBConnRef->__call('update', Array)
#4 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Connection/Database.php(536): Wikimedia\Rdbms\DBConnRef->update('smw_prop_stats', Array, Array, 'SMW\\SQLStore\\Pr...', Array)
#5 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php(210): SMW\MediaWiki\Connection\Database->update('smw_prop_stats', Array, Array, 'SMW\\SQLStore\\Pr...')
#6 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php(118): SMW\SQLStore\PropertyStatisticsStore->setUsageCount(13209, Array)
#7 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php(164): SMW\SQLStore\PropertyStatisticsStore->addToUsageCount(13209, -1)
#8 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyTableUpdater.php(97): SMW\SQLStore\PropertyStatisticsStore->addToUsageCounts(Array)
#9 /../w/extensions/SemanticMediaWiki/src/SQLStore/SQLStoreUpdater.php(374): SMW\SQLStore\PropertyTableUpdater->update(1493, Object(SMW\Parameters))
#10 /../w/extensions/SemanticMediaWiki/src/SQLStore/SQLStoreUpdater.php(226): SMW\SQLStore\SQLStoreUpdater->doFlatDataUpdate(Object(SMW\SemanticData))
#11 /../w/extensions/SemanticMediaWiki/src/SQLStore/SQLStore.php(316): SMW\SQLStore\SQLStoreUpdater->doDataUpdate(Object(SMW\SemanticData))
#12 /../w/extensions/SemanticMediaWiki/src/Store.php(234): SMW\SQLStore\SQLStore->doDataUpdate(Object(SMW\SemanticData))
#13 /../w/extensions/SemanticMediaWiki/src/DataUpdater.php(415): SMW\Store->updateData(Object(SMW\SemanticData))
#14 /../w/extensions/SemanticMediaWiki/src/DataUpdater.php(294): SMW\DataUpdater->updateData()
#15 /../w/extensions/SemanticMediaWiki/src/DataUpdater.php(198): SMW\DataUpdater->runUpdate()
#16 /../w/extensions/SemanticMediaWiki/src/ParserData.php(454): SMW\DataUpdater->doUpdate()
#17 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(303): SMW\ParserData->updateStore()
#18 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(216): SMW\MediaWiki\Jobs\UpdateJob->updateStore(Object(SMW\ParserData))
#19 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(137): SMW\MediaWiki\Jobs\UpdateJob->parse_content()
#20 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(94): SMW\MediaWiki\Jobs\UpdateJob->doUpdate()
#21 /../w/extensions/SemanticMediaWiki/src/SQLStore/Rebuilder/Rebuilder.php(239): SMW\MediaWiki\Jobs\UpdateJob->run()
#22 /../w/extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(466): SMW\SQLStore\Rebuilder\Rebuilder->rebuild(1493)
#23 /../w/extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(383): SMW\Maintenance\DataRebuilder->doUpdateById(1493)
#24 /../w/extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(222): SMW\Maintenance\DataRebuilder->rebuildAll()
#25 /../w/extensions/SemanticMediaWiki/maintenance/rebuildData.php(198): SMW\Maintenance\DataRebuilder->rebuild()
#26 /../w/maintenance/doMaintenance.php(107): SMW\Maintenance\rebuildData->execute()
#27 /../w/extensions/SemanticMediaWiki/maintenance/rebuildData.php(294): require_once('/srv/web/mediaw...')
#28 {main}



md5-f2d28094e82d112698baa5bcfca1e0e4



Wikimedia\Rdbms\DBQueryError from line 1699 of /../w/includes/libs/rdbms/database/Database.php: Error 1690: BIGINT UNSIGNED value is out of range in '`keenwiki`.`smw_prop_stats`.`usage_count` - 1' (localhost)
Function: SMW\SQLStore\PropertyStatisticsStore::addToUsageCount
Query: UPDATE  `smw_prop_stats` SET usage_count = usage_count - 1,null_count = null_count - 0 WHERE p_id = 13209



md5-f2d28094e82d112698baa5bcfca1e0e4



#0 /../w/includes/libs/rdbms/database/Database.php(1323): Wikimedia\Rdbms\Database->getQueryException('BIGINT UNSIGNED...', 1690, 'UPDATE  `smw_pr...', 'SMW\\SQLStore\\Pr...')
#1 /../w/includes/libs/rdbms/database/Database.php(1223): Wikimedia\Rdbms\Database->executeQuery('UPDATE  `smw_pr...', 'SMW\\SQLStore\\Pr...', 128)
#2 /../w/includes/libs/rdbms/database/Database.php(2457): Wikimedia\Rdbms\Database->query('UPDATE  `smw_pr...', 'SMW\\SQLStore\\Pr...', 128)
#3 /../w/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->update('`smw_prop_stats...', Array, 'p_id = 13209', 'SMW\\SQLStore\\Pr...', Array)
#4 /../w/includes/libs/rdbms/database/DBConnRef.php(375): Wikimedia\Rdbms\DBConnRef->__call('update', Array)
#5 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Connection/Database.php(536): Wikimedia\Rdbms\DBConnRef->update('smw_prop_stats', Array, Array, 'SMW\\SQLStore\\Pr...', Array)
#6 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php(112): SMW\MediaWiki\Connection\Database->update('smw_prop_stats', Array, Array, 'SMW\\SQLStore\\Pr...')
#7 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php(164): SMW\SQLStore\PropertyStatisticsStore->addToUsageCount(13209, -1)
#8 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyTableUpdater.php(97): SMW\SQLStore\PropertyStatisticsStore->addToUsageCounts(Array)
#9 /../w/extensions/SemanticMediaWiki/src/SQLStore/SQLStoreUpdater.php(374): SMW\SQLStore\PropertyTableUpdater->update(1493, Object(SMW\Parameters))
#10 /../w/extensions/SemanticMediaWiki/src/SQLStore/SQLStoreUpdater.php(226): SMW\SQLStore\SQLStoreUpdater->doFlatDataUpdate(Object(SMW\SemanticData))
#11 /../w/extensions/SemanticMediaWiki/src/SQLStore/SQLStore.php(316): SMW\SQLStore\SQLStoreUpdater->doDataUpdate(Object(SMW\SemanticData))
#12 /../w/extensions/SemanticMediaWiki/src/Store.php(234): SMW\SQLStore\SQLStore->doDataUpdate(Object(SMW\SemanticData))
#13 /../w/extensions/SemanticMediaWiki/src/DataUpdater.php(415): SMW\Store->updateData(Object(SMW\SemanticData))
#14 /../w/extensions/SemanticMediaWiki/src/DataUpdater.php(294): SMW\DataUpdater->updateData()
#15 /../w/extensions/SemanticMediaWiki/src/DataUpdater.php(198): SMW\DataUpdater->runUpdate()
#16 /../w/extensions/SemanticMediaWiki/src/ParserData.php(454): SMW\DataUpdater->doUpdate()
#17 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(303): SMW\ParserData->updateStore()
#18 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(216): SMW\MediaWiki\Jobs\UpdateJob->updateStore(Object(SMW\ParserData))
#19 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(137): SMW\MediaWiki\Jobs\UpdateJob->parse_content()
#20 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(94): SMW\MediaWiki\Jobs\UpdateJob->doUpdate()
#21 /../w/extensions/SemanticMediaWiki/src/SQLStore/Rebuilder/Rebuilder.php(239): SMW\MediaWiki\Jobs\UpdateJob->run()
#22 /../w/extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(466): SMW\SQLStore\Rebuilder\Rebuilder->rebuild(1493)
#23 /../w/extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(383): SMW\Maintenance\DataRebuilder->doUpdateById(1493)
#24 /../w/extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(222): SMW\Maintenance\DataRebuilder->rebuildAll()
#25 /../w/extensions/SemanticMediaWiki/maintenance/rebuildData.php(198): SMW\Maintenance\DataRebuilder->rebuild()
#26 /../w/maintenance/doMaintenance.php(107): SMW\Maintenance\rebuildData->execute()
#27 /../w/extensions/SemanticMediaWiki/maintenance/rebuildData.php(294): require_once('/srv/web/mediaw...')
#28 {main}
[52e992da63a5b67df4e6f3f0] [no req]   Wikimedia\Rdbms\DBTransactionError from line 1519 of /../w/includes/libs/rdbms/database/Database.php: Explicit transaction still active. A caller may have caught an error. Open transactions: sql/transaction/update
Backtrace:
#0 /../w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1642): Wikimedia\Rdbms\Database->assertNoOpenTransactions()
#1 /../w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2150): Wikimedia\Rdbms\LoadBalancer->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#2 /../w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1661): Wikimedia\Rdbms\LoadBalancer->forEachOpenMasterConnection(Closure)
#3 /../w/includes/libs/rdbms/lbfactory/LBFactory.php(244): Wikimedia\Rdbms\LoadBalancer->approveMasterChanges(array, string, integer)
#4 /../w/includes/libs/rdbms/lbfactory/LBFactorySimple.php(133): Wikimedia\Rdbms\LBFactory->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer, string, array)
#5 /../w/includes/libs/rdbms/lbfactory/LBFactory.php(246): Wikimedia\Rdbms\LBFactorySimple->forEachLB(Closure, array)
#6 /../w/includes/libs/rdbms/lbfactory/LBFactory.php(305): Wikimedia\Rdbms\LBFactory->forEachLBCallMethod(string, array)
#7 /../w/maintenance/doMaintenance.php(135): Wikimedia\Rdbms\LBFactory->commitMasterChanges(string)
#8 /../w/extensions/SemanticMediaWiki/maintenance/rebuildData.php(294): require_once(string)
#9 {main}
bug
Source
picture Malvineous
👀1

Most helpful comment

I tried altering extensions/SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php at line 164 where the error was, and added an extra if condition around it:

+if ($addition > 0) {
     $success = $this->addToUsageCount( $pid, $addition ) && $success;
+}

This fixed the error and allowed the process to complete. Not sure exactly what it has done or the implications of skipping that line for the negative values.

picture Malvineous on 7 Feb 2021
👍2 🎉1

All 5 comments

@Malvineous Did you follow these suggestions? https://www.semantic-mediawiki.org/wiki/Help:Upgrade/Upgrade_from_SMW_2.5_to_SMW_3.0

akuckartz picture akuckartz on 7 Feb 2021

I did, but I'm not sure how accurate the steps are because it tells you to run php extensions/SemanticMediaWiki/maintenance/setupStore.php -v but when I do that it complains Unexpected option v! I ran it without the -v which is where it came up with the same output as the end of update.php did, so I think update.php runs setupStore.php anyway. That page was where I found to run populateHashField.php to fix the update hash error, and everything works up until I get to the rebuildData.php step which fails with the above exception.

Malvineous picture Malvineous on 7 Feb 2021

I tried altering extensions/SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php at line 164 where the error was, and added an extra if condition around it:

+if ($addition > 0) {
     $success = $this->addToUsageCount( $pid, $addition ) && $success;
+}

This fixed the error and allowed the process to complete. Not sure exactly what it has done or the implications of skipping that line for the negative values.

Malvineous picture Malvineous on 7 Feb 2021
👍2 🎉1

I did, but I'm not sure how accurate the steps are because it tells you to run php extensions/SemanticMediaWiki/maintenance/setupStore.php -v but when I do that it complains Unexpected option v!

I ran into the same problem. Documentation bug?

cure picture cure on 15 Apr 2021

Indeed, "setupStore.php" does not have a v option. Sorry for this. Amended in the docu.

kghbln picture kghbln on 15 Apr 2021
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

WolfgangFahl picture WolfgangFahl  路  3Comments
OyvindLGjesdal picture OyvindLGjesdal  路  3Comments
mwjames picture mwjames  路  3Comments
JeroenDeDauw picture JeroenDeDauw  路  3Comments
Larivact picture Larivact  路  4Comments