Semanticmediawiki: Improved property change propagation

[mwjames]

Issue

As noted in [0] the process is to ensure that any relevant change to a property specification is being propagated and related entities are updated to reflect those changes (type, unit etc.). In recent MW releases relying on a sequential processing of updates and job queue is no longer guaranteed for processes that do not make that order explicit due to the deferring update logic.

Aside from the update sequence, an elevating factor is the amount of possible updates required and was already noted in [1] with "FIXME: this will kill large wikis! Use incremental updates!".

The issue is that once the type (or any other declarative property attribute) is changed, using Store::getAllPropertySubjects can no longer create a list of subjects that match the old specification (a new type means new table and storage etc.), so fetching all members has to happen before a new specification is "activated".

Approach

The solve the outlined insufficiencies, a new process is proposed where in case of a change, the property gets locked with the update being suspended until a ChangePropagationDispatchJob is scheduled which would first execute Store::getAllPropertySubjects (to find all related entities before the update) before updating the property with the new property specification.

• PropertyChangePropagationNotifier will detect any change to the listed declarative properties and suspend the update by setting property _CHGPRO and stores the new specification as serialized JSON blob
• A detected change will trigger a ChangePropagationDispatchJob
• The property page is being locked with the help of PermissionPthValidator and EditProtectionValidator and a notice will be shown as long as property _CHPRO remains part of the SemanticData annotation
• When ChangePropagationDispatchJob is run:
  
  
  
  • Fetch Store::getAllPropertySubjects
  
  
  • Schedule updates via ChangePropagationUpdateJob for all selected subjects and once finished
  
  
  • Schedule a new ChangePropagationUpdateJob to add the new SemanticData from the serialized _CHGPRO blob
  
  

The serialized JSON blob is stored with property _CHGPRO because attempts failed when using it as simple job parameter with something like [2] due to limited space job parameters can occupy and failed on a unserialized error during job execution.

[0] https://www.semantic-mediawiki.org/wiki/Property_type_change_propagation
[1] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/1.8.0.5/includes/SMW_ParseData.php#L334
[2] https://sandbox.semantic-mediawiki.org/wiki/Attribut:P106

[verdy-p]

Still no progress. This bug continues to affect more and more category page when they are added or recategorized.
And still no simple way to fix that without using dangerous admin-only commands that bypass all safety checks beside being restricted to admins (and on many wikis, the few authorized site-admins can't handle that trafic).

As well you still don't provide anyway to track the list of pages locked by this status: it's impossible to determine them with any standard Mediawiki query or in SemanticWiki-specific queries (there's no special page or no search option supporting this search, so the affetfed pages are found randomly and always isolately: this is a sever loss of time for site-admins).

[verdy-p]

Note also: the search API in SemanticMediawki fails with a fatal error:

https://translatewiki.net/wiki/Special:SearchByProperty

gives only:

[e4fa89a5f02f8089414446c0] 2020-09-07 08:55:08: Fatal exception of type "Error"

with no other info (the only thing that changes each time is the unique bug id at start, and the timestamp).

The "same" bug occurs as well on:

• https://translatewiki.net/wiki/Special:WantedProperties
• https://translatewiki.net/wiki/Special:Properties

And the "Processing error list" (listed in "Special pages" for maintenance) remains also empty:

• https://translatewiki.net/w/i.php?title=Special:Ask&q=%5B%5BHas+processing+error+text%3A%3A%2B%5D%5D&po=%3FHas+improper+value+for%7C%3FHas+processing+error+text&p=class%3Dsortable-20smwtable-2Dstriped-20smwtable-2Dclean%2Fsep%3Dul&eq=no&limit=20&bTitle=processingerrorlist&bHelp=smw-processingerrorlist-helplink&bMsg=smw-processingerrorlist-intro

[JeroenDeDauw]

@verdy-p can you file a new issue? This one is closed and it is not obvious to me how what you wrote relates to what James wrote

[verdy-p]

This is the same bug submitted multiple times (even separately), and that you systematically close without fixing it.

There's a fact: editing standard categories in Mediawiki causes the (spurious) "special property" named "Change propagation" to be added instantly on ALL categories after EACH of their edit (even just their creation) and causing the page to become instantly locked forever.

And we cannot remove at all this "special property" which SemanticMediaWiki systematically adds, but frequently forgets to remove (the associated job will never run, or it has already run before the special property got added).

This is not something new, it has been signaled again and again, since years (the list of affected pages that are locked because of this whose last effective edit or creation is now very old according to their page history).

[JeroenDeDauw]

Please create a new report or point me to one that I closed.

Or you can just comment more here, which I will ignore, and no one else will act upon either.

[verdy-p]

See also https://translatewiki.net/wiki/Translating:Localisation_for_developers.

NOTHING has changed with your fix, you did not not fix anything. And the bug continues with some pages added randomly to the list of locked page.

This is definitely a race condition between the time the jobn is setup in one thread (but the status is kept in a cache that will be stored later) and the background job that can run and terminate earlier, trying to delete the ```[[Change propagation::+]]```` pseudo-property for this page from the database '''before''' the status (kept in a cache of the main process) was actually commited to the database by the main thread. It seems to occur because the main thread may want to service other incoming requests from other visitors of the website.

[akuckartz]

@verdy-p Please create a new issue, or point to another one. Just linking to other issue trackers is not sufficient.

[verdy-p]

https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/4344 (opened since almost the same period on 27 Oct 2019)