Openrefine: Extend Wikidata extension to support arbitrary Wikibase instances

[susannaanas]

Roadmap for this feature (copied from @wetneb's Aug 13, 2018 comment https://github.com/OpenRefine/OpenRefine/issues/1640#issuecomment-412481683):

• [ ] Work with the MediaWiki developers to have the MediaWiki API endpoint expose configuration of extensions (such as Wikibase, Quality Constraints). This is a long term goal, I expect it will take ages so it should not be blocking. Task: https://phabricator.wikimedia.org/T155155
• [x] Draft and document an initial version of the manifest format. The format should include versioning so that we can switch to something lighter once MediaWiki exposes more info. Task: https://phabricator.wikimedia.org/T197588
• [x] Write Java classes to represent these manifests (with versioned implementation), with serialization and storage in OR's preferences. Each Wikibase instance should be linked to a default reconciliation service, used in the schema UI. The registry key should be something stable, such as the MediaWiki API endpoint or the manifest URI.
• [x] Create UI to list known instances, add a new one by providing the URL of a manifest.
• [x] Change schema serialization to include a mention of the Wikibase instance (not the full manifest). When not provided, fallback to Wikidata.
• [x] Percolate the instance through schema evaluation, quality assurance and editing.
• [x] Change the UI of the schema editor to let the user choose which instance to use.
• [x] Cleanup mentions to Wikidata -> Wikibase in the source, UI and translations.

Questions to be resolved:

• which key to use for the instances? Because the workflows must be reproducible, the manifest URL must appear in the operations, so using only the MediaWiki API URL does not seem to be enough. But using the manifest URL means pinning down the manifest version (including constraint configuration, and other things that might change fairly regularly).
• do we store only one overlay model or parallel schema for different instances? leaning to the former for simplicity of the UI.

[wetneb]

Right now it does not, but yes it should!

[thadguidry]

@susannaanas Is it a super urgent need ? Curious, Do you know of how many other folks that would benefit from this enhancement other than you ?

[susannaanas]

I would find use for it immediately, but I would not describe it as super urgent. There is a growing number of Wikibase instances, listed in this Wikibase https://wikibase-registry.wmflabs.org/. I could not do it myself, but I wonder if the component could be configured by a skilled developer to point to another Wikibase?

[wetneb]

@susannaanas it's a significant amount of work to do it properly. The first step would be to make the reconciliation interface run on any Wikibase, then create a notion of "profile of Wikibase instance" which contains all the information necessary to run the extension (MediaWiki API, reconciliation API, settings of WikibaseQualityConstraints, EditGroups…) and then forward that everywhere where this is needed.

[susannaanas]

Sounds much more complex than I imagined!

BTW, I think also Structured Data for Commons qualifies as one such Wikibase.

[wetneb]

sure, and that will need even more adaptations because the further tweaks in the data model (ids of the pages to edit will have a new format, but properties ids and items will be borrowed from Wikidata…)

[wetneb]

for reference, this is a feature request I have heard quite often (especially at the Wikimedia Hackathon) so it would be good to keep this in mind as a mid-term goal.

[wetneb]

@susannaanas it should now be easier to run the reconciliation interface for other Wikibase instances. I have added some docs and a Dockerfile at https://github.com/wetneb/openrefine-wikibase

[susannaanas]

Wow, we'll start testing it (soonish)!

Thanks!

[wetneb]

Roadmap for this feature:

_[copied to main issue body above]_

Questions to be resolved:

• which key to use for the instances? Because the workflows must be reproducible, the manifest URL must appear in the operations, so using only the MediaWiki API URL does not seem to be enough. But using the manifest URL means pinning down the manifest version (including constraint configuration, and other things that might change fairly regularly).
• do we store only one overlay model or parallel schema for different instances? leaning to the former for simplicity of the UI.

[ettorerizza]

Interesting diagram to better understand the current situation: https://github.com/HeardLibrary/linked-data/blob/master/wikibase/interaction-diagram.png

[msaby]

By the way : French national agencies will use wikidata for their prototype of national authorities file. So it could be useful for us

[wetneb]

Yes, and OCLC is experimenting with WIkibase too: https://www.oclc.org/research/publications/2019/oclcresearch-creating-library-linked-data-with-wikibase-project-passage.html

[belett]

Hi any update on this?

[wetneb]

No, this is still in the same state.

As much as I would like to solve this issue (and many others related to the Wikidata extension), I cannot spend all my time on Wikidata integration in OpenRefine - many other aspects of OpenRefine need my attention too! My hope was that the Wikibase community would get involved in this, but so far this has not really happened yet.

If anyone wants to work on this, I am available to help them.