Openrefine: Get ALL availble properties given a Wikidata Q id

[sharun-s]

Wikidata properties for the same type (eg person) differ quite a bit. One might have father. Another might not but might have child etc. Would be useful to pull in (as columns) whatever is available, as in ALL properties available, after completing reconciliation? Currently I do this one prop at a time.

I can see there is a wikidata extension in the UI but I can't find any help doc for that or whether it was meant to be used for this case. Can it be used to pull all available props and if not I'd be grateful for any docs or pointers of how to do this. Thanks.

[wetneb]

Hi!

First, let's make sure you are aware of the "Add columns by reconciled values", which makes it possible to pull more than one property at a time:

If you want to retrieve all properties on items, you can do this by calling the Wikidata API directly, using "Add column by fetching URLs":
"https://www.wikidata.org/w/api.php?action=wbgetentities&format=json&ids="+cell.recon.match.id

You can then retrieve the full properties by parsing the returned JSON.

[sharun-s]

Hey thanks. Yup sorry I mistyped that. Am aware of the feature to select properties.
I meant rather than picking columns, a 'get all available props' feature would be useful.

I have been playing with that mediawiki api, the problem is for each prop, value it returns the data type maybe different - list, dict, pointer to another wikidata item etc and some additional queries/parsing needs to be done. And Refine seems to have that functionality baked in somewhere. For example if I add 'child' column based on reconciled column, and child points to another wikidata item, rather than just being a string Refine does some cool magic making that cell into a link/reconciled item.

[wetneb]

Indeed! You could in theory reuse the "Add columns from reconciled values" operation if you craft a JSON representation of this operation manually, by including there all the Wikidata pids (from P5 to P7789 as of today). You could then apply this JSON transformation from the undo/redo tab and it would fetch all available properties on all your items.

Note that this will create thousands of columns in your table, which will make OpenRefine basically unusable.

[sharun-s]

wow :)) I might actually try it to see if it reduces all the other querying and parsing I am doing. Thanks!

[thadguidry]

Wikidata does not have Types like Freebase did. Only Properties.
Its more freeform for good reason, so that any entity can have any
statement with any properties that someone sees as appropriate and
reasonable.
This allows a more flexible data capturing process, rather than being rigid
and having to assign some Types with a finite set of properties to some
entities and then realizing you have other properties!!! you need and would
have to resort to making some other Type and assigning to the entity or
expanding some existing Types. Wikidata decided NO TYPES, let's let any
entity have any properties it needs.

So some PLACE might have 2 properties, and another PLACE might have 100+
A PERSON might have only 3 properties, and yet another famous PERSON might
have 30+

Thad
https://www.linkedin.com/in/thadguidry/

On Sat, Jan 11, 2020 at 8:13 AM Antonin Delpeuch [e-mail]
wrote:

Indeed! You could in theory reuse the "Add columns from reconciled values"
operation if you craft a JSON representation of this operation manually, by
including there all the Wikidata pids (from P5 to P7789 as of today). You
could then apply this JSON transformation from the undo/redo tab and it
would fetch all available properties on all your items.

Note that this will create thousands of columns in your table, which will
make OpenRefine basically unusable.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/OpenRefine/OpenRefine/issues/2283?email_source=notifications&email_token=AAHQ2RS3BOLPLC7PRARH4WTQ5HHYBA5CNFSM4KFRZ5ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIWCZUY#issuecomment-573320403,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHQ2RSBMCVNQ5Y6QYCO3HTQ5HHYBANCNFSM4KFRZ5ZA
.

[sharun-s]

hmm...I am not sure if I am using the right api.
The wikidata API and the mediawiki api return a bunch of types snaktype, value type, data type etc

https://www.wikidata.org/wiki/Special:EntityData/Q11816.json
https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q11816&format=json&languages=en&props=claims

And then I have to do another round of querying to figure out the names of each prop and another round if that prop has a wikibase-entityid type. So it doesnt just get the values father=John Adams. It returns P22 -> Q11806
Refine seems to take care of this automagically.

[wetneb]

Yes if you want to achieve the same result via the MediaWiki API, it's not as straightforward.

To do that with the "Add columns from reconciled column", you can apply this JSON (make sure you replace MY_COLUMN by the name of your column):

[
  {
    "op": "core/extend-reconciled-data",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "baseColumnName": "MY_COLUMN",
    "endpoint": "https://tools.wmflabs.org/openrefine-wikidata/en/api",
    "identifierSpace": "http://www.wikidata.org/entity/",
    "schemaSpace": "http://www.wikidata.org/prop/direct/",
    "extension": {
      "properties": [
        {"id": "P5"},{"id": "P6"},{"id": "P7"},{"id": "P8"},{"id": "P9"},{"id": "P10"},{"id": "P11"}
      ]
    },
    "columnInsertIndex": 1,
    "description": "Extend data at index 1 based on column test"
  }
]

This will fetch properties P5 to P11, you can adapt it to fetch more properties.

[sharun-s]

Thanks a lot @wetneb this is most useful!

[wetneb]

Closing this as I don't see something to improve here, feel free to reopen if you disagree.