Documentation: Indexing multilingual content in Fedora and Triplestore

[Natkeeran]

This is a ticket to followup on an issue posted by Zoltán Kanász-Nagy in the islandora-dev group.

The basis of the question is how to index multilingual content in Fedora and in Triplestore.

Jsonld does iterate over the language values. Note that it does not seem to indicate language by any attribute.

http://schema.org/name  
0   
@value  "Language 1 Title"
@type   "http://www.w3.org/2001/XMLSchema#string"
1   
@value  "Language 2 Title"
@type   "http://www.w3.org/2001/XMLSchema#string"

http://schema.org/category  
0   
@value  "Language 1 Category"
@type   "http://www.w3.org/2001/XMLSchema#string"
1   
@value  "Language 2 Category"
@type   "http://www.w3.org/2001/XMLSchema#string"

If language values are available, then they get indexed in triplestore as below. Note that both subject uri's contain all values.

<http://ip:8000/node/4?_format=jsonld> | schema:name | Language 2 Old Title
<http://ip:8000/node/4?_format=jsonld> | schema:name | Language 1 Title
<http://ip:8000/node/4?_format=jsonld> | schema:telephone | 111-222-3333
<http://ip:8000/node/4?_format=jsonld> | schema:category | Language 1 Category
<http://ip:8000/node/4?_format=jsonld> | rdf:type | schema:Organization

<http://ip:8000/ta/node/4?_format=jsonld> | schema:name | Language 1 Title
<http://ip:8000/ta/node/4?_format=jsonld> | schema:name | Language 2 Title
<http://ip:8000/ta/node/4?_format=jsonld> | schema:telephone | 111-222-3333
<http://ip:8000/ta/node/4?_format=jsonld> | schema:category | Language 1 Category
<http://ip:8000/ta/node/4?_format=jsonld> | schema:category | Language 2 Category
<http://ip:8000/ta/node/4?_format=jsonld> | rdf:type | schema:Organization

Fedora does not seem to have the logic to index Multilingual content. It creates the content in the default language. Not able to update fedora entries right now.

[DiegoPino]

@Natkeeran we should check on that. I remember there was an issue with JSON-LD specs and language attribute for typed values. I see I actually implemented (or maintained, won't attribute decisions I don't even remember why i made!) languages all over the place (and remember having tested) language attributes for the field normalizer, see https://github.com/Islandora-CLAW/jsonld/blame/36f37cb2bae7467fd6d3eacabd9e4a368ea77a7f/src/Normalizer/FieldItemNormalizer.php#L93, but maybe that has changed and we/i/you can uncomment that check.

[DiegoPino]

@Natkeeran also, maybe its not that only, maybe we need to make sure that the indexer (triplestore) actually enforces those? or maybe it is both... feeling so old...

[DiegoPino]

@Natkeeran Ok, found it: see https://json-ld.org/spec/latest/json-ld/#string-internationalization (NOTE at the bottom). So @whikloj @dannylamb @ (summoned by higher semantic needs), what would you say here? can I drop maybe the string typing in case of a string, since JSON-LD is already an explicitly typed language? (double quotes ==string, v/s integer no quotes?)

[DiegoPino]

Also, depending on how you folks are dealing with the JSON-LD to SPARQL update thing, would be
JSON "title": { "@id": "dc:title", "@container": "@language" } ```` followed by JSON
"title": {
"ja": "忍者",
"en": "Ninja",
"cs": "Nindža"
}
````
be also an option? in case of the same field, many languages? have anyone tried that in Drupal 8?

[Natkeeran]

@DiegoPino
We can possibly provide the language in the context. In that case, we would have to add logic to only get that language and all languages fields. Then add logic to separately index the uris in the triplestore and in Fedora!!

If there is a translated content available, then drupal sends both to the Triplestore (http://ip:8000/ja/node/4?_format=jsonld and http://ip:8000//node/4?_format=jsonld).

That seems to be appropriate! But, I also like the "title" : { "ja" : "忍者" } approach.

With respect to Fedora, I am not sure where to start!

[DiegoPino]

@Natkeeran you mean a whole translated node v/s per field? That makes sense but would bring some Linked data trouble because you will have for a same content different URIs right? Instead of a single URI (RDF) with multiple languages attributes/values for every predicate (drupal field). I bit confused because in case of a single, not English languages you are right! we need to have a general @context default language but still, that does not ensure fedora will get that. Maybe we can discuss this tomorrow on CLAW call? (is tomorrow the early morning version of the call or the middday one - East coast times?)

[whikloj]

Would an easy solution be to remove the default mapping of @type => xsd:string?

Then the field could be a considered a plain string and not a typed value which therefore allows the use of the @language attribute?

[DiegoPino]

@whikloj yes, that was my proposal for that (per field). But, do we have a way of setting integer, etc type mappings via YAML? Or are we basically saying, CLAW only handles by default strings?

[whikloj]

@DiegoPino does it have to be in the YAML, I thought the $field_mappings work you did in the ContextGenerator was to do that mapping, but perhaps I am incorrect.

I should probably be quiet until I have time to test this out more, but I just wondered if we stop assigning the xsd:string type to a large number of fields then they would become plain strings which would mean the $context['langcode'] would be added as their @language value.

[DiegoPino]

@whikloj yes, that one does the mapping $field_mappings (i had to read my own comments i sometimes not even remember what I code, gosh 😨 )
Ok, so all is here: https://github.com/Islandora-CLAW/jsonld/blob/36f37cb2bae7467fd6d3eacabd9e4a368ea77a7f/src/ContextGenerator/JsonldContextGenerator.php#L316-L319

The only issue I see with that is that YAML defined RDF mapping can't override the reality of an internal field and i wish i had added some alter mechanics there so other modules can contribute their mappings.. @todo probably.

Anyway: back. Dropping the default type is a good idea, but since i really need to basically drop type strings even if explicitly defined, from JSON-LD output completely (and keep the double quotes), should i better add that logic to the normalizer?
And for integers, also drop the type? but not allow language? or do i keep, for the sake of completeness the typed number stuff? So many questions

[whikloj]

For integers I would leave the type, if you drop it there is the chance it would get interpreted as a plain string. I think have the type declaration there for everything except strings makes the most sense.

[DiegoPino]

Ok, giving it a try. Do i need to make a test for this? please tell me not....

[whikloj]

Yeah sorry you probably will, not sure how extensive we need it to be to start. But eventually we want to ensure we can handle multiple copies of the same fields with different languages.

[DiegoPino]

👀 https://github.com/Islandora-CLAW/jsonld/pull/25 lots of love and fun code there. Enjoy

[DiegoPino]

[dannylamb]

Resolved via https://github.com/Islandora-CLAW/jsonld/commit/a93960ce1454d8350665581485b510835fa7f1a2

[Natkeeran]

Not sure if I should open a new issue or continue the discussions here. @DiegoPino solution works in producing a multilingual jsonld representation in drupal. However, how this gets indexed in Triplestore and Fedora still needs to be worked out.

http://purl.org/dc/terms/description |  
-- | --
0 |  
@value | "Collection 1 Description"
@language | "en"
1 |  
@value | "சேகரிப்பு 1 விபரிப்பு"
@language | "ta"

In Fedora

dcterms: description
    Collection 1 Description 
    சேகரிப்பு 1 விபரிப்பு 

In Triplestore:

<http://localhost:8000/node/1?_format=jsonld> | dcterm:description | Collection 1 Description
<http://localhost:8000/ta/node/1?_format=jsonld> | dcterm:description | Collection 1 Description
<http://localhost:8000/ta/node/1?_format=jsonld> | dcterm:description | சேகரிப்பு 1 விபரிப்பு

In Fedora, I do see a language @ta for the item. But, the @en does not seem to be there!

Not sure how language can be specified in Fedora. In Triplestore, we will need to add additional logic to handle language.

[whikloj]

@Natkeeran are you using curl to get the RDF graph from Fedora or viewing it through the HTML UI. Same question for the Triplestore. I feel like I've delved into this previously and the information is there, just not presented in the web UI stuff

[Natkeeran]

@whikloj @dannylamb

I reviewed this again. Please see my detail notes here: https://docs.google.com/document/d/1PyQLTbEyQjaZ_FO4w04gg2MWdV_uERtozEYl1X3YJwQ/edit?usp=sharing

It seems like the each url in drupal with the complete jsonld representation gets indexed into Triplestore. This is probably not what we want. We should index the canonical url only. (The language code is being applied properly.)

Please take a look at this paper and presentation for discussion around best practices with regards to supporting multilingual linked open data.

Some implementation examples include:

• yso.fi/onto/yso/p105377
• wikidata.org/wiki/Q16
• geonames.org/6251999
• vocabularies.unesco.org/thesaurus/concept766

[Natkeeran]

This issue has been resolved.