Dataverse: Search: Accented characters, ampersands, negative numbers and other special characters

[kcondon]

This was suggested by Eleni during discussions around search issues with accented characters.

The suggestion is to allow searching both using the original accented characters and without, in those cases where a user may not have access to accented keyboard, etc.

[pdurbin]

See also https://github.com/IQSS/dataverse/issues/818#issuecomment-52184409 and the search internationalization ticket at #326.

[pdurbin]

@mheppler pointed out an interesting answer at http://stackoverflow.com/questions/16627062/not-able-to-search-spanish-word-with-accent-in-solr/20657529#20657529

Here it is in full (note downsides though):

You can try using the ASCIIFoldingFilterFactory filter.

It converts characters with ascent into their no-ascent counterpart.
Put this in your schema.xml:

<filter class="solr.ASCIIFoldingFilterFactory"/>

_Note: The downside is that words like "cañon" and "canon" are now equivalent and both hit the same documents IIRC._

[mheppler]

Dusted off this oldie but goodie to be a representative issue for many other search bugs/feature requests. I have closed the following issues, to consolidate them here, and moved over any pertinent information in their comments.

• _Search: Support searching on accented words by typing unaccented characters instead_ (original title of this issue)
• _Search: Searching on a negative number returns a lot of false results. #819_ (-160 in Geographic Bounding Box for West Longitude and -65 for North Latitude, returns unexpected results, but 143 for East Longitude target dataset found)
• _Search: characters in (dataset names) return zero results #2156_ (search População Estrangeira na Europa e no Mundo Nôvo (%) - 1870 - 2001 is broken in Harvard Dataverse, remove "%" and it returns results)
• _Search: Text after and including Ampersand (&) is removed from Search Query #2702_ (search Economics & Politics in Harvard Dataverse, see Stack Overflow answer for potential fix)
• _Parameters issue in display on production #3993_ (example on Harvard Dataverse)

[pdurbin]

@mheppler I'm still haunted by the bug report at https://github.com/IQSS/dataverse/issues/1928#issuecomment-104471953 that a search for "Experiment" found datasets with "Experience" but I haven't tested lately. That was nearly five years ago. 😄

[BPeuch]

We here in Belgium second this, as our three official languages – Dutch, French and German – all three contains special characters (á, à, â, ç, é, è, ê, ë, í, ó, ö, ú, ù, û, ü…).

Here is an illustration of how accented characters can hinder the search for / the discovery of datasets:

———————————————————————————————————————————————

[pdurbin]

@BPeuch in a report from a French installation, switching from text_en to text_fr seems to have helped: https://groups.google.com/g/dataverse-community/c/9sjpBpPRuFk/m/uxH2KKJnAQAJ

Since you have three official languages, however, I'm not sure if this will work for you.

[BPeuch]

That's valuable information still. Thank you, @pdurbin!

I fear that indeed it might not work between because some characters are specific to some of these languages (e.g. á and ó for Dutch). It could be worth a try though.

[qqmyers]

FWIW: For QDR, we addressed some of this with changes in the solr schema.xml leveraging some filters:
solr.WordDelimiterGraphFilterFactory
solr.ASCIIFoldingFilterFactory
solr.PatternReplaceFilterFactory

I can dig up that code if it's helpful - I don't know much about solr so I think what we did was mostly to cut/paste from sources I found on the web though, so you might be better off searching for the latest on this issue - perhaps including some of those filter names. (Our interest was primarily in handling characters from other languages and contractions, so it may not be as general as others might want.)

One thing I think is helpful to convey though - I think this can be solved/significantly improved just making solr changes ,versus requiring Dataverse code changes. So looking for answers related to solr or solr expertise at our institutions might be a good approach.

[qqmyers]

FWIW: QDR's solution can be seen in https://github.com/QualitativeDataRepository/dataverse/blame/develop/conf/solr/7.7.2/schema.xml - see the qqmyers changes starting from https://github.com/QualitativeDataRepository/dataverse/blame/develop/conf/solr/7.7.2/schema.xml#L573.
The main thing for non-ASCII characters was to enable the solr.ASCIIFoldingFilterFactory filter during indexing and querying. We also used the solr.PatternReplaceFilterFactory filter to try to recognize contractions (e.g. qu'est que) - you can see from the pattern there that this is fairly limited. The use of the solr.ASCIIFoldingFilterFactory is probably generic and could be pulled into a separate PR if there's interest in having it as the default for the community.

(You'll also see some modifications to the solr.WordDelimiterGraphFilterFactory params - I think those are related to file names containing numbers - basically an unrelated but also potentially useful change).

[poikilotherm]

This might be linked to #6675 and #7375.