Openrefine: Improve use of Wikimedia i18n library to provide translated UI

[ostephens]

Following #1846 we need to make further modifications to make use of the more powerful translation options offered by this library

• [x] Amend the dictionary JSON returned by http://127.0.0.1:3333/command/core/load-language? (i.e. LoadLanguageCommand.java) to include multiple languages so the UI can fall back to an appropriate language if a term is not present in their preferred language. (Done - #2209)
• [x] Change uses of string concatenation to use parameterized templates e.g. https://github.com/OpenRefine/OpenRefine/blob/9d72de1874726be82e458335deef7cd4aee0302f/main/webapp/modules/core/scripts/views/data-table/menu-edit-column.js#L262 should be translated as a single phrase with parameter replacement like elmts.dialogHeader.text($.i18n('core-views/split-col-into-several-col', column.name')); with text like "Split column $1 into several columns". Note that phrases should also include any punctuation, so it can be localized correctly as well.
• [ ] Similarly review / change uses of HTML layout to provide visual concatenation of phrases that may need to be localized
• [x] Implement support for pluralization using the syntax described at (Wikimedia jQuery i18n plugin) anti-pattern looks like this: https://github.com/OpenRefine/OpenRefine/blob/ddd9bf9aa80a43c68bf6d8fb4dc9d0b8bb29dcd5/main/webapp/modules/core/scripts/dialogs/clustering-dialog.js#L274-L275

[wetneb]

We already have a language fallback (#2209) so the first point has been addressed.

We still need to improve our use of the localization library (Wikimedia jQuery i18n plugin), mostly:

• to use their support for templates rather than concatenating translated strings to other strings;
• to use support for plurals where appropriate.

For instance, we have things like
$.i18n('core-views/custom-numeric')+'...',
which is not ideal: there might be languages where adding ... at the end of such a message does not make sense!

Similarly, we have things like
$.i18n('core-views/split-col')+" " + column.name + " "+$.i18n('core-views/several-col')
which should rather use the template mechanism.

[wetneb]

The syntax for templates is described in jquery.i18n's documentation.

For instance: we currently have: $.i18n('core-views/split-col')+" " + column.name + " "+$.i18n('core-views/several-col') in the code and

   "core-views/split-col": "Split column",
   "core-views/several-col": "into several columns",

in the translations.

We should rather have $.i18n('core-views/split-column-into-several-columns', column.name) in the code and

   "core-views/split-column-into-several-columns": "Split column $1 into several columns"

[tfmorris]

Most packages support named parameters/placeholders, but it looks like the Wikimedia system only allows $1, $2, etc, which is unfortunate because the names are helpful to provide context for the translators in non-obvious cases.

[antoine2711]

I have a question:
If I have this in the json-lang files for FR:
"core-views/goto-page": "$1 de $2 {{plural:$2|page|pages}}"
I get an error.

This: "core-views/goto-page": "$1 de $2 page(s)" works.

And those always works:
(ES): "core-views/goto-page": "$1 de $2 {{plural:$2|página|paginas}}"
(EN): "core-views/goto-page": "$1 of $2 {{plural:$2|page|pages}}"

All JSON data called by the same:
https://github.com/OpenRefine/OpenRefine/blob/7274a21ee04e6480e50de39f1a7788164dd1981a/main/webapp/modules/core/scripts/views/data-table/data-table-view.js#L192-L195

Regards,
Antoine

[tfmorris]

@wetneb I've updated the main issue with your comments and turned it into a checklist
@antoine2711 You don't say what the error is, so it's hard to guess what's going on, but since it's language specific, I'd focus on what's different about the French file.

[antoine2711]

As a follow-up, here's a simple way to check it. Line 2 & 4 are EXACTLY the same. And the last (4th) line triggers the error.

$.i18n().locale = 'en';
$.i18n("$1 de $2 {{plural:$2|page|pages}}", 5, 10);
$.i18n().locale = 'fr';
$.i18n("$1 de $2 {{plural:$2|page|pages}}", 5, 10);

The last line gives this:

project-bundle.js:28751 Uncaught ReferenceError: pluralRuleParser is not defined
    at Object.getPluralForm (project-bundle.js:28751)
    at Object.convertPlural (project-bundle.js:28751)
    at Object.plural (project-bundle.js:28744)
    at Object.emit (project-bundle.js:28744)
    at project-bundle.js:28744
    at Function.map (project-bundle.js:504)
    at Object.emit (project-bundle.js:28744)
    at Object.parse (project-bundle.js:28739)
    at a.parse (project-bundle.js:28723)
    at Function.e.i18n (project-bundle.js:28723)

I doubt it's a file thing. I guess a library update would probably fix it.

Regards,
Antoine

[antoine2711]

I wonder about the plural… what if the value == 0, shouldn't it be singular?

Like with value == 1?

Regards,
Antoine

[thadguidry]

@antoine2711 if there's even 1 row...there is 1 page. If there are 0 rows... then "0 of 0 pages", I would expect?

[tfmorris]

The error in https://github.com/OpenRefine/OpenRefine/issues/1858#issuecomment-637274292 makes it clear that there is a bug. I'm not sure why a bug report didn't get created, but I've created #2700 .

On a completely separate topic, "0 pages" is correct English, but the plural form for zero will vary (greatly) by language.

[tfmorris]

PR #2717 includes some examples of Javascript string concatenation & pluralization and patterns for fixing them, in addition to the ones mentioned in the main issue.

[tfmorris]

I think the bulk of the Javascript based problems are fixed in #3019.

The remaining class of i18n issues are those that use layout-based juxtaposition to impose language ordering, typically by sandwiching an input field between two text strings e.g. "Parse every _1_ lines into one row" which requires translations like

JP - Combine multiple lines: _1_ Analyze as one line
HE - All analysis _1_ The rows into one row

Sure, some of that awkwardness is due to playing "telephone" with Google Translate, but you get the idea.

These should be changed to have a single descriptive label for the input field e.g. "Number of lines to treat as a single row: __"