Weblate: Weblate not outputing Android <plurals> fields properly, causing app crashes
With Hebrew, the user interface shows One and Other as the options:
But the _strings.xml_ then has One and Two filled out, then Many and Other blank:
<plurals name="details_last_update_weeks">
<item quantity="one">עודכן לפני שבוע</item>
<item quantity="two">עודכן לפני %1$d שבועות</item>
<item quantity="many"></item>
<item quantity="other"></item>
</plurals>
An empty Other string can cause crashes. Especially because we remove blank strings, since they cause other problems. Weblate should ensure that if any item of <plurals> has been translated, then <item quantity="other"> must be translated. Here's the Android source reference on the requirement for other:
https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/content/res/ResourcesImpl.java#268
Server configuration
eighthave
All 18 comments
352 <string name="updates__hide_updateable_apps">הסתרת יישומונים</string> 353 <string name="updates__show_updateable_apps">הצגת יישומונים</string> 354 355 <plurals name="updates__download_updates_for_apps"> 356 <item quantity="one">הורדת עדכון ליישומון אחד.</item> 357 <item quantity="other">הורדת עדכון ל־%1$d יישומונים.</item>../../src/main/res/values-iw/strings.xml:355:
352 <string name="updates__hide_updateable_apps">הסתרת יישומונים</string> 353 <string name="updates__show_updateable_apps">הצגת יישומונים</string> 354 355 <plurals name="updates__download_updates_for_apps"> 356 <item quantity="one">הורדת עדכון ליישומון אחד.</item> 357 <item quantity="other">הורדת עדכון ל־%1$d יישומונים.</item>../../src/main/res/values-he/strings.xml:437:
434 <string name="notification_content_single_downloading_update">מתבצעת הורדת עדכון עבור „%1$s“…</string> 435 <string name="notification_content_single_installing">ההתקנה של „%1$s“ מתבצעת כעת…</string> 436 <string name="notification_content_single_installed">ההתקנה הצליחה</string> 437 <plurals name="notification_summary_updates"> 438 <item quantity="one">עדכון אחד</item> 439 <item quantity="other">%1$d עדכונים</item>../../src/main/res/values-iw/strings.xml:437:
434 <string name="notification_content_single_downloading_update">מתבצעת הורדת עדכון עבור „%1$s“…</string> 435 <string name="notification_content_single_installing">ההתקנה של „%1$s“ מתבצעת כעת…</string> 436 <string name="notification_content_single_installed">ההתקנה הצליחה</string> 437 <plurals name="notification_summary_updates"> 438 <item quantity="one">עדכון אחד</item> 439 <item quantity="other">%1$d עדכונים</item>../../src/main/res/values-he/strings.xml:441:
438 <item quantity="one">עדכון אחד</item> 439 <item quantity="other">%1$d עדכונים</item> 440 </plurals> 441 <plurals name="notification_summary_installed"> 442 <item quantity="one">יישומון הותקן</item> 443 <item quantity="other">%1$d יישומונים הותקנו</item>+ 11 More Occurrences...
This lint check looks at each translation of a
<plural> and makes sure that all the quantity strings considered by the given language are provided by this translation.For example, an English translation must provide a string for
quantity="one". Similarly, a Czech translation must provide a string for quantity="few".
To suppress this error, use the issue id "MissingQuantity" as explained in the Suppressing Warnings and Errors section.
eighthave
on 7 Jul 2017
Apparently Gettext is not following CLDR plurals here and it causes problems in this case. Weblate is based on Gettext ones, where there are just two plural forms (n != 1), while CLDR (and Android) use four (one, two, many, other). This will be tricky to address...
nijel
on 9 Jul 2017
As a temporary workaround, maybe you could just stick the Other value into Two and Many?
eighthave
on 9 Jul 2017
It seems that some languages are including more than One/Other. For example, Russian has One, Few, and Many, but is missing Other:
<plurals name="details_last_update_days">
<item quantity="one">Обновлено %1$d день назад</item>
<item quantity="few">Обновлено %1$d дня назад</item>
<item quantity="many">Обновлено %1$d дней назад</item>
</plurals>
And Slovak includes One, Few, Many, and Other, but Many is apparently not supposed to be there:
<plurals name="updates__download_updates_for_apps">
<item quantity="one">Stiahnuť aktualizáciu pre %1$d aplikáciu.</item>
<item quantity="few">Stiahnuť aktualizácie pre %1$d aplikácie.</item>
<item quantity="many">Stiahnuť aktualizácie pre %1$d aplikácií.</item>
<item quantity="other">Stiahnuť aktualizácie pre %1$d aplikácií.</item>
</plurals>
Serbian seems to have the correct plurals (One, Few, Other):
<plurals name="tts_view_all_in_category">
<item quantity="one">Прикажи %1$d апликацију у категорији %2$s</item>
<item quantity="few">Прикажи све %1$d апликације у категорији %2$s</item>
<item quantity="other">Прикажи свих %1$d апликација у категорији %2$s</item>
</plurals>
This is problem only for languages where CLDR plurals do not match the ones commonly used by Gettext. I guess there won't be much of such languages...
PS: Slovak should have one/few/many/other, see http://www.unicode.org/cldr/charts/29/supplemental/language_plural_rules.html#sk.
nijel
on 13 Jul 2017
Thinking more about this we really have to add plural definitions per translation, which would be based on file format and fallback to language definition if nothing is defined for file format or within the file. This way we could also address different plural forms being used for one language (such as https://github.com/WeblateOrg/weblate/issues/901).
nijel
on 14 Jul 2017
Some formats will be easy to support properly:
- Use CLDR rules for Android (they can be probably extracted from Babel)
- Use
Plural-Formsheader for Gettext PO
Others might be a bit tricky and we will probably stick to current language defaults:
- XLIFF plurals are usually based on Gettext ones without mentioning the file format
- Qt TS rules themselves are not documented
Note: The Babel rules are not same as Gettext ones we're currently using, so it might need some more tweaks:
>>> import babel
>>> l = babel.Locale('cs')
>>> l.plural_form.tags
frozenset(['few', 'one', 'many'])
>>> l.plural_form.rules
{'few': 'i in 2..4 and v in 0', 'one': 'i in 1 and v in 0', 'many': 'v not in 0'}
>>> l.plural_form.abstract
[('few', ('and', (('relation', ('in', (u'i', ()), ('range_list', [(('value', (2,)), ('value', (4,)))]))), ('relation', ('in', (u'v', ()), ('range_list', [(('value', (0,)), ('value', (0,)))])))))), ('many', ('not', (('relation', ('in', (u'v', ()), ('range_list', [(('value', (0,)), ('value', (0,)))]))),))), ('one', ('and', (('relation', ('in', (u'i', ()), ('range_list', [(('value', (1,)), ('value', (1,)))]))), ('relation', ('in', (u'v', ()), ('range_list', [(('value', (0,)), ('value', (0,)))]))))))]
The variables are defined in the specification, there is also tool to convert them (written in PHP): https://github.com/mlocati/cldr-to-gettext-plural-rules
nijel
on 14 Jul 2017
After digging into this for few hours (trying to add CLDR plurals to Weblate), now I finally understand where the problem is - it's not that the CLDR plural rules would be that different (the differences are there, but they are limited to few languages).
The problem lies a bit deeper:
Translate-toolkit always adds other to the plurals:
It might sound weird, but apparently the root cause is in Babel not returning other for most of the languages it should. There is even bug reported for similar thing since April 21, see https://github.com/python-babel/babel/issues/495
nijel
on 9 Jan 2018
This should be fixed by following PR which replaces Babel usage in translate-toolkit by own plurals data: https://github.com/translate/translate/pull/3762
It would be really great if somebody with Android knowledge (eg. @eighthave , @TobiGr, @theScrabi ) could check if the tags are correct there - I was not able to find some official docs for Android covering this and the CLDR is still a bit magic for me as a result I'm not sure if the table I've generated correct (I've checked few languages which I know were problematic and they look correct).
nijel
on 9 Jan 2018
I don't quite understand what I should check. I looked at this diff of 5d853aa and I have no idea what that data is. I looked at https://github.com/translate/translate/pull/3762/commits/65b4000b01acc428289f09e8b5fc8f78877e5dd6, and I could see that some of the languages are missing "other". __All__ languages must have at least "other", that is a hard requirement. If a language has only one plurals form, it should be "other". I would include a CI test case that checks that all languages have "other" in them. I can see that _be_, _pl_, and _uk_ are missing "other".
eighthave
on 17 Jan 2018
No, not all languages use other for integers (that's what android cares for plurals), see for example Belarusian, Polish or Russian.
And yes https://github.com/translate/translate/commit/65b4000b01acc428289f09e8b5fc8f78877e5dd6 is think I'd like to get reviewed (or more precisly https://github.com/translate/translate/pull/3762/files).
nijel
on 19 Jan 2018
As far as I understand it, Android requires that "other" is always
present in the _strings.xml_ <plurals> blocks. Looking at your
examples, be/pl/ru use "other" for non-integers, so it is not wrong to
require an "other" block.
eighthave
on 19 Jan 2018
I agree with @eighthave. Android requires "other" and if it does not exist this can cause a crash.
TobiGr
on 20 Jan 2018
So far I thought that Android is simply following CLDR, but that doesn't seem to be the case, so this has to be implemented somewhere (apparently documentation doesn't seem to exist).
To me it really looks weird that it would require the "other" even for languages where it won't be used - the getQuantityString used to evaluate this accepts only integers...
Looking at the code I still think that "other" is not needed as long as all valid choices are there.
So can somebody actually test that other is really needed in all cases? Testing simple string with one of above mentioned languages (eg. Polish) should do that.
nijel
on 22 Jan 2018
I've test it and can confirm, some languages don't require the _other_ field, here's some tests that I did:
The test consists on loop through 0 to 220 and check if the the getQuantityString returns the language code or throws an error (using "language_code" + %1$din the strings).
| API | 25 | 23 | 19 | 16 |
|:---------------------:|:---------------------:|:--:|:--:|:--:|
| Without field "other" | only be, pl, ru, uk | same as 25 | same as 25, but some fallback | same as 19 |
| Without field "other"
(plural deleted if empty)
| same as above,
some fallback | same as 25 | same as 25 | same as 25|
| Following PR rules strictly | ✓ | ✓ | ✓, except bem, bez | same as 19|
- Exception when fail:
Resources$NotFoundException: Plural resource ID #id quantity=x item=other - You cannot have an empty plurals tag
- Deleted plurals just fallback to default
You can find the generator (to generate the strings.xml file and their contents) that follows the rules of PLURAL_TAGS that it's in your pull request (translate/translate#3762) and the app source code here:
https://github.com/mauriciocolli/LanguageTest
There's some already compiled apks if you want to test in your device.
mauriciocolli
on 24 Jan 2018
@mauriciocolli Thanks for verifying this! I've just deployed this change to Hosted Weblate, so that will give these changes some more real life testing.
There are still few languages whose plural rules in CLDR differ to what is usually used in Gettext, so these will need additional fixes on the Weblate side:
- Lithuanian (see also https://github.com/WeblateOrg/weblate/issues/901)
- Cornish
- Upper Sorbian
- Hebrew (see https://github.com/WeblateOrg/weblate/issues/1781#issuecomment-356652692)
- Lower Sorbian
nijel
on 25 Jan 2018
Thank you for your report, the issue you have reported has just been fixed in Weblate. The complete fix depends on https://github.com/translate/translate/pull/3762 (which is already deployed in Hosted Weblate).
- In case you see problem with the fix, please comment on this issue.
- In case you see similar problem, please open separate issue.
- If you are happy with the outcome, consider supporting Weblate by donating.
nijel
on 30 Jan 2018
Related issues
reloxx13
·
3Comments
tariver
·
4Comments
asereze
·
4Comments
nijel
·
3Comments
agaida
·
4Comments
Most helpful comment
I don't quite understand what I should check. I looked at this diff of 5d853aa and I have no idea what that data is. I looked at https://github.com/translate/translate/pull/3762/commits/65b4000b01acc428289f09e8b5fc8f78877e5dd6, and I could see that some of the languages are missing "other". __All__ languages must have at least "other", that is a hard requirement. If a language has only one plurals form, it should be "other". I would include a CI test case that checks that all languages have "other" in them. I can see that _be_, _pl_, and _uk_ are missing "other".