Fenix: [Bug] B+sat+olck in Language list

[ebalazs-sv]

Steps to reproduce

1. Launch Fenix and go to Settings - Language;
2. Scroll at the end of the Language list;
3. Observe the B+sat+olck language;
   

   Expected behavior

B+sat+olck language is not in the list.

Actual behavior

B+sat+olck language is in the list.

Notes

After selecting the B+sat+olck language, it will move at the top of the list, as displayed in the screenshot. Also, the app remains in English.

Device information

• Android device: Pixel 2 (Android 9), Nexus 5 (Android 5.1)
• Fenix version: Fenix version: Release 82.1.0-rc.2, Release 82.1.0-rc.1, Beta 82.0.0-beta.6, Beta 82.0.0-beta.5
  Not reproducible on Beta 82.0.0-beta.4 and Nightly 10/19.

[ebalazs-sv]

This issue is also reproducible on Nightly 201020 05:03 (Build #2015770729) GV 83.0a1 from 10/20 with Huawei P9 Lite (Android 7).

[liuche]

Some additional notes:

• can't search for B+sat+olck (sometimes it's on the bottom of the list, sometimes it's just somewhere in the middle)
• even though this locale (Santali) is almost at 97% completion, the app is all in English

@pocmo could you point me to somewhere in AC where we're doing locale renames? You mentioned that usually we need to do this for locales that Android already has but don't match, but would that be worth a try?

Also, if you have suggestions on what things to try, that would be appreciated.

[pocmo]

@pocmo could you point me to somewhere in AC where we're doing locale renames?

I don't think we do that in AC and I am not sure whether we ever did implement that for Fenix. But here's where we do that in Focus:
https://github.com/mozilla-mobile/focus-android/blob/master/app/src/main/java/org/mozilla/focus/widget/LocaleListPreference.java#L40-L75

[pocmo]

Since this is the first language that uses a BCP 47 language tag (which also seems to only be supported from Android 7+), there may be something going wrong when we are constructing the locale - since we haven't been expecting this format. I can take a quick look and see if I can find something.

[pocmo]

Some findings:

• The order of languages is determined by the gradle script that looks at the resource folders, so heavily depends on the OS and how it iterates over the folder contents

• We seem to build a Locale object with language set to b+sat+Olck, which seems wrong. That format seems only be used for the resource folders. Internally I assume we want to build a Locale with language sat and script Olck.

• If I build such a Locale object manually then I see only one String change, but nothing else. 🤔

[mcarare]

@Delphine As I mentioned before, AFAIK this can only be solved by changing the folder resource name for "b+sat+olck" to "sat". I have tested locally and it works just fine. The change must also be made in Pontoon, so future imports would be made to the same file/ folder.

Apart from the fact that @pocmo mentioned, that BCP 47 language tags are supported from SDK 24+, the language tag b+sat+olck does not follow the standard, as the code after b+ should be a 2 letter code, an ISO 639-1 Code. As we can see here: https://www.loc.gov/standards/iso639-2/php/code_list.php and other ISO 639-1 Code lists, Santali does not have an ISO 639-1 Code, so the BCP 47 language tag could never be used in Android for this language.

Mapping does not help here, as it could change just the name in the language lists.
Like @pocmo said, adding the locale for Santali, using Locale.Builder().setLanguage("sat").setScript("olck").build() solves only the language name, but when selecting the locale the strings from the strings.xml file in "b+sat+olck" folder are not used. There is one string change that also disappears when restarting the app, even without changing the locale.

Maybe the folder name could be changed at build time, but IMO the straightforward approach would be importing it correctly.

[Delphine]

@mcarare: Since the localizers practically finished their work on Fenix, I'm going to create a separate "sat" locale in Pontoon at this point, and have their existing work migrated there. Seems like this is the quickest thing I can do to get them all set up.

Note: my team looked a bit into this, and it seems that it is in fact possible that the language code be 3 letters in BCP 47: https://tools.ietf.org/html/bcp47#section-2.2.1

In fact, we do have other locales that use ISO 639-2 codes, like "cak", and they seem to load resources just fine.
Since BCP 47 should accept ISO 639-2 codes, could this be a bug in Android’s implementation of BCP 47? Should an issue be filed for this?

[mcarare]

I am unaware of the reasons the Android implementation of the standard does not accept other variants of it. Also, I think there are also problems with "cak" https://github.com/mozilla-mobile/fenix/issues/12528 and it might be a good idea to only stick with ISO 639 codes for language tags.

[gueroJeff]

We've been shipping ISO 639-2 locales for years without issue. If this is new, let's do the due diligence to triage why it exists before abandoning these locales. Additionally, the screenshot you reference displays the issue exists for ISO 639-1 codes as well (e.g., "an"), so the issue is broader and requires further triage.

[mcarare]

@gueroJeff Sorry, I meant ISO 639. I did not suggest abandoning the locale, just importing it with a proper language tag.

[Delphine]

Heads-up: I just changed sat-Olck locale code to sat on Pontoon, and made necessary changes in the repo here: https://github.com/mozilla-l10n/android-l10n/pull/286

Let's see if this helps solve part of the problem.

[pocmo]

Looks like the last import moved everything to sat:
https://github.com/mozilla-mobile/fenix/commit/bf69519dc96dfadd05940a0518d56c3c7e835243

[mcarare]

Tested on some devices (Android 6, 9, 10) and it works as expected: The Santali language shows up in the language list, and the app displays translated strings when selected. Also tagging this for QA check.

[ebalazs-sv]

Verified as fixed on Nightly 201103 05:01 (Build #2015773417) GV 84 from 11/3 with

• Motorola Moto G6 (Android 8);
• Pixel 2 (Android 9),

and the Santali language is in the Language list and when selected the app displays the translated strings.

Also, the B+sat+olck language is not in the Language list.
I will remove the qa:needed label and close this issue.