Wordpress-android: Missing Characters for Tajik Cyrilic language

[maxme]

Note: missing characters appears in the Post List, but probably everywhere else

Example of failing characters:

• Ҳ
• Ҷ

as in ҲИҶРАТ – ОРЗУ ИСО

I don't think it comes from the font. I tried this sentence in https://typecast.com/preview/google/Merriweather and everything was correctly displayed. I wonder if that comes from the font renderer.

Ref https://github.com/wordpress-mobile/WordPress-Android/issues/2460 (vietnamese characters bug)

[maxme]

Example coming from https://mazmun.wordpress.com/ - https://mazmun.wordpress.com/2015/10/20/%D2%B3%D0%B8%D2%B7%D1%80%D0%B0%D1%82-%D0%BE%D1%80%D0%B7%D1%83-%D0%B8%D1%81%D0%BE/

[maxme]

On wpandroid 4.7 - we don't have the same experience on a 4.1.1 device or on 6.0 device:

https://cloudup.com/c7hUauL4n48

[rachelmcr]

This is also an issue for Cyrillic characters in Uzbek, for example:
Ў
Қ
Ғ

Reported for wpandroid 4.6, using an android 4.3 device.

[rachelmcr]

This is also an issue for the letter č, which is used in a number of languages (e.g. Czech). It doesn't show up correctly in the list of blog posts, but it does appear elsewhere (such as in the editor and in the post preview).

Reported by a beta tester and reproduced for Android 4.4.4 (API 19), but works ok on 6.0 (API 23).

[maxme]

I think I found the source of this issue, it comes from our class org.wordpress.android.widgets.WPTextView, extending android.support.v7.widget.AppCompatTextView - when I replace it by AppCompatTextView I don't get this font problem. (need to dig more).

[maxme]

When I update the Merriweather font to v1.584, the problem is solved on all devices. Here is a 4.1.1 screenshot:

[maxme]

Updating to the latest Merriweather font also fixes this issue https://github.com/wordpress-mobile/WordPress-Android/issues/2460

@mattmiklic I know we should stick with the "old" Merriweather version, but this is a real problem for some users (we get a lot of comments from vietnamese users in Play Store comments) and I don't see any other way to fix this issue (fix the old version font? herculean task).

[maxme]

I updated the fonts in issue/3358-update-merriweather-font if anyone wants to try.

[mattmiklic]

@mattmiklic I know we should stick with the "old" Merriweather version, but this is a real problem for some users (we get a lot of comments from vietnamese users in Play Store comments) and I don't see any other way to fix this issue (fix the old version font? herculean task).

I've tested version 1.584 of Merriweather on the web, and while it fixed problems in Cyrillic it didn't really improve any of the Vietnamese examples I looked at, there are still a significant number of missing characters in the newer version.

It's definitely something that needs to be figured out, but because of the significant difference between the two versions of Merriweather I think we should keep the existing one in place while we figure out a fix for the issue in Calypso on all platforms, web included.

[maxme]

@mattmiklic thanks for looking at this.

I think we can't compare what we see on the Web and what we get in the Android app with the old version of Merriweather. Let me explain:

• Using a 6.0 device and the "old" (ie current) merriweather version: I see no problem in Vietnamese and Tajik Cyrillic.
• Using a 4.1.1 or a 5.0 device and the "old" (ie current) merriweather version: about 5% of Vietnamese and Tajik Cyrillic characters are not rendered.
• Using a 6.0, 4.1.1 or 5.0 device and the "new" (ie current) merriweather version: I haven't seen any missing character during my test with random Tajik and Vietnamese.

So my guess is there is an incompatibility with the old font and how Android renders it and even if the new version is missing a significant number of characters it fixes that "4.1.1 and 5.0 Android devices can't render 5% of the some character set" issue.

[maxme]

Also happens in greek:

[hypest]

@mattmiklic , any news about a fix on the web-side? This looks like a fix that the mobile app needs since we're embedding the font in the app itself instead of relying on system to provide (and update) it.

I'd propose to update and get issue/3358-update-merriweather-font merged as @maxme has suggested.

[mattmiklic]

No fix available yet; I don't think anyone has had a chance to work on it.

I do not recommend switching to the new version of Merriweather—it's an altogether different typeface than the one we're currently using, and isn't a visual match.

The best option we've come up with so far (the only one that doesn't involve finding an entirely new typeface for all languages) is to revert to system serif fonts (Georgia, Noto Serif) if the language uses a character set that is unsupported by Merriweather. This will need a developer to help figure out how to make it work, though.

[hypest]

@mattmiklic , I see, thanks for elaborating!

FWIW, I tried @maxme 's branch and unfortunately it doesn't fix anything for greek texts 😞 .

While at it, the problem with greek text is quite severe as none of the accented characters are displayed correctly. That practically means that pretty much all greek posts are broken.

[hypest]

I did a survey of how greek text looks depending on the Android API level.

1. Chars look good in all screens on API 21
2. Chars have misaligned diacritics (should be above chars, not up-and-right) in posts list on API 19
3. Chars missing or displayed as "x" blocks in post preview/editing on API 18
4. Chars missing or displayed as "x" blocks in all screens on APIs 16, 17

From left to right are API 16, 17, 18, 19, 21. (_Latest released API level at the time of writing: 24_)

Posts list:

Preview Post:

Edit Post:

[hypest]

The best option we've come up with so far (the only one that doesn't involve finding an entirely new typeface for all languages) is to revert to system serif fonts (Georgia, Noto Serif) if the language uses a character set that is unsupported by Merriweather.

@mattmiklic , had a look and it seems we cannot rely on the system to drop back to the system serif fonts on older Android versions on a per-character basis. Only API levels 21 and above handle that gracefully and without any software hacks.

I tried out only setting the font family to serif and I verified it looks OK on all API levels.

BTW, notice that on the shots linked above the chars are not rendered with Merriweather even on API 21. The font seems to have only a few of the greek letters anyway (it has the "μ" for example, for math expressions I guess).

Pulling data from the Google Play console, it seems that well above 50% of our users are on API 21 and later. At the end of the day, it seems that on Android, it's only a section of our userbase that can enjoy our Merriweather right now and for others the text can be broken or weird. Unless I'm missing some better solution, I'd suggest we consider setting a custom font (Merriweather) only on API 21+ and use the system serif fonts below that. WDYT?

[mattmiklic]

@mattmiklic , had a look and it seems we cannot rely on the system to drop back to the system serif fonts on older Android versions on a per-character basis. Only API levels 21 and above handle that gracefully and without any software hacks.

BTW, notice that on the shots linked above the chars are not rendered with Merriweather even on API 21. The font seems to have only a few of the greek letters anyway (it has the "μ" for example, for math expressions I guess).

Hey @hypest, thanks for putting those test screenshots together. I was aware of the second issue you mentioned, where individual characters from the system serif font are being substituted for Merriweather characters; I didn't realize that they fail to appear altogether on older versions of Android.

In my earlier comment, I was thinking about whether it'd be possible to detect the user's set language, and fall back to system serif fonts if they're using a language that includes characters that Merriweather doesn't support (this appears to be languages using an extended latin character set). Thinking about that more this morning though, I'm not sure it's possible—even though I use the app in English, I could happen across a post in the Reader that's in Vietnamese, for example, and would still experience the same issue with Merriweather.

Let me ping some designers about this issue—we've been discussing Merriweather's character support issues recently, and we need to find a solution for Calypso and iOS, as well.

[maxme]

@mattmiklic this is still an issue in the app, any news about a solution in Calypso?

[mattmiklic]

I don't think there's been any movement here on the Calypso side, but will try to give a nudge to some of the folks working on that side of things, to try to move us toward a solution here.

[mattmiklic]

We're working on testing a new typeface that may solve this soon; will update when we come to a conclusion.

[Ahrary]

Hello! How about Noto fonts?

[mattmiklic]

@Ahrary thanks for the suggestion! I think we're pretty close to landing on a solution. 🙂