Libgdx: Support for complex font rendering (Chinese, Arabic, ...)

[adrianmay]

For languages like Chinese, it's not practical for an app to deliver its own font. Therefore there's a desire for libgdx to be able to use fonts delivered by the platform.

Some discussion already occurred here:
Roll-your-own Chinese font?
and here:
Development Forum: Supporting Asian fonts in BitmapFont

Earlier attempts at the same challenge have run into difficulties but there doesn't seem to be an exhaustive list of those difficulties. Perhaps that should be compiled here before attempting to develop a new plan.

[badlogic]

Took the freedom to modify your issues title,

I believe we could create an extension that uses system facilities to render strings to textures. I'm not sure what an interface for something like that would look like.

Another alternative would be a LRU glyph cache with the same interface as BitmapFont. Open for other suggestions (and not saying i'm gonna implement this!)

[adrianmay]

If it's only about making textures, we could give Gdx.graphics a getSystemFont method taking the font characteristics and returning a Font which had a method like "Pixmap write(String s)".

But perhaps the whole edifice of widgets and dialogs ought to work with this, so how about introducing a Font interface or base class and then deriving BitmapFont and SystemFont from it. That would require changing all the BitmapFont references into Font references, but there's only about 600 of those ;-)

[jrenner]

how would the glyphs from the LRU cache be stored? Individual textures? in an atlas?

Is there anyway to do this that doesn't bind textures like crazy? I've been brainstorming but I haven't come up with anything.

Could we somehow use freetype font generator to dynamically add glyphs to a pre-rendered font, possibly with an internal texture atlas. for example:
myfont.png contains 100 chars I know I will use for buttons.
Users inputs n chars not in the original 100.
myfont2.png is created and generates those n chars into the first spots of the png, keeping track of position for writing next dynamically generated char.

It is just brainstorming, I don't know if it is a good idea.

[badlogic]

Your suggestion comes close to the system i wrote at work a few years ago. It's a big mess, especially if it has to be general purpose.

We should make this a higher priority imo, i want to be able to support the asian and arabic markets. For what it's worth, Unity has zero support for asian and arabic scripts...

Ultimately i think we need to expose a simple method that takes a string in utf and spits out a layouted texture/pixmap. Any glyph caching will fail on mobile due to layouting, The layouting and rendering can cache glyphs though via LRU on the heap. It's still going to be slow to generate textures for new strings.

based on this functionality, we can implement a special label in scene2d. for long blocks of text we'll need to be able to split the text up into multiple textures, which is going to be a pain. and of course, there's a limit to the amount of Ram se have available. the benefit of this method is that we can implement it per platform using systme libraries, including gwt.

i'll also try to find Androids implementation of text rendering and will try to get inspired by it. they actually have a glyph cache and freetype. we'll see.

[badlogic]

oh, and i think this functionality belongs in core.

[badlogic]

Here's Android GLES 2 based font renderer for inspiration

https://github.com/android/platform_frameworks_base/blob/a85db5ab785c59726c3644999801e48f449714f3/libs/hwui/FontRenderer.cpp

[NathanSweet]

From the discussion in the topic on the dev forum it seems layout is simple enough for CJK that we can use an atlas approach, same as we do for Latin languages. If that is true, we can get quite far by adding glyphs to the atlas on the fly. A lot of glyphs can fit on a 2048 texture. There can only be so many glyphs on the screen at once so if the texture were to get filled, simply clearing it would likely be sufficient.

For Arabic and RTL, I dunno, cache full strings I guess.

[davebaol]

AFAIK the number of letters/glyphs in the Arabic language is not a real problem.
They have 28 letters and their glyphs are about 4x since each letter can have up to 4 different shapes depending on whether it will be connecting with a preceding and/or a succeeding letter.
So the main problem is to properly render the letters in one of their conditional forms, depending on whether they are at the beginning, middle or end of a word. In particular, they may have 4 distinct forms: initial, medial, final and isolated.
A few letters have only the isolated or the final form though.
Also one more problem to take into account is that Arabic language is RTL but their numbers are LTR, just like ours. So a sentence containing numbers is a mix of RTL and LTR.

[cypherdare]

An idea following what Nate said...At startup, seed a glyph map with 2000 or so most common characters for the language, which seems to cover most text for these languages. Reserve the rest of the space for filling in additional glyphs (maybe reserve a fixed square for each of them to make it easy to overwrite them). As new strings are passed in for rendering, the font manager fills new characters into the empty spaces. (Not sure, but would an FBO be faster than new Pixmap-to-Texture if you are just writing a few sprites to it when it's updated? It wouldn't be getting cleared, so there'd still be a buffer restore each time.) This could minimize regeneration of the glyph texture...as the font manager is fed more and more strings, it becomes less and less likely to require updates to the texture.

Maybe also keep a usage count for each character with some kind of comparitor, so it will know which characters to overwrite first. Maybe even push overwritten ones to a second texture and smartly order the drawing of glyphs for only two texture binds if necessary.

[badlogic]

I'd like to start out with the simpler version that doesn't do an LRU glyph cache but instead renders/layouts an entire string to a texture. It's wasteful, but appearently good enough for cocos2d-x (Unity has zero support for CJK/arabic btw. And people pay for it...).

Questions:

• What should the API be like?
• Does this belong in core?
• Should we try to make it somewhat interchangeable with BitmapFont? Or we could just create a new FixedLabel actor or something similar that folks can use instead of the BitmapFont based actor
• Should we support system fonts?

I have a bit of time this week, so i'd like to get going with this asap.

[badlogic]

For reference, here's what Cocos2D-X does http://www.cocos2d-x.org/wiki/Text_Labels

Funny that that works out well for them. On a related note, we need to come up with a better way to do CJK/arabic CJK. Returning keycodes doesn't cut it with most custom keyboards on Android anymore. Should open a separate issue for that.

[xeoshow]

support system fonts: vote +1, since this could reduce the size of apk and
make development easier ...

2014-05-29 20:29 GMT+08:00 Mario Zechner [email protected]:

For reference, here's what Cocos2D-X does
http://www.cocos2d-x.org/wiki/Text_Labels

Funny that that works out well for them. On a related note, we need to
come up with a better way to do CJK/arabic CJK. Returning keycodes doesn't
cut it with most custom keyboards on Android anymore. Should open a
separate issue for that.

—
Reply to this email directly or view it on GitHub
https://github.com/libgdx/libgdx/issues/1589#issuecomment-44526350.

[sridharsundaram]

I did some work using harfbuzz to support Indic and Arabic languages in Bitmap fonts in libgdx. This worked for Desktop and for Android. Not sure about IOS.

1. BitmapFontCache had to be modified - there is not a 1-1 mapping between chars and glyphs.
2. the charsequence is first converted into a glyphsequence, then the glyphs are shaped (reordered according to language specific rules), then rendered - this is done in a JNI call.
3. harfbuzz + freetype library needs to be added.
4. textwrapping was an issue.

[sabotagedninja]

Where system fonts are concerned, Java AWT's TextLayout classes can be used to render just about any language and styles, inclusing full RTL Arabic. I successfully used this to support Arabic text in our product based on MT4j. We only need it to run on desktop (win/mac), so I can't say if it'll work on mobile devices. The implementation generates one texture for every string requested on the fly, so it's not efficient in the least, but it does work splendidly. Maybe this is another point of inspiration.

@sridharsundaram I'd be very interested to learn about the changes you had to make in libgdx to support Arabic. Is there any way you can share that?

[NathanSweet]

If you want to use AWT for desktop-only font layout, you might see libgdx's UnicodeFont:
https://github.com/libgdx/libgdx/blob/master/extensions/gdx-tools/src/com/badlogic/gdx/tools/hiero/unicodefont/UnicodeFont.java
It renders glyphs to an atlas texture as needed and renders strings glyph by glyph. It doesn't cache strings. It doesn't do RTL. Hiero makes for a nice demo:
https://github.com/libgdx/libgdx/wiki/Hiero

[sridharsundaram]

I had to modify BitmapFontCache in addToCache

....
if (isComplexScriptLayoutEnabled()) {
List glyphs = getGlyphsAfterShaping(str, start, end, data);
requireGlyphSequence(glyphs);
for (Glyph g: glyphs) {
addGlyph(g, x + g.xoffset * scaleX,
y + g.yoffset * scaleY,
g.width * scaleX,
g.height * scaleY);
x += g.xadvance * scaleX;
}
return x - startX;
}
....

// Here we make a JNI call to get the complex script shaping done
private List getGlyphsAfterShaping(CharSequence str, int start,
int end, BitmapFontData data) {
List glyphs = new ArrayList();
int[] glyphIndices = complexScriptLayout.getGlyphsAfterShaping(str,
start, end);
for (int glyphIndex: glyphIndices) {
Glyph glyph = data.getGlyph((char) glyphIndex);
if (glyph != null) {
glyphs.add(glyph);
}
}
return glyphs;

}

complexScriptLayout is a JNI based class which simply calls into Harfbuzz
to reorder the glyphs.
I can send more details if there is interest.

Entire strings are processed at a time because the glyphs need to get
reordered and multiple
glyphs are composed to render a single screen character.

On Sun, Dec 7, 2014 at 1:12 AM, Bastiaan de Jong [email protected]
wrote:

Where system fonts are concerned, Java AWT's TextLayout classes can be
used to render just about any language and styles, inclusing full RTL
Arabic. I successfully implemented this to support Arabic text in our
product based on MT4j. We only need it to run on desktop (win/mac), so I
can't say if it'll work on mobile devices. The implementation generates one
texture for every string requested on the fly, so it's not efficient in the
least, but it does work splendidly. Maybe this is another point of
inspiration.

@sridharsundaram https://github.com/sridharsundaram I'd be very
interested to learn about the changes you had to make in libgdx to support
Arabic. Is there any way you can share that?

—
Reply to this email directly or view it on GitHub
https://github.com/libgdx/libgdx/issues/1589#issuecomment-65910840.

[bill-lin]

+1 to using system font
I am developing an app in Chinese and it is very painful to handle chinese input and cross platform.

[tdlrobin]

+1 to using system font

[abdlquadri]

@NathanSweet I edited Hiero to handle RTL. Details here. Tested only Arabic though. http://www.badlogicgames.com/forum/viewtopic.php?f=15&t=18592

But it seems that success does not extend to the core libraries because my test still renders LTR even on desktop.

[abdlquadri]

@sridharsundaram I will be interested in how you implemented that. Also does if work for the other platforms?

[abdlquadri]

Hi All,

Just wondering what progress has been made on using System Font and how I can contribute to this. Any pointer to where to start would be great.

Regards.

[sridharsundaram]

I had to modify BitmapFontCache in addToCache

....
if (isComplexScriptLayoutEnabled()) {
List glyphs = getGlyphsAfterShaping(str, start, end, data);
requireGlyphSequence(glyphs);
for (Glyph g: glyphs) {
addGlyph(g, x + g.xoffset * scaleX,
y + g.yoffset * scaleY,
g.width * scaleX,
g.height * scaleY);
x += g.xadvance * scaleX;
}
return x - startX;
}
....

// Here we make a JNI call to get the complex script shaping done
private List getGlyphsAfterShaping(CharSequence str, int start,
int end, BitmapFontData data) {
List glyphs = new ArrayList();
int[] glyphIndices = complexScriptLayout.getGlyphsAfterShaping(str,
start, end);
for (int glyphIndex: glyphIndices) {
Glyph glyph = data.getGlyph((char) glyphIndex);
if (glyph != null) {
glyphs.add(glyph);
}
}
return glyphs;

}

complexScriptLayout is a JNI based class which simply calls into Harfbuzz
to reorder the glyphs.
I can send more details if there is interest.

Entire strings are processed at a time because the glyphs need to get
reordered and multiple
glyphs are composed to render a single screen character.

On Mon, Mar 9, 2015 at 3:42 PM, abdlquadri [email protected] wrote:

@sridharsundaram https://github.com/sridharsundaram I will be
interested in how you implemented that. Also does if work for the other
platforms?

—
Reply to this email directly or view it on GitHub
https://github.com/libgdx/libgdx/issues/1589#issuecomment-77827922.

[abdlquadri]

"I can send more details if there is interest." Please send more detail.

[NathanSweet]

@sridharsundaram if you post a project that builds the HarfBuzz JNI stuff with gdx-jnigen, I'd like to take a look.

[NathanSweet]

2955 adds support for shaping characters into glyphs. It sets the stage to use HarfBuzz. Once merged I have code ready which enables FreeTypeFontData to render glyphs on the fly:

The red boxes are the backing textures, so the font has 4 pages of glyphs. The page size was made very small on purpose for testing.

Packing glyphs on the fly isn't super efficient with PixmapPacker's tree packing, since they aren't sorted. Might be better to use line by line packing, maybe using the max glyph height.

[Masterhame]

Is this problem solved? any clear solution or any package to have the completed methods for writing with complex script languages?

[NathanSweet]

Not as far as I know. libgdx has on the fly glyph rendering, so Latin, Cyrillic, CJK, and similar languages are supported. RTL languages would need something like Harfbuzz for their complex layout.

[CrowniAPIs]

RTL languages like Arabic not just direction from RIGHT to LEFT, but how connect their letters with each other and some letters as a block with 2 or more letters.
So, I do it with fully supporting RTL language.
https://github.com/CrowniAPIs/libGDX-RTL-Language

[Kieaer]

Even in October 2019, the problem is not solved.
I still don't see the combination characters I'm typing when I type them, and I can't solve the problem that only appears when the combination characters are completed.