Libgdx: Discussion: HarfBuzz shaping extension design

[Darkyenus]

In #5244 I have mentioned wanting to create HarfBuzz-based font shaping extension for libGDX. Since then I've been working on it, in my limited spare time. I have already created a JNI-Gen binding for HarfBuzz, and now I'm designing the shaper itself. It is not yet in a PR form, but I want to discuss some things I have encountered so far and how to tackle them.

First off, current BitmapFont is not well suited for being part of this, so I am creating a new "Font stack" around FreeType and HarfBuzz, which may eventually replace current one, if successful. BitmapFont does a lot of things which are not really needed in the new system, and more importantly assumes some things which are no longer true in complex layout, for example, that glyph ID maps 1:1 to char.

Separation of glyph generation and shaping

Even if the new font stack would be better than old one in every way, it could never replace it, unless it supports all platforms. And in this respect, the only one that is problematic, is GWT. It could be possible to emscripten all native libraries for JS, but it is not something I would really want to do. And since we're probably not dropping support for GWT either, I am planning to split the new stack around these three responsibilities:

• Shaping (with implementations: )
  
  
  
  • HarfBuzz
  
  
  • current approach, enough for Latin scripts, possibly extended to support widely used scripts like Arabic
  
  
• Supplying glyphs to shaper
  
  
  
  • FreeType creation on demand ("incremental")
  
  
  • Pre-rendered, cached from incremental, or even hybrid
  
  
• Rendering shaped runs to (Sprite)Batch
  
  
  
  • BitmapFontCache reimplementation
  
  
  • maybe DistanceFieldFontCache also?
  
  

This division feels more appropriate than BitmapFont/BitmapFontData, which now do "too little and too much at the same time" respectively, as BitmapFont is basically just a wrapper for BitmapFontCache + BitmapFontData, and BitmapFontData is too tightly coupled to shaping and to the idea of pre-rendered glyph pages.

While on this topic: I could be wrong in my judgement, but I'd like to simplify the new equivalent of BitmapFontCache. Does it really need the ability to add multiple unrelated text layouts and to render only a part of the text? Latter will become almost impossible to implement fast & correct with the complex layout which can omit, add or even reorder characters.

Scaling and flipping

I don't think that font should care about its scaling (BitmapFontData.setScale()) nor direction of Y coordinate (BitmapFont.isFlipped()). These things should be handled in layer one above (possibly in aforementioned BitmapFontCache) - currently it only complicates already pretty complicated code. IMO font should have only 2 size-related settings:

• size in pixels - this is what shaping works in (HarfBuzz works only on integers anyway) and this is also the size of glyphs on backing texture. Both things should be pixel perfect for pretty results.
• size in points - this is what (for example) UI system that uses the font works in. Ideally, these sizes would match on standard screen and be multiples of each other on high DPI screens (e.g. 100 points = 200px on 2x retina).

Maybe it would be best to specify only (float) size in points and (float) scaling factor to get size in pixels (int) from it. But I'm not sure on this yet.

Unicode

HarfBuzz does shaping, but does not help with BiDirectional text layout, nor deciding where line should be wrapped, nor between which chars can editing caret go (which seems to be a surprisingly tricky problem in HarfBuzz-ed unicode). Thankfully, all this can be solved with ICU library. However, it weighs around 12MB, which is quite a lot, but it should be possible to trim it as it contains a lot of unnecessary data (for this purpose).

Input text

While the current markup [BLUE]language[] is easy to use, it does not play well with most parts of the text stack. So for this font stack, I am playing with the other approach (used at least in iOS and possibly in Android): annotated text, i.e. object that associates text characters with their attributes.

This should make it easier to generate colors programatically, as text does not have to be modified anymore, and it will allow for colored text in input text fields, as markup tags previously got in the way and messed up mapping between GlyphRun indices and source text indices. Converter from current markup strings to annotated strings is, of course, planned.

Above, I mentioned attributes and not just color - I am playing with mixing different fonts in as well. This not only means multiple different typefaces in single string, but also italics, bolds, underlines (courtesy of FreeType) and also different sizes.

I am also considering to add a special support for tabulators (\t). Initially they will probably work like in terminal console ("move the run beginning with 't' to the right up to the nearest multiple of 4 spaces worth of space") but it would be pretty easy to add support for more complex behavior, customizable when invoking the text layout.

---

What are your thoughts? Are there any other things that should be changed about the current system? Some things that I want to omit which are actually very useful? Or some other features which were overlooked, but should be factored in from the start?

I'll keep this issue open until I have a PR ready.

[obigu]

Looks like an ambitious plan on what I believe is a quite complex topic.

You say:

Even if the new font stack would be better than old one in every way[...]

For those who are not familiar on the topic such as myself, even if there are hints all through your post, it's not clear which issues/limitations is trying to fix and how it would be better to the current implementation. Adding some bullet points could help identifying them more clearly.

[Darkyenus]

Good point, let me elaborate.

Why do we want HarfBuzz/complex shaping engine:

Current font system can't handle anything more complex than mapping each character of input text to single texture region, and then laying these regions next to each other. This is enough for English and similar languages, but falls apart when dealing with more complex languages. For example, á can be represented by character á (U+00E1) or by a followed by ' combining mark (U+0301). Current system can (with proper font) render only the first case, as second relies on glyph composition, which is currently not done/easily possible. However, this is just a tip of the iceberg. Other feature which is missing, is the ability to choose correct glyph variant, based on context. This is crucial for scripts like Arabic, where each letter has multiple forms (represented by different glyph IDs in font, but same unicode codepoint in text), and which one is used depends on whether the character appears at the beginning of the word, end, middle or is solitary (IIRC, I'm not entirely familiar with it). Arabic is also special, because it is written right to left, instead of left to right. Combining LTR and RTL scripts is not easy, but should get "solved" by including the ICU library (among other things, other include the problem of caret placement, when you have a followed by ' - can you place care between them?). Another feature are ligatures, but those are mostly for aesthetics and I am not aware of their use in some language's script.

All of these features are basically free, once we switch to using HarfBuzz internally. (And it may make libGDX a game framework with the best font rendering overnight :D )

There should not be a noticeable performance impact. HarfBuzz is an optimized C(++) library used in a lot of software today. There is a good chance that the text you are reading right now was shaped by HarfBuzz.

Why the new font stack

The current one (BitmapFont) is not well suited for complicated text layouts. Even this sentence is out of reach, because it has italics, which require use of different fonts inside single block of text, which would be pretty hard to get into the current system without rewriting substantial chunks of it. If we are to create a new system, it would be great if it was flexible enough to support at least markdown levels of text customization.

All of this could be left purely as an extension that would provide "HarfBuzzFont", but migrating from one system to another during development would be painful (for example, almost all scene2d.ui widgets would have to have 2 variants, BitmapFont ones and HarfBuzzFont ones). So the more logical choice would be to create shared base, which would replace the current system, and would allow multiple font rendering/shaping backends to be chosen from, based on platforms' capabilities and user requirements. (I suspect that this was original plan when creating BitmapFont - why specify that it is Bitmap when you never have anything else? But then it probably became ubiquitous and other systems (such as FreeType and distance field font) had to be "hacked" inside the same shell. But this is just a speculation.)

[obigu]

Thanks for the elabotate explanation (you don't like bullet points I see :P).

I can see how this feature is critical for apps that need support for Arabic language (are there any other "major" languages not supported by libGDX?) and regarding the text customization it's definitely a nice to have.

A couple of thoughts:

• RoboVM backend already uses a trimmed version of the ICU library which could probably either shared or reused
• I think it makes sense as an extension. I understand the problem with Scene 2D but, unless the impact on Scene 2D API is minimal and the impact of including the trimmed ICU library (which is quite heavy anyways) is not a problem, it will be quite hard to add it as part of libGDX core. I guess @NathanSweet will have a say on this :)

[Darkyenus]

Bullet points were planned, but it was too late at night, so they didn't happen :D

I completely failed to mention this before, but the plan was to only include the basis of the system (i.e. abstract classes) and a simple GWT-compatible implementation of them in the core libGDX. So the core would only see a different API and support for multiple fonts in single GlyphLayout. The FreeType + HarfBuzz implementation would stay as an extension (either replacing or depending on the FreeType extension).

Another feature that would be nice, but won't make it into the initial implementation, are fallback fonts. That is, the ability to specify, for example, Noto fonts as fallback for your own fancy font, so that users can type in any language they want and they will never encounter � character. But I don't see how to implement that easily, so it will have to wait for later versions.

I have opened this discussion, because I think that this extension would be beneficial to all libGDX users and I want to design it so that it has a chance to be merged into the mainline libGDX, eventually. But if it turns out that it should stay a 3rd party extension, that is also fine.

[NathanSweet]

BitmapFontCache. Does it really need the ability to add multiple unrelated text layouts and to render only a part of the text?

I doubt that it really needs to do either of those things.

Scaling and flipping

I never liked the scaling, but people wanted it. The flipping is a bit out of place, but it is needed somewhere along the way to rendering if y-down is to be supported. That is not something I care about, usually it's for someone new to the y-up world who doesn't want to change.

I agree BitmapFont and friends have collected a lot of cruft over the years. IIRC, BitmapFontData was introduced solely to support AssetManager.

It makes sense to introduce interfaces that BitmapFont and the new system implement. When changing libgdx usage (mostly scene2d.ui) the LCD interface may be limiting (eg no scaling). That could be special cased, though I wouldn't like that the new system makes things too ugly.

I guess whether it can live in master or is better off as a 3rd party extension is mostly up to how many others use it and are willing to maintain it. Either way, Mario is likely willing to add it to the build servers.

This is quite a big project! While I do think it would be super cool, the current system gets us pretty far. I'd say the main benefits of the new system are less about supporting more languages (we'd get Arabic and probably others, which is great, but we already have the most widely used languages) and more about the other features, like attributes and multiple fonts. Fancier layout would also be super useful, eg inline or floating images. You've obviously put a lot of thought into a new system and likely have a good grasp on the difficulty level. Just a thought, but have you considered lower effort improvements of the current system to achieve some of the same goals (of course excluding Arabic and similar languages)?

[Darkyenus]

[...] LCD interface may be limiting (eg no scaling)

I'm not entirely sure what you mean by this, but if it is about the "pixel-perfect" part, it shouldn't be a problem. IIRC FreeType does not do any "subpixel" rendering that would be messed up by scaling and non-pixel-perfect translations. Even in current system, fonts look good only when they are rendered pixel-perfect and I believe that with omission of scaling at font level, it should be even easier to achieve that in the current system.

Font level scaling seems to be (in libGDX) only used in scene2d.ui.Label for fontScale property. It is already implemented through a workaround (changing BitmapFontData scale on each frame does not seem particularly clean), so replacing it with a different, possibly cleaner workaround doesn't seem to be that bad. For pretty font rendering it shouldn't be used much anyway, but I can see its utility, the slight visual degradation is usually more tolerable than setting up something more complex, like distance field font.

The scaling would be easy to implement in the BitmapFontCache. It would also be a better fit API wise.

have you considered lower effort improvements of the current system to achieve some of the same goals (of course excluding Arabic and similar languages)?

I originally wanted to do mainly the HarfBuzz, the other improvements (multiple font layout) came later. But it seems to me that the non-harfbuzz layout also has its place in libGDX (for GWT, if not for anything else), so the current plan is to create generic font interface which would expose relevant layout features, and two implementations:

1. Simple, Java only, GWT compatible, similar to the current BitmapFont, "low effort"
2. Full HarfBuzz & FreeType, for those that want full foreign language/unicode support

I still need to experiment a bit with the general API design, but when I have something I'm happy with, I'll post it online for further API feedback.

[NathanSweet]

When changing libgdx usage (mostly scene2d.ui) the LCD interface may be limiting (eg no scaling).

Sorry, by this I was trying to say the lowest common denominator interface probably doesn't have font scaling like BitmapFont, so scene2d.ui Label or other code relying on that might need special cases to keep the same functionality.

What happens to GlyphLayout? Does it fall into the simple/GWT compatible bucket? Currently it is used for measuring, so it is tied pretty closely to shaping.

[Darkyenus]

Since GlyphLayout does the shaping, it will have one implementation per "font backend". Each version will implement shared interface, which will expose some common functionality, e.g. setText(text, width, wrap, etc.), measuring, and drawText(BitmapFontCache) (not real signatures).
Normal user code will likely access GlyphLayout directly to do the font layout, so that we don't have to duplicate the lengthy setText signature on each class of the font stack. GlyphLayout will also provide special API for caret measurement (for text fields), as it is no longer completely straightforward, especially around ligatures and text with mixed directionality.

[NathanSweet]

Sounds cool! Our text area and even text field support has always been a bit janky. I wish you the best of luck, I know it's a huge project.

[terefang]

hmm ...
just a side note why do you want to use harfbuzz directly,
if you can use Pango (https://www.pango.org/),
which also lets you do attributed text (https://developer.gnome.org/pango/unstable/PangoMarkupFormat.html)
aside from handling RTL/LTR.

[Darkyenus]

I have considered Pango initially, but it seemed to too much. However, in retrospect, it may be a better option, as we actually want a lot of the "too much". And the API seems somewhat sane, although it seems to depend on fontconfig, which we probably don't really want.

[terefang]

hi

looking at the configure scripts it seams not only is fontconfig+freetype
required, but also at least cairo+freetype.

i am with you that fontconfig is an entirely other universe to tackle as it
duplicates much what is within java or android.

webkit for android (https://github.com/naver/webkit-android-libraries)
already does it (fontconfig+freetype+harfbuzz+cairo)

hmm ... could be an alternative route to downstrip webkit to bare needed
essentials for libgdx.

On Wed, Jul 4, 2018 at 12:11 PM Jan Polák [e-mail] wrote:

I have considered Pango initially, but it seemed to too much. However, in
retrospect, it may be a better option, as we actually want a lot of the
"too much". And the API seems somewhat sane, although it seems to depend on
fontconfig, which we probably don't really want.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/libgdx/libgdx/issues/5278#issuecomment-402433219, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAeCehG0A5Xeubal19UN-IDWLvLfFYaHks5uDJTHgaJpZM4U099y
.

--

Document My Code? Why do you think they call it "code" ?

App developers spend too much time debugging errors in production systems

https://betanews.com/2016/11/03/developers-debugging-production-errors/

[Darkyenus]

I believe that it would be possible to use Pango without Cairo, but after some further investigation, I'll probably not use Pango, as it pulls too many redundant dependencies and seems to be a pain to build. In addition, I have discovered that Java already exposes APIs for things that would need ICU library (Bidi and BreakIterator). And working around fontconfig would also be annoying, as it seems to assume that everybody is building an OS or office suite, not small game/program with well defined fonts.

Doing anything with webkit seems a bit... excessive, it would be much easier to make pango work than to do anything with webkit. And I don't really want to modify any native code, because it would only inflate the maintenance load on libGDX maintainers - especially as most contributors are well versed in Java, but PRs for native code are very rare.

And even personally, I'd rather write more code in Java, than to do few C hacks, so after some investigation into Pango, I'm going back to pure HarfBuzz.

[terefang]

according to this (https://bugs.openjdk.java.net/browse/JDK-8064530)
harfbuzz should be in jdk9, maybe you can short-circuit from there.

On Fri, Jul 6, 2018 at 1:16 PM Jan Polák [e-mail] wrote:

I believe that it would be possible to use Pango without Cairo, but after
some further investigation, I'll probably not use Pango, as it pulls too
many redundant dependencies and seems to be a pain to build. In addition, I
have discovered that Java already exposes APIs for things that would need
ICU library (Bidi
https://docs.oracle.com/javase/7/docs/api/java/text/Bidi.html and
BreakIterator
https://docs.oracle.com/javase/7/docs/api/java/text/BreakIterator.html).
And working around fontconfig would also be annoying, as it seems to assume
that everybody is building an OS or office suite, not small game/program
with well defined fonts.

Doing anything with webkit seems a bit... excessive, it would be much
easier to make pango work than to do anything with webkit. And I don't
really want to modify any native code, because it would only inflate the
maintenance load on libGDX maintainers - especially as most contributors
are well versed in Java, but PRs for native code are very rare.

And even personally, I'd rather write more code in Java, than to do few C
hacks, so after some investigation into Pango, I'm going back to pure
HarfBuzz.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/libgdx/libgdx/issues/5278#issuecomment-403005157, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAeCegKTYLCbHMceusfMTHL4eybuTgeuks5uD0cqgaJpZM4U099y
.

--

Document My Code? Why do you think they call it "code" ?

App developers spend too much time debugging errors in production systems

https://betanews.com/2016/11/03/developers-debugging-production-errors/

[Darkyenus]

I am aware of that, but it is very unlikely that we would be able to use it, as it won't be publicly accessible (at least without some hacks). And I don't think that it is worth trying to pursue, as it would limit our users to openjdk >= 9, which is only a relatively small subset of JVMs that we run on.

[terefang]

no, i mean you could use the code as inspiration

actually ... the code is available here:
http://hg.openjdk.java.net/harfbuzz/jdk9/jdk/rev/8c09472c3de2

On Sun, Jul 8, 2018 at 1:55 PM Jan Polák [e-mail] wrote:
>

I am aware of that, but it is very unlikely that we would be able to use it, as it won't be publicly accessible (at least without some hacks). And I don't think that it is worth trying to pursue, as it would limit our users to openjdk >= 9, which is only a relatively small subset of JVMs that we run on.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

--

Document My Code? Why do you think they call it "code" ?

App developers spend too much time debugging errors in production systems

https://betanews.com/2016/11/03/developers-debugging-production-errors/

[terefang]

and having poked around some more:

/src/java.desktop/share/native/libfontmanager/harfbuzz contains the
harfbuzz source
/src/java.desktop/share/native/libfontmanager/layout contains the icu
opentype layout engine source applicable for harfbuzz
/src/java.desktop/share/native/libfontmanager/ contains the glue code
for java/awt/font
On Sun, Jul 8, 2018 at 8:36 PM Terefang Verigorn [e-mail] wrote:
>

no, i mean you could use the code as inspiration

actually ... the code is available here:
http://hg.openjdk.java.net/harfbuzz/jdk9/jdk/rev/8c09472c3de2

On Sun, Jul 8, 2018 at 1:55 PM Jan Polák [e-mail] wrote:
>

I am aware of that, but it is very unlikely that we would be able to use it, as it won't be publicly accessible (at least without some hacks). And I don't think that it is worth trying to pursue, as it would limit our users to openjdk >= 9, which is only a relatively small subset of JVMs that we run on.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

--

Document My Code? Why do you think they call it "code" ?

App developers spend too much time debugging errors in production systems

https://betanews.com/2016/11/03/developers-debugging-production-errors/

--

Document My Code? Why do you think they call it "code" ?

App developers spend too much time debugging errors in production systems

https://betanews.com/2016/11/03/developers-debugging-production-errors/

[Darkyenus]

I have published what I currently have to https://github.com/Darkyenus/libgdx-new-font-api .

Feel free to check it out and provide feedback about the API, but please read the README first.

[Darkyenus]

Quick update: pure Java implementation (BitmapFontSystem) is almost complete, it just needs some optimization for long texts and I will move to the HarfBuzz backend. (Old BitmapFont lays out my long testing lorem ipsum in about 0.3 ms, new does it in about 1.3 ms. This is unacceptable, but should be pretty easy to get down under 0.5 ms and I'll see what can be done from there. That said, it already has slightly better memory requirements, but not by much.)

Check out the readme of the repo, it has a tame preview of the multi-font functionality.

[MrStahlfelge]

Closing this, I suppose the discussion has come to an end. :) @Darkyenus I think adding a link to your repo on the wiki would be a good idea. A suggestion to the readme of the repo: For me as a developer using fonts (and being quite unhappy with them), but not being invested in the topic, I miss information if I can use the repo in my projects or not. I see there is a list of tasks not done yet, but I don't know which drawbacks I will have using the repo.

[Darkyenus]

My repo is here and while it works as a proof of concept, it is not yet finished and ready for general use. The main problems from the usability point of view right now are:

• No integration into existing font infrastructure (no scene2d, etc. integration)
• No pre-built artifacts of natives (HarfBuzz)

And the code is not 100% finished either, although most of the "hard problems" are already solved. So it is not something any random user can just start using. But it can serve as a robust base for any further development, if you are prepared to put some effort in.

I am currently not planning on working on it further, but I will be able to help and answer questions about it, should somebody want to continue working on it.