Supertux: Arabic Not Displayed Correctly

[Fahad-Alsaidi]

After reading in changelog about adding Arabic font to supertux in this commit:
https://github.com/supertux/supertux/commit/54c3cb298de0e1bfce6d512764e08ccfdf72c22f

I tested the dev version in Ubuntu 14.04. what I found is that arabic letters are not connected together and have wrong order (from left to right). They are useless in current status.

[tobbi]

Ah, thanks for the bug report. So the order of the letters is incorrect, but the orientation is correct, right?

[Fahad-Alsaidi]

the orientation is incorrect also.

[tobbi]

So, basically it needs to be mirrored on the x-axis and the order of the letters reversed?

[Fahad-Alsaidi]

Yes, but I am not sure if the letter will get connected by these step only.
However here is a project you may find it interested
https://github.com/lxnt/ex-sdl-freetype-harfbuzz

[tobbi]

Can you please check out the latest master, and see whether at least the orientation is correct? I will deal with connecting the letters in a separate commit.

[Fahad-Alsaidi]

I didn't install required packages to build it from the master. I have to wait the next daily build for ubuntu to test it.
If you can just take a screenshot and post here, I can tell you whether it is correct or not.

[tobbi]

See http://screencast.com/t/DHZEQAt9

[Fahad-Alsaidi]

actually no thing change !

[tobbi]

Interesting. apparently, the rtl detection code doesn't work... I'll look into it.

[tobbi]

The problem is that we set check the rtl flag globally, that means: If a non-arabic font is loaded after the arabic one, the flag gets overridden.

I hope you're still there to test any changes I make to fix this problem?

[Fahad-Alsaidi]

Is there any update regarding this issue?

[tobbi]

Yes, the update is that I have "kind of" found the cause. And we are debating switching to a different font because of this. Currently, the font glyphs are inside image files, which is a pain to handle. In order to reverse the order of a string, I tried reversing it using methods from std namespace, which didn't work due to the unicode nature of the text string.

So, I will probably try thinking through SDL_ttf or harfbuzz, as you mentioned above, before the next release.

[Fahad-Alsaidi]

ok. let me know if you want help in testing. I know little c++ coding, so I think I can help.

[tobbi]

We won't make this for 0.4.0, unfortunately. It's only 2 months time and I doubt we (or I) can make it in this time. Thus postponing it till after 0.4.0. Sorry about this.

[tobbi]

I hope that after all this time you still read this issue. Since SDL_ttf support is nearly complete, I'd like you to please find a suitable font we can use. Requirement is that it's under an open-content license, such as the SIL Free Open Source license.

[Fahad-Alsaidi]

Thanks for your work to implement SDL_ttf. Unfortunately, SDL_ttf dosen't support any RTL languages. The good news is there is a patch to make SDL_ttf support arabic here 3211.
I don't know what is the right approach to include it in superTux.

[maxteufel]

@Fahad-Alsaidi: We're only saying SDL_ttf support (font rendering) is complete. The most important parts for this to be fixed still need work:

• use GNU FriBidi (or any other implementation of the Unicode bidi algorithm) for RTL language support
• use HarfBuzz for text shaping

I think that should fix all issues mentioned.

If you'd like to help, feel free to submit any patches in form of a pull request against the feature/modern_fonts branch.

[Fahad-Alsaidi]

@maxteufel I got it and what I am saying is there is a patch for SDL_ttf which makes it uses FriBidi & HarfBuzz and it provides a very decent support for CTL. Actually, the patch uses a library called Raqm to do the heavy work.
Now, the way forward up to you guys, what I can think about it right now is either make a local version of SDL_ttf patched to support CTL, use SDL & freetype directly without using SDL_ttf at all (something similar to pygame does), or talk to SDL_ttf dev to find a solution.

[maxteufel]

Well, I'm not sure right now: can't we use HarfBuzz with SDL_ttf or do we need FreeType for that? I admit this is something we hadn't evaluated before choosing to start with SDL_ttf, but I really don't want to depend on more local libraries and a mess of submodules, so using FreeType might be the better solution.

[Fahad-Alsaidi]

@maxteufel but this means redoing all the work done recently !!

[maxteufel]

Is there a problem with that? The only disadvantage is (partly) throwing the old work away. Which is not really a disadvantage since FreeType doesn't seem to be much different from SDL_ttf (at least the API part). The advantages, which are not having to take care of in-tree dependencies and not using unsupported patches, clearly overweigh.

[Fahad-Alsaidi]

in that case, I suggest to use libraqm to deal with FriBidi & HarfBuzz .

[maxteufel]

libraqm is not packaged by all distributions and just adds another 3rd-party we have to rely on to the chain.

[Fahad-Alsaidi]

libraqm is packaged for archLinux, the work in progress to package it for Debian & Fedora. There is ppa for Ubuntu 14.04. Using libraqm will save you a lot of time and energy. For packaging problem, you can use a fallback strategy in case libraqm not available, something similar what ImageMagick did.

However, writing something similar to libraqm is not that difficult and I am sure you can do it.

[maxteufel]

• https://aur.archlinux.org/packages/libraqm-git/ is not an official package, just a PKGBUILD in the AUR
• Debian and Ubuntu: Third-party PPAs are shady, and the packaged version will probably only appear in later releases of both distributions (which can take a very long time for Debian)
• Same concerns for Fedora's Copr build system and it will take too long to appear in RHEL, CentOS and Scientific
• Other distros I checked that do not include libraqm packages: Gentoo, Exherbo, Void Linux, NixOS, Alpine
• Also unclear is how libraqm will affect portability to the *BSDs, to Windows, and to OS X

[moceap]

@Fahad-Alsaidi @maxteufel
It will be in Fedora (and Redhat EPEL) soon.

https://bugzilla.redhat.com/show_bug.cgi?id=1329807

Edit (by @maxteufel, 2015-04-23 15:33:15 UTC+2): Merged comments.

[maxteufel]

Fedora and RedHat are not the majority of distributions. Additionally, this is completely useless for older versions of the distros (it should be possible to compile new releases with as few unpackaged dependencies as possible).

Still unclear is how this will affect portability.

[moceap]

Everything needs time to done ^_^

To solve this issue add Raqm support as an option during building .

[maxteufel]

There is clearly no need to discriminate against users of certain distributions when we can support the same features without having to do that.

[maxteufel]

@Fahad-Alsaidi @moceap Since I don't know Arabic, could you tell me if this is the correct orientation+shaping: https://urandom.eu.org/i/oDGhumZN.png

Edit: This uses SDL_ttf with the libraqm patch, and is available in the feature/modern_fonts_v2 branch.

[Fahad-Alsaidi]

Yes, they are correct. Great !!

[maxteufel]

Sadly this will probably not be in our next release since it needs some more improvements (fallback fonts), build system tweaks (since we're shipping a custom SDL_ttf), and much more testing (0.4.0 shipped so many issues, we're a bit more careful now).

[tobbi]

I'd like to point out that the Arabic translation is at 3% [_Edit by @maxteufel:_ after some recent decisions in the project, this most likely wouldn't even ship in the release]. @Fahad-Alsaidi If you want to help get SuperTux ready for 0.6.0 (whenever that'll be released), I can add you on transifex.

[Fahad-Alsaidi]

@tobbi sorry to late reply, sure I can help but there is not benefit from translating something broken!!. Please let me know when the project is ready to translate to Arabic.

[maxteufel]

@Fahad-Alsaidi It does make sense to translate it, since then there is less work to do when the feature is merged! We expect to release 0.5 before the end of this year, after that is done, we will merge our SDL_ttf branch into the main development tree as fast as possible to allow broad testing. It would be great if we could already show some translations at that point!

[Fahad-Alsaidi]

Ok then you can add me on transifex.

[tobbi]

You're already on Transifex as a coordinator for Arabic. You can go ahead and start translating.

[tobbi]

Just to give you guys a quick update: We're currently in the process of merging together different fonts into one general SuperTux font. Updated screenshot:
https://prnt.sc/krthel

[qwertychouskie]

Looking nice!

[tobbi]

As the SDL ttf branch is now merged and libraqm is used for all OSes, I'm closing this!

[Fahad-Alsaidi]

I just want to say thanks for the great work.