Godot: Add support for gettext plural forms

[Calinou]

Following discussion in https://github.com/godotengine/godot/pull/31423, we should eventually implement support for pluralization as provided by gettext. Many languages have more than 2 plural forms, which means the plural forms cannot be hardcoded. Instead, the localization file should be able to define as many plural forms as the language requires.

The main focus would be to make it available in the editor, but making it usable in projects would be welcome as well. For projects, this would likely require using PO files for localization, as CSV is too limited for this.

[remram44]

Is there a particular reason a library such as gettext can't be used directly, instead of using a custom PO file parser?

That would add:

• Less code to maintain here
• Support for MO files (compiled binary message catalogs)
• Support for plurals and context, which is not there yet

Is the problem simply a build concern, it's hard to link it statically on some platforms?

[remram44]

Looking around, this would require multiple changes:

• Update the PO parser to read the plural forms from config (it's a conditional expression that takes n and evaluate to an integer, the index of the form to use, for example (Polish) plural=(n==1 ? 0 : n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2);)
• Update the PO parser to read multiple msgstr variants for each string
• Update the Translation class to store the multiple forms (and change Translation::add_message())
• Update the API, Object.tr() should take a number as argument (and for proper xgettext support, it should probably take two forms for the msgid). Those arguments can be added to tr() with proper defaults or it can be a new function (tr_plural/trs/tr_n)

[Calinou]

Is there a particular reason a library such as gettext can't be used directly, instead of using a custom PO file parser?

I believe licensing would be an issue, as gettext libraries appear to be LGPL-licensed, which we can't integrate in Godot. (The reason is that we'd like to link binaries statically without restrictions, especially on iOS where static linking is the only option.)

[akien-mga]

Indeed, LGPL is problematic since we'd have to provide a way for users to change the version of the LGPL'ed work even in the statically compiled version, which is far from trivial.

We used to have a library under LGPLv2.1 (libwebsockets), but it had a static linking exception. We're also better off without its legalese now.

Note: great resource on the LGPLv2.1: https://copyleft.org/guide/comprehensive-gpl-guidech11.html
We bundled LGPL'ed code for cross platform convenience would make things even tricker as we may be seen as a "work based on the library".

[blockspacer]

"provide a way for users to change the version of the LGPL'ed work even in the statically compiled version," - What is version of the LGPL'ed work?

What do you think about linking gettext as shared lib in community provided module? Anyone tested that approach before?

[blockspacer]

BTW, that do you think about gettext-compatible library, under BSD license?

http://wiki.netbsd.org/projects/project/libintl/
http://ftpmirror.your.org/pub/NetBSD/NetBSD-release-8/src/lib/libintl/

attempt at modernizing and updating NetBSD's BSD-licensed libintl
implementation:
https://gitlab.com/worr/libintl

[Calinou]

What do you think about linking gettext as shared lib in community provided module? Anyone tested that approach before?

Nothing prevents you from doing this, but we don't want to do it officially :slightly_smiling_face:

Keep in mind some platforms such as iOS don't support dynamic linking at all.

BTW, that do you think about gettext-compatible library, under BSD license?

http://wiki.netbsd.org/projects/project/libintl

I didn't know there were permissively-licensed, feature-complete libraries for handling PO files. Someone needs to give it a try, I guess.

[blockspacer]

BTW I can see gettext switched to LGPLv2.1, with static linking exception . Anyone can confirm it?

Based on comment from code https://github.com/autotools-mirror/gettext/blob/3c82d1a522c0b4e79168ff479d90eeadc109d9f6/gettext-runtime/intl/xsize.h

"GNU Lesser General Public License as published by
the Free Software Foundation; either version 2.1, or (at your option)
any later version."

UPD:
it gives the right to use libintl as a static library or to incorporate libintl into another library only to free software.
https://www.gnu.org/software/gettext/manual/html_node/Discussions.html

[Calinou]

@blockspacer I don't see a static linking exception in that license notice.

[blockspacer]

What do you think about ICU Message syntax
https://phrase.com/blog/posts/guide-to-the-icu-message-format/#CC
https://support.crowdin.com/icu-message-syntax/#plural
https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/classMessageFormat.html

See icu::Formattable, icu::PluralRules and base/i18n from chromium (BSD license)

https://github.com/chromium/chromium/blob/2ca8c5037021c9d2ecc00b787d58a31ed8fc8bcb/base/i18n/message_formatter.cc#L64

https://github.com/chromium/chromium/blob/2ca8c5037021c9d2ecc00b787d58a31ed8fc8bcb/ui/base/l10n/formatter.cc#L166

https://codereview.chromium.org/2736543002/

http://userguide.icu-project.org/formatparse/messages

[blockspacer]

BTW tinygettext has expression parser for plural forms, zlib license.
https://github.com/tinygettext/tinygettext/pull/23
https://github.com/tinygettext/tinygettext/blob/master/src/plural_forms.cpp

[manavspg2]

What do you think about ICU Message syntax
https://phrase.com/blog/posts/guide-to-the-icu-message-format/#CC
https://support.crowdin.com/icu-message-syntax/#plural
https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/classMessageFormat.html

See icu::Formattable, icu::PluralRules and base/i18n from chromium (BSD license)

https://github.com/chromium/chromium/blob/2ca8c5037021c9d2ecc00b787d58a31ed8fc8bcb/base/i18n/message_formatter.cc#L64

https://github.com/chromium/chromium/blob/2ca8c5037021c9d2ecc00b787d58a31ed8fc8bcb/ui/base/l10n/formatter.cc#L166

https://codereview.chromium.org/2736543002/

http://userguide.icu-project.org/formatparse/messages

I think ICU Message Syntax is a very good method of involving the use of messaging in Godot, but since we need a library for the same, we still need PO file translational method,i.e, POT file generation to enable the same.

[manavspg2]

BTW tinygettext has expression parser for plural forms, zlib license.
tinygettext/tinygettext#23
https://github.com/tinygettext/tinygettext/blob/master/src/plural_forms.cpp

Yes. tinygettext has expression parser for plural forms, but there are some conflicting files. Therefore it is recommended to built a more powerful expression parser to work in bigger files and softwares.
So in terms of support I think we can go for xgettext, but that wont be compatible in Godot. So using gettext and adding translational platform Transifex is a better option.

[remram44]

@manavspg2 you seem to be a bit confused about what this issue is about.

xgettext is just the extraction tool, it's not a translation library (which would be gettext). It should be compatible with GDScript since that's very close to Python. Or you can use the Python Babel tool, for which I wrote a plugin to handle .tscn files.

The question here is which library to use in Godot to read the PO or ICU files.

Once you have the PO files, which localization platform you use (Transifex, Weblate, or plain GitHub) doesn't really matter. If they handle PO files, they'll all be "compatible in Godot".

[mhilbrunner]

Feature and improvement proposals for the Godot Engine are now being discussed and reviewed in a dedicated Godot Improvement Proposals (GIP) (godotengine/godot-proposals) issue tracker. The GIP tracker has a detailed issue template designed so that proposals include all the relevant information to start a productive discussion and help the community assess the validity of the proposal for the engine.

The main (godotengine/godot) tracker is now solely dedicated to bug reports and Pull Requests, enabling contributors to have a better focus on bug fixing work. Therefore, we are now closing all older feature proposals on the main issue tracker.

If you are interested in this feature proposal, please open a new proposal on the GIP tracker following the given issue template (after checking that it doesn't exist already). Be sure to reference this closed issue if it includes any relevant discussion (which you are also encouraged to summarize in the new proposal). Thanks in advance!