Privatebin: Consider migrating localization to a translation platform

[yarons]

I would like to offer using a localization platform for translation management.

There are numerous solutions for that such as Weblate, Crowdin, Transifex, TranslateWiki, etc.

These systems has many benefits such as:

• Translators are not using git
• There are less worries about breaking the json files
• The translators are getting updates when there are new strings
• Translators can collaborate, suggest and watch screenshots of the string in question

I would love to assist with the migration if necessary.

[elrido]

I am all in favour of making translation as simple as possible. How do these platforms integrate with the git & pull request based process? In the end we release the software via git archive, so ideally such a solution could raise a pull request for the CI/CD to validate them and one of the maintainers to sign off.

[rugk]

Usually it syncs the files with git commits or so.
But what data format they support is the big question…

[yarons]

Well Weblate has a tight git integration so it's not a problem, regarding fine formats:
https://docs.weblate.org/en/latest/formats.html

[elrido]

So, shall we set up a proof of concept in master using weblate? I assume we would need to create some organization account on weblate and integrate it with the github based repo. Anything else?

[yarons]

You are correct, if there's anything I can help with I'd love to.

Furthermore, I'd love to assist with mapping the strings to screenshots (needs admin rights).

Thanks!

[elrido]

I looked into Weblate. The hosted solution isn't free (even for OSS projects), but it is OSS itself and one can self-host it. They provide a docker compose recipe and container images, so I was able to test it on my local machine.

They did automatically detect our translation files and imported them as "JSON-Nested-Structure". Here is how the plurals get displayed in the translation screen:

While this looked promising, changing and saving one of the translation strings resulted in a mess. Here are some examples of the results:

{
    "PrivateBin": "PrivateBin",
    "%s is a minimalist, open source online pastebin where the server has zero knowledge of pasted data": {
        " Data is encrypted/decrypted <i>in the browser</i> using 256 bits AES": {
            " More information on the <a href=\"https://privatebin": {
                "info/\">project page</a>": {
                    "": "%s это минималистичный Open Source проект для создания заметок, где сервер не знает ничего о сохраняемых данных. Данные шифруются/расшифровываются <i>в браузере</i> с использованием 256 битного шифрования AES. Подробнее можно узнать на <a href=\"https://privatebin.info/\">сайте проекта</a>."
                }
            }
        }
    },
    "Because ignorance is bliss": "Потому что неведение - благо",
    "en": "ru",
    "Paste does not exist, has expired or has been deleted": {
        "": "Запись не существует, просрочена или была удалена."
    },
    "%s requires php %s or above to work": {
        " Sorry": {
            "": "Для работы %s требуется php %s или выше. Извините."
        }
    },
[...]

For reference, the original looks like this:
https://github.com/PrivateBin/PrivateBin/blob/633ec4b199896afb5f62f43db45e426c7bd3bae6/i18n/ru.json#L1-L11

So, unfortunately, weblate doesn't deal with our custom, bilingual and plural JSON and we would need to host and maintain our own instance as we have no budget.

Are there other such services that we could try?

[rugk]

The hosted solution isn't free (even for OSS projects)

the last time I checked, you could/had to apply there as an FLOSS project and would then get an account.

[elrido]

the last time I checked, you could/had to apply there as an FLOSS project and would then get an account.

That is nice of them, but that doesn't solve that our format isn't supported by them. Other services you would suggest?

[yarons]

Is it documented?
Is it supported by translate-toolkit?

We can open a bug on any of them and ask them to support this format, then it will be supported officially (for many other projects as well).

[elrido]

We use a custom format, not one of the supported ones. Ours is JSON, but bilingual and with plurals - basically like Gettext, but stored as JSON instead of a binary representation.

When I implemented translations, there weren't yet any standardized formats for JSON translations. And we do want to use JSON so we can use single file format in the frontend and backend and not have to support multiple translation formats or a JS library that converts XMLs or gettext po's into something that works in the browser.

Changing the format at this point would be very labour intensive, as the JSON formats supported on weblate all are monolingual, which means you also have to change all of the source code that uses translations in both JS and PHP to switch from human readable messages to machine readable message IDs.

I am sorry, but I do not want to spend that much work, just to be able to use this tool. Are there other tools that do work with bilingual, plural JSON files out of the box?

[yarons]

I don't know all the dirty details of i18next but it might be a relatively simple solution to implement:
https://www.i18next.com/overview/supported-frameworks

If not Weblate, can you try and use Crowdin, it also has a pretty wide support of formats (although not fully open source).

[elrido]

I had looked at it and it is a monolingual format, hence the difficulty of conversion as outlined above.

Can we please focus on finding a solution that works for us as is and that doesn't require someone having to rewrite the translation system from scratch, including all of the translation files?

[yarons]

You resent the Weblate link but I suggested Crowdin :)

Are you sure that i18next is a complete rewrite? It looks pretty similar to what you were trying to achieve and supports both JS and PHP.

[elrido]

Ok, I'll give Crowdin a try next weekend.

Yes, i18next is monolingual, not bilingual, which means that not only the translation file format needs to be converted and a i18next library implemented, but also the human readable messages would need to be converted to machine readable message IDs in both JS and PHP code. That would be a substantial piece of work - it would also IMHO make translation more difficult when using the JSON files in git (as I prefer to do).

[elrido]

I got the integration setup and can edit the translations in Crowdin at https://crowdin.com/project/privatebin

It syncs changes into the i18n branch, so I assume we can then pick up changes from there back into master.

Haven't figured out the correct mapping yet, it seems to always create everything under preset language folders instead of changing the files in place. The machine translation is helpful and I could use it to translate a few sentences in Italian by just clicking on the suggestion that sounded the best: https://github.com/PrivateBin/PrivateBin/commit/56dd17560fb74c00d353e93999de0f86a32ae673

I'm not to happy that these commits are all done in my users name. Organizations (in Github) don't have an account themselves, so we may create a dedicated bot account for this purpose. I'll try to fix the mapping issue, first.

[elrido]

With some help from the crowdin support I might finally have figured out the mapping of the translations. They need a single source file with the English strings, so I needed to generate an i18n/en.json for that purpose, even though that won't be used in our translation system otherwise.

Now I need to tackle the account issue, so this doesn't all get committed on my users account, see above.

[elrido]

Ok, another change required by Crowdin is that now all translations files contain all possible plural forms, even if a particular language only has one plural. The additional plurals will increase the filesize but should not be accessed as the plural to use is restricted by language (logic below for reference).

https://github.com/PrivateBin/PrivateBin/blob/91062a28e35c328316076107fc40c54b90e668ad/js/privatebin.js#L767-L796

[yarons]

Hmmmm does that mean you'll have to manage the plural forms yourself?
There should be a repository for that, either in CLDR or the gettext base.

[elrido]

We always had to manage the plural forms in our translations. Many languages have complex rules for them and so this can't be automated. A common mistake of us programmers is to assume that plurals can just be created by appending a suffix to the singular, like the "s" for plurals in English. But if you look, for example, at Slovene, the words can change quite drastically from plural to plural and the different time units seem to follow different rules on how the singular changes into the different plural forms:
https://github.com/PrivateBin/PrivateBin/blob/5ec72f1d89790e7e31922d492b14f0ce9cefe414/i18n/sl.json#L49-L72

[elrido]

Ok, I think it finally works with a minimal back and forth. I do like the suggested auto-translations of the strings in crowdin. It also tells you when there is non-matching punctuations like a question mark in the translation, or an additional dot where there is none in the source or extra brackets or white spaces around the translated string. Stepping through some translation files I did notice some inconsistencies that I didn't before. It does have a discussion feature, so translators could discuss over there outside of the github issues.

So it was a bit of a struggle, but I think it might pay off to lower the translation barrier like that.