Meshcentral: Multi-language Support

[tasagore]

Hi

Not really a issue but I think that the project should support more languages. As far as I've seen the texts are embedded in the code (html files in views folder?) so I suggest as first step to separate it to be able to have different languages with a common code. For example a easy way is to have in XML files, example:

english.xml
<data name="CommandRelease" xml:space="preserve"> <value>Release</value> </data> <data name="CommandAcquire" xml:space="preserve"> <value>Acquire</value> </data> <data name="ConnectToGuest" xml:space="preserve"> <value>Click here to connect to guest</value> </data>

spanish.xml
<data name="CommandRelease" xml:space="preserve"> <value>Liberar</value> </data> <data name="CommandAcquire" xml:space="preserve"> <value>Adquirir</value> </data> <data name="ConnectToGuest" xml:space="preserve"> <value>Pulse aquí para conectar con el invitado</value> </data>

and so on.

The code should be modified to support that xml file but probably now it's affordable before the project becomes bigger.

It's just an idea, what do you think?

[Ylianst]

Sorry for the detail, just got back from traveling. Yes, I am supportive of this and your absolutely right that this is the first step. It will take some time to catch up on a bunch of requests, but I do want to start small with a bunch of string and grow this list.

[Floche87]

I can help for french translation.

[waldek1984]

I can help for polish translation :)

[Ylianst]

I am been working today on multi-language support. It's going to take a while, but once done, it should make it easy to translate to other languages.

[MailYouLater]

@Ylianst: What tools are you using to do this?
I started playing with this a few days ago, and I currently have it working (minus moving strings into translation files and actually having strings for multiple languages) using i18next, i18next-express-middleware and i18next-node-fs-backend. I'm not sure I did it the best way, or even 'right', but it does function.
Can I get your thoughts?

[MailYouLater]

@Ylianst: Ping!

[jsastriawan]

I think @Ylianst is working on that infrastructure. He used to have translation tools for the old Meshcentral. Let's wait. He has a lot on his plate. 😀

[Ylianst]

Oh! Sorry, just noticed the "Ping!". So, an update on the multi-language support. I am been hard at work on it for 3 days and making good progress. As a result, I have not been posting much. First, I have a new tool that pulls strings out of web pages and JavaScript and generates a JSON file that I can load in an updated version of the "Resource Translator Tool" (Built in C#). To pull strings out of the JavaScript, I changed many of the pages to use double quotes "" for English text and single quotes '' for anything else. This way, the extractor works really well.

I need to do more work on the Resource Translator Tool, it used to be able to call Google Translate on a batch of strings, it's broken now, I need to fix this feature. In the screen below I just use a mode where I swap letters A->Z, Z->A and generate a test translation in seconds.

I then have the same extractor tool run in a different mode and take the original web page and translation file and generate a new translate page. So far, it works great. I still need to get MeshCentral to send out translated pages based on browser settings and, as you may see below, I forgot to pull some English strings that are in element attributes. Like HTML button value="Group Action" and things like that. So, still some work required on the extractor/translator.

Fixing the default.handlebar to use "" and '' at the right places was a real pain, took many hours to do. I still have the mobile web site left. I also changed a bunch of strings around to make them easier to translate using format("This is a number {0}", x). Hope to have a release next week.

[Ylianst]

Maybe I am going to feel really dumb doing all this because I did not know about i18next, i18next-express-middleware and i18next-node-fs-backend. I will take a look at these and see if they can be of use. I generally think I will end up with pre-translated pages with nothing done at runtime, however, I will investigate these new modules.

Sorry for not answering your "ping" earlier...

[MailYouLater]

Don't worry too much about missing my ping, I know you have a lot on your plate already.

So anyway, I don't have it in front of me right now, but basically I was able (after some trial & error, and a steep learning curve because I'm not that familiar with node) to add around 10 or so lines of code across about three or so strategic places in the webserver.js file, add a .json file containing a few test strings for each of a couple of languages, switch out some text in some of the .handlebars files for a reference to where I'd put them in the .json files, and it worked pretty well, except that for some reason I couldn't get the saveMissing option to work. (It's supposed to operate while the server runs and make a file next to the main .json translation file containing an overview of strings that were used in the page but didn't exist in the current language, and thus had to use the strings from the fallback language.)

My current code uses the middleware's browser detection to automatically choose the language, and can be overridden with ?lng=<language code> in the URL, but it shouldn't be too hard to set it to use a cookie and set that somewhere in the UI, that way you don't have to always have ?lng=fr (or de, etc.) in the URL if you don't want to use what it auto-detects.

It's also possible to load the translations on the client side instead of the server side (though I haven't looked into it too much 'cause I was having enough difficulty getting as far as I did), which should allow for more dynamic language switching, so you don't have to refresh the page to switch to a new language.

If you want to see what I have so far, I can post it here tomorrow, but seeing as you're much more familiar with both the MeshCentral codebase, and node/express/handlebars than I am, you can probably glance at the docs and have something working better than I do in much less time.

[MailYouLater]

Note: I really feel like having separate views (I think that's the right term) for each language is not the right way to go, as it will make it much more difficult to maintain (for both you and translators) in the long term than a single .handlebars file (per view) referencing language specific .json files.

[Ylianst]

On the separate views things, yes we will have many "default-[lang].handlebars" files but they are all auto-generated. There will only be one default.handlebars and one translation dictionary to really manage. I don't expect anyone to be making changes to the auto-generated files if that makes sense. This way, there is no processing of the files are run time.

I think there are currently over 1500 strings to translate, most of them will hopefully be done using a machine translators and then edited manually when needed. I am thinking I will be adding the extract/translate feature to MeshCtrl. I think it will all make sense in a few days when I get this published.

[Ylianst]

Just published MeshCentral v0.4.2-w along with a new blog on multi-language support. Basically, all you need to do is:

• Download the Resource Translator Tool for Windows.
• Load the "translations.json" file from "node_modules/meshcentral/translate".
• Select a language, translate from strings and save the file again.
• Run "node node_modules/meshcentral/translate.js translateall" to re-translate the pages.
• Refresh MeshCentral to see you changes.
• Send me your latest translations, so everyone can benefit.

It's pretty simple and works well. I have not done page minification yet. If someone can try this out and comment on their experience, that would be wonderful. Thanks.

[Floche87]

Is there a linux version of "Resource Translator Tool" ? Another tool ? or can i directly edit the json file ?

[Floche87]

I have an error when running "node node_modules/meshcentral/translate.js translateall"

internal/modules/cjs/loader.js:583
throw err;
^

Error: Cannot find module '/opt/meshcentral/node_modules/meshcentral/translate.js'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
at Function.Module._load (internal/modules/cjs/loader.js:507:25)
at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
at startup (internal/bootstrap/node.js:283:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:743:3)

I thing the subfolder "translate" is missing

if i add it, i have another error :

MeshCentral web site translator
fs.js:115
throw err;
^

Error: ENOENT: no such file or directory, mkdir '../views/translations'
at Object.mkdirSync (fs.js:753:3)
at start (/opt/meshcentral/node_modules/meshcentral/translate/translate.js:95:67)
at InstallModules (/opt/meshcentral/node_modules/meshcentral/translate/translate.js:358:121)
at Object. (/opt/meshcentral/node_modules/meshcentral/translate/translate.js:35:1)
at Module._compile (internal/modules/cjs/loader.js:689:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Function.Module._load (internal/modules/cjs/loader.js:530:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)

Any ideas ?

[Ylianst]

Hi. Just update your server again. I forgot to package the "translate" folder in MeshCentral v0.4.2-w, but added it in later releases. That folder should include a "readme.txt", "translate.js" and "translate.json" files as shown below.

I don't have a Linux/MacOS version of the "Resource Translator Tool". This tool is an update of what I used for MeshCentralv1, so it's an old tool. I could take a look at having a web interface version of the same tool at some point.

[MailYouLater]

And we're already seeing issues that wouldn't be occurring if you'd used i18next. Oh well.
Plus with i18next you could easily set up a project at any number of online translation management tools for people to have a web interface that even tracks how complete each translation is.

[Ylianst]

@MailYouLater Ok... will look into it. I am traveling starting next week for two weeks and wanted to get this out the door. I need to take a better look at i18next.

[Ylianst]

So, looking at i18next, like the sample here. I would have to change all my strings in every web page to i18next.t('key'); where 'key' is the string key name. That would be massive amounts of work. I still feel it in my arm of having to type ' and " all weekend long (to separate English strings). Maybe if i started that way. Willing to take a look at this again down the road.

This said, what I can do next is generate a standard file format to load into the translation tools instead of the custom JSON format I have now. That way, I can ditch the really old Resource Translator Tool which is not great. I have to research a translation tool and file format to support. Suggestions on this welcome.

[MailYouLater]

I didn't use i18next.t('key'); in my working copy. I saw it in the docs, but didn't know where and how to use it in that way. I followed some different examples, and used {{t 'key' }} in the .handlebars files instead.

Anyway, method aside, internationalization is now present, and while perhaps the way it's done will be changed in the future, that shouldn't prevent this issue from being closed now that the new feature has been added. Thanks for all your work @Ylianst.

For anyone trying to switch languages, it looks like it detects the language set in your browser by default, and if you want to change it manually you have to set it in your account settings, and it also seems you need to refresh the page to load the new language strings.

[Ylianst]

Ok, I will close this one and as time permits, I may keep adding more French translations. I am still looking at options, for example this page makes a lot of sense to me. I am sure I will keep improving things.

[MailYouLater]

In practice that actually looks very similar to i18next. Either of these look good to me, or really anything where the translations are kept separate from the web page formatting. I just feel like the way you have it working right now is a bit too complex and application specific for a lot of potential translators to be able to just jump in and help out. It would be really nice to be able to use something like Zanata or Weblate for translators to submit their translations, this would lower the barrier to entry even more.