Docker-stacks: Translating documentation

[parente]

We have good US English documentation in this project thanks to the efforts of quite a few people. It would be excellent if we had translations of the docs to promote broader use of and contribution to the docker stacks. Motivated by @leportella comments, I'm opening this issue as an invitation to anyone who would be interested in helping out, either with the translations or figuring out how best to setup the translation-build-deploy pipeline.

Full disclosure: I am an English-only speaker. Creating a translation is not something I can do, but setting up the infrastructure to support translation is.

Some things to research and consider:

• Read The Docs supports translations via .pot/.po files and subprojects linked to the main project. (https://docs.readthedocs.io/en/stable/guides/manage-translations.html) It might be worthwhile setting up a dummy project using one or more of the workflows they suggest to see how approachable the process is.
• Jupyter Notebook has Korean translation of some of its docs. From a few minutes of investigation, it looks like it's not setup to build on Read The Docs, but rather committed to a subfolder and meant to be viewed on GitHub. (https://github.com/jupyter/notebook/tree/master/docs-translations/ko-KR)
• Per @betatim, we should think about whether tools like Google Translate might prove useful for "patching" translations in the future, when someone submits a minor change to documentation and we need to update all of the translated docs to match. For the record, I do not think we should rely on such tools for creating an initial translation of the entire docs site to any language, and only use them as a stopgap for minor changes when a native speaker is unavailable to help.

TODOs

• [x] Initialize config and docs/locale/en following https://docs.transifex.com/integrations/sphinx-doc
• [x] Test that Transifex automatically sync source language files and updates strings for translation in https://www.transifex.com/project-jupyter/jupyter-docker-stacks-1/dashboard/ (it works! see #844)
• [x] Setup Travis to update docs -> pots -> locale/en on every merge to master including a docs change (#863, but not tested yet)
• [ ] Test that Transifex properly opens pull requests here (initial try had them open under the wrong subfolder, trying https://community.transifex.com/t/github-pr-goes-to-wrong-folder/366 next)
• [x] Improve build process so that docker build does not occur when all changes are under docs/
• [x] Configure ReadTheDocs to handle all languages for which we have a translation
• [ ] Figure out attribution model, whether Transifex tracks it automatically or if we need to do more to call out the hard work of translators
• [x] Add a page to docs about how to contribute translations (#864)
• [ ] Document everything we figured out here for reproduction in other Jupyter projects!

[michiboo]

Hi I am native Chinese speaker and would like to help translate some docs!

[parente]

@michiboo Awesome. Thanks for reaching out. I'm a little swamped with day-job work at the moment, but I hope to follow-through on setting up the build infrastructure necessary to publish translated docs to ReadTheDocs following one of the paths noted in the description. (Of course, if you have any experience in that setup, please do share!)

[Allanfs]

Hey, I'm native brazilian Portuguese speaker. I would like to help too!

[parente]

Doing a little research today while closing old issues here:

• Django is using the Transifex + ReadTheDocs approach according to https://docs.djangoproject.com/en/dev/internals/contributing/localizing/#documentation and documented by RTD here: https://docs.readthedocs.io/en/stable/guides/manage-translations.html
• Python is also using Transifex (https://www.transifex.com/python-doc/public/)

[parente]

Toying around with sphinx-build -b gettext based on the RTD instructions. It looks like only paragraph level text is extracted into .pot files for translation. Bulleted lists are omitted, for example, even when they contain human readable text. I wonder if this is a problem with trying to use the sphinx tools against markdown files instead of proper restructuredtext. 🤔

[michiboo]

Hi I try to use sphinx-build, Bulleted lists are omitted where in the table of content where following format are used:
example:
`.. toctree::
:maxdepth: 2
:caption: User Guide

using/selecting
using/running``

[michiboo]

Is there a way around this, or we have to manually put the text and link there?

Maybe we can copy and paste it and it will reference to the new translation?

[parente]

Right now I think we're just experimenting with options. I was mostly focused on getting files to and from transifex for translation (https://www.transifex.com/project-jupyter/jupyter-docker-stacks-1/dashboard/).

Two things we might try next to resolve the "missing text" problem:

• I saw that transifex supports whole document translation which would bypass the use of pot/po files and the sphinx-build process. That might skirt the issue we're seeing but may make it harder for people to contribute (i.e., they have to know what the translate out of full docs and what to leave alone ) and may require a branch + config setup for every translation on ReadTheDocs.
• The sphinx-build extraction process might work better if all the documentation were reStructuredText. We could try porting one of the markdown files to reST and see if that pans out. If so, we could move all the doc to reST. The downside would be the effort involved and decrease in community familiarity with the documentation syntax moving from Markdown to reST, I think.

[betatim]

Slightly off topic: https://medium.com/@RaoOfPhysics/a-guide-to-collaborative-translation-workflows-48c259100614 is the workflow I was trying to think of at the team meeting a few weeks back.

[Nico769]

The sphinx-build extraction process might work better if all the documentation were reStructuredText. We could try porting one of the markdown files to reST and see if that pans out. If so, we could move all the doc to reST. The downside would be the effort involved and decrease in community familiarity with the documentation syntax moving from Markdown to reST, I think.

Hi, I'm an Italian native speaker and I'd love to help. Although I'm no expert on this, I'd suggest porting the entire doc to reST to leverage better compatibility with Sphinx (+ Transifex) and future development.

This blog post gives a good outline on why to do so.

Meanwhile, this might also help.

[parente]

Upgrading to recommonmark 0.5.0 seems to have corrected the problem with sphinx-build gettext missing bulleted lists. 🎉

I've uploaded the latest portable object template (.pot) files to Transifex and enabled languages zh, pt_BR, and it for translation in the Transifex project here:

https://www.transifex.com/project-jupyter/jupyter-docker-stacks-1

@michiboo, I accepted your request to become a translator for the project. @Allanfs, @Nico769 I'm guessing that if you follow the link above, it will have a link or button for you to request the same. If you all would like to start using Transifex to translate a few strings:

1. that would be awesome / much appreciated / really cool ⭐️ ,
2. we can see what happens on Transifex after you submit a few translations (e.g., I think there's some optional review process),
3. I / we can continue figuring out the rest of the workflow: pulling translations from transifex, committing to git, and building ReadTheDocs localized sites.

[parente]

Slightly off topic: https://medium.com/@RaoOfPhysics/a-guide-to-collaborative-translation-workflows-48c259100614 is the workflow I was trying to think of at the team meeting a few weeks back.

Thanks for sharing. I noticed in passing on Transifex that you can configure automatic translation using Google, AWS, etc. with an API key. There also seem to be options for paid translation and crowd sourced translation with voting in Enterprise accounts as well.

[Allanfs]

@michiboo, I accepted your request to become a translator for the project. @Allanfs, @Nico769 I'm guessing that if you follow the link above, it will have a link or button for you to request the same. If you all would like to start using Transifex to translate a few strings:

I've sended a join request to the team. As soon as you accept me, i'll make some translations and comment back here any question or sugestion.

Also, i'll read the links that the collegues shared above. 📖

[parente]

I accepted you both. Thanks for helping out!

I've also added a TODO list to the description of this issue of things needing to get done to streamline the process and make it reproducible in other Jupyter projects.

[parente]

After merging #844, I also see that Transifex automatically pulled the source language template files and updated the available strings for translation in the zh, pt_BR, and it projects. There was one hiccup I noted in the PR but it shouldn't be a problem from here on out.

When translated resources become available, we should see Transifex open a pull request here attempting to merge them into docs/locale/<lang>. 🤞

[parente]

When translated resources become available, we should see Transifex open a pull request here attempting to merge them into docs/locale/. 🤞

I took another pass through Transifex this afternoon and paid closer attention to what I was clicking. I noticed that automated pull requests only appear when the English strings are either 100% translated to another language or 100% reviewed.

It's not a problem, just noting for the record why we don't see any PRs from Transifex yet.

[betatim]

Transifex costs (a lot) of money and I couldn't find a "free for open-source" option. How did you organise for this Peter?

[parente]

@betatim there’s a free for open source option when you add a new project. You provide a repo URL and someone or something (maybe) confirms it on the Transifex side.

I’ve setup an org in Transifex for Jupyter and would be happy to invite you and other maintainers who would like to poke around and/or configure it for other Jupyter projects. I was only holding off on suggesting that until we’re a bit further into the experiment here and have time to document the entire workflow. Let me know if you would like org access now.

[Nico769]

I've translated the index.po. Hopefully it's polished enough! Let me know what you think.

[Allanfs]

@parente . I've seen that would be usefull for the readers(especially those who are entering the Docker world) if there were a link referencing the Docker documentation, for exemple: talking about the use of volumes and working directory it would have a link to it in the Docker docs.

May be in a future work, but what do you think about it?

[parente]

@Nico769 I see Transifex opened a pull request with your translation! It's not quite in the right directory, but that's nothing you did wrong. I'll get the misconfiguration sorted out. Thanks for doing the heavy lifting. Hopefully someone else who can read/write Italian fluently can help out with reviewing and translating more of the docs.

@Allanfs Links to additional resources are always welcome. Feel free to send a pull request against the English docs with the links you'd like to add. That'll give us a test case for automating the pushes to Transifex so that the new material can be translated as well.

I'm heads down on my day job this week, but I hope to continue working through the TODO list in the description here soon.

[parente]

I setup ReadTheDocs subprojects following the docs here (https://docs.readthedocs.io/en/stable/localization.html#project-with-multiple-translations). It works!

https://jupyter-docker-stacks.readthedocs.io/pt_BR/latest/
https://jupyter-docker-stacks.readthedocs.io/it/latest/
https://jupyter-docker-stacks.readthedocs.io/zh/latest/

All of these are accessible from the popup menu inserted by RTD in the bottom right:

[Nico769]

That's great! I've fixed some minor typos to index.po (e.g. :doc: tag not displaying correctly).

I'll try to go through the other pages as time permits :).