Terminal: Support Right to Left (RTL) text

[MahdiGhiasi]

• What you're doing and what's happening: When writing some right-to-left characters (Farsi, Arabic, ...) after some left-to-right characters, the rtl text goes from the cursor position into the left, causing it to be mixed with the ltr text written before.

• Steps to reproduce: Open a Windows Terminal window. write some english characters, like abcdef. Then write some Farsi/Arabic text, like:

  سلام
  

  Notice that the Farsi text goes inside the english text.

• Screenshot:

• what should be happening instead: As console is generally left to right, the rtl text should begin more on the right (cursor should go to right), and it should end exactly when the ltr text ended. See how Notepad (correctly) handles this situation:

• Windows build number: 10.0.18362.86

[jalchr]

30 years and we still can't render Arabic characters inside Windows terminal, while Ubuntu and Mac can do !

[DHowett-MSFT]

30 years and we still can't render Arabic characters inside Windows terminal, while Ubuntu and Mac can do !

@jalchr Thanks for reaching out. Comments like this are not substantive and _do not add to the discussion._

[jalchr]

Thanks for getting the notice.
I have been reading the issues and people are shouting out for the Arabic support, while the responses were not encouraging.
We are building a new terminal right?
It should be at least like if not better than what is already on the table

[zadjii-msft]

We're all for adding RTL support, but know that doing that would require a _lot_ of work.

Much of the new Terminal's codebase is directly shared with the old console codebase, and there are lots of places in that code that assume the text is all LTR. Any fixes for the Terminal would need to make sure to not break existing console behavior in this regard.

I'm not saying it's impossible to support RTL, or that we won't. I'm just saying that based on the volume of work it'd require, it's probably not a 1.0 feature, unless we get a lot of help from the community.

[jalchr]

It might suffice for v1.0 to at least render Arabic characters correctly

[mtlive]

Old codebases aren't that bad 😁:

[egmontkob]

while Ubuntu and Mac can do

Ubuntu doesn't refer to a particular terminal emulator. Some of its terminal emulators (e.g. konsole, pterm, mlterm) do _some_ right-to-left rendering, and so does macOS's Terminal.app.

And they all do it wrong.

Just to give you an idea, let me mention here one of the many problems they all suffer from. They unconditionally "reverse" every Farsi/Arabic/Hebrew/etc. text. This fixes them in the command line and the in the output of simple utilities, and at the same time, hopelessly, unfixably breaks them for more complex applications (e.g. Emacs).

See my work at https://terminal-wg.pages.freedesktop.org/bidi/ (along with a WIP implementation in VTE / GNOME Terminal) about a more detailed description of the problems and a proposed specification for the desired behavior.

[XVilka]

Added this issue in my gist to track: https://gist.github.com/XVilka/a0e49e1c65370ba11c17

[schorrm]

Partial fix, in #1873

[XVilka]

Note, that with the release of GNOME 3.34 the support of BiDi by @egmontkob is available in Gnome Terminal out of the box, which makes comparing your implementation and testing it against valid one way easier.

[j4james]

For the record, I don't think the VTE proposal is the right approach, at least not as a default behaviour. In my opinion we should be doing our best to match the way the DEC terminals functioned, implementing the DEC RTL modes and escape sequences, and making sure we work with existing RTL/bidirectional applications.

It's not our job to try and fix applications that aren't RTL aware. That approach is doomed to failure, and just gets in the way of applications that are actually trying to do the right thing. That said, there are things we can do to improve the user-experience when working with applications that don't support RTL, but any functionality like that should be up to the user to activate.

[adueck]

Using terminal preview Version: 1.2.2234.0

The RTL now works! 🥳 Although there are some unsightly joining gaps... it pretty much renders correctly!

This is RTL -> عربي فارسی

But in Vim version 8.1.2269 (Ubuntu 20.04) it doesn't quite work. 😢 The RTL ordering shows up correctly, but unfortunately the joining is broken.

This is RTL in vim not joining properly -> عربي فارسی

EDIT: The joining in Vim can be fixed by either:

• setting termbidi to true (:set termbidi)
  
  
  
  • default is false but true for mlterm
  
  
• setting arabicshape to false (:set noarabicshape)
  
  
  
  • default is true
  
  

It does work in nano though...

[schorrm]

@adueck - yup, this would've been my fix in #7190.
It looks to me like Vim is corntolling its own thing, I don't think there's a real issue here.