Terminal: Add emoji support to Windows Console

[HLFH]

Please support emoji within Windows Console.

Very useful when you code in vim newsletters for startups or when you categorize stuff by emoji.

[zadjii-msft]

Already on the backlog :)

[mikemaccana]

Sweet! Another use case: I have a command line app that outputs warnings using ⚠.

[mqudsi]

@zadjii-msft will this include support for non-latin unicode characters? i.e. can Arabic or Japanese characters not found in the currently deployed console font be displayed with a different font face?

[zadjii-msft]

Hypothetically, yes.

Arabic is kinda it's own problem though - there's pretty much no chance we'll be implementing left to right language support any time soon, bt the characters might be able to render correctly.

@adiviness can speak more on the topic if there's more to share.

[adiviness]

Arabic is a difficult one because of the additional support required for bidirectional text and ligatures. As far as I’m aware not many terminal emulators currently support it.

There is also a difference between the console being able to render emoji and supporting font fallback in the case of the current font not having a glyph. They are both items on our backlog.

[SidShetye]

+1 Our build and CI scripts have emojis for success ✔️, warning ⚠ and errors ❌ for quick, productive eyeballing of logs.

[miniksa]

Can we please not +1 issues? Please use reactions or subscribe to notifications.

[SidShetye]

@miniksa , I wouldn’t get too distracted by the +1. You could entirely ignore it. The main purpose of typing it out was to articulate the use case. Presumably platform/product owners ought to be very interested in customer use cases ... thought granted in this case it’s not too extravagant.

[bitcrazed]

@SidShetye - appreciate your comment - your use-case is not unusual, but is useful to hear about.

In general, we ask people not to +1 because we'd like to avoid people +1-ing (esp. with no further comment), which just ends up as noise, and makes threads more difficult to parse and manage.

Sharing additional comments, context, observations, issues, etc. is FAR more valuable than a +1 ;)

[tyranteon]

Weird that nobody mentioned it, but the Yarn package manager uses Emoji's and it's a bit annoying that they are only displayed as squares :/

[bitcrazed]

Thanks @destructive-dragon - there are many tools that contain/emit emoji, but the Console is not yet able to render them.

[kavdev]

@bitcrazed You mentioned in this twitter thread on the conpty release that we still need to wait for a new buffer and a new renderer (DirectWrite). Are those the only two major blockers left?

[bitcrazed]

@kavdev Essentially, yes. To display emoji glyphs, we first have to be able to store (potentially compound) Unicode code-points for each glyph (E.g. Ninjacats), but we also have to be able to render them, which requires font-fallback, which GDI doesn't support.

We'll be working on improvements to both the Console's text buffer implementation, and also the Renderer in future releases.

[be5invis]

@bitcrazed
Many Emoji are composites, i.e., having multiple code points joined together (using ZWJ or VS, or whatever), and in most cases they do not fit in one Console cell. So your problem is not "1 cell to n characters" but "m cells to n characters"...

[be5invis]

[sedwards2009]

FWIW, my terminal emulator and iTerm2 both render emoji by basically treating them as "full width" (2 cell) CJK style characters. I don't know about iTerm2 but I don't make any attempt to support unicode "modifiers" on emoji or any other characters. Each character has to be one unicode code point although it may be normal width or full width.

[be5invis]

@sedwards2009
The ultimate solution should be a Unicode or whatever specification that tell us how to handle complex script on a character grid.

[be5invis]

The closest idea may be applying some concepts from justification (like inserting spaces between East Asian characters, and inserting Kashida when fitting Arabic characters into the grid), but justification implementation is a total mess. AFAIK DWRITE does the best job so far, but some implementations (like Kashida) is still very hacky.

[benc-uk]

Just one comment which might be useful - certain emoji and unicode symbols used to work, I'd display some on my WSL login banner, e.g. 🍰 (U+1F370), since 1809 update they no longer display in any terminal (WSL bash, Hyper, VS Code)
Some symbols do work in 1809 however, such as ☕ (U+2615) but I think they are in a different parts of the Unicode spectrum i.e. much lower codepoints

[noxabellus]

October update seems to bring a regression in this regard rather than progress. I was previously able to use all unicode symbols I have tested in VS Code integrated terminal (Which uses powershell), however after the October update all emoji and certain foreign language characters do not render correctly.

This actually seems to be a semi-system-wide font issue as even console emulators like ConEmu now fail to render emoji correctly.

[MartinMa]

@Ben-Hope @noxabellus same here.

I updated Windows 10 to 1809 and emojis are gone (PowerShell and Visual Studio Code; integrated terminal).

See command vue ui of vue-cli as an example:
🚀 Starting GUI...

[benc-uk]

Yeah I miss that little rocket when starting the vue-cli 😢

[mqudsi]

@bitcrazed @zadjii-msft is there an issue tracking the regression found in 1809? I'm not talking about full-on emoji support, just getting back the basic foreign language/unicode glyphs that were supported under previous releases. Do you know if the issue persists in later beta releases?

[fz568573448]

Same here. I updated my Win10 from 1803 to 1809 several days ago, and now all characters >= U+10000 (UTF-8 with 4 bytes or more) no longer display. I have also tried the newest insider version(Windows 10 Insider Preview 18358.1 (19h1_release)), unfortunately, this bug still exists.

Since 19H1 will be released soon, could you please fix it, or report it because it could be a bug in other project?

[dmitweb]

Same =(

[MartinMa]

Emoji rendering seems to have improved.

I tried a recent build of CascadiaPackage (Windows; x64) and get this (see image):

Running Windows 10, Build 1903.

[mdtauk]

Is there any merit to having an option to disable colour font emoji?

So any emoji used will be single colour using the font colour settings?

[zadjii-msft]

@mdtuak yep, in fact thats a setting @miniksa and I have previously discussed adding. I just filed #956 to track that work :)

[sylveon]

@MartinMa that's an entirely different thing than the existing conhost.exe, if you tried to run OpenConsolePackage (which is the OSS conhost) you should still have the issue.

[DHowett-MSFT]

@MartinMa that's an entirely different thing than the existing conhost.exe, if you tried to run OpenConsolePackage (which is the OSS conhost) you should still have the issue.

Don't be so sure! The DirectWrite renderer that's used in Windows Terminal _is also part of OpenConsole!_ You just need to set a registry key (HKCU\Console\UseDx = DWORD(1)) before it'll use it.

[ysc3839]

@DHowett-MSFT The console throws an exception when pasting an emoji string.
It's here.
https://github.com/microsoft/terminal/blob/2fdcb679ab1f1f1edc542e3b86327dacea78f7ac/src/buffer/out/CharRowCellReference.cpp#L15

[miniksa]

@DHowett-MSFT Dustin Howett FTE The console throws an exception when pasting an emoji string.
It's here.
https://github.com/microsoft/terminal/blob/2fdcb679ab1f1f1edc542e3b86327dacea78f7ac/src/buffer/out/CharRowCellReference.cpp#L15

I am 80-90% sure that either @adiviness or I have a bug covering that already somewhere around here.

[adiviness]

Yeah, the work I'm doing on the /dev/austdi/NewCookedRead probably affects the crash there. We don't support emoji's being pasted (or typed) quite yet in all shells.

[fcharlie]

@MartinMa that's an entirely different thing than the existing conhost.exe, if you tried to run OpenConsolePackage (which is the OSS conhost) you should still have the issue.

Don't be so sure! The DirectWrite renderer that's used in Windows Terminal _is also part of OpenConsole!_ You just need to set a registry key (HKCU\Console\UseDx = DWORD(1)) before it'll use it.

I didn't even know. I am going home to try it at night. But does this registry affect the default conhost.exe?

Update 2019-07-19 20:47 UTC+8

Although OpenConsole can open DirectWrite rendering with HKCU\Console\UseDx, it still seems to be unable to display emoji.

Maybe related to https://github.com/microsoft/terminal/issues/2053

[DHowett-MSFT]

Yes, it's almost certainly because of #2053.

[fcharlie]

@DHowett-MSFT Will my commit fix be appropriate, if there is no problem, I will create a PR.

https://github.com/fcharlie/terminal/commit/4c6280ca35fff9eac0041c94385574bedc5f2a27

[innovoix]

I saw this Youtube video with @cinnamon-msft. She showed emojis support. Is this fixed? 😄

[ExE-Boss]

@innovoix That’s the Windows Terminal, this is about the Windows Console.

[innovoix]

@ExE-Boss Ah OK, my bad.

[jasonhargrove]

When I setup a Windows workstation for the first time in 10 years, now very used to OSX, I was very disturbed to lose the beloved and extremely useful emojis. Sort of funny it's not already a long ago default.

What was UnfundedPillow2 and ShutUpCon saying? We'll now never know 😢😢

Also big shout out to iTerm. May we find you and your greatness on this platform sometime soon. No split panes? What a pain 😱😱

[DHowett-MSFT]

@jasonhargrove thanks for the feedback. You may want to check out _Windows Terminal_, which is (incidentally) also built out of this repository. It does support the things you're looking for.

[bitcrazed]

@jasonhargrove We do plan on integrating many of the improvements to the core engine underlying both Windows Terminal, and Windows Console back into Windows Console in the future. Such areas include the internal text buffer, text rendering engine, etc. - things that won't generally impact backward-compatibility.

Why the "backward compat'" caveat? Windows Console's job (along with Cmd), is to remain backward-compatible wherever possible. Thus, there will be many features (e.g. tabs, split panes, etc.) that will only be available in Windows Terminal, and will never make it back into Console.

We strongly recommend that users start evaluating and testing Windows Terminal, file any unexpected issues here so that we can triage and fix them as quickly as possible as we drive Terminal towards v1.0 (~Q2 2020).

FWIW, as of v0.7 Windows Terminal DOES support split panes, as well as multiple tabs, UTF-8, emoji, GPU-accelerated text rendering, a huge number of configuration options, several selection/copy and paste improvements, etc.

[jsilvermist]

One thing that drives me crazy is the inability to Shift+Insert paste in the Windows Terminal.

[DHowett-MSFT]

@jsilvermist you can add shift+ins as a key binding in your Terminal settings.

[bitcrazed]

@jsilvermist Polite request - please keep issues to relevant threads. The issue you describe relates to key-bindings, not emoji ;)

[jsilvermist]

@bitcrazed True, my bad! Also, thanks @DHowett-MSFT!

Edit: forgot obligatory emoji :wink:

[bitcrazed]

@jsilvermist No problem - we all do it now and again - many thanks 😜 👍

[jasonhargrove]

Demonstrating emojis and panes in Windows Terminal:

Ok Christmas comes early! Great start to this and what a major leap forward, congrats. And thanks for letting me know about it 📈

[AmericanY]

is it normal to see ?? if i paste the emoji. please confirm. thanks in advance.

[DHowett-MSFT]

@AmericanY yeah, this is an issue with PowerShell 5.1.

[AmericanY]

@DHowett-MSFT that's happens on CMD, PowerShell and windows new terminal .

[Vikr-182]

My WSL is printing some unusual characters after it has rendered the unicode string. I was trying to build an interactive game and wasn't able to display the characters due to this. Looks like I have to restrict myself to 128-bit ASCII

[DHowett-MSFT]

I just want to say, with apologies, that this looks hilarious.

[luke1961]

Is the src code for fmt_test.exe available somewhere? (screenshot by @fcharlie earlier in this thread) I am having hard time duplicating the result.

[fcharlie]

@luke1961 see https://github.com/fcharlie/bela/blob/master/test/fmt/main.cc

[luke1961]

Thanks!

[remidebette]

Hi,
To add on the comment of @AmericanY , when pasting in the Windows Terminal an emoji (say from a preexisting output) it is broken in the input and correct in the output.
Also if doing an up arrow right after, then it is correct in the input this time.

Is this expected ?

Windows Terminal
Powershell Core 6.2.1
Meslo LG M for Powerline

[am11]

@AmericanY yeah, this is an issue with PowerShell 5.1.

Besides that one, the space between two emojis get converted to linefeed in posh 5.1, but not in other shells like ash, bash and so on:

[zadjii-msft]

@remidebette You'll want to see #1503

[remidebette]

@zadjii-msft , this seems quite hard, but will be really nice once you crack it.
Hang in there!

[gund]

I've tested emoji on a few shells under Windows Terminal (Version: 0.11.1191.0) and here are results:

• CMD
  
• Powershell
  
• Cmder
  
• WSL with ZSH
  
• Cygwin with ZSH
  

Looks like the only shell that works perfectly so far is WSL (both bash and zsh)...

[adiviness]

I’m working with an old set of information, but it at least used to be that the console stores the prompt line in a separate buffer from the output buffer and that it didn’t support more than ucs2 encoding. WSL works bc it’s encoded in utf8. I was working on a branch at one point that fixed this, if I find the time this weekend I’ll take a peek. Hmm

edit: WSL also might be working if it doesn't use the cooked read like the others, I don't remember if it does or not.

[louanbastos]

How can i fix the emoji?

[hwongbaul]

I'm encountering the same problem; it seems no an up-to-date solution afaik.

[schuelermine]

Sweet! Another use case: I have a command line app that outputs warnings using ⚠.

Output of Emoji works fine in my experience.

[Raibows]

So it can be sure that the problem is input characters not rendering correctly in windows terminal. I have tried Git Bash(one kind fo shell), It works well with Git Bash GUI, but not working with windows terminal same as the bugs mentioned as others.

1. Git Bash GUI
   
2. Git Bash with Windows Terminal
   

My windows terminal version is 1.1.2233.0.
Hope could fix it as soon as possible! 👍 And is there an estimated schedule?

[DHowett]

Please file a separate bug for this issue and include the contents of your Windows Terminal settings file.

[mamayadi]

Please file a separate bug for this issue and include the contents of your Windows Terminal settings file.

I have the same issue:
{ "guid": "{00000000-0000-0000-ba54-000000000002}", "acrylicOpacity": 0.75, "closeOnExit": true, "colorScheme": "Campbell", "commandline": "\"%PROGRAMFILES%\\git\\usr\\bin\\bash.exe\" -i -l", "cursorColor": "#FFFFFF", "cursorShape": "bar", "fontFace": "Consolas", "fontSize": 10, "historySize": 9001, "icon": "%PROGRAMFILES%\\Git\\mingw64\\share\\git\\git-for-windows.ico", "name": "Bash", "padding": "10, 0", "snapOnInput": true, "startingDirectory": "%USERPROFILE%", "useAcrylic": true },
Windows Terminal: v1.3.2651.0
Git: v2.16.2.windows.1