Psreadline: Some emoji, eg '😃' don't work, whereas others like '⏰' do

[mikemaccana]

Moved from https://github.com/microsoft/terminal/issues/1606

Using app store preview build of Windows Terminal, with win+. to send 😃 and then ⏰:

@DHowett-MSFT mentioned it's likely not a Windows Terminal issue:

Powershell has trouble with high unicode input ... and there's not much we can do to help that.

Name                           Value
----                           -----
PSVersion                      6.2.1
PSEdition                      Core
GitCommitId                    6.2.1
OS                             Microsoft Windows 10.0.18362
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[vexx32]

I wonder if this is due to PS itself or PSReadLine...

Can you Remove-Module PSReadline and try it again?

[mikemaccana]

After Remove-Module PSReadline

[vexx32]

Interesting. Input doesn't like it much, but output is OK.

I'd venture to say this is probably an issue both for PS itself in places and the PSReadLine module as well,

@daxian-dbw any thoughts on this?

[iSazonov]

/cc @daxian-dbw @SteveL-MSFT Should we track the issue here or in PSREadline repo?

[SteveL-MSFT]

Seems like there is an issue in both PS and PSRL. We can keep this issue here for now. Removing PSRL and on macOS, simply pasting "😃 results in �"😃�. Quite strange.

[SteveL-MSFT]

@chuanjiao10 that may be true on Windows, but on macOS there is still an issue although perhaps different from the original issue

[daxian-dbw]

In Windows Terminal, garbage characters may appear when deleting the emoji character.

[iSazonov]

@daxian-dbw Is any updates in PSRL with the issue?

[daxian-dbw]

@iSazonov Unfortunately, no news from the PSReadLine side. also, as mentioned in this issue, the same repros without PSReadLine.

[DHowett-MSFT]

PSReadline uses raw input and non-PSReadline uses COOKED_READ in the console host. One of these things can never be fixed :smile:

[iSazonov]

I'd prefer to get updated COOKED_READ in Core.

[DHowett-MSFT]

Cooked read is provided by conhost.exe, not Core, and we cannot change it without significantly impacting application compatibility all across Windows. Raw input with a readline-like library is the _correct_ thing to do.

[iSazonov]

Raw input with a readline-like library is the correct thing to do.

I mean that this must be in Core so that we do not re-implement this in every application. Is there a tracking issue?

[lzybkr]

@iSazonov - see https://github.com/PowerShell/PSReadLine/issues/1045

[iSazonov]

I opened https://github.com/dotnet/runtime/issues/800, maybe not entirely correct, but the presence of these features in Core seems desirable.

[daxian-dbw]

@DHowett-MSFT You mentioned in microsoft/terminal#1606 that:

Powershell has trouble with high unicode input ... and there's not much we can do to help that.

For all high unicode emoji input, such as 😀 (surrogates: D83D, DE04), Console.Readkey() seems to only return a high surrogates character, like ConsoleKeyInfo { Key = 18, KeyChar = '\ud83d', Modifiers = 0 }, but not the low surrogates. Could you please provide me some pointers on how to correctly read all surrogates? An in general, can you please provide me some pointers on how should a readline library handle emojis?

[SeeminglyScience]

@daxian-dbw it's sent as two separate keys. If you do this you'll see both

while ($true) {
    '0x{0:X4}' -f [int][Console]::ReadKey($true).KeyChar
}

[SeeminglyScience]

Also, I think escape sequences are being written between the surrogates:

[Console]::WriteLine("`u{D83D}`u{DE00}")
# Shows emoji

[Console]::WriteLine("`u{D83D}`e[30m`u{DE00}")
# Shows two separate question mark characters

[daxian-dbw]

[Console]::WriteLine("u{D83D}u{DE00}") shows 2 question mark character for me in both Windows terminal and legacy console host:

[SeeminglyScience]

Yeah you gotta change output encoding first:

[Console]::OutputEncoding = [Text.Encoding]::Unicode

[iSazonov]

Oh, is it a time to change [Console]::OutputEncoding to Utf8 for PowerShell?

[SeeminglyScience]

Also Delete should delete two characters if the target is a high surrogate. @daxian-dbw let me know if you want a new issue for that.

[daxian-dbw]

@SeeminglyScience Yes, please open a new issue for Delete. Thanks!

[daxian-dbw]

For me later: https://github.com/microsoft/terminal/issues/1503#issuecomment-605311026

[rayphi]

I've got also an Exception for a 💄in my commit message.

Exception:

System.Text.EncoderFallbackException: Unable to translate Unicode character \\uD83D at index 10 to specified code page.
   at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
   at System.Text.EncoderFallbackBuffer.InternalFallback(ReadOnlySpan`1 chars, Int32& charsConsumed)
   at System.Text.Encoding.GetBytesWithFallback(ReadOnlySpan`1 chars, Int32 originalCharsLength, Span`1 bytes, Int32 originalBytesLength, EncoderNLS encoder)
   at System.Text.Encoding.GetBytesWithFallback(Char* pOriginalChars, Int32 originalCharCount, Byte* pOriginalBytes, Int32 originalByteCount, Int32 charsConsumedSoFar, Int32 bytesWrittenSoFar, EncoderNLS encoder)
   at System.Text.Encoding.GetBytes(Char* pChars, Int32 charCount, Byte* pBytes, Int32 byteCount, EncoderNLS encoder)
   at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass81_0.<WriteHistoryRange>b__0()
   at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
   at Microsoft.PowerShell.PSConsoleReadLine.WriteHistoryRange(Int32 start, Int32 end, Func`2 fileOpener)
   at Microsoft.PowerShell.PSConsoleReadLine.IncrementalHistoryWrite()
   at Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)
   at Microsoft.PowerShell.PSConsoleReadLine.InputLoop()
   at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics, CancellationToken cancellationToken)

[chenshuai2144]

I have same error