Psreadline: Crash received by inputting Platform Dependent character in PowerShell

[mkontani]

Environment data

PS version: 5.1.18362.1
PSReadline version: 2.0.0-beta2
os: 10.0.18362.236 (WinBuild.160101.0800)
PS file version: 10.0.18362.236 (WinBuild.160101.0800)
BufferWidth: 182
BufferHeight: 46

Steps to reproduce or exception report

I saw crash by inputing a Platform Dependent character (example 🍣) and one backspace, then enter.
The crash log is below.

直前 200 個のキー:
h o m e \ $ n a m e ) . V e r s i o n I n f o . F i l e V e r s i o n ) " Enter
Ctrl+Enter Space Space Space Space " B u f f e r W i d t h : Space $ ( [ c o n s o l e ] : : B u f f e r W i d t h ) " Enter
Ctrl+Enter Space Space Space Space " B u f f e r H e i g h t : Space $ ( [ c o n s o l e ] : : B u f f e r H e i g h t ) " Enter
Ctrl+Enter } Enter 
Ctrl+c Backspace Backspace Ctrl+l Ctrl+p Ctrl+n Backspace Backspace Backspace Backspace Backspace s u s i Backspace Backspace Backspace Backspace ? ? Enter
Ctrl+c Ctrl+l ? ? Backspace Enter
Enter
Ctrl+c Ctrl+l Ctrl+p Ctrl+p Ctrl+p UpArrow UpArrow UpArrow Enter
UpArrow DownArrow DownArrow DownArrow s u Backspace Backspace Backspace Backspace UpArrow Enter
 ? ? Backspace Enter
 UpArrow UpArrow Enter
 UpArrow DownArrow DownArrow DownArrow ? ? Backspace Enter


例外:
System.Text.EncoderFallbackException: インデックス 0 にある Unicode 文字 \uD83C を指定されたコード ページに変換できません。
   場所 System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
   場所 System.Text.EncoderFallbackBuffer.InternalFallback(Char ch, Char*& chars)
   場所 System.Text.UTF8Encoding.GetBytes(Char* chars, Int32 charCount, Byte* bytes, Int32 byteCount, EncoderNLS baseEncoder)
   場所 System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
   場所 System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   場所 System.IO.StreamWriter.Dispose(Boolean disposing)
   場所 System.IO.TextWriter.Dispose()
   場所 Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass81_0.<WriteHistoryRange>b__0()
   場所 Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
   場所 Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)
   場所 Microsoft.PowerShell.PSConsoleReadLine.InputLoop()
   場所 Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics)
-----------------------------------------------------------------------

[Morilli]

I just got a similar issue, at least it's throwing the same **EncoderFallbackException``. In my case, I was trying to delete a file from the recycle bin that wasn't shown by windows. Specifically, it looked like this:

Environment data

PS version: 5.1.18362.145
PSReadline version: 2.0.0-beta2
os: 10.0.18362.236 (WinBuild.160101.0800)
PS file version: 10.0.18362.236 (WinBuild.160101.0800)
BufferWidth: 130
BufferHeight: 9999

Steps to reproduce or exception report

Letzte 18 Tasten:
 c d Space Tab Tab Tab Shift+Tab Enter
 d i r Enter
 d e l Space Tab Enter


Ausnahme:
System.Text.EncoderFallbackException: Das Unicode-Zeichen \uDC73 im Index 7 kann nicht in die angegebene Codepage übersetzt werden.
   bei System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
   bei System.Text.EncoderFallbackBuffer.InternalFallback(Char ch, Char*& chars)
   bei System.Text.UTF8Encoding.GetBytes(Char* chars, Int32 charCount, Byte* bytes, Int32 byteCount, EncoderNLS baseEncoder)
   bei System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
   bei System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   bei System.IO.StreamWriter.Dispose(Boolean disposing)
   bei System.IO.TextWriter.Dispose()
   bei Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass81_0.<WriteHistoryRange>b__0()
   bei Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
   bei Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)
   bei Microsoft.PowerShell.PSConsoleReadLine.InputLoop()
   bei Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics)

I really have no idea where that file came from, probably a simple case of overwriting an executable while it is running, that has caused similar unshown recycle bin entries for me in the past.

[micka-gsoft]

Environment data

PS version: 5.1.18362.145
PSReadline version: 2.0.0-beta2
os: 10.0.18362.1 (WinBuild.160101.0800)
PS file version: 10.0.18362.1 (WinBuild.160101.0800)
BufferWidth: 120
BufferHeight: 3000

Steps to reproduce or exception report

I ran these command and I know for a fact that few of the returned group contains unicode characters)

PS C:\Windows\system32> $unifiedGroups = Get-UnifiedGroup
PS C:\Windows\system32> $unifiedGroups | ? DisplayName -Like "*?*"

---

c l s Enter
$ u n i f i e d G r o u p s Space = Space G e t - U n i f i e d G r o u Tab Enter
$ u n i f i e d Tab Enter
$ u n f i Tab Backspace Backspace i f i Tab Space | Space ? Space Space D i s p l a y N a m e Space - l i Tab Space " * ? * " Enter
UpArrow LeftArrow LeftArrow Backspace ? ? Enter
UpArrow LeftArrow LeftArrow Backspace Enter

Exception:
System.Text.EncoderFallbackException: Unable to translate Unicode character \uD83E at index 39 to specified code page.
at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
at System.Text.EncoderFallbackBuffer.InternalFallback(Char ch, Char& chars)
at System.Text.UTF8Encoding.GetBytes(Char chars, Int32 charCount, Byte* bytes, Int32 byteCount, EncoderNLS baseEncoder)
at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.IO.StreamWriter.Dispose(Boolean disposing)
at System.IO.TextWriter.Dispose()
at Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass81_0.b__0()
at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
at Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)
at Microsoft.PowerShell.PSConsoleReadLine.InputLoop()
at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics)

[daxian-dbw]

This is because the characters \uD83C, \uDC73 and \uD83E are invalid Unicode characters. See https://www.fileformat.info/info/unicode/char/d83c/index.htm for an example.

For those characters, even if we change the file encoding to Encoding.Unicode or explicit passing Encoding.UTF8 to StreamWriter and StreamReader to suppress the exception (no exception when writing to the file with those explicit encoding), the string read back from the file will be different from the original string, so it's not useful for command line history purpose.

I think the best PSReadLine can do here is to write out an error to let the user know that history saving failed because of the invalid Unicode character.

[tats-u]

This module corrupts non-BMP characters, in addition to reporting error for them.

Environment data

PS version: 6.2.3
PSReadline version: 2.0.0-beta3
os: 10.0.19038.1 (WinBuild.160101.0800)
PS file version: 6.2.3.0
HostName: ConsoleHost (Windows Terminal)
BufferWidth: 119
BufferHeight: 30

Steps to reproduce or exception report

Trying to input echo 𠮷 (𠮷 is U+20BB7), got:

````
Oops, something went wrong. Please report this bug with the details below.

Report on GitHub: https://github.com/lzybkr/PSReadLine/issues/new

Last 7 Keys:

 e c h o Space � Enter

Exception:

System.Text.EncoderFallbackException: Unable to translate Unicode character \\uD842 at index 5 to specified code page.
   at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
   at System.Text.EncoderFallbackBuffer.InternalFallback(Char ch, Char*& chars)
   at System.Text.UTF8Encoding.GetBytes(Char* chars, Int32 charCount, Byte* bytes, Int32 byteCount, EncoderNLS baseEncoder)
   at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass81_0.<WriteHistoryRange>b__0()
   at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
   at Microsoft.PowerShell.PSConsoleReadLine.WriteHistoryRange(Int32 start, Int32 end, Func`2 fileOpener)
   at Microsoft.PowerShell.PSConsoleReadLine.IncrementalHistoryWrite()
   at Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)
   at Microsoft.PowerShell.PSConsoleReadLine.InputLoop()
   at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics, CancellationToken cancellationToken)

---

````

After the above message:

PS> echo �
�
PS> Remove-Module PSReadLine
 PS> echo ��
𠮷

[BananaAcid]

Seems to be same issue #1282

Emoji inserted by win + .

[beary]

I got the same error when echo '😄'

---

PS D:> echo '�'

Oops, something went wrong. Please report this bug with the details below.
Report on GitHub: https://github.com/PowerShell/PSReadLine/issues/new

Environment

PSReadLine: 2.0.0
PowerShell: 7.0.1
OS: Microsoft Windows 10.0.19631
Last 55 Keys

 e c h o Spacebar ' ' LeftArrow � Enter
 Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Enter
 Backspace Backspace e c h o Spacebar ' ' LeftArrow 1 2 3 4 5 6 Enter
 c d Backspace Backspace D : Enter
 c l s Enter
 UpArrow UpArrow UpArrow UpArrow Enter

Exception

System.Text.EncoderFallbackException: Unable to translate Unicode character \\uD83D at index 6 to specified code page.
   at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
   at System.Text.EncoderFallbackBuffer.InternalFallback(ReadOnlySpan`1 chars, Int32& charsConsumed)
   at System.Text.Encoding.GetBytesWithFallback(ReadOnlySpan`1 chars, Int32 originalCharsLength, Span`1 bytes, Int32 originalBytesLength, EncoderNLS encoder)
   at System.Text.Encoding.GetBytesWithFallback(Char* pOriginalChars, Int32 originalCharCount, Byte* pOriginalBytes, Int32 originalByteCount, Int32 charsConsumedSoFar, Int32 bytesWrittenSoFar, EncoderNLS encoder)
   at System.Text.Encoding.GetBytes(Char* pChars, Int32 charCount, Byte* pBytes, Int32 byteCount, EncoderNLS encoder)
   at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass83_0.<WriteHistoryRange>b__0()
   at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
   at Microsoft.PowerShell.PSConsoleReadLine.WriteHistoryRange(Int32 start, Int32 end, Func`2 fileOpener)
   at Microsoft.PowerShell.PSConsoleReadLine.IncrementalHistoryWrite()
   at Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)
   at Microsoft.PowerShell.PSConsoleReadLine.InputLoop()
   at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics, CancellationToken cancellationToken)

[BananaAcid]

Does any1 know, if there was some progress?

[daxian-dbw]

@BananaAcid Sorry, no work has been done in this area yet. But I'm moving this issue higher in my to-do list as there are more and more similar reports.

[robharman]

Unclear if this is helpful at this point since the exception doesn't seem to offer anything new. However, here's one with an upside down smiley. 🙃

Happens in both Windows Terminal, and in pwsh.exe

�### Environment
PSReadLine: 2.0.0
PowerShell: 7.0.2
OS: Microsoft Windows 10.0.18362
Last 58 Keys

 " ` u { 1 F 4 A 9 } " Enter
<Cliipped private info> Spacebar Enter
 UpArrow � � Enter

Exception

System.Text.EncoderFallbackException: Unable to translate Unicode character \\uD83D at index 41 to specified code page.
   at System.Text.EncoderExceptionFallba�ckBuffer.Fallback(Char charUnknown, Int32 index)
   at System.Text.EncoderFallbackBuffer.InternalFallback(ReadOnlySpan`1 chars, Int32& charsConsumed)
   at System.Text.Encoding.GetBytesWithFallback(ReadOnlySpan`1 chars, Int32 originalCharsLength, Span`1 �bytes, Int32 originalBytesLength, EncoderNLS encoder)
   at System.Text.Encoding.GetBytesWithFallback(Char* pOriginalChars, Int32 originalCharCount, Byte* pOriginalBytes, Int32 originalByteCount, Int32 charsConsumedSoFar, Int32 bytesWrittenSoFar, EncoderN�LS encoder)
   at System.Text.Encoding.GetBytes(Char* pChars, Int32 charCount, Byte* pBytes, Int32 byteCount, EncoderNLS encoder)
   at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean� flush)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass83_0.<Write�HistoryRange>b__0()
   at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
   at Microsoft.PowerShell.PSConsoleReadLine.WriteHistoryRange(Int32 start, Int32 end, Func`2 fileOpener)
   at Microsoft.PowerShell.PS�ConsoleReadLine.IncrementalHistoryWrite()
   at Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)
   at Microsoft.PowerShell.PSConsoleReadLine�.InputLoop()
   at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics, CancellationToken cancellationToken)