Fsharp: Typing / sending unicode characters in FSI is broken

Created on 15 Oct 2019  Β·  9Comments  Β·  Source: dotnet/fsharp

Since F# 4.7, typing or sending unicode chars in FSI results not in the typed char, but in an unexpected one.

Example:

Typing let ΓΌber = 42;; in F# 4.5 works; in F# 4.7, it doesn't.

image

The behavior is also present using VSCode + Ionide (sending text via Alt ENTER).

Area-FSI Severity-Medium bug
Source
picture ronaldschlenker
😕3

Most helpful comment

@KevinRansom any chance you can look at this? This is really annoying in languages that are not English

picture forki on 18 Jan 2020
👍3

All 9 comments

Any news on this one? This breaks quite a lot of my working scripts.

ronaldschlenker picture ronaldschlenker on 18 Dec 2019

@KevinRansom any chance you can look at this? This is really annoying in languages that are not English

forki picture forki on 18 Jan 2020
👍3

Another case here: https://developercommunity.visualstudio.com/content/problem/911485/bad-unicode-encoding-from-literal.html

Likely the same underlying issue. We'll treat this as in scope for .NET 5 probably.

cartermp picture cartermp on 18 Feb 2020

Note that FSI in VS doesn't have this issue.

image

cartermp picture cartermp on 18 Feb 2020

In Rider, it also works:

image

See also: https://twitter.com/auduchinok/status/1207260258483286021?s=20

Eugene Auduchinok
@auduchinok

  1. Dez. 2019
    They seem to use a different fsi console integration mode, while we're using the same one as in VS.
ronaldschlenker picture ronaldschlenker on 21 Feb 2020

With F# 5.0 and a freshly installed net 5.0 I still get the error. It is not exactly the same though. It seems the keys I type gets mistranslated in a different way:
image

However, if I change the code pages when calling fsi it works!:
image

Edit:
Codepage 28591 is better than 1252. It works in net 5.0 also:
image

larjo picture larjo on 12 Nov 2020
👍1

And with this setting it works in vscode + ionide as well:

  "FSharp.fsiExtraParameters": [
    "--fsi-server-input-codepage:28591",
    "--fsi-server-output-codepage:65001"
  ]
larjo picture larjo on 12 Nov 2020

The problem persist in F# 5.0. If you paste a string that contains larger then 51 symbol, all Unicode symbols starting from 52st position substituted with several instances of '\uFFFD'.

Microsoft (R) F# Interactive version 11.0.0.0 for F# 5.0
Copyright (c) Microsoft Corporation. All Rights Reserved.

For help type #help;;

> "β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’";;
val it : string = "β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’"

// string that was pasted: "β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’"
> "β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’οΏ½οΏ½οΏ½";;
val it : string = "β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’οΏ½οΏ½οΏ½"

> let ustr = "β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’";;
val ustr : string = "β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’"

// string that was pasted: let ustr = "β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’";;
> let ustr = "β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’οΏ½οΏ½οΏ½";;
val ustr : string = "β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’οΏ½οΏ½οΏ½"

> β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’;;

  β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’;;
  ^

/stdin(5,1): error FS0010: Unexpected character 'β†’' in interaction

> β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’οΏ½οΏ½οΏ½ 

// string that was pasted: let ustr = "β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’ΓΌ";;
> let ustr = "β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’οΏ½οΏ½";;
val ustr : string = "β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’β†’οΏ½οΏ½"
wildart picture wildart on 8 Dec 2020

It looks like the bug is related to terminal emulation. I got the this problem in VS Code internal terminal and Jupyter web terminal.

I run ReadLineConsole.ReadLine function of current console.fs file (slightly modified version to show bad output after Console.ReadKey call), and got the error in VS Code internal terminal.

vs-code

But when I'm running the same function in Alacritty terminal, everything is ok.

alacritty

wildart picture wildart on 8 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jackfoxy picture jackfoxy  Β·  4Comments
enricosada picture enricosada  Β·  3Comments
TysonMN picture TysonMN  Β·  3Comments
Tarmil picture Tarmil  Β·  4Comments
Szer picture Szer  Β·  3Comments