Ptvs: Saving files with CJK characters inside makes them unreadable

[KexyBiscuit]

Files created within Visual Studio are vulnerable to this.

If the file was created from 3rd party editor(like Visual Studio Code), and saved into UTF-8, then adding CJK characters into it with Visual Studio is OK.

I guess PTVS saves file in "local codepage", which on Windows en-US is codepage 437, instead of UTF-8(codepage 65001).

Credit: A neko who doesn't want to tell her name.

[zooba]

By default, VS will save files into your current code page (1252 for en-US).

However, if there are any characters that can't be represented in that codepage, VS should prompt to change to Unicode or automatically change to Unicode (depending on your setting for Tools, Options, Environment, Documents, Save documents as Unicode when data cannot be saved in codepage).

Do any prompts appear when saving the file? If so, possibly that dialog is being suppressed some other way.

Also, can you share the specific characters that are being added? It's possible that we're getting into surrogate pairs in UTF-16, which may be less reliable than characters encoded into a single word.

[KexyBiscuit]

@zooba While saving C# project or pure text files, there is a prompt. But nothing pops up for Python projects.
Any CJK character is affected, for example, "我能吞下玻璃而不伤身体", "中国智造惠及全球" and "ギリギリ爱 ギリギリ舞".

[KexyBiscuit]

@zooba Any idea?

[zooba]

Perhaps we have ended up bypassing the code that displays the prompt. That may have happened when we switched the default encoding to UTF-8 (No signature).

How are you creating the file? And if you go to Advanced Save Options immediately after creating it, what encoding is listed there?

[KexyBiscuit]

@zooba I created the file within Visual Studio. Actually, editing any empty file can reproduce this.

Even if I entered Chinese characters, the encoding listed in Advanced Save Options is "Western European (Windows) - Codepage 1252".

[zooba]

Okay, our approach to fixing this will be:

• ensure the default encoding for Python files is UTF-8 (no signature) or the encoding specified by the coding comment (this is certainly the case in the latest releases, but was not reliable prior to that)
• detect when characters cannot be encoded in the current encoding and prompt before saving (this has apparently regressed, as it used to work)

Given there exist workarounds and we have to be highly selective about bugs that we fix at this stage, I don't think we'll have it fixed for VS 2017's initial release. It should appear in a later update.

[zooba]

See also #2421.

Currently we have fixed the new files having the wrong encoding (VS would skip our default if they were truly empty, so we added a newline), and now they should always be UTF-8. However, we still have to add the encoding dialog.

[mhtvsSFrpHdE]

@KexyBiscuit
Should this fix be applied either Visual Studio 2015 / 2017?

[zooba]

@Incubator85653 It won't be applied to VS 2015, since we're not releasing updates to that version anymore. But we plan to get the popup into a VS 2017 update at some point in the future.