Powershell: wrong file Encoding

[Skybeat]

Steps to reproduce

• create text file in UTF-8
• run the follow command in powershell 5
  Get-Content <path to UTF8 file> -Encoding UTF8 | Set-Content <path to new output file> -Encoding default

• the output file will be encoded in ANSI (what is correct)

• run the follow command in powershell core (6.2.1)
  Get-Content <path to UTF8 file> -Encoding UTF8 | Set-Content <path to new output file> -Encoding default

• the output file will be encoded in UTF8I (what is NOT correct)

Expected behavior

the output file from Set-Content with parameter "-Encoding ANSI" have to bee encoded in ANSI like it does in Powershell 5 and so on

Actual behavior

the output file from Set-Content with parameter "-Encoding default" is encoded in UTF8 what is wrong

Environment data

Name Value
---- -----
PSVersion 6.2.1
PSEdition Core
GitCommitId 6.2.1
OS Microsoft Windows 6.1.7601 S
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0.}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0

[Skybeat]

One more thing what shows that there is a issue:
If i run the command [System.Text.Encoding]::Default.EncodingName in PowerShell 5 it shows "Western Europen (Windows)" what is correct. If i run the same command in Powershell Core 6.2.1, the outcome is "Unicode (UTF-8)" what is totally wrong.

[alexandair]

Default value for the Encoding parameter has been changed in PowerShell Core. Here is the excerpt from the Set-Content help doc:

Specifies the type of encoding for the target file. The default value is UTF8NoBOM.

[jhoneill]

If i run the command [System.Text.Encoding]::Default.EncodingName in PowerShell 5 it shows "Western Europen (Windows)" what is correct. If i run the same command in Powershell Core 6.2.1, the outcome is "Unicode (UTF-8)"

Different versions of .NET have different default text encodings. (That's not a PowerShell thing)
See https://docs.microsoft.com/en-us/dotnet/api/system.text.encoding.default?view=netframework-4.8
There's a big warning and then an explanation of how it is set on .NET frame work followed by this:
_On .NET Core, the Default property always returns the UTF8Encoding._

[Skybeat]

If i run the command [System.Text.Encoding]::Default.EncodingName in PowerShell 5 it shows "Western Europen (Windows)" what is correct. If i run the same command in Powershell Core 6.2.1, the outcome is "Unicode (UTF-8)"

Different versions of .NET have different default text encodings. (That's not a PowerShell thing)
See https://docs.microsoft.com/en-us/dotnet/api/system.text.encoding.default?view=netframework-4.8
There's a big warning and then an explanation of how it is set on .NET frame work followed by this:
_On .NET Core, the Default property always returns the UTF8Encoding._

Default value for the Encoding parameter has been changed in PowerShell Core.

This is in the OutputEncoding preference variable, which starts as [System.Text.Encoding]::Default

Thank you for the information!

So is there a way that i can change / overwrite the default encoding in powershel core, back to the "old" behavior we have had in PowerShell till 5.1?
If so, how can i do this in PowerShell Core?
I mean i like it mutch more the way that default means the system-wide settings and not fixed UTF8, especially if I specify a special parameter like -Encoding.
That make absolutly no sense for me. In my opinion it wold be better to make the encoding for all this cmdleds default to UTF8 and as far as the parameter -Encoding is set it use the given codepage or if the value for -Encoding is default, it use the system-wide default setting, from the OS.

Encoding.Default Property (System.Text)

[vexx32]

You can set $PSDefaultParameterValues['*:Encoding'] = 'ANSI' or whatever value you happen to prefer. If you add that line to a $profile script, it will be effectively persistent. 🙂

[jhoneill]

Sorry , what I said above about Output encoding is wrong. It is not used as the default value for encoding. For more details on what it does see get-help about_preferenceVariables, and
get-help out-file -Parameter encoding says the defaults are set in cmdlets and gives the right default information on different versions. @vexx32 suggests one good solution, if the encoding matters to a given script/module, specify it explicitly in the code.

[Skybeat]

if the encoding matters to a given script/module, specify it explicitly in the code.

Well, you are right that is a way but, now with this -Encoding default is all the time UTF8 and i have to explicit set the codepage, it goes from easy to complex. All scripts what comes from PowerShell 5 or below and worked very well, are now don't work as expected. It will coast a lot of time to find the issue because of PowerShell core. There is a nice sentence: Never change a simple running system.
In my opinion, users questions will now increase because of the crazy behavior of PowerShell Core when it comes to encoding.

Perhaps it should be considered, for the parameter -Encoding, to introduce a value such as OSdefault, like -Encoding OSdefault

But any ways, now that i know how complex i have to do it, because PowerShell Core are not willing to use the OS default settings any more, I have to use the new way.

[jhoneill]

@Skybeat We may have been talking at cross-purposes.

@SteveL-MSFT this might not be answered after all.

In Windows PowerShell the helps says
" _Unicode is the default.
Default uses the encoding of the system's current ANSI code page_."

The current source has an ArgumentToEncodingTransformationAttribute class which transforms "Default" "Utf8" and "Utf8NoBom" to the same thing.

So the meaning of "Default" has changed from "Current default ANSI code page" to "default for this command"

To get the equivalent of -Encoding Default on Windows PowerShell one has to run
-Encoding $ExecutionContext.Host.CurrentCulture.TextInfo.ANSICodePage on PowerShell 6/7
but this means you can't have one script which works on all versions

Really, specifying -encoding Default should revert to to what it did in 5.
Currently it's a change which might break scripts which target "5 and later".
However rolling it back should not break anything for scripts which target "6 and later" because no-one should be writing -Encoding Default in V6/7: tab completion doesn't suggest it, the help excludes it, and writing nothing at all achieves the same thing.
If someone wants to be explicit about UTF8 encoding in 6/7 they shouldn't be using "default" to do it but "UTF8" or "UTF8NoBOM".

[SteveL-MSFT]

cc @JamesWTruher your thoughts on this? Calling Default to mean Windows PowerShell would solve the compatibility concern, but the default is UTF8NoBOM so that would introduce confusion.

[Skybeat]

Really, specifying -encoding Default should revert to to what it did in 5.
Currently it's a change which might break scripts which target "5 and later".
However rolling it back should not break anything for scripts which target "6 and later" because no-one should be writing -Encoding Default in V6/7: tab completion doesn't suggest it, the help excludes it, and writing nothing at all achieves the same thing.
If someone wants to be explicit about UTF8 encoding in 6/7 they shouldn't be using "default" to do it but "UTF8" or "UTF8NoBOM".

That is what i mean.
for now scripts till Powershell 5.1 will not work as expected in PowerShell 6/7 (i have this issue by my own with currently 6 scripts).
I change like this with "-Encoding default" will make all older scripts from Powershell 5.1 and older incompatible.

[vexx32]

I doubt telling people that the "default" setting for -Encoding (which isn't the _real_ default value for those cmdlets, which would still be UTF8NoBom) is WinPS's default is a good idea.

Folks can just specify their encodings. This is a major version bump. Expecting folks to review and update their scripts before upgrading really ought to be a _given_, especially the ease of which you could do this with, for example, a hashtable variable for splatting into anything that needs an -Encoding parameter.

[Skybeat]

I doubt telling people that the "default" setting for -Encoding (which isn't the _real_ default value for those cmdlets, which would still be UTF8NoBom) is WinPS's default is a good idea.

Folks can just specify their encodings. This is a major version bump. Expecting folks to review and update their scripts before upgrading really ought to be a _given_, especially the ease of which you could do this with, for example, a hashtable variable for splatting into anything that needs an -Encoding parameter.

On one hand your are true when you say the user have to check his scripts before. On the other handy, make a move from WinPS to PS Core as simple as posible and get more acceptance and enjoyment from the end users. It is not always good to reinvent the wheel! It has been so over 3 Powershell versions that will frustrate end users when everything is re-invented and will not promote the acceptance to PS Core right now.

[vexx32]

I appreciate the desire for convenience, but this is not a matter of reinventing the wheel. It's more a matter of aligning PowerShell to established standards in the majority of operational infrastructure, especially in (ever more common) environments where Unix operating systems run the server infrastructure. UTF8 without BOM is largely the standard encoding for text files at present.

[Skybeat]

I appreciate the desire for convenience, but this is not a matter of reinventing the wheel. It's more a matter of aligning PowerShell to established standards in the majority of operational infrastructure, especially in (ever more common) environments where Unix operating systems run the server infrastructure. UTF8 without BOM is largely the standard encoding for text files at present.

So that means that Powershell, what comes from Windows platformes, must now go "standarts" from Unix world?!?! Why don't we say the Windows / Microsoft way is a standart and Unix is out of standart?!?!

Don't get me wrong but i can't see any reason of the Encoding change. I can see only a lot of problems / issues for the end users special those who have been running scripts for years without any problems, but these will no longer work properly with Powershell core.

For example, the "-Encoding default" is changed to UTF8 all the time, but the Parameter Name -Encoding himself is not changed. Why you don't change the parameter name from "-Encoding" to "-Encode"? Because the reason is that PS 5 and older scripts want be work.
So please tell me the reason to change a running system to a system what brings so many issues for end users. And please don't say Unix is standart, a lot of users will say Windows is standard. Who has determined the Unix standard and everything else out of the standard?

[vexx32]

See here for lengthy discussions on various reasonings for or against BOM. The general consensus is that it's not worth adding the BOM to UTF8 files.

At the end of the day, we're here now. And while I appreciate it might be an inconvenience to you to have to update your scripts, I don't think one person is a sufficient driving force to change the standard for all of PowerShell.

If it makes life difficult for you, you can easily set your own defaults for these parameters in a profile script or simply add one easy line to affected scripts to set the default for those scripts using my previous suggestion. 🙂

[jhoneill]

@vexx32

I doubt telling people that the "default" setting for -Encoding (which isn't the _real_ default value for those cmdlets, which would still be UTF8NoBom) is WinPS's default is a good idea.

Folks can just specify their encodings.

Joel, you're right. "Default" was a bad choice of name in Windows PowerShell. It doesn't come up with the argument completer, it's not listed in the help. So people are no longer told about "Default" And if people want the default Ansi code page they can put 1252 or $ExecutionContext.Host.CurrentCulture.TextInfo.ANSICodePage

But there is undocumented support for "default", which seems specifically so that old scripts don't break. Either that should go, or it should keep those old scripts working in the way they used to work. Silently changing "default" mean what most people would expect is wrong.

It's one place. In EncodingUtils.cs there is this

internal static Dictionary<string, Encoding> encodingMap = new Dictionary<string, Encoding>(StringComparer.OrdinalIgnoreCase)
        {
            { Ascii, System.Text.Encoding.ASCII },
            { BigEndianUnicode, System.Text.Encoding.BigEndianUnicode },
            { Default, ClrFacade.GetDefaultEncoding() },
            { OEM, ClrFacade.GetOEMEncoding() },
            { Unicode, System.Text.Encoding.Unicode },
            { Utf7, System.Text.Encoding.UTF7 },
            { Utf8, ClrFacade.GetDefaultEncoding() },
            { Utf8Bom, System.Text.Encoding.UTF8 },
            { Utf8NoBom, ClrFacade.GetDefaultEncoding() },
            { Utf32, System.Text.Encoding.UTF32 },
            { String, System.Text.Encoding.Unicode },
            { Unknown, System.Text.Encoding.Unicode },
        };

Either delete the "default" entry and make people re-write to get the old behaviour or replace it with a suitable System.Text.Encoding.something which will give the old behaviour (without documenting it).

[vexx32]

My personal opinion is (if anything) we should delete the Default entry and just document in the parameter help which of the standard options corresponds to the default value in pwsh, perhaps adding a note that it differs from Windows PowerShell.

That way we have a more clear and obvious sign to users coming from Windows PowerShell that this has changed, and we don't have inexplicable failures coming from somewhere else all of a sudden. 🙂

[jhoneill]

I think deleting it is the second best choice. The ideal would be to keep it working but print a warning that it is deprecated, and delete it from a later version.

[SteveL-MSFT]

@PowerShell/powershell-committee to discuss removing Default.

[Skybeat]

@PowerShell/powershell-committee to discuss removing Default.

Why you want discuss a remove? A remove will also break all WinPS scripts.
Why don't go back do the behavior of WinPS because you have allready a value UTF8 for param -Encoding?

Can you please tell me the reason what makes iit so hard for you to let the old behavior from WinPS , in PS Core, as it is in WinPS?
Why is there a strong attempt to change or delete the value "default", which has existed for years and has been used and proven?
That makes no sense to me and I can not understand that.
I mean PowerShell is a name of a strong and usefull product since years, and if you want change even such simple things like this, then maybe it is better you have to change the name of the product as well and let PowerShell die.
I have the feeling that just this important part is completely ignored.

But ok, I'm only a small single user, we'll see what the future holds and how the stack overflow and co entries pile up because of such changes and end users are suffering.

But these are the different perspectives between developers of PowerShell and users of PowerShell. Maybe the focus should not only be on development but also on usability

[jhoneill]

The alternative to delete is to make that line in EncodingUtils.cs

{ Default, System.Text.Encoding.GetEncoding(Host.CurrentCulture.TextInfo.ANSICodePage) },

This, delete and "no change" are the options the committee should discuss :-) The choice between breaking change and non-breaking change are as simple as 3 options for one line.

[SteveL-MSFT]

@Skybeat the @PowerShell/powershell-committee has already discussed and approved the RFC on default encoding. Looking at it again, it appears we did not implement $PSDefaultEncoding where there is a WindowsLegacy value that would resolve your issue, but only if you knew about it. The concept behind removing Default is that anyone relying on it would know immediately they were not getting the Windows PowerShell default encoding. The absence of specifying -Encoding would be UTF8NoBOM.

[Skybeat]

@Skybeat the @PowerShell/powershell-committee has already discussed and approved the RFC on default encoding. Looking at it again, it appears we did not implement $PSDefaultEncoding where there is a WindowsLegacy value that would resolve your issue, but only if you knew about it. The concept behind removing Default is that anyone relying on it would know immediately they were not getting the Windows PowerShell default encoding. The absence of specifying -Encoding would be UTF8NoBOM.

@SteveL-MSFT
thank you for your answer.
Now after you remove the value default, will you also document (an example) how user can get back there funktionality as it was in WinPS?
For the parameter -Encoding, there is no value like OSdefault (like default was in WinPS).
Can you may also discuss with the comittee, to implement a value like OSdefault with the value of default OS encoding ?
What i mean is that as long as OSdefault is set as value for -Encoding it will use the OS default settings no matter if it is Unix or Windows. OSdefault force to read the OS default endocing and use it. That would be a very good and especially and easy usability and on top, the end user can easy see on the name of the value what it means.

Or can you add a value for ANSI like ANSI and implement it with the code from @jhoneil
{ ANSI, System.Text.Encoding.GetEncoding(Host.CurrentCulture.TextInfo.ANSICodePage) },

Thank you.

[SteveL-MSFT]

I understand this is a pain point, the @PowerShell/powershell-committee will consider different options. Removing Default is just one option.

[SteveL-MSFT]

@PowerShell/powershell-committee discussed this. Despite the pain of Default having different meanings on Windows PowerShell and PowerShell Core, we made a deliberate decision to enable cross platform support of PowerShell Core for the broader community. At this point, no further changes are planned to be made.

[Skybeat]

@PowerShell/powershell-committee discussed this. Despite the pain of Default having different meanings on Windows PowerShell and PowerShell Core, we made a deliberate decision to enable cross platform support of PowerShell Core for the broader community. At this point, no further changes are planned to be made.

@SteveL-MSFT
That means / you will say that "to enable PowerShell Core for the broader community" means you have to change the value "default" in the parameter "_Encoding" because to enable PS Core for a broader Community?!
Please read your own sentence again, that makes zero sence! Since WinPS there was a value "UTF8" for param "-Encoding" so it makes zero sence to change it with Core so that value "default" do exactly the same like "UTF8".

And you would have reached the wider community if I had left the value "Default" on OS Encoding setting. Then you would reach even more in the community because it would be even more flexible as before and even more as it is now. Now there are 2 different values that the same do but no longer value to ANSI encoded. For all those who want to encode to ANSI it gets totally complicated starting with PS Core.
Thanks for opening PS Core to the broad community.

Sorry but you to a very big mistake on basic thinks! If this is how PS core gose to, than i have to tell all my customers to don't use PS Core because it is not longer a descendant of Powershell and not compatible to WinPS at all.

Thank you!

[Skybeat]

It's a shame the issue has just been closed and the bug just stays in PS Core.
I will explain the link to this issue and show the users (especially those who use Powershzell for many years in Windows) how important Microsoft's own product is now to their own operating system. Microsoft already counts alleged standards of Linux / Unix more than an existing and functinating standart functionality in Powershell.

[daxian-dbw]

@Skybeat You can use ANSI encoding by specifying -Encoding oem. It uses GetOEMCP to get the current OEM code page identifier for the OS on windows platform.

[Skybeat]

@Skybeat You can use ANSI encoding by specifying -Encoding oem. It uses GetOEMCP to get the current OEM code page identifier for the OS on windows platform.

yes but it is still not compatible with WinPS 5.1 an older.

In this blog post https://devblogs.microsoft.com/powershell/the-next-release-of-powershell-powershell-7/ from @SteveL-MSFT he was written the follow:

This means that Windows PowerShell and PowerShell Core users will be able to use the same version of PowerShell to automate across Windows, Linux, and macOS and on Windows, and PowerShell 7 users will have a very high level of compatibility with Windows PowerShell modules they rely on today.

So with the current situation, the statement of @SteveL-MSFT is wrong.
I can run a script where i have -Encoding default inside with WinPS5.1 and PS Core but the result is totally different.

For example, I have run a script with -Encoding default for a long time in WinPS 5.1 and the file what was created via the script was encoded very well in ANSI. This file was hand over to a 3. party application (this application needs the data in ANSI) which imports the data.

Since I have run the exact same script with PS Core, half of the 45000 employee data sets the 3. party application imported were destroyed and the employee can't work in this application. Thats because of this big bug / issue in PS Core.

Once again, don't think only as a developer, think more as end user who use this application maybe since PS 3.0 (in this version the -Encoding default was implemented).

In my case, more than 20000 employee base data in the shift planing system was destroyed and my client was very angry. Till i found the issue i takes 1,5 days. In this 1,5 days this more than 20000 employees can't book ther times or request absences and stuff like this.

From my point of view, there must not be such changes in PS Core and these changes contradict the official statements that PS Core is compatible with WinPS.

Maybe for the developers of Powershell Core it is not a big deal, to change some simple thinks like this, but for the people who use Powershell Core it is a big change and let break the scripts who use stuff like this.

And the more crazy thing for me is why it is so hard for the dev team of PS Core and Microsoft to understand this and why theye can not admit that it was a mistake to make that change and say ok, we'll undo that change.

[jhoneill]

@PowerShell/powershell-committee discussed this. Despite the pain of Default having different meanings on Windows PowerShell and PowerShell Core, we made a deliberate decision to enable cross platform support of PowerShell Core for the broader community. At this point, no further changes are planned to be made.

@SteveL-MSFT could you clarify what you mean by "a deliberate decision to enable cross platform support".

Neither the on-line help nor intellisense suggest "default". _No one should ever use -encoding default in code written for PowerShell 6 or 7,_ so it isn't a cross platform issue, as far as I can see.

I get the feeling that @Skybeat manages systems which aren't running in English, and "default" gets the right code page but "ASCII" does not, a problem someone working in English doesn't have.

There is one line of code in PowerShell core which seems to be there to ensure that if a script author put -encoding "default" in a scripts for PowerShell 3,4,5 it doesn't throw an error don't throw an error, on 6 & 7. But, it appears that the person who put that line in did not understand that "default" here didn't mean what it usually means. Form @Skybeat 's point of view that misunderstanding means New PowerShell = Corrupt Output.

Deleting that one line would force anyone using "default" to get their national code page to make a change, and changing it as I outlined in a previous post would all @Skybeat 's produced the expected out put. Either this was not clear to committee when it was discussed, or if was the benefit of silently corrupting output if a legacy choice is selected needs a little more explanation.

[SteveL-MSFT]

@jhoneill the original change to Encoding was really to solve several problems:

1. many cmdlets had their own -Encoding parameter which was using a ValidateSet so you had different values that could be used. The fix here is to have all of them use the actual [Encoding] type consistently.
2. the default Encoding for these cmdlets were not consistent. Fix here is to align with direction of Win10 and also Unix systems which is UTF8NoBOM.

Both of these were breaking changes that @PowerShell/powershell-committee discussed at length and agreed for the future of PowerShell, these were the right changes even though there would be some pain points for users transitioning between Windows PowerShell and PowerShell Core.

[Skybeat]

@jhoneill the original change to Encoding was really to solve several problems:

1. many cmdlets had their own -Encoding parameter which was using a ValidateSet so you had different values that could be used. The fix here is to have all of them use the actual [Encoding] type consistently.
2. the default Encoding for these cmdlets were not consistent. Fix here is to align with direction of Win10 and also Unix systems which is UTF8NoBOM.

Both of these were breaking changes that @PowerShell/powershell-committee discussed at length and agreed for the future of PowerShell, these were the right changes even though there would be some pain points for users transitioning between Windows PowerShell and PowerShell Core.

@SteveL-MSFT
May be you have done a mistake with the current change in PS Core and the default, because in all online docus (including the official MS docu for Win-Powershell example: https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/set-content?view=powershell-5.1 )
the value default is documentat as follow: Default Uses the encoding that corresponds to the system's active code page (usually ANSI).
why did not you just add UTF8NoBOM as a new value and leave the rest as it was?

these were the right changes even though there would be some pain points for users transitioning between Windows PowerShell and PowerShell Core.

So you call it "some pain for users" when such a useless change destroys millions of data in thousands of systems from thousands of companies at once?
Is such a statement really the new thinking of Microsoft and the Powershell Core development team?
To make you understand what such rash changes have effects: In my company we have 50 employees, of which half use Powershell for years. Now I have asked my colleagues who uses the code "Set-Content -Encoding default" in their scripts. Everyone uses it.
Now imagine the scriptre with such erroneous behavior, e.g. be used in banking or social work.
Could you as a developer really not look over the counter and think of any effects before you start discussing something that is not worth discussing as it has been rooted in Powershell way too long?
Have you ever considered discussing it with end users and listening to it before you know better about making such significant changes?

Sorry if I say so directly but it's a shame how you deal with the end users. You leave them alone with the problem and mistakes that you know and which have been reported to you.
You come with more and more excuses just to not have to admit a mistake and as always the users have to be this discharge and stand straight for it. Big movie @Microsoft

[Skybeat]

@all
is there a way to override the incorrect behavior (bug) of -encoding default in Powershell core?
I mean not in every script himself, i mean system-wide!

[jhoneill]

@SteveL-MSFT
OK got it now.
[Encoding] objects don't work with V5 because of the validate sets in use. That needs to be fixed ... but just typing the parameter as [system.text.encoding] won't allow code pages and "OEM" which leads toencodingMapin the code which gets to the right values but _looks_ wrong.

@Skybeat Posts like that long one don't help. As far as anyone here knows, you are the only person impacted by this. You say 25 people in your company use set-content -encoding default. If you ask them what it does , how many would say selecting a different code page ? Can any of them can give an example of something which will break if they put ASCII or UTF8 .
When you talk "about millions of data in thousands of companies" (which assumes that those companies switch from Windows PowerShell to PowerShell core without testing their scripts first) _but_ no-one else has reported the problem, so it reads like the ranting of a crazy person and instead of helping you people ignore a crazy person. You took a dependency on an old mistake and the fix creates extra work for you (like testing), people should be sympathetic but again, ranting crazy people don't get sympathy.

to the short post. Yes. you can over-ride it with a proxy function. If the version is >5 replace "default" with a code page. Done properly it will generate a warning when it does that so you get told where to check your scripts.

[SteveL-MSFT]

@Skybeat I believe I've been more than responsive to you so please respect our Code of Conduct.

[vipiny35]

@Skybeat You can use ANSI encoding by specifying -Encoding oem. It uses GetOEMCP to get the current OEM code page identifier for the OS on windows platform.

Setting -Encoding oem on Windows 10 machine writes file in utf-8 whereas using -Encoding utf8 itself doesn't work