Runtime: Proposal: Obsolete UTF-7 encoding in the framework

[GrabYourPitchforks]

Motivation

The UTF-7 transcoding logic in the framework is problematic. UTF-7 itself has been deprecated for quite some time, and MSDN even warns against using the UTF7Encoding type due to the possibility of introducing security and reliability problems.

ASP.NET Full Framework 4.5+ and ASP.NET Core (all versions) forbid processing UTF-7 encoded requests by default due to the problems that UTF-7 has historically caused.

Ideally we'd be able to change application code not to consider UTF-7 as a default candidate for text processing, but that's not always practical. There are scenarios where library code passes untrusted _charset_ values as inputs to the Encoding.GetEncoding API, and they're not necessarily hardened against these APIs returning UTF7Encoding or some other undesirable instance. A proposal which properly deprecates UTF-7 framework-wide also needs to take this into account.

Proposed API and behavioral changes

namespace System.Text
{
    public class Encoding
    {
        [Obsolete(...)] // ** NEW attribute ** on existing property
        public static Encoding UTF7 { get; }
    }

    public class UTF7Encoding
    {
        [Obsolete(...)] // ** NEW attribute ** on existing ctor
        public UTF7Encoding();

        [Obsolete(...)] // ** NEW attribute ** on existing ctor
        public UTF7Encoding(bool);
    }
}

• Add [Obsolete] (as warning) to the System.Text.UTF7Encoding ctors; this allows things like typeof(UTF7Encoding) to continue to work
• Add [Obsolete] (as warning) to the static property System.Text.Encoding.UTF7
• Disallow System.Text.Encoding.GetEncoding("utf-7") from returning a UTF7Encoding instance by default

If an application needs Encoding.GetEncoding("utf-7") to return a UTF7Encoding instance, this can be done by using AppContext at app start, re-introducing UTF-7 knowledge into the system, as shown below.

AppContext.SetSwitch("System.Text.Encoding.EnableUnsafeUTF7Encoding", true);

[GrabYourPitchforks]

I also wouldn't be opposed to introducing a new "convenience" API UTF7Encoding.Register (or something) so that app devs can easily bring this back without needing to learn how the EncodingProvider plumbing works.

[GrabYourPitchforks]

Note to self: also investigate what other libraries like ICU do with regard to UTF-7.

[tarekgh]

can we have a config switch for anyone who wants to turn the old behavior back at runtime? I am fine with the change but worried about the app compact for anyone using this encoding today.

[GrabYourPitchforks]

@tarekgh What if we used AppContext for this? Seems to be the way to do things nowadays. We can use the switch name "System.Text.Encoding.EnableUTF7".

[Joe4evr]

@tarekgh Better Obsoletion is supposed to solve this. Projects that are still dependent on this legacy encoding can ignore/#pragma the unique warning ID without compromising Obsolete diagnostics from other sources.

[tarekgh]

@GrabYourPitchforks

What if we used AppContext for this? Seems to be the way to do things nowadays. We can use the switch name "System.Text.Encoding.EnableUTF7".

This looks good to me.

@Joe4evr

Better Obsoletion is supposed to solve this. Projects that are still dependent on this legacy encoding can ignore/#pragma the unique warning ID without compromising Obsolete diagnostics from other sources.

If I understand the Better Obsoletion correctly, it is a compile-time and not runtime enforcement. apps can use code like Encoding.GetEncoding(encodingName) which the compiler will not be able to catch. at runtime, encodingName can be set to the UTF7 encoding name.
Also, the other scenario I am worried about is, if the app is using 3rd party library which is enabling UTF7. I am not sure if the app will have any control over that at that time.

[GrabYourPitchforks]

Right, the app context switch is to revert back to the older runtime behavior. New obsoletions like this only affect source. We'd still want to use a diagnostic id for this so that devs can opt out of this obsoletion without opting out of obsoletions as a whole.

[GrabYourPitchforks]

I updated the proposal to use AppContext.

[tarekgh]

Could you please also hide UTF-7 encoding from the encoding enumeration Encoding.GetEncodings()?

[GrabYourPitchforks]

Breaking change notification: https://github.com/dotnet/docs/issues/19274