Runtime: HttpClient not able to handle UTF8 Headers

[tsproesser]

I've noticed this issue when I first tried to request a service that just answered me with a cookie containing UTF8 strings (and obvisously got UTF8 encoded by the server; probably IIS). As soon as I send the next request I get the following exception:

HttpRequestException: Request headers must contain only ASCII characters.

Sure headers need to be ASCII-encoded, but the real world already started to make an exception for cookies (e.g. FF, Chrome, Opera seam to do). As far as I can see I am not able to set a encoding for the headers so I am not able to communicate with this server.

[tsproesser]

Just found that in https://github.com/dotnet/runtime/blob/master/src/libraries/System.Net.Http/src/System/Net/Http/Headers/HeaderDescriptor.cs#L140 already the Location is read as UTF8 - probably for redirect. Guess the same should be done for set-cookie?

[alnikola]

Could you please clarify whether you encountered the error on receiving first response or sending the second request?

[tsproesser]

The following code can fix the issue:

Encoding.UTF8.GetString(Encoding.GetEncoding(28591).GetBytes((response as HttpWebResponse).Cookies[1].Value))

If any request receives a UTF8 encoded Cookie the second will fail (because a non ASCII character is set in die cookies)

[alnikola]

Did I get it right that server responded with non-ASCII characters? It's prohibited by the RFC.
Additionally, could you please point me to some official documentation for Chrome saying that it can accept UTF-8 encoded cookies and sent them back to the server? I tried searching for something, but with no luck.

[tsproesser]

You can check this guy who has done several tests: https://stackoverflow.com/questions/25665703/encoding-scheme-used-for-cookies. All I can tell you is that Chrome works for me with a page sending e.g. a German "Ü" in a cookie's content - the .NetCore WebClient gets an exception. And as I've already mentioned: the same fix was done for the Location, too.

[tsproesser]

Since the service I'm querying is an ASPX-service I'd guess that ASPX encodes the Cookies UTF8 in general.

[alnikola]

I read that post on SO, but it seems the author was talking about cookie storage on the client inside browser whereas the transfer encoding is discussed in this comment where everything is properly encoded.
Anyways, thank you for your suggestion. We will consider it on triage.

[tsproesser]

Yes, that's correct ... But somehow the cookies must have come there.

Any idea how I can fix this right now relying on the WebRequest-class? Can I somehow parse new cookies the first time I receive them?

[tsproesser]

If anyone else faces this problem, I've written an extension method that fixes the cookies and Url-encodes them (since you fixed cookie can't be sent containing non-ASCII characters):

public static WebResponse FixResponseUTF8Cookies(this WebResponse paramHttpWebResponse)
{
    foreach (var c in (paramHttpWebResponse as HttpWebResponse)?.Cookies.OfType<Cookie>())
    {
        var newValue = Encoding.UTF8.GetString(Encoding.GetEncoding(28591).GetBytes(c.Value));
        Debug.Assert(newValue == c.Value);
        if (newValue != c.Value) { c.Value = HttpUtility.UrlEncode(newValue); }
    }

    return paramHttpWebResponse;
}

Sure the server needs to understand this, but guess the most do.

One more note for the dotnet-team, if you consider allowing utf8 encoded cookies (like you do for the Location): remember that you also need to allow utf8 encoded cookies for the request!

[tsproesser]

I read that post on SO, but it seems the author was talking about cookie storage on the client inside browser whereas the transfer encoding is discussed in this comment where everything is properly encoded.
Anyways, thank you for your suggestion. We will consider it on triage.

That's the best SO I've found so far: Browser cookie encoding

[davidsh]

Did I get it right that server responded with non-ASCII characters? It's prohibited by the RFC.

Just found that in https://github.com/dotnet/runtime/blob/master/src/libraries/System.Net.Http/src/System/Net/Http/Headers/HeaderDescriptor.cs#L140 already the Location is read as UTF8 - probably for redirect. Guess the same should be done for set-cookie?

In general, non-ASCII characters are disallowed by the RFC in header values. That doesn't stop browsers, however, from creating 'defacto' standards and accept UTF8 in certain headers.

While we generally try to adhere strictly to RFC standards, we also try to be flexible in accepting/receiving certain non-standard formats that are being used. The recent change to the 'Location' header to allow UTF8 is one such example.

We will have to take a closer look at the use of UTF8 in cookie header values before making a decision about any change to .NET Core behavior.

[tsproesser]

Did I get it right that server responded with non-ASCII characters? It's prohibited by the RFC.

Just found that in https://github.com/dotnet/runtime/blob/master/src/libraries/System.Net.Http/src/System/Net/Http/Headers/HeaderDescriptor.cs#L140 already the Location is read as UTF8 - probably for redirect. Guess the same should be done for set-cookie?

In general, non-ASCII characters are disallowed by the RFC in header values. That doesn't stop browsers, however, from creating 'defacto' standards and accept UTF8 in certain headers.

While we generally try to adhere strictly to RFC standards, we also try to be flexible in accepting/receiving certain non-standard formats that are being used. The recent change to the 'Location' header to allow UTF8 is one such example.

We will have to a closer look at the use of UTF8 in cookie header values before making a decision about any change to .NET Core behavior.

Sure, no problem! And thank you very much for your kind answer! For now I can go with the extension method I've posted - and I'd love to see the feature implemented in future. Since it's exactly the same "wrong or right" like the location is, I guess it should be implemented - or at least be optionally possible. But sure, that's your decision! Anyway, thanks in advance!

Edit: one more thing: if maybe just the HttpRequestException: Request headers must contain only ASCII characters. - exception can be disabled I could just use the library out of the box? Since the server will understand what the mangled cookie I send him wants to say.