Runtime: HttpRequestHeaders unable to add malformed User-Agent header

[doggy8088]

The following code snippet will produce IndexOutOfRangeException. Is this a bug?

using System.Net.Http;
using System.Net.Http.Headers;

HttpRequestHeaders headers = new HttpClient().DefaultRequestHeaders;
headers.Add("User-Agent", "Mozilla/4.0 (compatible (compatible; MSIE 8.0; Windows NT 6.1; Trident/7.0)");

[davidsh]

The following code snippet will produce IndexOutOfRangeException. Is this a bug?

The header value you are adding is malformed. So, you would expect an exception.

If you're trying to add malformed headers and don't want validation, you can do this:

```c#
using System.Net.Http;
using System.Net.Http.Headers;

HttpRequestHeaders headers = new HttpClient().DefaultRequestHeaders;
headers.TryAddWithoutValidation("User-Agent", "Mozilla/4.0 (compatible (compatible; MSIE 8.0; Windows NT 6.1; Trident/7.0)");
```

[doggy8088]

@davidsh Thanks! After I added to the headers, I can't read it out. It will produce IndexOutOfRangeException too. What can I do?

[davidsh]

@davidsh Thanks! After I added to the headers, I can't read it out. It will produce IndexOutOfRangeException too. What can I do?

How are you actually accessing the header? What is the code you're using?

Usually if you are using the strongly-typed properties such as HttpRequestHeaders.UserAgent then it will return null is there is no header value or the header value cannot be parsed into a valid value/strong type. In that case (i.e. for malformed headers), you use methods like GetValues or TryGetValues which will return raw strings for the header value(s).

[doggy8088]

My case is in a Proxy server. I have to copy the context's Headers to forwarded Headers. I also have to determine which headers have the required data to write conditions such as User-Agent header. If I can't read it out, I can't check the header value.

I was just checking the following code in my LinqPad. The IndexOutOfRangeException still happen:

void Main()
{
    HttpRequestHeaders headers = new HttpClient().DefaultRequestHeaders;

    headers.TryAddWithoutValidation("User-Agent", "Mozilla/4.0 (compatible (compatible; MSIE 8.0; Windows NT 6.1; Trident/7.0)");

    headers.TryGetValues("User-Agent", out IEnumerable<string> ua);  // IndexOutOfRangeException

    headers.GetValues("User-Agent");  // IndexOutOfRangeException
}

[doggy8088]

Due to many of the users use some weird Browsers who sent the malformed User-Agent header, that cause my ASP.NET Core app throw IndexOutOfRangeException. I don't know how to workaround this.

How can I read the headers safely without any Exception happen?

[davidsh]

What is the version of .NET Core you are using? Can you show output from "dotnet --info"?

[doggy8088]

.NET Core SDK (reflecting any global.json):
 Version:   2.2.200-preview-009748
 Commit:    2d1cbdca8f

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.200-preview-009748\

Host (useful for support):
  Version: 2.2.1
  Commit:  878dd11e62

.NET Core SDKs installed:
  1.1.0 [C:\Program Files\dotnet\sdk]
  2.0.0 [C:\Program Files\dotnet\sdk]
  2.0.2 [C:\Program Files\dotnet\sdk]
  2.0.3 [C:\Program Files\dotnet\sdk]
  2.1.2 [C:\Program Files\dotnet\sdk]
  2.1.4 [C:\Program Files\dotnet\sdk]
  2.1.101 [C:\Program Files\dotnet\sdk]
  2.1.200 [C:\Program Files\dotnet\sdk]
  2.1.201 [C:\Program Files\dotnet\sdk]
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.300 [C:\Program Files\dotnet\sdk]
  2.1.302 [C:\Program Files\dotnet\sdk]
  2.1.400 [C:\Program Files\dotnet\sdk]
  2.1.401 [C:\Program Files\dotnet\sdk]
  2.1.402 [C:\Program Files\dotnet\sdk]
  2.1.403 [C:\Program Files\dotnet\sdk]
  2.1.500 [C:\Program Files\dotnet\sdk]
  2.1.502 [C:\Program Files\dotnet\sdk]
  2.1.503 [C:\Program Files\dotnet\sdk]
  2.1.600-preview-009426 [C:\Program Files\dotnet\sdk]
  2.1.600-preview-009472 [C:\Program Files\dotnet\sdk]
  2.2.100 [C:\Program Files\dotnet\sdk]
  2.2.101 [C:\Program Files\dotnet\sdk]
  2.2.200-preview-009648 [C:\Program Files\dotnet\sdk]
  2.2.200-preview-009748 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

[damianh]

This is a bit yucky alright. Not being able to read the header that was added without validation I would consider to be a bug.

Also HTTPClient not being able to send a request who had a header added without validation is also a bug.

var request = new HttpRequestMessage(HttpMethod.Get, "https://example.com");
request.Headers.Add("User-Agent", "Mozilla/4.0 (compatible (compatible; MSIE 8.0; Windows NT 6.1; Trident/7.0)");
var client = new HttpClient();
await client.SendAsync(request); // IndexOutOfRangeException

[davidsh]

@doggy8088 It is possible to try and reproduce this on .NET Framework.

We did have some header parsing logic changes in the latest versions of .NET Core. And it's possible that some things got broken w.r.t. malformed headers.

This is a bit yucky alright. Not being able to read the header that was added without validation I would consider to be a bug.
Also HTTPClient not being able to send a request whose had a header added without validation is also a bug.

I agree these things are bugs.

@geoffkizer @stephentoub

[damianh]

I can confirm that TryGetValues throws on IndexOutOfRangeException occurs on .NET Framework (net471).

request.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/4.0 (compatible (compatible; MSIE 8.0; Windows NT 6.1; Trident/7.0)");
request.Headers.TryGetValues("User-Agent", out var _); // IndexOutOfRangeException

However, client.SendAsync(request) does not throw on .NET Framework

[damianh]

Also shows up via in AspNet Core TestHost ClientHandler:

System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at System.Net.Http.HttpRuleParser.GetExpressionLength(String input, Int32 startIndex, Char openChar, Char closeChar, Boolean supportsNesting, Int32& nestedCount, Int32& length)
   at System.Net.Http.Headers.ProductInfoHeaderValue.GetProductInfoLength(String input, Int32 startIndex, ProductInfoHeaderValue& parsedValue)
   at System.Net.Http.Headers.ProductInfoHeaderParser.TryParseValue(String value, Object storeValue, Int32& index, Object& parsedValue)
   at System.Net.Http.Headers.HttpHeaders.TryParseAndAddRawHeaderValue(String name, HeaderStoreItemInfo info, String value, Boolean addWhenInvalid)
   at System.Net.Http.Headers.HttpHeaders.ParseSingleRawHeaderValue(String name, HeaderStoreItemInfo info)
   at System.Net.Http.Headers.HttpHeaders.ParseRawHeaderValues(String name, HeaderStoreItemInfo info, Boolean removeEmptyHeader)
   at System.Net.Http.Headers.HttpHeaders.<GetEnumeratorCore>d__21.MoveNext()
   at Microsoft.AspNetCore.TestHost.ClientHandler.<>c__DisplayClass3_0.<SendAsync>b__0(HttpContext context)
   at Microsoft.AspNetCore.TestHost.ClientHandler.<SendAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()