Aspnetcore: System.NullReferenceException on Microsoft.AspNetCore.Http.Internal.DefaultHttpRequest.get_Query

[MatthewLymer]

Operating System: Ubuntu 16.04

dotnet --info

Host (useful for support):
  Version: 2.1.0
  Commit:  caa7b7e2ba

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.NETCore.App 2.1.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Dependencies:

<PackageReference Include="AutoMapper" Version="6.1.1" />
<PackageReference Include="JetBrains.Annotations" Version="11.1.0" />
<PackageReference Include="Microsoft.AspNetCore.Authentication" Version="2.0.0" />
<PackageReference Include="Microsoft.AspNetCore.Diagnostics" Version="2.0.0" />
<PackageReference Include="Microsoft.AspNetCore.Hosting" Version="2.0.0" />
<PackageReference Include="Microsoft.AspNetCore.Mvc" Version="2.0.0" />
<PackageReference Include="Microsoft.AspNetCore.Server.Kestrel" Version="2.0.0" />
<PackageReference Include="Microsoft.Extensions.Caching.Redis" Version="2.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.CommandLine" Version="2.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="2.0.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Console" Version="2.0.0" />
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="2.0.0" />
<PackageReference Include="Microsoft.Net.Http.Server" Version="1.1.2" />
<PackageReference Include="Serilog.Extensions.Logging.File" Version="1.1.0" />
<PackageReference Include="FluentValidation.AspNetCore" Version="7.1.1" />
<PackageReference Include="Microsoft.Extensions.Logging.Filter" Version="1.1.2" />
<PackageReference Include="System.Text.Encoding.CodePages" Version="4.5.0" />

Exception:

System.NullReferenceException

Stack Trace:

   at Microsoft.AspNetCore.Http.Internal.DefaultHttpRequest.get_Query()
   at Microsoft.AspNetCore.Mvc.ModelBinding.QueryStringValueProviderFactory.CreateValueProviderAsync(ValueProviderFactoryContext context)
   at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ActionContext actionContext, IList``1 factories)
   at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ControllerContext controllerContext)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame`1.ProcessRequestsAsync()

[Tratcher]

Request url? Route definitions? Action signature?

[MatthewLymer]

[Authorize]
public sealed class SearchController : Controller
{
    [HttpPost]
    [Route("Search")]
    public async Task<IActionResult> PostAsync([FromBody] SearchRequest request)
    {
        // impl
    }
}

public sealed class SearchRequest
{
    [JsonProperty(PropertyName = "Name")]
    public string Name { get; set; } 

    [JsonProperty(PropertyName = "SessionId")]
    public string SessionId { get; set; }

    [JsonProperty(PropertyName = "Tags")]
    public IDictionary<string,string> Tags { get; set; }

    [JsonProperty(PropertyName = "SearchField")]
    public string Search { get; set; }
}

The request was a POST /search HTTP/1.1 and a JSON body.

[Tratcher]

Side note: symptoms like this are most commonly caused by concurrently accessing the HttpContext on multiple threads. I don't see any other obvious signs of that in the call stack or code, but keep an eye out for it.

How consistent is the error? Every request?

[MatthewLymer]

In ~54,000 requests, it has happened ~20 times.

It looks to coincide to including authorization to my API, so I'll try and deduce from that whether it is related. I'm implementing AuthenticationHandler<T> and it's using Request.Headers and Response.Headers, not the query string specifically though.

[MatthewLymer]

I did a redeploy without authentication middleware and the issue disappeared.

[MatthewLymer]

I commented out the services.AddAuthentication(...) and app.UseAuthentication() to prevent the issue.

In my application configuration:

public void ConfigureServices(IServiceCollection services)
    {        
        services
            .AddAuthentication("Bearer")
            .AddScheme<BearerAuthenticationOptions, BearerAuthenticationHandler>(
                "Bearer", 
                settings => settings.ApiKey = Configuration["ApiKey"]);
    }



    public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory, IServiceProvider serviceProvider)
    {
        app.UseCors(b => b.AllowAnyOrigin().AllowAnyHeader().AllowAnyMethod());

        app.UseAuthentication();

        app.UseMvc();
    }



internal sealed class BearerAuthenticationHandler : AuthenticationHandler<BearerAuthenticationOptions>
{
    public BearerAuthenticationHandler(
        IOptionsMonitor<BearerAuthenticationOptions> options, 
        ILoggerFactory logger, 
        UrlEncoder encoder, 
        ISystemClock clock) 
        : base(options, logger, encoder, clock)
    {

    }

    protected override Task HandleChallengeAsync(AuthenticationProperties properties)
    {
        Response.Headers["WWW-Authenticate"] = "Bearer realm=\"myapp\"";

        return base.HandleChallengeAsync(properties);
    }

    protected override Task<AuthenticateResult> HandleAuthenticateAsync()
    {
        var expectedApiKey = OptionsMonitor.Get(Scheme.Name).ApiKey;

        if (expectedApiKey != "*")
        {
            if (!TryGetApiKey(Request, out var actualApiKey))
            {
                return Task.FromResult(AuthenticateResult.NoResult());
            }

            if (expectedApiKey != actualApiKey)
            {
                return Task.FromResult(AuthenticateResult.Fail("Bad API key."));
            }
        }

        var ticket = CreateTicket();

        return Task.FromResult(AuthenticateResult.Success(ticket));
    }

    private static bool TryGetApiKey(HttpRequest request, out string value)
    {
        const string prefix = "Bearer ";

        var authorizationValue = request.Headers["Authorization"].SingleOrDefault();

        if (authorizationValue == null)
        {
            value = null;
            return false;
        }

        if (!authorizationValue.StartsWith(prefix, StringComparison.OrdinalIgnoreCase))
        {
            value = null;
            return false;
        }

        value = authorizationValue.Substring(prefix.Length);
        return true;
    }

    private static AuthenticationTicket CreateTicket()
    {
        var identities = new[]
        {
            new ClaimsIdentity("ApiUser")
        };

        return new AuthenticationTicket(new ClaimsPrincipal(identities), "Bearer");
    }
}

[Eilon]

@MatthewLymer were you able to make any progress on this? Do you have any new information?

What @Tratcher said is pretty much always the case: some code somewhere is incorrectly holding onto an HttpContext and using it from the wrong thread or concurrently, neither of which is supported.

[Eilon]

Previous report that ended up with no firm conclusion: https://github.com/aspnet/Home/issues/2806

[allevyMS]

I am also hitting this issue, I access the HttpContext in my authorization handlers, in an action filter OnActionExecuted and in the controller method itself. I am not using the HttpContextAccessor. I assume my usage should not be concurrent?

[Eilon]

@allevyMS - can you share some relevant snippets of your code so we can take a look? Unfortunately with these kinds of issues it's really hard to diagnose without the details.

[srmorin]

We see this issue too. For us, it occurs infrequently but when it does, it always occurs in the middleware processing when trying to set the user from the claims principal in the request:

"message": "Object reference not set to an instance of an object.",
"stackTrace": "
at Microsoft.AspNetCore.Http.DefaultHttpContext.set_User(ClaimsPrincipal value)\n
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\n
at Common.Logging.Middleware.ManageLoggingContext.Invoke(HttpContext context)\n

[Eilon]

@srmorin does any other code in your app hold onto the HttpContext (or related object) and potentially access it from another thread?

[srmorin]

Yes. We use some information from the HttpContext in our log messages, and we get this in the Invoke method in the AspDotNet core middleware.

---

From: Eilon Lipton [e-mail]
Sent: Friday, July 13, 2018 4:54 PM
To: aspnet/Home
Cc: srmorin; Mention
Subject: Re: [aspnet/Home] System.NullReferenceException on Microsoft.AspNetCore.Http.Internal.DefaultHttpRequest.get_Query (#3219)

@srmorinhttps://github.com/srmorin does any other code in your app hold onto the HttpContext (or related object) and potentially access it from another thread?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/aspnet/Home/issues/3219#issuecomment-404951489, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AUFZmBwdP7P7hMFTjaOULdijtGZ8mS9Jks5uGQkjgaJpZM4UlLAj.

[davidfowl]

Can you be more specific? Are you storing the http context somewhere?

[srmorin]

Yes, we store specific information from the HttpContext into a separate data structure that is used when the request is complete. We primarily pull this information from the Request property and store it in a separate structure and call Invoke(). At the conclusion of the call, we write out this information by calling the API to our logging infrastructure.

---

From: David Fowler [e-mail]
Sent: Monday, July 16, 2018 11:20 AM
To: aspnet/Home
Cc: srmorin; Mention
Subject: Re: [aspnet/Home] System.NullReferenceException on Microsoft.AspNetCore.Http.Internal.DefaultHttpRequest.get_Query (#3219)

Can you be more specific? Are you storing the http context somewhere?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/aspnet/Home/issues/3219#issuecomment-405283567, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AUFZmPPJ3PdTkdEa4f-Kesp5SatRailqks5uHK9YgaJpZM4UlLAj.

[Tratcher]

Can you share that method? It's possible you're capturing something that's not safe to access after the request completes.

[allevyMS]

@Tratcher question, does the httpcontext need to be modified in a separate thread or simply accessed for this error to appear? If I have the httpcontext already available through one of the various context objects in asp net core, is it safe to use it? Should this issue only appear when using the HttpContextAccessor?

[Tratcher]

Modification on at least one thread while being concurrently accessed or modified by another thread. The catch is that many HttpContext properties are lazy and change internal state even when accessed, so it's not always obvious if your modifying it.

HttpContext is safe to use in the main request processing flow, e.g. as it travels through the pipeline. What context objects did you have in mind?

HttpContextAccessor is not the only way to cause this, but it's certainly the most common.

[allevyMS]

Here are all of my usages:
The Context objects I use are:

• ActionExecutingContext, ActionExecutedContext (in both cases to get the Request HashCode) like this OnActionExecutingContext and OnActionExecuted calls of an IActionFilter like so:

       actionContext.HttpContext.Request.GetHashCode()

• In an AuthorizationHandlerContext supplied to a HandleRequirementAsync call of an AuthorizationHandler to get the User and request Path like this:

      var filterContext = context.Resource as AuthorizationFilterContext;
      var request = filterContext.HttpContext.Request;
      var user = filterContext.HttpContext.User;
  

• in an ExceptionFilterAttribute OnException call to setReasonPhrase and Response.StatusCode and to retrieve user claims

HttpContextAccessor: used in an app insights ITelemetryInitializer like this:

        var httpContext = _httpContextAccessor.HttpContext;
        var headers = httpContext?.Request?.Headers;

neither the context or the headers are modified.

Does something seem suspicious in these usages?

[Tratcher]

What does ITelemetryInitializer do with the the httpContext and headers? Do you have this problem if you remove that?

[davidfowl]

It’s likely a problem caused by accessing the HttpContext in telemetry initlializers. It’s pretty fundamentally broken as those run when the App insights pipeline writes any data (from any thread). The “fix” that the app insights team have used in the past is to lock on the http context object. When trying to access it inside of the telemetry initializer.

[allevyMS]

This is the entire Initialize snippet:

    public void Initialize(ITelemetry telemetry)
    {
        var httpContext = _httpContextAccessor.HttpContext;
        var headers = httpContext?.Request?.Headers;

        if (headers == null)
        {
            return;
        }

        if (headers.ContainsKey(_client_ip_header))
        {
            var headerValue = headers[_client_ip_header];
            if (!string.IsNullOrWhiteSpace(headerValue))
            {
                telemetry.Context.Location.Ip = headerValue;
            }
        }
    }

It's hard to verify since this issue is transient even with the app insights instrumentation.

Does the above usage seem suspect? I can disable it and run continuous benchmark tests to see if we still hit this null reference.

[davidfowl]

Yes disable it and see if it works. The problem is the code that any code that calls through app insights API might be running on different threads causing this code to be executed concurrently.

The most common occurrence of this problem happens when using the APIs after an outbound call (http request).

[allevyMS]

I really appreciate the input here @davidfowl

So this is a plausible cause if I understand you correctly:

The initialize may happen on multiple threads, if it happens after an outbound call (which happens as part of handling the request - which we have many) we may hit this null reference exception?

What is the significance of the outbound call that breaks it?

[davidfowl]

The initialize may happen on multiple threads, if it happens after an outbound call (which happens as part of handling the request - which we have many) we may hit this null reference exception?

That's correct.

What is the significance of the outbound call that breaks it?

The thread you return to is a random thread pool thread.

Here's the type of code that would cause that:

```C#
using Microsoft.Extensions.Logging;

namespace Talk1Samples
{
public class ParallelAccessController : Controller
{
private readonly ILogger _logger;

    public ParallelAccessController(ILogger<ParallelAccessController> logger)
    {
        _logger = logger;
    }

    [Route("/parallel")]
    public async Task<string> ParallelAccess()
    {
        var task1 = GetAsync("www.google.com");
        var task2 = GetAsync("www.bing.com");

        await Task.WhenAll(task1, task2);

        return "Hello World";
    }

    private async Task GetAsync(string url)
    {
        try
        {
            _logger.LogInformation("Before {url}:{contenttype}", url, Request.ContentType);

            await Task.Delay(100);
        }
        finally
        {
            _logger.LogInformation("After {url}:{contenttype}", url, Request.ContentType);
        }
    }
}

}
```

PS: We gave a talk on this at NDC London https://vimeo.com/254635669. It shows this specific example and how it can crash and burn 😄.

[MatthewLymer]

@Eilon
I updated my code to pass in a NullLoggerFactory.Instance to the AuthenticationHandler constructor, this resolved the problem for me. I looked at my logger providers, but I don't see any of them using IHttpContextAccessor, though they're 3rd party providers, so I can't be certain.

[allevyMS]

@davidfowl Disabling the App Insights instrumentation (and thus the use of httpcontext accessor from the ITelemetryInitializer) solved the null reference. Thank you!

[Eilon]

Closing because it does not appear there is any further action required for ASP.NET Core.