Aspnetcore: Templates do not work in Antares Linux due to missing scheme forwarders

[Tratcher]

The 2.1 templates include UseHttpsRedirection and UseHsts by default. These stop working and put your site into an infinite loop if you deploy to Antares Linux (or behind any other reverse proxy besides IIS). The problem is that TLS is terminated by the reverse proxy and Kestrel doesn't know the correct request scheme. OAuth and OIDC also fail in this configuration as they generate incorrect redirects.

The fix is to add and configure the ForwardedHeaders middleware to fix up the scheme as forwarded from the proxy. UseIISIntegration adds and configures this middleware when running behind IIS but we have no matching lightup and config for Linux e.g. UseApacheIntegration, UseNginxIntegration, etc..

Related:
https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer
https://github.com/aspnet/Docs/issues/5806

Proposal:
See what it would take to detect and light up in Antares Linux, and add that code to CreateDefaultBuilder. Also see if this can be generalized to other Apache and NGinx scenarios.

@DamianEdwards for 2.1 rc1 consideration

[muratg]

Moving out due to schedule constraints. @DamianEdwards @glennc FYI.

[muratg]

@shirhatti Could you please sync up with @Tratcher on this one?

[Tratcher]

Another hit: https://github.com/aspnet/Security/issues/1901

[shirhatti]

cc @bradygaster

[muratg]

Moving out since preview2 already shipped.

[bradygaster]

@muratg i tried this tonight using 2.1 templates to antares linux and it lit up right away (hero path was clean). i will also try on AKS, as that uses, if memory serves me correctly, nginx, and update this with the result. if this also works i'd like to make sure i understand what's awry to help make sure the experiences (kubernetes with nginx, or an AKS out-of-the-box deployment and app service linux).

[muratg]

@bradygaster interesting. The template does have UseHttpsRedirection, right? I wonder if there's no reverse proxy involved in antares linux then (would be surprising on its own.)

On AKS, I believe you'd pass through Azure load balancer. (You may also install an internal reverse proxy, e.g. via nginx-ingress.) Not sure if those would be broken or not.

[bradygaster]

@muratg yes, they do have UseHttpsRedirection in Startup.cs. The e2e "hero scenario" to go from dotnet new razor to a working https://dotnetcoreonlinux.azurewebsites.net/ (real url) took about 10 minutes. i'll try it again for kicks now using the new az webapp up command, just for kicks. 👍

Worked on getting our demo aks cluster working tonight with nginx per this article tonight and didn't get too far, so i'm going to rebuild the cluster and try via draft up with a few yaml tweaks, with no customization to the cluster - just out-of-the-box AKS. will report back on that tomorrow by eod.

[Tratcher]

Did you echo out the scheme to make sure? HTTPS can be enforced on the front end, but that doesn't help redirect generation for things like auth.

[bradygaster]

@Tratcher I'll want to talk to you and @muratg to understand the scenario. :) The out-of-the-box scenario works, so I presume i'm missing some setup steps that would expose the issue on Antares linux.

[Tratcher]

Another hit: https://github.com/MicrosoftDocs/azure-docs/issues/11104

[bradygaster]

reproduced this issue, and with feedback from @cephalin was able to validate a potential solution, which is represented by code from Startup.cs in a working project below.

This has been validated in a few new web apps running with a home-built docker image of a .net core 2.2 app.

In the Configure(IApplicationBuilder app, IHostingEnvironment env) method of Startup.cs, we'd need to add code like this that would effectively enable HTTP in a container on App Service:

var options = new ForwardedHeadersOptions {
      ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto    
};

options.KnownProxies.Add(IPAddress.Parse("::ffff:172.19.0.1"));

app.UseForwardedHeaders(options);

A proper fix for this would be simple, but we need to answer questions on how the argument(s) passed to IPAddress.Parse should be loaded or discovered. Another concern is preloading these with values appropriate for App Service customers, Kubernetes customers, and potential variations of these scenarios.

[DamianEdwards]

Seems worthy of a design meeting with a few folks, e.g. @Tratcher, @davidfowl, @shirhatti, myself, etc.

[anurse]

Who is driving this design meeting?

[bradygaster]

see ur inbox

[anurse]

Triage decision: I'll leave this here for now since the meeting is scheduled for this week.

[anurse]

@Tratcher needs something that doesn't involve Domain Controllers to tinker with.

[anurse]

The plan:

• [3.0] Make Forwarded Headers middleware idempotent
• [3.0] Add code to CreateDefaultBuilder that enables forwarded headers if environment variable ASPNETCORE_FORWARDEDHEADERS_ENABLED=true (name can be bikeshed)
• [3.0] Change Antares Linux container image to set this variable - Azure-App-Service/dotnetcore-template#1
• Blog about doing it for your own containers, and for doing this in 2.x

[anurse]

@bradygaster assigned to own blog content

[anurse]

2.x solution blog post outline:

• Add this code to top of startup

if(string.Equals(Environment.GetEnvironmentVariable("ASPNETCORE_FORWARDEDHEADERS_ENABLED"), "true", StringComparison.OrdinalIgnoreCase))
{
    app.UseForwardedHeaders(...);
}

• and set the app setting ASPNETCORE_FORWARDEDHEADERS_ENABLED=true

[bradygaster]

Talked to ORYX team and App Service team and all agree ORYX would be the best place to dial in the environment variable.

[cephalin]

@anurse Does this method work right now or would I need to wait for an update to .NET Core?

[Tratcher]

The approach Brady gives is the recommendation, but you'll need to use an existing environment variable to make it work now, ASPNETCORE_FORWARDEDHEADERS_ENABLED won't be available yet.

[cephalin]

@Tratcher sounds good. Thanks.

[bradygaster]

@cephalin the nice people who build the container images will be setting that EV for us. Estimate is early summer for release of the images inclusive with that EV.

[anurse]

@cephalin FYI we don't plan to backport this to 2.x builds, but the code is very simple (just enabling the forwarded headers middleware when an environment variable is set). @bradygaster will be writing up a blog post on how to enable this on 2.x builds.

[cephalin]

@bradygaster That sounds great! @anurse Or I can just modify the guidance here: https://docs.microsoft.com/en-us/azure/app-service/containers/configure-language-dotnetcore#detect-https-session

[bradygaster]

Yes, @cephalin that would be fantastic for you to complement the upcoming blog post (once we fix this) with the doc update. thanks so much.

[Tratcher]

https://github.com/Azure-App-Service/dotnetcore-template/pull/1 ASPNETCORE_FORWARDEDHEADERS_ENABLED=true added to the docker files.

[Tratcher]

@bradygaster let's sync up about the 2.x blog post.

[Tratcher]

Recommended 2.x mitigation that cleanly upgrades to 3.0:

                // ConfigureServices
                if (string.Equals("true", hostingContext.Configuration["ForwardedHeaders_Enabled"], StringComparison.OrdinalIgnoreCase))
                {
                    services.Configure<ForwardedHeadersOptions>(options =>
                    {
                        options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
                        // Only loopback proxies are allowed by default. Clear that restriction because forwarders are
                        // being enabled by explicit configuration.
                        options.KnownNetworks.Clear();
                        options.KnownProxies.Clear();
                    });
                }

                // Configure:
                app.UseForwardedHeaders();

[anurse]

Closing, since this is done for 3.0. Let's make getting the blog post written part of "accepting" the issue.

[Tratcher]

Verified with 3.0.100-preview6-012234. Note we don't have the new container images so you need to add ASPNETCORE_FORWARDEDHEADERS_ENABLED=true as an app setting.

@bradygaster, how's the blog post?

[bradygaster]

in progress amidst conf prep and post-move. will have a review to you two by EOW