Aspnetcore: ConfigureServices in WebApplicationFactory invoked after Startup.ConfigureServices when use IHostBuilder in 3.0.0-preview6

[branhill]

Describe the bug

ConfigureServices in WebApplicationFactory.ConfigureWebHost invoked after Startup.ConfigureServices has been invoked when use generic IHostBuilder CreateHostBuilder in Program.

To Reproduce

Steps to reproduce the behavior:

1. Using this version of ASP.NET Core '3.0.0-preview6'
2. Create a new ASP.NET Core MVC project with individual auth, in Program.cs the generated code use generic host builder by default in 3.0.

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
        });

1. Create a unit test project with Microsoft.AspNetCore.Mvc.Testing package, inherit WebApplicationFactory and override ConfigureWebHost follow the docs.
2. Set breakpoints both inner scopes of ConfigureServices in Startup and CustomWebApplicationFactory.
3. Debug the test, see breakpoint in Startup hit before CustomWebApplicationFactory. Also the DbContext cannot be overridden like the docs shows.
4. Replace CreateWebHostBuilder in Startup with old CreateWebHostBuilder

public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
    WebHost.CreateDefaultBuilder(args)
        .UseStartup<Startup>();

1. See breakpoint in CustomWebApplicationFactory hit before Startup.

Expected behavior

ConfigureServices in WebApplicationFactory should be invoked before Startup.ConfigureServices.

Additional context

.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview6-012264
 Commit:    be3f0c1a03

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18362
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.0.100-preview6-012264\

Host (useful for support):
  Version: 3.0.0-preview6-27804-01
  Commit:  fdf81c6faf

.NET Core SDKs installed:
  2.1.800-preview-009696 [C:\Program Files\dotnet\sdk]
  2.1.800 [C:\Program Files\dotnet\sdk]
  2.2.400-preview-010219 [C:\Program Files\dotnet\sdk]
  2.2.400 [C:\Program Files\dotnet\sdk]
  3.0.100-preview6-012264 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview6.19307.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview6-27804-01 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0-preview6-27804-01 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

[davidfowl]

cc @javiercn

[javiercn]

@davidfowl This is a result of the move to generic host. Where Startup.ConfigureServices gets called inmmediately. I don't think there's anything we can do about it.

Should you, @Tratcher and me get some time to chat about these changes and what guidance to provide?

I remember having to do some gymnastics to get the Identity UI tests to work with generic host.

I think the biggest change is that Startup.ConfigureServices runs inmmediately, so given that we setup the tests by invoking your builder as the first thing, you can't simply override it later.

• This is what I do in the Identity UI test project: https://github.com/aspnet/AspNetCore/blob/master/src/Identity/testassets/Identity.DefaultUI.WebSite/Program.cs#L20
• This is what I do in my custom factory in order to use a custom Startup class so that I don't get repeated calls https://github.com/aspnet/AspNetCore/blob/master/src/Identity/test/Identity.FunctionalTests/Infrastructure/ServerFactory.cs#L34-L35 https://github.com/aspnet/AspNetCore/blob/master/src/Identity/test/Identity.FunctionalTests/Infrastructure/ServerFactory.cs#L41

For reference, this is how I override the database now:
https://github.com/aspnet/AspNetCore/blob/b9823ed41c8ff23b6dbd233f5cc2a8e5078d2339/src/Identity/test/Identity.FunctionalTests/Infrastructure/FunctionalTestsServiceCollectionExtensions.cs#L24-L55

The main reason to use a custom TStartup is if you need to make incompatible service registrations that are not easy to override (like in our case where we test multiple Identity configurations and call AddIdentity with multiple generic parameters (or not at all)) or when you need to completely replace the Configure pipeline.

I'm not sure at this point if it would make sense to recommend using a different environment, by calling UseEnvironment("Testing") and tell people to write ConfigureTestServices or have them write something like

void ConfigureServices(IServiceCollection collection)
{
    if(!HostEnvironment.Environment == "Testing")
    {
        // Register your services normally
    }
}

That way you could theoretically get away with just doing UseStartup again on the WAF, but I think the main point is that you could do this before without any code change (in IWebHostBuilder) and it is not possible anymore with HostBuilder.

I'm not sure there's anything we can do about it, other than what I did in our own tests. Is there alternative guidance we can provide customers?

[davidfowl]

So this is about overriding startup completely? Is it about replacing calls to Configure or a ConfigureServices?

[javiercn]

@davidfowl In the past, calling UseStartup would replace the original UseStartup class completely, and when you called ConfigureServices on the host builder after UseStartup those would get called before ConfigureServices in UseStartup. (That's why we had ConfigureTestServices)

Now the problem is that users can't simply call UseStartup in their tests without both Startup and TestStartup ConfigureServices running and you can't override services by relying in TryAdd (as we used to do in the past), For example, EF would do TryAdd so you would do ConfigureServices(services => services.Add) and that would force the one in the regular app startup to NOOP.

I don't think that specifically is a big deal as given that now you always run after you can simply replace the descriptor in the container. (Although a bit inconvenient sometimes).

But the issue remains that there is nothing to prevent the original Startup.ConfigureServices from running, which makes using a TestStartup in your test code painful without changes.

What I did in my code was to take control of calling startup in all my tests and simply add a static flag to Program.cs that I could set before invoking CreateHostBuilder.

That's good for our tests, but it has two problems:

• I did it because I tracked down what was going on (which wasn't obvious either).
• It requires modifying your app for testing purposes, which is something we tried really hard to avoid as a starting point.

At some point, I think we can discourage people to use TestStartup classes in their tests and simply override the services, but this is not a small change for people with current test suites.

[anurse]

We should do some docs stuff here, but there's no code changed planned for 3.0

[guardrex]

I was working on doc updates for 3.0 (aspnet/AspNetCore.Docs #13353). There are a few issues at the moment unrelated to the discussion here that I'm dealing with [e.g., Identity doesn't play well with an in-memory dB, which affects all of our doc samps that use Identity, and the samp is fighting with me 🥊 to get UseInternalServiceProvider to light up).

Notwithstanding those unrelated issues, it sounds like here that the work on this topic+sample should wait until post Pre9. Is that correct? If so, I'll work on those unrelated issues now but I'll sit on my branch until later ... until after this engineering issue is resolved.

[a-vishar]

What is the guidance for this now?

Using .NET Core 3.0, I would like to override/replace services configured in my Startup. Previously I could inherit WebApplicationFactory<Startup> and use the factories CreateClient to get a HttpClient for calling into the test environment.

[niemyjski]

I'm in the same boat...

System.InvalidOperationException : The TestServer constructor was not called with a IWebHostBuilder so IWebHost is not available.
   at Microsoft.AspNetCore.TestHost.TestServer.get_Host()

[Tratcher]

@niemyjski that looks unrelated to overriding services, please open a different issue. TestServer's relationship to the host is a bit different when working with generic host.

[javiercn]

I agree with @Tratcher, this is not the same issue.

[javiercn]

I'm closing this issue as the only remaining thing here is to document the new behavior (which is by design) and how to approach the problem when using generic host; and that is being covered by the docs issue https://github.com/aspnet/AspNetCore.Docs/issues/13353