Aspnetcore.docs: Use of .ConfigureServices() versus .ConfigureTestServices() in the integration tests

[lnaie]

It would be great if the use of these 2 helpers will be thoroughly clarified.
My understanding is that the latter is used to replace services while the former to add more services. Both are run after Startup has been dealt with.

Following that logic, in the example below, I see the following code added in the .ConfigureServices() when it feels that it belongs to the .ConfigureTestServices() setup:
var serviceProvider = new ServiceCollection() .AddEntityFrameworkInMemoryDatabase() .BuildServiceProvider(); services.AddDbContext<ApplicationDbContext>(options => { options.UseInMemoryDatabase("InMemoryDbForTesting"); options.UseInternalServiceProvider(serviceProvider); });
The original sample:
````
public class CustomWebApplicationFactory
: WebApplicationFactory
{
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
builder.ConfigureServices(services =>
{
// Create a new service provider.
var serviceProvider = new ServiceCollection()
.AddEntityFrameworkInMemoryDatabase()
.BuildServiceProvider();

        // Add a database context (ApplicationDbContext) using an in-memory 
        // database for testing.
        services.AddDbContext<ApplicationDbContext>(options => 
        {
            options.UseInMemoryDatabase("InMemoryDbForTesting");
            options.UseInternalServiceProvider(serviceProvider);
        });

        // Build the service provider.
        var sp = services.BuildServiceProvider();

        // Create a scope to obtain a reference to the database
        // context (ApplicationDbContext).
        using (var scope = sp.CreateScope())
        {
            var scopedServices = scope.ServiceProvider;
            var db = scopedServices.GetRequiredService<ApplicationDbContext>();
            var logger = scopedServices
                .GetRequiredService<ILogger<CustomWebApplicationFactory<TStartup>>>();

            // Ensure the database is created.
            db.Database.EnsureCreated();

            try
            {
                // Seed the database with test data.
                Utilities.InitializeDbForTests(db);
            }
            catch (Exception ex)
            {
                logger.LogError(ex, $"An error occurred seeding the " +
                    "database with test messages. Error: {ex.Message}");
            }
        }
    });
}

}
````
Could someone explain why DbContext & Co are exceptions?

Thanks

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 77889d43-fedb-f8b4-8fea-379ec946555c
• Version Independent ID: f499b131-8b14-7e10-1728-543fdd5a5656
• Content: Integration tests in ASP.NET Core
• Content Source: aspnetcore/test/integration-tests.md
• Product: aspnet-core
• GitHub Login: @guardrex
• Microsoft Alias: riande

[guardrex]

@javiercn

[guardrex]

@lnaie Let's wait on this until the new year. Engineering has been 🏃🏃🏃🏃😅, and Javier might be OOF for a bit. I've assigned myself so that I don't lose track of this. We'll flesh the concepts out when he gets back and make some improvements to the coverage. Thanks for opening the issue. :+1:

[lnaie]

nice. thanks!

[andmos]

Haha spent some hours wrapping my head around why my Startup's ConfigureServices got called _after_ the CustomWebApplicationFactory setup had run. Turned out to be ConfigureTestServices I was after... As @lnaie says, would be great if the doc clarified this.
Found this issue that helped me out: https://github.com/aspnet/Mvc/issues/8262

[guardrex]

Thanks for that cross-reference, @andmos. That will be helpful if they decide to work this issue.

They told me to hold off on working on issues pertaining to Integration Testing until engineering triages them. They're on the Integration Testing project, including this one. As soon as I hear back from them, I'll work the issues into the schedule.

[ajtowf]

The docs is clearly missleading, quite frankly wrong in this case (https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-2.2), you definitely want to replace the db config/seed, which is done from ConfigureTestServices.

What happens otherwise is that ConfigureServices in the CustomWebAppFactory fires first, where you setup an inmemory db or whatever, then ConfigureServices in your Startup fires which replaces your test config with local dev settings.

Just got tricked by following the docs :-)

[guardrex]

@ajtowf It's more nuanced than that. It depends on your approach to "seeding." The SUT (when it runs outside of tests) seeds in Program.Main, and that code doesn't run under testing, so it doesn't replace/modify the seeding performed by initialization code in the tests.

Don't get me wrong there tho: I agree with your remarks on ConfigureTestServices ... it's covered ... but it needs better coverage. It probably could use a clear, explicit remark about ordering. It probably would be better with an additional example ... something like configuration. We've collected quite a bit of feedback on the project (https://github.com/aspnet/AspNetCore.Docs/projects/31), so the engineers will be able to see what devs are asking about.

Just curious ... How are you seeding? ... There are a few different ways that it can be done.

[VictorioBerra]

PostConfigureServices would have been a much better name I think.

[InDieTasten]

This is a HUGE issue. Luckily, I was only trying to create a sample project for this. The current documentation will NOT use in-memory, and when the original app has a valid SqlServer provider configured, you will easily wipe your sql database trying to setup test data.

I've spent hours now trying to find the issue on my end, when I realised, that the sample app provided in the docs is using in-memory also for the application. That's why the tests don't break in the sample.

As soon as you put a real db behind that, and Assert.True(dbContext.Database.IsInMemory()) to your CustomWebApplicationFactory, all tests fail, because it will in fact still use the real DB when you only meant to use in-memory for your integration tests.

So I would consider this not only an enhancement. This causes problems for anyone brave enough to do integration tests today using any real application (using a db, and not in-memory for normal execution, unlike the sample given).

The entire section about "Customize WebApplicationFactory" with the given example is currently wrong and might lead readers to mutate a database unknowingly during an integration test run, because the documentation made them believe a temporary in-memory db would be used.

[pedropablo]

This is a HUGE issue. Luckily, I was only trying to create a sample project for this. The current documentation will NOT use in-memory, and when the original app has a valid SqlServer provider configured, you will easily wipe your sql database trying to setup test data.

I've spent hours now trying to find the issue on my end, when I realised, that the sample app provided in the docs is using in-memory also for the application. That's why the tests don't break in the sample.

As soon as you put a real db behind that, and Assert.True(dbContext.Database.IsInMemory()) to your CustomWebApplicationFactory, all tests fail, because it will in fact still use the real DB when you only meant to use in-memory for your integration tests.

So I would consider this not only an enhancement. This causes problems for anyone brave enough to do integration tests today using any real application (using a db, and not in-memory for normal execution, unlike the sample given).

The entire section about "Customize WebApplicationFactory" with the given example is currently wrong and might lead readers to mutate a database unknowingly during an integration test run, because the documentation made them believe a temporary in-memory db would be used.

This happened to me! Also discussed this on another issue about the documentation of that section. I removed and added back those services to actually use the InMemoryDB. Now I'm struggling with Auth Middleware and logging a user without writing more code than my whole project has so far :D

[guardrex]

I think that this has been resolved by the latest updates to the topic+sample, especially https://github.com/aspnet/AspNetCore.Docs/pull/15096, which addressed the order of service registrations to make sure that the tests don't use the dB context set up by the app. There have been a few more updates, including the most recent auth + environment updates on https://github.com/aspnet/AspNetCore.Docs/pull/15340.

Since a lot of time has passed with updates, let's close this. If you hit continued problems with the doc, please open a new issue using the This page feedback button/form at the bottom of the integration testing topic.