Aspnetcore: How to cancel starting up a web application that is doing some initial startup logic (like seeding a DB)?

[PureKrome]

👋 Hi!

I'm trying to figure out how I can stop starting up my web application when I'm doing some initial application-startup logic, like seeding a database or loading some data from some URI/DB/etc into memory, etc.

The database seeding is an easy example, but it could be any "dependency" that fails. And for this example, imagine that dependency is unavailable (e.g. DB is starting up because all these services start at the 'same time' with docker-compose up).

In the public void Configure(..) method (in startup.cs) this is a great place to start DB seeding.

So what I do is:

• Seed data.
• if error, retry.
• repeat for 15 times .. which takes a total of about 30 seconds max.
• else .. crash/exception if 15 attempts all failed.

The 'retry' action is using Polly.

In the middle of the Polly retries I hit CTRL-C and the application says "stopping" .. but Polly doesn't know about this and keeps retrying.

I thought the trick was to pass the applicationLifetime.ApplicationStopping CancellationToken down to the logic which calls/retries. Doing this, the token is always false (ie. nothing to cancel .. keep on trucking).

I tried asking this in the Polly repo but we're also stuck .. because we _feel_ like this is more related to the underlying framework or more specifically, what I'm not coding correctly.

So lets look at some code and some pwitty pictures...

public void Configure(IApplicationBuilder app,
                        IHostingEnvironment env,
                        IDocumentStore documentStore,
                        IOptions<CorsSettings> corsSettings,
                        IApplicationLifetime applicationLifetime,   // <-- TAKE NOTE
                        ILogger<Startup> logger)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }

    app.UseProblemDetails()
        .UseCustomCors(corsSettings.Value)

        // OK -> here we go. lets 'UseRavenDb' which will seed some data.
        .UseRavenDb(env.IsDevelopment(), 
                    documentStore, 
                    applicationLifetime.ApplicationStopping,
                    logger)

        .UseAuthentication()
        .UseCustomSwagger("accounts", SwaggerTitle)
        .UseMvc();
}

So first up, lets seed some data to the DB (in this case RavenDB) and pass along the ApplicationStopping token from an IApplicationLifetime which was injected in.

this eventually calls this check to see if the database exists, before i throw data into it ....

DatabaseStatistics existingDatabaseStatistics = null;
try
{
    var checkRavenDbPolicy = CheckRavenDbPolicy(cancellationToken, logger);
    checkRavenDbPolicy.Execute(token =>
    {
        existingDatabaseStatistics = documentStore.Maintenance.Send(new GetStatisticsOperation());
    }, cancellationToken);
}
... <snipped the catches>

The policy is a retry 15x with a 2 second delay between retries. Simple. (oldish code in the image below.)

And this is what it looks like when I hit CTRL-C in the middle of the retry's because I didn't start the DB (on purpose) :

Ok .. application is shutting down....

Ok - so we've paused inside when an exception is thrown and the policy is handling it. Lets check out what the cancellationToken is ...

and it looks like the cancellationToken is saying 'nothing to handle. nothing cancelled. keep on cruising'.

So I'm not sure how to leverage the applicationLifetime.ApplicationStopping property, correctly. Or if that's the correct property to leverage.

[NinoFloris]

Stopping seems to be the correct event to latch onto. Could you try to register a Console.Writeline callback on that cancellationtoken like shown here? http://www.binaryintellect.net/articles/535d4ca7-9726-46cf-bc64-64e94ec55f6e.aspx That may give you some clue if it actually ever fired

[poke]

In the public void Configure(..) method (in startup.cs) this is a great place to start DB seeding.

I disagree. The Configure method is a place to set up the application but not to establish pre-conditions about the application. It is further executed by other tools, like dotnet ef or the WebApplicationFactory, which makes it sub-optimal for preparing stuff for actually launching the application.

My usual recommendation is to seed the database outside of the Startup at the host or web host level. By doing that, you are seeding before you actually start the application which also gives you the benefit of not serving the application early although it is not yet ready.

So by following this approach, you have a lot more control about when the application is actually started and you can use that to properly seed your database _first_ and only start the application once that is successfully completed. And since you are now outside of complex lifetimes, you can also control cancelling a lot easier.

In general, this would look something like this:

```c#
public async Task Main(string[] args)
{
var host = CreateWebHostBuilder(args).Build();

using (var scope = host.Services.CreateScope())
{
    // retrieve the seed service and attempt to seed the database
    var databaseSeeder = scope.ServiceProvider.GetRequiredService<IDatabaseSeeder>();

    if (!await databaseSeeder.TrySeedDatabase())
    {
        Console.WriteLine("Seeding was not successful");
        return;
    }
}

// run the application
await host.RunAsync();

}
```

And since this runs now as a normal console application without anything magic, you can also simply check for the Console.CancelKeyPress to check for Ctrl+K and pass that on to a cancellation token for example that then interrupts your seed process.

[newline-coder]

@poke Really good advice but not sure I would advocate checking for key press.

[poke]

@DAllanCarr I am checking for the cancel key press which is what the ConsoleLifetime in ASP.NET Core does as well when gracefully shutting down.

[PureKrome]

@poke So using your suggestion, this means we setup (but not start) the web app _hosting_ yet because CreateWebHostBuilder(args).Build() [Link is from the Archived GH repo] does call ConfigureServices [i.e. sets up DI] and then does also call Configure [which doesn't mean much, right now].

So now that we have DI setup, we can then pull down our database 'stuff' (i.e. instances in the DI Store) and start seeding.

What your example is missing is connecting applicationLifetime.ApplicationStopping to your sample code.

Lastly, if this is what you're recommending @poke (which is an interesting suggestion) ... can the MS team make a call on what they recommend also (e.g. poke's suggestion is a great way) so this can get documented in the ASPNET.Docs please (CC @Rick-Anderson @guardrex). I feel like this is a pretty common scenario so it could be a valid thing to document _officially_.

[guardrex]

@PureKrome (Just pertaining to the 'seeding' part of the discussion ...) I think they decided to punt that over to the EF doc folks in terms of covering it in our doc set.

https://docs.microsoft.com/ef/core/modeling/data-seeding

... and there's a call to improve that topic at https://github.com/aspnet/EntityFramework.Docs/issues/882.

We have tutorials that use variations of the Program>Main>CreateScope approach. (Example: https://docs.microsoft.com/aspnet/core/data/ef-rp/intro#update-main)

[poke]

@PureKrome Yeah, the application only starts when you call Run(). The Build() will only use the host builder to configure the host but not actually run it. Since the Startup is part of that configuration, it will be run properly but the application will still not launch. So you can use that to delay serving any requests until you are done with preparation.

I am not sure what you would expect the applicationLifetime.ApplicationStopping to do there. The point of this approach is that you are _not_ launching the application yet, so you do not have to check for it to start/stop. You only start it once you are done with your migration.

Of course, if you need to check for application shutdown, you could still do that, although then I would suggest you to use an IHostedService instead. Those will be terminated properly when the application stops (for whatever reason).

@guardrex While the EF docs cover how to seed in general, when to do that in ASP.NET Core would still be an appropriate thing to document. But yeah, that example you linked does cover that properly.

[guardrex]

My recommendation is for them to cover it and for us to link to it. It's hard to keep up with externally moving :cheese:. :smile:

[anurse]

It _looks_ like there isn't actionable work in Hosting to do for this, so I'm putting it in the Discussions category. Let me know if I'm misreading or something does come up.

[PureKrome]

@anurse thanks Andrew - _really appreciate_ you popping into this thread.

When I started looking at what @poke suggested and also specifically the EF Core with Razor Pages tutorial by @guardrex which suggests we do any custom logic (like data seeding) after the host builder has been built (so DI is ready) but _before_ we start/run our host ... I was looking into _how_ the application can handle a CTRL-C, etc. The host can handle this, but it needs to be running.

So then I thought "Ok, lets just manually wire-up handling when CTRL-C is pressed" but that's like making us wire it up 2x.

• manually before we run
• inside the low level Host code which is already provided.

It feels like a waste and disconnect. I tried to see what happens when i just hit CTRL-C while debugging _and_ while i was seeding some data and I didn't really get anything working.

So maybe the:

• Console.CancelKeyPress += (sender, e) =>
• AppDomain.CurrentDomain.ProcessExit += (sender, eventArgs) =>

might get wired up earlier on and not only when the app is WaitForStartAsync / ran?

It _feels_ easier to let things like data-seeding to occur after the app has started / starting up .. but I totally understand the perspective that the app shouldn't be able to receive requests until it's properly ready, included all data-seeded (if that is offered).

Would love to hear other peoples thoughts on when in the pipeline things like data-seeding should occur and of course, handle an unexpected termination of the app during this process.

[anurse]

The data seeding scenario has definitely come up before (https://github.com/aspnet/Hosting/issues/1085).

I believe we came to the conclusion that for data seeding you should use an IHostedService and block the task returned by StartAsync until the seeding completes (and just ignore StopAsync). If you do that, you'll run asynchronously and get to respond to shutdown (via the CancellationToken in StartAsync).

Now, if you register that Hosted Service in Startup.ConfigureServices I believe it will end up after the server itself in the list of hosted services so you could receive requests before you finish seeding data. There are two ways around that:

1. If a request comes through before seeding finishes, respond with some error status code indicating the server isn't actually ready yet. If you have load-balancers or health-check systems (like Docker) in place they can be configured not to route traffic to you until these checks succeed.

2. (3.0 Generic Host Only) Register your "data seeding" hosted service before the server hosted service. In 3.0, when using Generic Host, the server is just another hosted service and is registered in DI during ConfigureWebHost or ConfigureWebHostDefaults. Hosted services are run in order and wait for each StartAsync to complete before calling the next one. So in 3.0 you can have async code block the server from starting by using Generic Host and calling .ConfigureServices on the HostBuilder before the call to .ConfigureWebHostDefaults:

Host.CreateDefaultBuilder(args)
    .ConfigureServices(services => 
    {
        services.AddSingleton<IHostedService, InitializationService>();
    })
    .ConfigureWebHostDefaults(webBuilder =>
    {
        webBuilder.UseStartup<Startup>();
    });

I believe option 2 will work, though I haven't fully tested it 😸 .

[PureKrome]

Oh wow - 24th May 2017 was when this discussion point was first raised by you @anurse in aspnet/Hosting#1085 ! Kewl - this isn't a new problem.

So if things "get better" in 3.0, then I'm totally kewl to wait until that. I've waited a while aready, so I can still wait a bit longer.

So using your example above, this means we should be using Generic Host.

Described (and emphasis, mine):

Generic Host is new in ASP.NET Core 2.1 and isn't suitable for web hosting scenarios. For web hosting scenarios, use the Web Host. Generic Host will replace Web Host in a future release and act as the primary host API in both HTTP and non-HTTP scenarios.

So does this mean in 3.0 that WebHost will be gone/replaced?

Hosted services are run in order and wait for each StartAsync to complete before calling the next one

So in the example above, are you saying:

• First we create our own IHostedService which just does some custom stuff (e.g. InitializationService) like seed data or preload data from a file into memory/redis/whatever, etc..
• Once that finished, we now create a Web-like-Host and off we go, run that which starts accepting requests, etc.

Is that what I'm reading?

[anurse]

So does this mean in 3.0 that WebHost will be gone/replaced?

Yep, in fact if you look at the version of that doc for 3.0, that guidance has been updated:

Starting in ASP.NET Core 3.0, Generic Host is recommended for both HTTP and non-HTTP workloads. An HTTP server implementation, if included, runs as an implementation of IHostedService. IHostedService is an interface that can be used for other workloads as well.

Yeah, I think what you've described is correct. You do need to configure things a little differently in your Program.cs. See https://docs.microsoft.com/en-us/aspnet/core/fundamentals/index?view=aspnetcore-3.0&tabs=windows#the-host for some docs

[anurse]

If you can try this on the latest previews that would really help us understand if this is going to work for your scenario!

[PureKrome]

For you and the team:

I'll try this in the next few days with v3.0.0-preview5 and put it up to GH for review, etc.

(No pressure, PK .... 😅 )

[PureKrome]

Ok - appologies for the late reply .. life/family/kids/kids/kids/kids/kids/kids/kids/life/kids have been pulling at me.

So i sat down tonight and tried to give this a crack and I did come up with something ... but not 100%.

=> Experimental Repo up on GitHub.

So I created a new IHostingService like you suggested. And then in program.cs I've wired it up _before_ the webhost kicks in, like you said.

works great.... until I try to hit CTRL-C during the hardcoded Task.Delay. The ideas were:

• Initialize some code BEFORE the web host kicks in, etc.
• Break out of the initialize code, if we hit CTRL-C. (eg. something is wrong and we need to quit asap).

The first step works. The CTRL-C doesn't break out of the first host's StartAsync method. It's like the cancellation token isn't getting set correctly OR i'm not using the correct cancellation token.

So - we're back to the initial question/issue: How to cancel a web app during some initial startup logic.

(My gut feeling is that the code in the sample GH repo I've provided is really crap and wrong :blush: )

[poke]

The problem here is that the cancellation token that gets passed to the hosted service is the same that gets passed to Host.StartAsync() which is default by default and as such isn’t used.

I’m not sure if this is desired or not, but it seems that interrupting the start phase of hosted services is not supported by default. Or at least not hooked to the actual application lifetime.

[PureKrome]

Thanks @poke for looking into what I did, etc. Now lets see what @anurse thinks. Cheers!! 🍾

[PureKrome]

Polite ping to @anurse (I know you're busy etc ... just don't want this issue to get lost in the noise)

[anurse]

life/family/kids/kids/kids/kids/kids/kids/kids/life/kids have been pulling at me.

Oh I hear that!

It does seem odd that we wouldn't cancel the token passed to IHostedService.StartAsync. Could you try injecting IApplicationLifetime and seeing what the tokens there do when you cancel here?

[poke]

Ctrl+K does stop the IApplicationLifetime which triggers the shutdown behavior, so I would assume that this would be the same behavior then. What remains is that the StartAsync behavior isn’t affected by the application lifetime but will only come into effect _after_ the startup is completed.

In practice that seems perfectly fine since the usual use case is probably not to interrupt/terminate the application during its startup, especially since the web host usually starts rather quickly. However, when looking at startup behaviors that do take a bit longer, which will likely become a more common thing in the future with the generic host and increased usage of IHostedService, cancelling the startup may become a more common use case.

So maybe we should look at whether we could make the StartAsync cancellable by the IApplicationLifetime. Maybe we could come up with a wrapper CTS that is cancelled by either the token passed to IHost.StartAsync or the IApplicationLifetime event which is then passed down to the hosted services.

[anurse]

Yeah, it seems reasonable to me to make a change to have the IHostedService.StartAsync token cancelled in this scenario. Thoughts @davidfowl @Tratcher ?

Any interest in a PR @poke or @PureKrome ?

[Tratcher]

Sounds plausible. @anurse want to move this issue to Extensions?

You'd want to make a linked CTS right about here:
https://github.com/aspnet/Extensions/blob/7105fe76f50728597dcb49dd50f82ba9d3401790/src/Hosting/Hosting/src/Internal/Host.cs#L42

[poke]

@anurse Sure, I can give it a try.

@Tratcher Yeah, that’s where I was looking at too. Do we have any kind of linked CTS already, or should I just create a normal CTS and hook it up with the two source events?

[Tratcher]

No, just the normal CancellationTokenSource.CreateLinkedTokenSource.

[poke]

Good that I asked – TIL CreateLinkedTokenSource 😁

[anurse]

@Tratcher GitHub is not letting me transfer right now :(. I'll try again later.

[anurse]

I won't bother transferring now that it's closed ;P. Thanks @poke!

[PureKrome]

Thank you to everyone participating in this issue and discussion. Lots of respect to you all 🍰