Aspnetcore: Q: Is it possible to trigger a drainstop & restart on configuration changes

[grahamehorner]

I'm looking for a way to drainstop a kestrel web host and restart the service via a configuration change; is this possible ? how would one inform the kestrel process to stop accepting request without impacting existing requests in process/queue?

[anurse]

We don't have anything built in to wire these up. You can use the ReloadToken property on IConfiguration to get signalled whenever configuration changes and shut the server down yourself though.

I believe you can use the StopApplication (which you can inject from DI) method on IApplicationLifetime to trigger a "graceful" shutdown (where existing requests continue to be processed).
@Tratcher @halter73 am I correct here?

As for restarting, there's nothing built-in to trigger a restart. You have a few options here. You could write code in your Program.Main to automatically restart the server if the shutdown is due to a configuration reload (you'll have to flow this information from the config reload callback yourself). Or you could have a "watchdog" process running that auto-restarts the server when it shuts down.

[Tratcher]

Correct, with a default 5 second graceful shutdown timeout.

That said, config has change detection and some components respond to it. What are you re-configuring?

[halter73]

If you do still need to restart the app to react to config changes, and the default 5 second graceful shutdown timeout is insufficient, you can configure it. https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/web-host?view=aspnetcore-2.2#shutdown-timeout

[grahamehorner]

@anurse @Tratcher @halter73 thanks for the information; in regards to stopping/pausing the acceptance of new request and/or checking the request/response queue so you may prevent termination of the requests in process ?

[Tratcher]

I don't understand how that's different from your original question.

[grahamehorner]

@Tratcher the usecase/flow here is:-

1. a change in configuration is detected
2. inform kestrel not to accept anymore requests
3. check how many request or in process
4. wait until request in process complete
5. restart kestrel

my follow-up question is how can I hook into kestrel to identify how many requests are in-process or have been queued? and/or how long a request as been in-process/queued?

I'm not sure of what the shutdown timeout would be? as all queued request must be processed prior to shutdown/restart.

[grahamehorner]

@Tratcher with regards to what is getting re-configured this could be anything from digital certificates to database connection string and anything in-between and therefore we opted for a process where a configuration change from a given source would restart the main webhost task.

[Tratcher]

inform kestrel not to accept anymore requests

StopApplication handles this.

check how many request or in process

That's not something you need to worry about, Kestrel takes care of it.

wait until request in process complete

Kestrel will do this, but it will abort requests after a given timeout (5 seconds by default). You can raise that timeout as needed, but I don't suggest disabling it. You wouldn't want a slow request to prevent your site from restarting indefinitely.
https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/web-host?view=aspnetcore-2.2#shutdown-timeout

[grahamehorner]

@Tratcher hmmm, we don't what a restart to terminate any request; if a request is stuck or slow to complete, then manual intervention is required or we need some way of capturing these prior to killing the request and restarting. Q: is there any logging with regard to kestrel when a StopApplication is received ie. the status of the request queue time shutdown received etc. ?

[Tratcher]

At this point I suggest trying StopApplication to see how well it works for you.

[grahamehorner]

@Tratcher Yes, thanks we are looking at this at present and will update soon with regards to our usecase

[grahamehorner]

@Tratcher I've got a simple application listening for configuration changes and calling the StopApplication as discussed, however the configuration change event is fired many times on a single simple change? which seems odd? are you aware of any issues around IChangeToken events ?

simple log with a count outputs :=

WebApplicationWithRestart.Startup:Information: configuration change detected 1
WebApplicationWithRestart.Startup:Information: application stopping.
WebApplicationWithRestart.Startup:Information: configuration change detected 2
WebApplicationWithRestart.Startup:Information: configuration change detected 3
WebApplicationWithRestart.Startup:Information: configuration change detected 4
WebApplicationWithRestart.Startup:Information: configuration change detected 5
WebApplicationWithRestart.Startup:Information: configuration change detected 6
WebApplicationWithRestart.Startup:Information: configuration change detected 7
WebApplicationWithRestart.Startup:Information: configuration change detected 8
WebApplicationWithRestart.Startup:Information: configuration change detected 9
WebApplicationWithRestart.Startup:Information: configuration change detected 10
WebApplicationWithRestart.Startup:Information: configuration change detected 11
WebApplicationWithRestart.Startup:Information: configuration change detected 12
WebApplicationWithRestart.Startup:Information: configuration change detected 13
WebApplicationWithRestart.Startup:Information: configuration change detected 14
WebApplicationWithRestart.Startup:Information: configuration change detected 15
WebApplicationWithRestart.Startup:Information: configuration change detected 16
WebApplicationWithRestart.Startup:Information: configuration change detected 17
WebApplicationWithRestart.Startup:Information: configuration change detected 18
WebApplicationWithRestart.Startup:Information: configuration change detected 19
WebApplicationWithRestart.Startup:Information: configuration change detected 20
WebApplicationWithRestart.Startup:Information: configuration change detected 21
WebApplicationWithRestart.Startup:Information: configuration change detected 22
WebApplicationWithRestart.Startup:Information: configuration change detected 23
WebApplicationWithRestart.Startup:Information: configuration change detected 24
WebApplicationWithRestart.Startup:Information: configuration change detected 25
WebApplicationWithRestart.Startup:Information: configuration change detected 26
WebApplicationWithRestart.Startup:Information: configuration change detected 27
WebApplicationWithRestart.Startup:Information: configuration change detected 28
WebApplicationWithRestart.Startup:Information: configuration change detected 29
WebApplicationWithRestart.Startup:Information: configuration change detected 30
WebApplicationWithRestart.Startup:Information: configuration change detected 31
WebApplicationWithRestart.Startup:Information: configuration change detected 32
WebApplicationWithRestart.Startup:Information: configuration change detected 33
WebApplicationWithRestart.Startup:Information: configuration change detected 34
WebApplicationWithRestart.Startup:Information: configuration change detected 35
WebApplicationWithRestart.Startup:Information: configuration change detected 36
WebApplicationWithRestart.Startup:Information: configuration change detected 37
WebApplicationWithRestart.Startup:Information: configuration change detected 38
WebApplicationWithRestart.Startup:Information: configuration change detected 39
WebApplicationWithRestart.Startup:Information: configuration change detected 40
WebApplicationWithRestart.Startup:Information: configuration change detected 41
WebApplicationWithRestart.Startup:Information: configuration change detected 42
WebApplicationWithRestart.Startup:Information: configuration change detected 43
WebApplicationWithRestart.Startup:Information: configuration change detected 44
WebApplicationWithRestart.Startup:Information: configuration change detected 45
WebApplicationWithRestart.Startup:Information: configuration change detected 46
WebApplicationWithRestart.Startup:Information: configuration change detected 47
WebApplicationWithRestart.Startup:Information: configuration change detected 48
WebApplicationWithRestart.Startup:Information: configuration change detected 49
WebApplicationWithRestart.Startup:Information: application stopped.
WebApplicationWithRestart.Startup:Information: configuration change detected 50
WebApplicationWithRestart.Startup:Information: configuration change detected 51
WebApplicationWithRestart.Startup:Information: configuration change detected 52
WebApplicationWithRestart.Startup:Information: configuration change detected 53
WebApplicationWithRestart.Startup:Information: configuration change detected 54
WebApplicationWithRestart.Startup:Information: configuration change detected 55
WebApplicationWithRestart.Startup:Information: configuration change detected 56
WebApplicationWithRestart.Startup:Information: configuration change detected 57
WebApplicationWithRestart.Startup:Information: configuration change detected 58
WebApplicationWithRestart.Startup:Information: configuration change detected 59
WebApplicationWithRestart.Startup:Information: configuration change detected 60
WebApplicationWithRestart.Startup:Information: configuration change detected 61
WebApplicationWithRestart.Startup:Information: configuration change detected 62
WebApplicationWithRestart.Startup:Information: configuration change detected 63
WebApplicationWithRestart.Startup:Information: configuration change detected 64
WebApplicationWithRestart.Startup:Information: configuration change detected 65
WebApplicationWithRestart.Startup:Information: configuration change detected 66
WebApplicationWithRestart.Startup:Information: configuration change detected 67
WebApplicationWithRestart.Startup:Information: configuration change detected 68
WebApplicationWithRestart.Startup:Information: configuration change detected 69
WebApplicationWithRestart.Startup:Information: configuration change detected 70
WebApplicationWithRestart.Startup:Information: configuration change detected 71
WebApplicationWithRestart.Startup:Information: configuration change detected 72
WebApplicationWithRestart.Startup:Information: configuration change detected 73
WebApplicationWithRestart.Startup:Information: configuration change detected 74
WebApplicationWithRestart.Startup:Information: configuration change detected 75
WebApplicationWithRestart.Startup:Information: configuration change detected 76
WebApplicationWithRestart.Startup:Information: configuration change detected 77
WebApplicationWithRestart.Startup:Information: configuration change detected 78
WebApplicationWithRestart.Startup:Information: configuration change detected 79
WebApplicationWithRestart.Startup:Information: configuration change detected 80
WebApplicationWithRestart.Startup:Information: configuration change detected 81
WebApplicationWithRestart.Startup:Information: configuration change detected 82
WebApplicationWithRestart.Startup:Information: configuration change detected 83
WebApplicationWithRestart.Startup:Information: configuration change detected 84
WebApplicationWithRestart.Startup:Information: configuration change detected 85
WebApplicationWithRestart.Startup:Information: configuration change detected 86
WebApplicationWithRestart.Startup:Information: configuration change detected 87
WebApplicationWithRestart.Startup:Information: configuration change detected 88

[Tratcher]

That's a lot of changes 😁. What config source actually changed? A file?

@HaoK ?

[grahamehorner]

@Tratcher I changed the appsettings.json logging level from within VS 2019 when running the code in debug

[HaoK]

Yeah the config change stuff is super noisy unfortunately, you can get notified many times for one actual file change, its due to using change tokens directly, we don't have any higher level concept that represents a 'file was updated' instead you get a bunch of file watcher notifications for changes that might not be the entire operation

[grahamehorner]

@HaoK is there any way to dial down these events via configuration or are these fix in code, also I'm wondering if its possible to only receive the event if a given section changes as opposed to a file/set of files loaded by the configuration builder ?

[HaoK]

Not really, the best thing to do to better control things might be to turn off the built in reload on change and manually watch for file changes and call reload on the config root yourself

[anurse]

is there any way to dial down these events via configuration

@HaoK can correct me if I'm wrong but I believe the chattiness here originates in the OS and how the app you use to edit the file writes to it. There really isn't much we can do. You could definitely add some logic to "debounce" the notifications though (i.e. wait a few seconds for the notifications to settle).

[HaoK]

Correct, these notifications are coming from the file system abstractions directly, there's nothing we can really do at the config layer for this https://github.com/aspnet/Extensions/blob/master/src/FileProviders/Abstractions/src/IFileProvider.cs#L32

[HaoK]

We already added some throttling in the reload logic to prevent spamming reloads, but it doesn't look that that helps a ton:

https://github.com/aspnet/Extensions/blob/master/src/Configuration/Config.FileExtensions/src/FileConfigurationProvider.cs#L37

[msftbot[bot]]

Thank you for contacting us. Due to a lack of activity on this discussion issue we're closing it in an effort to keep our backlog clean. If you believe there is a concern related to the ASP.NET Core framework, which hasn't been addressed yet, please file a new issue.

This issue will be locked after 30 more days of inactivity. If you still wish to discuss this subject after then, please create a new issue!