Identityserver4: How to prevent logging for EnableTokenCleanup

[akiander]

Question

Has anyone run into an issue where TokenCleanupHost.StartInternalAsync causes thread starvation?

Minimal working example

I have a token server running the latest version of IdentityServer4 on .net core 3.1 and using EntityFramework for the operational store only (codes, tokens, consents).

The configuration for that looks like this:

                   .AddOperationalStore(options => {
                    options.ConfigureDbContext = builder =>
                        builder.UseSqlServer(ssoConnection,
                            sqlOptions => {
                                sqlOptions.MigrationsAssembly(migrationsAssembly);
                                sqlOptions.EnableRetryOnFailure();
                            });
                    // this enables automatic token cleanup. this is optional.
                    options.EnableTokenCleanup = true;
                    options.TokenCleanupInterval = 30;
                })

Relevant debugging information

Over time, we can watch the thread count to rise until the application crashes due to thread starvation.

When we take a memory dump from the thread-starved process, we see hundreds of instances of threads waiting on Application Insights logging. We think they are rooted by the calls to TokenCleanupHost.StartInternalAsync although truthfully I am not enough of an expert to say for sure.

here's the tasks view of the memory dump

and here's the threading view showing the App Insights logging threads apparently waiting

My guess is that we have App Insights logging configured incorrectly but even stripping it down to what I think matches guidance, I still see the thread starvation issue.

Wondering if anyone else has run into this or has advice.

[brockallen]

Bizarre. I'd almost ask them (meaning AppInsights or MSFT) what's going on? The background cleanup is only working on a single (logical) thread, so I don't why app insights is doing their own? I'd suggest removing app insights to see if the problem persists (since you can repo it). That'd narrow down whose issue it is :)

[akiander]

I removed Application Insights so I'll keep an eye on it for the next few days.

[akiander]

I received some help from Microsoft and it sounds like the real issue here is multiple calls to BuildServiceProvider in startup and some need for better object disposal. I will close this out once I confirm.

[akiander]

TokenCleanupHost.StartInternalAsync was not the cause for this issue. It was just a casualty. This was corrected by eliminating the per-request calls to BuildServiceProvider() in the Startup.cs. Apologies for the distraction.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.