Efcore: Timeout expired

[mayconbeserra]

Timeout expired

From time to time, I get this issue on Production. The timeout lasts 8 hours. To solve the issue, I restarted the containers for the application and it worked fine.

Some fun facts are - we don't have a high load during the night (only our monitoring tools) and after restarting the application everything works.

Is the connection pool per user, database and application instance (cluster mode)?

The max connection pool parameter isn't set in the connection string.

Exception

Exception message: System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
Stack trace:
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.BufferlessMoveNext(DbContext _, Boolean buffer)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()
   at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source)
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ResultEnumerable`1.GetEnumerator()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass15_1`1.<CompileQueryCore>b__0(QueryContext qc)"

Code

Registering DBContext

            services.AddScoped(context =>
            {
                return new DbContextOptionsBuilder<DbContext>()
                    .UseSqlServer
                    (
                        Env.Variables.DB_CONNECTION
                    )
                    .UseLoggerFactory(context.GetService<ILoggerFactory>())
                    .Options;
            });

Further technical details

EF Core version: 2.1
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: Linux Containers

[ajcvickers]

@mayconbeserra Is the app using connection resiliency?

[mayconbeserra]

No, It is not using the retry.

Will that make the difference for the pool size exception?

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Arthur Vickers [e-mail]
Sent: Monday, May 27, 2019 7:57:30 PM
To: aspnet/EntityFrameworkCore
Cc: Maycon Beserra; Mention
Subject: Re: [aspnet/EntityFrameworkCore] Timeout expired (#15814)

@mayconbeserrahttps://github.com/mayconbeserra Is that app using connection resiliencyhttps://docs.microsoft.com/en-us/ef/core/miscellaneous/connection-resiliency?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/aspnet/EntityFrameworkCore/issues/15814?email_source=notifications&email_token=AAG6RBO6IS45ZYMEC7AWVZ3PXQOIVA5CNFSM4HPWLF62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWKJEWQ#issuecomment-496276058, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAG6RBMHTKIZPBFNP7ZF5QTPXQOIVANCNFSM4HPWLF6Q.

[ajcvickers]

@mayconbeserra I'm not 100% sure. @divega? @AndriySvyryd?

[akselarzuman]

We have the same issue as well.

We're using dotnet core 2.2 and all our Microsoft packages are up to date.
Our database is SqlServer 2012.

We have been looking for a solution and could not do much about it. We suspected that the error could be from Entity Framework Core.

Today, we created a dummy API that only has one GET endpoint in which all configurations are made from Microsoft's official documentation. We sent 1000 requests per second and realized that EF Core opens 15 new connections and does not close any of them after all requests are completed. After 20 secs, API stopped giving response also for health check and we got timeout exception for all requests and the app stopped.

We tried this scenario in Docker, Linux instance(CentOS) and Windows instance. All the results are the same.

[divega]

@akselarzuman I suspect the issue you are seeing may be different from @mayconbeserra's. There can be multiple different causes for this exception. Yours actually look similar to https://github.com/dotnet/SqlClient/issues/18, while @mayconbeserra's doesn't occur under load.

We sent 1000 requests per second and realized that EF Core opens 15 new connections and does not close any of them after all requests are completed.

Can you elaborate a bit on how you are observing this behavior?

According to the analysis at https://github.com/dotnet/SqlClient/issues/18#issuecomment-493754228, in that particular case, the application code is causing thread starvation by mixing async calls with blocking code, like task.GetAwaiter().GetResult().

If you think you don't have this issue in your code, we will need a repro, but please create a new issue.

[Xepe]

Actually I have the same issue using EF 2.2.4, I just created an api and when I make 10 rps I got the same exception with CentOS and dotnet 2.2.300

[ajcvickers]

@mayconbeserra If you're setting ConnectRetryCount in your connection string, can you make sure it's set to some value >= 1. (If it's not there, then it defaults to 1, so this only matters if you're currently setting it to zero.)

[ajcvickers]

Note from triage: we are still following up with the SQL Server folks on this.

[mayconbeserra]

Hi, @ajcvickers sorry for the delay. I changed to retry six times and started using DbContextPool.

The performance improved, but I still get the issue from time to time.

I'm looking forward to hearing any news about that

[divega]

@mayconbeserra, @Xepe we re-discussed this issue in triage today, and in order to make progress on it (regardless of whether the root cause is on EF Core, on SqlClient, or elsewhere) we would need a small repro we can run. If you can provide it, that would help a lot.

As has been mentioned in this thread, there are other known causes for this exception, but you both mentioned you can repro:

1. Not under a heavy load
2. On Linux

Also, one of you mentioned Linux containers and the other one CentOS. Could you please confirm whether you are both running on containers and what version of Linux you are using?

[ajcvickers]

EF Team Triage: Closing this issue as the requested additional details have not been provided and we have been unable to reproduce it.

BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.