Pomelo.entityframeworkcore.mysql: Master Connection String sometimes sets Database=''

[bakester14]

Steps to reproduce

Unfortunately my boss won't let me spend any more time on this, so I'm not sure what the steps to reproduce actually are.

The issue

The connection string used for the Master Connection is initialized in such a way that in some conditions the connection string reads like this:

Server=localhost;Database=;Username=root;

This causes connection errors when running commands like myContext.Database.Migrate(). I wish I had more info on when this can occur but I've been struggling to repro it for over a day now and my boss is asking me to move on. I'll provide more info about my specific use case below, but I couldn't get this to repro outside of proprietary code.

An error occurred using the connection to database '' on server 'qi-primary-db.cluster-ccg4yb3eyk72.us-west-2.rds.amazonaws.com'.
MySql.Data.MySqlClient.MySqlException (0x80004005): Connect Timeout expired. ---> System.ObjectDisposedException: Safe handle has been closed
   at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)
   at System.StubHelpers.StubHelpers.SafeHandleAddRef(SafeHandle pHandle, Boolean& success)
   at Interop.Sys.TryChangeSocketEventRegistration(IntPtr port, SafeHandle socket, SocketEvents currentEvents, SocketEvents newEvents, IntPtr data)
   at System.Net.Sockets.SocketAsyncContext.Register()
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.StartAsyncOperation(SocketAsyncContext context, TOperation operation, Int32 observedSequenceNumber)
   at System.Net.Sockets.SocketAsyncContext.PerformSyncOperation[TOperation](OperationQueue`1& queue, TOperation operation, Int32 timeout, Int32 observedSequenceNumber)
   at System.Net.Sockets.SocketAsyncContext.Connect(Byte[] socketAddress, Int32 socketAddressLen)
   at System.Net.Sockets.SocketPal.Connect(SafeCloseSocket handle, Byte[] socketAddress, Int32 socketAddressLen)
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.Net.Sockets.TcpClient.Connect(IPEndPoint remoteEP)
   at System.Net.Sockets.TcpClient.Connect(IPAddress address, Int32 port)
   at MySqlConnector.Core.ServerSession.OpenTcpSocketAsync(ConnectionSettings cs, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 775
   at MySqlConnector.Core.ServerSession.OpenTcpSocketAsync(ConnectionSettings cs, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 785
   at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 274
   at MySql.Data.MySqlClient.MySqlConnection.CreateSessionAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 518
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at MySql.Data.MySqlClient.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 232
   at MySql.Data.MySqlClient.MySqlConnection.Open() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 218
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)
warn: Hangfire.AutomaticRetryAttribute[0]
      Failed to process the job '362fe1e8-7062-4833-b295-3053f3619c9f': an exception occurred. Retry attempt 1 of 10 will be performed in 00:00:36.
MySql.Data.MySqlClient.MySqlException (0x80004005): Connect Timeout expired. ---> System.ObjectDisposedException: Safe handle has been closed
   at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)
   at System.StubHelpers.StubHelpers.SafeHandleAddRef(SafeHandle pHandle, Boolean& success)
   at Interop.Sys.TryChangeSocketEventRegistration(IntPtr port, SafeHandle socket, SocketEvents currentEvents, SocketEvents newEvents, IntPtr data)
   at System.Net.Sockets.SocketAsyncContext.Register()
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.StartAsyncOperation(SocketAsyncContext context, TOperation operation, Int32 observedSequenceNumber)
   at System.Net.Sockets.SocketAsyncContext.PerformSyncOperation[TOperation](OperationQueue`1& queue, TOperation operation, Int32 timeout, Int32 observedSequenceNumber)
   at System.Net.Sockets.SocketAsyncContext.Connect(Byte[] socketAddress, Int32 socketAddressLen)
   at System.Net.Sockets.SocketPal.Connect(SafeCloseSocket handle, Byte[] socketAddress, Int32 socketAddressLen)
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.Net.Sockets.TcpClient.Connect(IPEndPoint remoteEP)
   at System.Net.Sockets.TcpClient.Connect(IPAddress address, Int32 port)
   at MySqlConnector.Core.ServerSession.OpenTcpSocketAsync(ConnectionSettings cs, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 775
   at MySqlConnector.Core.ServerSession.OpenTcpSocketAsync(ConnectionSettings cs, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 785
   at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 274
   at MySql.Data.MySqlClient.MySqlConnection.CreateSessionAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 518
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at MySql.Data.MySqlClient.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 232
   at MySql.Data.MySqlClient.MySqlConnection.Open() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 218
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
   at Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlDatabaseCreator.<>c__DisplayClass18_0.<Exists>b__0(DateTime giveUp)
   at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.<>c__DisplayClass12_0`2.<Execute>b__0(DbContext c, TState s)
   at Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
   at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.Execute[TState,TResult](IExecutionStrategy strategy, Func`2 operation, Func`2 verifySucceeded, TState state)
   at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.Execute[TState,TResult](IExecutionStrategy strategy, TState state, Func`2 operation)
   at Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlDatabaseCreator.Exists(Boolean retryOnNotExists)
   at Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlDatabaseCreator.Exists()
   at Microsoft.EntityFrameworkCore.Migrations.HistoryRepository.Exists()
   at Microsoft.EntityFrameworkCore.Migrations.Internal.Migrator.Migrate(String targetMigration)
   at Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.Migrate(DatabaseFacade databaseFacade)
. . .

Further technical details

MySQL version: 8.0.1.15
Operating system: Windows 10 (latest update as of writing)
Pomelo.EntityFrameworkCore.MySql version: 2.2.0
Microsoft.AspNetCore.App version: 2.2.104

Other details about my project setup:

I use a scheduled task via Hangfire (an open source task scheduler) to initialize the DB context and run a migration with that context. My Startup.cs file looks _something_ like this:

public class Startup
    {
        private readonly IConfiguration Config;
        private readonly IHostingEnvironment HostingEnvironment;

        public Startup( IConfiguration config, IHostingEnvironment hostingEnvironment )
        {
            this.Config = config;
            this.HostingEnvironment = hostingEnvironment;
        }

        // This method gets called by the runtime. Use this method to add services to the container.
        // For more information on how to configure your application, visit https://go.microsoft.com/fwlink/?LinkID=398940
        public void ConfigureServices( IServiceCollection services )
        {
            services.AddDbContextPool<MyContext>(
                options => options
                    .UseMySql( this.Config.GetConnectionString( "MyContext" ) )
                    .EnableSensitiveDataLogging(this.HostingEnvironment.IsDevelopment())
            );

            services.AddMemoryCache();
            services.AddHangfire( x => x.UseMemoryStorage() );
            services.AddMvc();
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure( IApplicationBuilder app, IHostingEnvironment env, IServiceProvider serviceProvider )
        {
            if ( env.IsDevelopment() )
            {
                app.UseDeveloperExceptionPage();
            }

            app.UseHangfireServer();
            app.UseHangfireDashboard();
            app.UseMvc();

            GlobalConfiguration.Configuration.UseActivator( new MyActivator() );

            BackgroundJob.Enqueue<MyTask>( task => task.Execute() );
        }
    }

This _doesn't occur when I initialize my config variables from file_, but it _does_ occur when I pull from my environment variables.

[bakester14]

Oh I meant to add: I think the issue is here: https://github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql/blob/c58e00c7b3b6bc8f2eeffebd4d10f2e1ff16b790/src/EFCore.MySql/Storage/Internal/MySqlRelationalConnection.cs#L30

We should be setting Database to null here, or refactoring to use the Remove method of the underlying MySqlConnectionStringBuilder.

[mguinness]

This doesn't occur when I initialize my config variables from file, but it does occur when I pull from my environment variables.

Would seem to be more of a configuration issue if that's the case. See Why isn't my ASP.NET Core environment-specific configuration loading?

[bakester14]

To be clear - the connection string loads just fine via env variable and I can perform read/write operations with that connection string. The issue occurs when I try to perform an operation that requires a master connection, such as Migrate - for whatever reason the connection string builder ends up returning a master connection string that is bugged.

[mguinness]

In your DbContext is the OnConfiguring method overridden?

[bakester14]

No, OnModelCreating is though.

[thangchung]

@bakester14 Have you resolved or worked around on it? I have got the same issue when I tried to do a migration :( That should not set empty string to the master database actually

[caleblloyd]

Looking at MySqlConnector:

https://github.com/mysql-net/MySqlConnector/blob/14ac31ed2200ba67378ffe9e766c08ae6dbce1d0/src/MySqlConnector/Protocol/Payloads/HandshakeResponse41Payload.cs#L24

It should not matter if the Database is set to null or "" because the connect with database flag is only set if !string.IsNullOrWhiteSpace(database)

I see you are using RDS. Maybe this is a bug in RDS implementation of the protocol?

@bgrainger any ideas?

[caleblloyd]

It seems that the error message is misleading. The error message comes from Entity Framework here, it is logging the database as ''

https://github.com/aspnet/EntityFrameworkCore/blob/6dfb3dbdb249adcae0941561a66cc94876400f37/src/EFCore.Relational/Properties/RelationalStrings.resx#L177

The real problem that you have is Connect Timeout expired however. A connection to the database was never made.

You probably need to review Security groups as your app server is not able to communicate with your database

[bakester14]

I think you have misread my post.

Please read again and note that I have no trouble connecting, reading/writing to the database - only to when running .Migrate() or something similar. I could also run the migrations just fine via command line, as in dotnet ef migrate. I'm not familiar with the internals there but I believe it would use the same master connection, but initialized a different way.

I really believe the problem is exactly where I pointed it out. I wish I could spend more time on this to prove it but at this point I've moved on from MySql and this framework and don't really have an incentive to go back.

[qkhanhpro]

We run into the same issue connecting using Pomelo to AWS RDS Today (It worked perfectly and today it just fail without any change ) . Specifically Aurora Serverless RDS ( mySQL 5.6.10a )
Swapping Pomelo out for mysql connector and it works fine.
I will gather more information and update here in case someone is interested

[majidazn]

Could you please explain your solution? @qkhanhpro

[lauxjpn]

@majidazn What is the issue you are experiencing (include version information)?

[majidazn]

@lauxjpn I have .netCore api application and it installed on some linux server and running with kestrel. In some of these servers it stop with "connect timeout expired" error

MySQL version: MariaDb 10, 0, 29
Operating system: Linux
Pomelo.EntityFrameworkCore.MySql version: 2.2.0
Microsoft.AspNetCore.App version: 2.2

[lauxjpn]

@majidazn First of all, MariaDB 10.0.29 is not supported anymore. Neither by MariaDB itself, nor by Pomelo (see Compatibility: Supported DBMS and Versions).

Also, Pomelo version 2.2.0 is quite old. Please consider upgrading to Pomelo 3.1.1. It is compatible with .NET Core 2.0+ (using EF Core 3.1.x). See Compatibility: EF Core and .NET Standard for further information.

In your concrete case, we need more information:

• Is the MariaDB server hosted on the same server as the website, or hosted on another server?
• Can you successfully log into the MariaDB server using the mysql.exe client (from the sever the website is running on)?
• How does your connection string look like (replace sensitive information like real IP addresses, username and passwords).
• What is the exact exception (including stack trace), you are getting?