Sqlclient: System.IO.IOException: Unable to write data to the transport connection: Broken pipe.

[kieronlanning]

When running a .NET Core 2.1 app in Docker, after a few minutes, the app exits with the following exception:

identity-api         |    Unhandled Exception: System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.) ---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe
identity-api         |    at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
identity-api         |    --- End of inner exception stack trace ---
identity-api         |    at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
identity-api         |    at System.Data.SqlClient.SNI.SslOverTdsStream.WriteInternal(Byte[] buffer, Int32 offset, Int32 count, CancellationToken token, Boolean async)
identity-api         |    --- End of inner exception stack trace ---
identity-api         |    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
identity-api         |    at System.Threading.Tasks.Task.Wait()
identity-api         |    at System.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count)
identity-api         |    at System.Net.Security.SslStreamInternal.WriteSingleChunk[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
identity-api         |    at System.Net.Security.SslStreamInternal.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
identity-api         |    at System.Net.Security.SslStreamInternal.Write(Byte[] buffer, Int32 offset, Int32 count)
identity-api         |    at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
identity-api         |    at System.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet)
identity-api         |    at System.Data.SqlClient.SNI.SNIMarsConnection.Send(SNIPacket packet)
identity-api         |    at System.Data.SqlClient.SNI.SNIMarsHandle.SendControlPacket(SNISMUXFlags flags)
identity-api         |    at System.Data.SqlClient.SNI.SNIMarsHandle.Dispose()
identity-api         |    at System.Data.SqlClient.SNI.TdsParserStateObjectManaged.Dispose()
identity-api         |    at System.Data.SqlClient.TdsParserSessionPool.Dispose()
identity-api         |    at System.Data.SqlClient.TdsParser.Disconnect()
identity-api         |    at System.Data.SqlClient.SqlInternalConnectionTds.Dispose()
identity-api         |    at System.Data.ProviderBase.DbConnectionPool.DestroyObject(DbConnectionInternal obj)
identity-api         |    at System.Data.ProviderBase.DbConnectionPool.CleanupCallback(Object state)
identity-api         |    at System.Threading.TimerQueueTimer.CallCallback()
identity-api         |    at System.Threading.TimerQueueTimer.Fire()
identity-api         |    at System.Threading.TimerQueue.FireNextTimers()

There are two containers that access SQL, they both exit around the same time.

This happens consistently. I've tried Alpine and non-Alpine images. The apps are both MVC and the DataContexts have been unused other than running the migrations.

[tmds]

From the stacktrace it seems the issue occurs when the connection to the sql server is removed from the pool.
The IOException is thrown because the server has already closed the connection. Because no-one catches it, it crashes your app.

[kieronlanning]

@tmds What I don't understand I guess is, well, why.

I'm not explicity catching it because I don't explicity do anything with it.

In-fact, it's an MVC app that catches and logs pretty much everything to prevent downtime and bad requests. But this, only in Docker, takes the app out completey.

[tmds]

Your not supposed to catch this one, corefx is.

[SeymourRu]

Same here!
2.0 runs without error
2.1 runs with error

[AfsanehR]

@kieronlanning Would you be able to provide a repro?

[kieronlanning]

@AfsanehR I'm afraid not. We were in the dark for nearly 5 months before this was picked up by the .NET Team a few days ago, in that time we had to re-engineer (and re-host) the solution because we couldn't get past that problem.

[Hartdrooz]

Same issue here, running .Net Core 2.2 on Linux Container and this happen frequently, we are using SQL Server Azure don't know if this is a factor. Any update on this ?

[stephentoub]

@AfsanehR, this seems fairly problematic if it's actually happening frequently for multiple apps. I realize it could be challenging to address without a repro, but I've moved it from Future to 3.0 for now so that it doesn't slip through the cracks.
cc: @karelz

[karelz]

cc @David-Engel @tarikulsabbir @Gary-Zh

[tmds]

The stack trace is saying a lot:

A timeout occurs:

identity-api         |    at System.Threading.TimerQueue.FireNextTimers()

It cleans up a connection from the connection pool:

identity-api         |    at System.Data.ProviderBase.DbConnectionPool.CleanupCallback(Object state)

The connection gets disposed:

identity-api         |    at System.Data.SqlClient.SqlInternalConnectionTds.Dispose()

A control packet is sent through the connection:

identity-api         |    at System.Data.SqlClient.SNI.SNIMarsHandle.SendControlPacket(SNISMUXFlags flags)

But the connection was already closed:

identity-api         |    Unhandled Exception: System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.) ---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe

[TerryAspen]

I'm also encountering this exact same error including the stack trace running .Net Core 2.2.
Linux docker container accessing Azure SQL server.

Has there been any resolution to this?

[DenKn]

I have similar problem. _Unable to read data from the transport connection: Broken pipe._
But I don't use SQL server.
This error sometimes happens on console app (.net core 2.2) which works with server app (.net core 2.2, run as docker container in Azure on Linux).

[mifurm]

Same in here. Can give repo if required by all has been said already!

[DaEkstrim]

I don't know if my stack trace will help. I'm running a DotNet Core 2.1.2 app in Linux docker container on Kubernetes accessing Azure SQL. If this is noise, feel free to delete my comment.

System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.) ---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Data.SqlClient.SNI.SslOverTdsStream.WriteInternal(Byte[] buffer, Int32 offset, Int32 count, CancellationToken token, Boolean async)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslStreamInternal.WriteSingleChunk[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStreamInternal.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStreamInternal.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet)
   at System.Data.SqlClient.SNI.SNIMarsConnection.Send(SNIPacket packet)
   at System.Data.SqlClient.SNI.SNIMarsHandle.SendControlPacket(SNISMUXFlags flags)
   at System.Data.SqlClient.SNI.SNIMarsHandle.Dispose()
   at System.Data.SqlClient.SNI.TdsParserStateObjectManaged.Dispose()
   at System.Data.SqlClient.TdsParserSessionPool.Dispose()
   at System.Data.SqlClient.TdsParser.Disconnect()
   at System.Data.SqlClient.SqlInternalConnectionTds.Dispose()
   at System.Data.ProviderBase.DbConnectionPool.DestroyObject(DbConnectionInternal obj)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.OpenAsync(CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at Geode.Services.Cache.SQLMemoryOLTP.SQLMemoryCacheManager.OpenSqlConnection() in /src/Geode.Services.Cache.SQLMemoryOLTP/SQLMemoryCacheManager.cs:line 301
   at Geode.Services.Cache.SQLMemoryOLTP.SQLMemoryCacheManager.ExecuteNonQueryCommandAsync(SqlCommand command) in /src/Geode.Services.Cache.SQLMemoryOLTP/SQLMemoryCacheManager.cs:line 223
---> (Inner Exception #0) System.IO.IOException: Unable to write data to the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Data.SqlClient.SNI.SslOverTdsStream.WriteInternal(Byte[] buffer, Int32 offset, Int32 count, CancellationToken token, Boolean async)<---

[divega]

As recently announced in the .NET Blog, focus on new SqlClient features an improvements is moving to the new Microsoft.Data.SqlClient package. For this reason, we are moving this issue to the new repo at https://github.com/dotnet/SqlClient. We will still use https://github.com/dotnet/corefx to track issues on other providers like System.Data.Odbc and System.Data.OleDB, and general ADO.NET and .NET data access issues.

[alexavrutin]

Also having exactly the same issue in the ASP.NET Core 2.2 / Linux / Kubernetes / SQL Azure environment.

Unhandled Exception: System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.) ---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size) --- End of inner exception stack trace --- at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size) at System.Data.SqlClient.SNI.SslOverTdsStream.WriteInternal(Byte[] buffer, Int32 offset, Int32 count, CancellationToken token, Boolean async) --- End of inner exception stack trace --- at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at System.Threading.Tasks.Task.Wait() at System.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count) at System.Net.Security.SslStreamInternal.WriteSingleChunk[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory1 buffer)
at System.Net.Security.SslStreamInternal.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory1 buffer) at System.Net.Security.SslStreamInternal.Write(Byte[] buffer, Int32 offset, Int32 count) at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count) at System.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet) at System.Data.SqlClient.SNI.SNIMarsConnection.Send(SNIPacket packet) at System.Data.SqlClient.SNI.SNIMarsHandle.SendControlPacket(SNISMUXFlags flags) at System.Data.SqlClient.SNI.SNIMarsHandle.Dispose() at System.Data.SqlClient.SNI.TdsParserStateObjectManaged.Dispose() at System.Data.SqlClient.TdsParserSessionPool.Dispose() at System.Data.SqlClient.TdsParser.Disconnect() at System.Data.SqlClient.SqlInternalConnectionTds.Dispose() at System.Data.ProviderBase.DbConnectionPool.DestroyObject(DbConnectionInternal obj) at System.Data.ProviderBase.DbConnectionPool.CleanupCallback(Object state) at System.Threading.TimerQueueTimer.CallCallback() at System.Threading.TimerQueueTimer.Fire() at System.Threading.TimerQueue.FireNextTimers()

[grecosoft]

Having this same issue with a Docker service hosted within Azure as a Web App for Containers resource. The WebApi hosted in a container is connecting to an Azure SQL Database. This was not an issue when the WebApi was hosted internally on a Docker server pointed to an internally hosted SQL Server instance. After moving the container to be hosted in Azure as a Web App for Containers and using an Azure SQL database, the issue occurred. When developing local with the WebApi running as a local process this is noticed after a few minutes of running the service. However, running the container within Azure, the issue only happens once every day or so. When running in Azure as a Web Container, the container restarts after receiving the exception. Following is the application log and it is exactly the same as others listed:

2019-07-07T12:07:21.144590240Z
2019-07-07T12:07:21.354835247Z Unhandled Exception: System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.) ---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe
2019-07-07T12:07:21.354873348Z at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
2019-07-07T12:07:21.354879249Z --- End of inner exception stack trace ---
2019-07-07T12:07:21.354883049Z at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
2019-07-07T12:07:21.354886949Z at System.Data.SqlClient.SNI.SslOverTdsStream.WriteInternal(Byte[] buffer, Int32 offset, Int32 count, CancellationToken token, Boolean async)
2019-07-07T12:07:21.354891049Z --- End of inner exception stack trace ---
2019-07-07T12:07:21.354894749Z at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
2019-07-07T12:07:21.354910050Z at System.Threading.Tasks.Task.Wait()
2019-07-07T12:07:21.354914050Z at System.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count)
2019-07-07T12:07:21.354918050Z at System.Net.Security.SslStreamInternal.WriteSingleChunkTWriteAdapter
2019-07-07T12:07:21.354937751Z at System.Net.Security.SslStreamInternal.Write(Byte[] buffer, Int32 offset, Int32 count)
2019-07-07T12:07:21.354941851Z at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
2019-07-07T12:07:21.354945851Z at System.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet)
2019-07-07T12:07:21.354949751Z at System.Data.SqlClient.SNI.SNIMarsConnection.Send(SNIPacket packet)
2019-07-07T12:07:21.354953752Z at System.Data.SqlClient.SNI.SNIMarsHandle.SendControlPacket(SNISMUXFlags flags)
2019-07-07T12:07:21.354957752Z at System.Data.SqlClient.SNI.SNIMarsHandle.Dispose()
2019-07-07T12:07:21.354961852Z at System.Data.SqlClient.SNI.TdsParserStateObjectManaged.Dispose()
2019-07-07T12:07:21.354965752Z at System.Data.SqlClient.TdsParserSessionPool.Dispose()
2019-07-07T12:07:21.354969752Z at System.Data.SqlClient.TdsParser.Disconnect()
2019-07-07T12:07:21.354981753Z at System.Data.SqlClient.SqlInternalConnectionTds.Dispose()
2019-07-07T12:07:21.354986053Z at System.Data.ProviderBase.DbConnectionPool.DestroyObject(DbConnectionInternal obj)
2019-07-07T12:07:21.354989853Z at System.Data.ProviderBase.DbConnectionPool.CleanupCallback(Object state)
2019-07-07T12:07:21.354994453Z at System.Threading.TimerQueueTimer.CallCallback()
2019-07-07T12:07:21.354998453Z at System.Threading.TimerQueueTimer.Fire()
2019-07-07T12:07:21.358832809Z at System.Threading.TimerQueue.FireNextTimers()

2019-07-07T12:08:49.664615781Z warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
2019-07-07T12:08:49.664684484Z No XML encryptor configured. Key {008e5fce-9c78-423e-bcd4-bcf0cfa63614} may be persisted to storage in unencrypted form.
2019-07-07T12:08:50.055775592Z Hosting environment: Test
2019-07-07T12:08:50.083971732Z Content root path: /opt/apis/xxxxxxx.xxxxx.WebApi
2019-07-07T12:08:50.113987245Z Now listening on: http://[::]:80
2019-07-07T12:08:50.114015047Z Application started. Press Ctrl+C to shut down.

[gl1tch]

Getting the same error running the following stack.

ASP.NET Core 2.2 / Windows with Linux containers / SQL Azure environment.

Unhandled Exception: System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.) ---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Data.SqlClient.SNI.SslOverTdsStream.WriteInternal(Byte[] buffer, Int32 offset, Int32 count, CancellationToken token, Boolean async)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at System.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslStreamInternal.WriteSingleChunkTWriteAdapter
at System.Net.Security.SslStreamInternal.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet)
at System.Data.SqlClient.SNI.SNIMarsConnection.Send(SNIPacket packet)
at System.Data.SqlClient.SNI.SNIMarsHandle.SendControlPacket(SNISMUXFlags flags)
at System.Data.SqlClient.SNI.SNIMarsHandle.Dispose()
at System.Data.SqlClient.SNI.TdsParserStateObjectManaged.Dispose()
at System.Data.SqlClient.TdsParserSessionPool.Dispose()
at System.Data.SqlClient.TdsParser.Disconnect()
at System.Data.SqlClient.SqlInternalConnectionTds.Dispose()
at System.Data.ProviderBase.DbConnectionPool.DestroyObject(DbConnectionInternal obj)
at System.Data.ProviderBase.DbConnectionPool.CleanupCallback(Object state)
at System.Threading.TimerQueueTimer.CallCallback()
at System.Threading.TimerQueueTimer.Fire()
at System.Threading.TimerQueue.FireNextTimers()
The program 'dotnet' has exited with code 0 (0x0).

[johnsimons]

Getting the same error running the following:

Netcoreapp2.2 / Linux container (mcr.microsoft.com/dotnet/core/runtime:2.2) / Kubernetes 0.13.5 / SQL Azure ( Elastic Pools, Standard Plan )

Unhandled Exception: System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.) ---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Data.SqlClient.SNI.SslOverTdsStream.WriteInternal(Byte[] buffer, Int32 offset, Int32 count, CancellationToken token, Boolean async)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslStreamInternal.WriteSingleChunk[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStreamInternal.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStreamInternal.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet)
   at System.Data.SqlClient.SNI.SNIMarsConnection.Send(SNIPacket packet)
   at System.Data.SqlClient.SNI.SNIMarsHandle.SendControlPacket(SNISMUXFlags flags)
   at System.Data.SqlClient.SNI.SNIMarsHandle.Dispose()
   at System.Data.SqlClient.SNI.TdsParserStateObjectManaged.Dispose()
   at System.Data.SqlClient.TdsParserSessionPool.Dispose()
   at System.Data.SqlClient.TdsParser.Disconnect()
   at System.Data.SqlClient.SqlInternalConnectionTds.Dispose()
   at System.Data.ProviderBase.DbConnectionPool.DestroyObject(DbConnectionInternal obj)
   at System.Data.ProviderBase.DbConnectionPool.CleanupCallback(Object state)
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()

This causes the pods to restart with a 139 exit code, and we can't prevent it 😞

Currently, this is a show stopper for us, are there any workarounds?

[cheenamalhotra]

Could you let us know what authentication mode is in use?
Is it possible to test with Non-SNI dependent authentication mode, e.g. SqlPassword Authentication and let us know if there's still a problem? Just trying to understand the source of the problem..

[johnsimons]

@cheenamalhotra we are using username/password in our connection string to authenticate to sql server.
I am not too sure how to make it a Non-SNI dependent authentication mode?
I looked at https://docs.microsoft.com/en-us/dotnet/api/system.data.sqlclient.sqlconnectionstringbuilder.authentication and that seems to be only available for netfull not netstandard/netcore ?
So can you please give us more info?

[johnsimons]

@cheenamalhotra here is our connection string

Server=tcp:#{SqlServerName}.database.windows.net,1433;Initial Catalog=#{InstanceId};Persist Security Info=False;User ID=#{InstanceId};Password=#{SqlLoginPassword};MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;ConnectRetryCount=3;

[johnsimons]

Forgot to mention what we also set in code:

var connectionStringBuilder = new SqlConnectionStringBuilder(connectionString)
{
                MultipleActiveResultSets = true,
                ApplicationName = applicationName  
};

[cheenamalhotra]

If you just google "Kubernetes broken pipe", it turns out to be an issue for many Kubernetes TCP applications. Looks like more of a Kubernetes networking issue to me.

Have you tried port forwarding and ensure you're using Kubernetes State Metrics version > 1.5.0?
Similar Reported Issue seems to be fixed in one of their PRs.

[cheenamalhotra]

Same for Docker images.. there are many reported issues on the same topic..

• Ensure Port forwarding is applied correctly.
• Look for any Networking Service running in background that is looking for idle TCP connections and times them out, e.g. vpnkit.
• Ensure you're using latest docker/Kubernetes versions of environments to avoid any issues that have already been addressed.

Similar issues that have been addressed:

• https://github.com/docker/for-mac/issues/2442
• https://github.com/kubernetes/kube-state-metrics/issues/550

[droyad]

Even if K8 is causing the connection to fail, there could be many other reasons as well (e.g. network cable unplugged). I think the application should not exit due to that happening.

[pawelpabich]

If I'm reading the stack trace correctly then this error occurs when a connection is being disposed/destroyed. System.Data.ProviderBase.DbConnectionPool.DestroyObject(DbConnectionInternal obj).

I would expect this operation to execute in a way that doesn't return any errors to the application.

[cheenamalhotra]

@pawelpabich @mifurm @droyad @johnsimons
Is it possible to provide a link to repro app to reproduce the issue locally and study traces?

I agree that the application should not exit leading to Pod restart if the trigger is this IO exception. I can take a deeper look with user-scenario handy, and then provide some insights.

[johnsimons]

This issue seems to be related to https://github.com/dotnet/SqlClient/issues/60

[mcasperson]

@cheenamalhotra

We are seeing this issue running in an Azure AKS cluster version 1.13.5, which was until recently the latest non-preview version supplied by Azure.

If there is any idle connection timeout functionality, it would have been implemented by AKS as we have not added any custom functionality here.

Port mapping is all done using the standard Kubernetes functionality via services.

AKS uses the metrics-server rather than kube-state-metrics.

We have relied on AKS as much as possible to provide sensible defaults for our environment, and would prefer not to have to tweak this managed service to support .NET Core apps (indeed if such tweaks are possible and supported by AKS at all).

[johnsimons]

@cheenamalhotra https://github.com/dotnet/SqlClient/issues/60 seems to have a repro for this issue, is that enough for you to chase this one?

[misterspeedy]

Exactly the same issue for us. It's a showstopper and I'm having to migrate the service onto a classic Azure worker.

microsoft/dotnet:2.2-sdk-alpine

Trace:

Unhandled Exception: System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.) ---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Data.SqlClient.SNI.SslOverTdsStream.WriteInternal(Byte[] buffer, Int32 offset, Int32 count, CancellationToken token, Boolean async)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslStreamInternal.WriteSingleChunkTWriteAdapter
at System.Net.Security.SslStreamInternal.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet)
at System.Data.SqlClient.SNI.SNIMarsConnection.Send(SNIPacket packet)
at System.Data.SqlClient.SNI.SNIMarsHandle.SendControlPacket(SNISMUXFlags flags)
at System.Data.SqlClient.SNI.SNIMarsHandle.Dispose()
at System.Data.SqlClient.SNI.TdsParserStateObjectManaged.Dispose()
at System.Data.SqlClient.TdsParserSessionPool.Dispose()
at System.Data.SqlClient.TdsParser.Disconnect()
at System.Data.SqlClient.SqlInternalConnectionTds.Dispose()
at System.Data.ProviderBase.DbConnectionPool.DestroyObject(DbConnectionInternal obj)
at System.Data.ProviderBase.DbConnectionPool.CleanupCallback(Object state)
at System.Threading.TimerQueueTimer.CallCallback()
at System.Threading.TimerQueueTimer.Fire()
at System.Threading.TimerQueue.FireNextTimers()

[David-Engel]

For those experiencing this issue, are you able to disable MARS (MultipleActiveResultSets = false) and see if the exceptions go away?
@kieronlanning @misterspeedy @mcasperson @johnsimons @gl1tch @grecosoft @alexavrutin @DaEkstrim @mifurm @SeymourRu @Hartdrooz @TerryAspen @DenKn

@mifurm You said you could provide a repro. (Hopefully a simple one.) Would that still be possible?

[johnsimons]

@david-Engel as I said before https://github.com/dotnet/SqlClient/issues/60 seems to repro it

[johnsimons]

@david-Engel I have also created https://github.com/dotnet/SqlClient/issues/141 which it may be related

[David-Engel]

@johnsimons I looked at #60 and I don't see the similarities with this issue. #141 also appears to be different to me, though I doubt it is the same as 129, as that was specific to Async and CommandBehavior.CloseConnection.

[misterspeedy]

@David-Engel Thanks for jumping in! I'll try turning MARS off and report back. If that doesn't work I'll set up a repo with a min reproducible case and share here.

[misterspeedy]

Ok, quick update: I've written a repro which I think is doing the same thing as my real app and... the issue does not occur. (Works fine with MARS either turned off or turned on.) Doing some work now to try and work out what is different about my real app that is triggering the problem. Will update here when there is news.

[misterspeedy]

Couldn't work out what the difference was between my attempted repro project (which doesn't show the issue) and the real app (which does). Therefore I've tried disabling MARS on the live app and have left it running. It's a super-low-volume (but important!) service so it'll be a while before I know whether disabling MARS fixed the issue. Will report back when I know.

[DaEkstrim]

@David-Engel Unfortunately, we can't disable MARS without rewriting portions of the application. I'll see whether we can redesign these segments. Issue is that the last time we faced this error was on 28th July (it's sporadic and not frequent), so we have no way of knowing for sure whether removing MARS circumvents the error without waiting for months.

[misterspeedy]

OK, our live system has just successfully processed several events (and written to SQL database), so it seems that turning MARS off (when possible) does work around the problem.

[David-Engel]

I noticed a very suspicious throw introduced long ago that would easily cause this issue if disposing a broken, mars-enabled connection:
https://github.com/dotnet/corefx/pull/4594/files#diff-a8e6146000f306b97ce4e48207f7cd9fR69
I plan to remove the throw for the next M.D.SqlClient preview.

[misterspeedy]

Nice one, thanks @David-Engel !

[cheenamalhotra]

Preview release v1.0.19221.1 has been published containing fix. Closing issue.

[speciesunknown]

The issue persists for me with 1.0.19239.1

I have this stack trace:

Unhandled Exception: System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.) ---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Data.SqlClient.SNI.SslOverTdsStream.WriteInternal(Byte[] buffer, Int32 offset, Int32 count, CancellationToken token, Boolean async)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslStreamInternal.WriteSingleChunk[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStreamInternal.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStreamInternal.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet)
   at System.Data.SqlClient.SNI.SNIMarsConnection.Send(SNIPacket packet)
   at System.Data.SqlClient.SNI.SNIMarsHandle.SendControlPacket(SNISMUXFlags flags)
   at System.Data.SqlClient.SNI.SNIMarsHandle.Dispose()
   at System.Data.SqlClient.SNI.TdsParserStateObjectManaged.Dispose()
   at System.Data.SqlClient.TdsParserSessionPool.Dispose()
   at System.Data.SqlClient.TdsParser.Disconnect()
   at System.Data.SqlClient.SqlInternalConnectionTds.Dispose()
   at System.Data.ProviderBase.DbConnectionPool.DestroyObject(DbConnectionInternal obj)
   at System.Data.ProviderBase.DbConnectionPool.CleanupCallback(Object state)
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()

I traced through the code side by side and got as far as SNITcpHandle.cs
https://github.com/dotnet/SqlClient/blob/master/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SNI/SNITcpHandle.cs#L420

I can see handles for 4 possible exception types, but nothing for the exception type of my stack trace which is System.AggregateException

I propose that unusual exceptions which reach back to System.Data.ProviderBase.DbConnectionPool.CleanupCallback be handled gracefully, becuase allowing the exception to fall through at this point will crash the main thread.

[speciesunknown]

@cheenamalhotra Should I create a new issue or can you re-open this?

[speciesunknown]

I've realised that my second crash was because I had another service in the application using an old version of this package, as its using Microsoft.EntityFrameworkCore.SqlServer which uses System.Data.SqlClient

Its most likely that the error is in fact fixed by the aforementioned patch.

[marcelcandidoferreira]

Hi there, I'm having the same issue.
I'm using Microsoft.EntityFrameworkCore.SqlServer version 2.2.1 already.
Any other clues on how to fix this issue?

[yukiwongky]

@marcelaldecoa From the dependencies listed in the Microsoft.EntityFrameworkCore.SqlServer 2.2.1 nuget page looks like it's still using System.Data.SqlClient. Can you try Microsoft.EntityFrameworkCore.SqlServer 3.0.0, which has a dependency on Microsoft.Data.SqlClient driver version >= 1.0.19249.1.

[marcelcandidoferreira]

Hi @yukiwongky thanks for you message.

Unfortunately I can't upgrade easily Microsoft.EntityFrameworkCore.SqlServer to version 3.0.0, it would require me to update to netstandard2.1, which requires .net core 3.0 and my stack is running with .net core 2.2.

I've updated Microsoft.EntityFrameworkCore.SqlServer from 2.2.1 to 2.2.6 but I don't see it fixed the issue either.

[yukiwongky]

Hi @marcelcandidoferreira we're planning to port the fix to System.Data.SqlClient. We'll let you know once it's ready.

[WeihanLi]

Hi @marcelcandidoferreira we're planning to port the fix to System.Data.SqlClient. We'll let you know once it's ready.

had the fix released? which version could be used? @yukiwongky

[cheenamalhotra]

@WeihanLi

Yes, it would be System.Data.SqlClient v4.8.1

[WeihanLi]

@WeihanLi

Yes, it would be System.Data.SqlClient v4.8.1

Thanks @cheenamalhotra