Sqlclient: [SQL Server 2005] System.Data.SqlClient: pre-login fails with SSL error even with Encryption=false

[kostix]

Platform : Debian 8 "Jessie"
Runtime : 1.0.0-preview1-002702,coreclr,x64,linux
System.Data.SqlClient : 4.1.0-rc2-24027

ConnectionString : Server=tcp:server.domain.lan,1433;User ID=XXXX;Password=XXXX;Encrypt=False"

An attempt to connect to a Microsoft SQL Server 2005 instance results in the following exception:

Project foo (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
Unhandled Exception: System.Data.SqlClient.SqlException: A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: SSL Provider, error: 31 - Encryption(ssl/tls) handshake failed) ---> System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: size
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Data.SqlClient.SNI.SslOverTdsStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.<>c.<BeginReadInternal>b__39_0(Object <arg>)
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.TaskToApm.End[TResult](IAsyncResult asyncResult)
   at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Data.SqlClient.SNI.SNITCPHandle.EnableSsl(UInt32 options)
   at System.Data.SqlClient.SNI.SNIProxy.EnableSsl(SNIHandle handle, UInt32 options)
   --- End of inner exception stack trace ---
   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, Object providerInfo, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling)
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at ConsoleApplication.Program.Main(String[] args)

To me, the error looks as if the Encrypt=false just does not get communicated properly in the PRELOGIN packet because the exception is the same no matter if I set Encrypt to true (the default) or false.

Adding TrustServerCertificate=true with the Encrypt setting enabled or disabled has no effect.

The Encrypt setting is parsed as if I specify an invalid value for it, I get an error about it.

[kostix]

I'd say the second line in the exception stack trace, System.Data.SqlClient.SNI.SslOverTdsStream.Read(... looks especially suspicious: it suggests SSL reader is used even though we told the client to not use it.

[kostix]

Here's the capture file created by tcpdump of a sample session under Encrypt=false.

The interesting thing is that the client is sending its PRELOGIN packet twice, and MS-TDS dissector in Wireshark considers the server's response to the first PRELOGIN packet as invalid.

[saurabh500]

cc @corivera

[kostix]

In the meantime I have converted the capture to a Network Monitor v2 format using the editcap utility and inspected the generated capture in the Microsoft Network Monitor 3.4 (which has much better dissector for MS-TDS).

Inspecting the capture in NM, I see:

• The dissector fails to properly parse the initial PRELOGIN packet we sent. Specifically it fails to parse the encryption option's _data._
• The server's response packet explicitly has encryption option set to "off".
• Still, the second PRELOGIN packet we sent appears to contain the TLS 1.2 handshake, and the server apparently disconnects us after receiving it (supposedly failing to parse what should not have been a TLS handshake as "plain" payload).

Hence if we assume the NM's dissector is correct, its output might actually indicate several problems:

• We might be generating our PRELOGIN packet incorrectly.
• We fail to deal with the server telling us not to use encryption.
• The exception raised appears to be misleading as we did not actually read anything from the server after it disconnected us.

Here's that capture file.

[saurabh500]

@kostix Does this connectivity work from Windows using the same app?

[kostix]

Yes. Verified using dotnet restore + dotnet run on a Windows 7 SP1 x64 using standalone SDK.

Here are archived lockfiles from both projects (that one whose name starts with "win7" is from the Windows build, which works).

[saurabh500]

Yes. Verified using dotnet restore + dotnet run on a Windows 7 SP1 x64 using standalone SDK.

Is the Yes is for successful connection from Windows ?
We are looking into this.

[kostix]

Yes, the connection works.

From the dump taken on that host using NM, it appears that the TLS session is still being used while authenticating, just it manages to happen OK on Windows.

Here's the .

There, the 192.168.2.145 is the client and .25 is the server.
Highlighted, is the server's PRELOGIN response packet indicating encryption turned off.
As you can see, what follows is the TLS-protected exchange.

[joshfree]

@saurabh500 @corivera any update on this? Are you planning to have a fix ready to review today or tomorrow?

[corivera]

@joshfree Currently trying to diagnose the problem.

[corivera]

@kostix Do you get this error on any other Linux distro, or is it limited to just Debian? If you haven't tested this, that's fine; just wondering. I'm actually getting generic TCP timeouts when connecting to SQL 2005 from Ubuntu (which is still unexpected).

[kostix]

Sorry, I happen to use Debian whereever I run GNU/Linux systems.

I have a couple of Wheezy (7.x) installations around but since corefx
RC2 explicitly states it's not a supported platform I see little reason
trying things there.

[kostix]

@corivera, a dumb question: did you try telnet mssql2005.server.host 1433 to verify TCP connectivity to that box is actually OK?

You can also install tcpdump and wireshark-common there, capture the traffic to your server and then run editcap on the resulting dump to produce a file suitable for digestion by NM on your development workstation. Or just run Wireshark right there on Ubuntu if you have X running there: I'm sure it should be enough to see if the server fails to respond to some MS-TDS message in time (which could mean it's incomplete or whatnot).

[saurabh500]

@kostix The TCP connectivity to the DB machine that we are using is fine.

We have however facing a timeout in the Post Login phase of the connection

Unhandled Exception: System.Data.SqlClient.SqlException: Connection Timeout Expired. The timeout period elapsed during the post-login phase. The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections. The duration spent while attempting to connect to this server was - [Pre-Login] initialization=141; handshake=293; [Login] initialization=3; authentication=18; [Post-Login] complete=29025;

The connection string is something like
server=tcp:Some2005Server,1433;uid=UID;pwd=Pass;connect timeout=30;Encrypt=False

[kostix]

@saurabh500, is it possible to enable debug logging on the server? It's interesting what the server thinks about the malfunctioning client.

I'd also try to see at the traffic dump captured on Ubuntu in NM: it could happen you are merely sending malformed TDS message. Say, in the message header you advertise it has N bytes of payload and then send M < N, so the server waits for the rest and eventually either end times out.

[saurabh500]

We will continue to investigate this issue but may not be targeting a fix for RTM. I verified that the connectivity from Sql Server 2008 and above is working.

[D155Y]

Hi
I am new to .Net Core and I am having the same issue as mentioned above. Just wanted to know, is this fix available in current .net core version or its still being looked at.

[paalmoest]

Will the fix for this issue be present in core 1.1 ? @saurabh500

[markmcdowell]

I've just hit this on centos 7. Any workarounds?

[Denterl]

The same issue reproduced for .net core 2.0 preview 1, tried to connect from ubuntu x64 16.04 lts. The same code successfully connects from Windows 10 Creators Update

[sakopov]

Any progress?

[sshinault]

I am having the same problem connecting from an Amazon Linux AMI EC2 host to a Sql Server 2005 instance. I am using the mssql-cli client to test why my AWS Lamdba functions (C# .NET Core) were getting connection timeouts.

The packet captures look almost exactly the same as the original poster's - 2nd PRELOGIN request appears to attempt to send TLS/SSL payload and server responds by closing connection.

mssql-cli login attempts from Windows 10 host to this server succeed. SQL Server has encryption turned OFF.

[brokenthorn]

I have never had this problem and have never upgraded SQL Operations Studio yet the same saved connections I have been using before now suddenly don't work anymore. I'm on macOS High Sierra.

[LandonCampbell]

@sshinault Did you ever find a solution? I'm getting this from C# AWS lamba functions too.

[sshinault]

@LandonCampbell No, we decided to upgrade to SQL Server 2008 R2 which works fine.

[LandonCampbell]

@sshinault Thanks, I appreciate the feedback.

[divega]

As recently announced in the .NET Blog, focus on new SqlClient features an improvements is moving to the new Microsoft.Data.SqlClient package. For this reason, we are moving this issue to the new repo at https://github.com/dotnet/SqlClient. We will still use https://github.com/dotnet/corefx to track issues on other providers like System.Data.Odbc and System.Data.OleDB, and general ADO.NET and .NET data access issues.

[David-Engel]

Assuming all above issues are related to SQL Server 2005 and not a currently supported SQL version and closing this issue. Feel free to re-open if it is present against a currently supported version.