Sqlclient: SqlConnection.Open doesn't seem to work on Ubuntu 20.04

[zanfranceschi]

I've recently upgraded to Ubuntu 20.04 (clean install) and found out the SqlConnection.Open doesn't seem to work using dotnet core sdk 3.1 and Microsoft SQL Server 2012 (SP4) versions.

It does not produce an error ─ it simply gets stuck and blocks execution. No timeout, not apparent error. I've even sniffed tcp traffic with wireshark to see if I could get something. Nothing.

It was working normally while using Ubuntu 19.10.

The steps to reproduce is to simply instantiate an SqlConnection object and try invoke "Open" on it.

I've also installed dotnet core 2.1 and tried System.Data.SqlClient. Same behaviour.

Any ideas or more information needed?

[ErikEJ]

Are you using the 3.1.4 SDK, that is required for Ubuntu 20.04 support?

[zanfranceschi]

[edited]
Mmmm, I'm not using it:

dotnet --list-sdks
2.1.806 [/usr/share/dotnet/sdk]
3.1.202 [/usr/share/dotnet/sdk]

I've used the apt repository.

[edited second time]
I'm confused now.

I'm not finding this SDK version. I however have this specific runtime version:

dotnet --list-runtimes
Microsoft.AspNetCore.All 2.1.18 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.18 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.18 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.4 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

[karinazhou]

Hi @zanfranceschi ,

Thank you for reporting this. I will take a look at it.

Just to confirm the following behaviors with you again:

• neither of net core 2.1 and 3.1 works on Ubuntu 20.04
• both of net core 2.1 and 3.1 works on Ubuntu 19.10

Is it correct?

[zanfranceschi]

Hi @zanfranceschi ,

Thank you for reporting this. I will take a look at it.

Just to confirm the following behaviors with you again:

• neither of net core 2.1 and 3.1 works on Ubuntu 20.04
• both of net core 2.1 and 3.1 works on Ubuntu 19.10

Is it correct?

Correct. That's it.

And just to add that the CPU consumption increases and takes 100% of one of the cores in a few seconds. But again, no dotnet error.

[cheenamalhotra]

It looks related to #538

[karinazhou]

@zanfranceschi
I tried these docker images which are for Ubuntu 20.04 with the docker-compose project in SqlClient\ repository:
mcr.microsoft.com/dotnet/core/runtime:3.1-focal
mcr.microsoft.com/dotnet/core/runtime:2.1-focal

When I connect to a SQL Server 2005, which doesn't support TLS 1.2, it gives me the following error instead of hanging:
OpenSslCryptographicException: error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol

I tested the repository from the SqlClient master branch. Have you tried that out?

As @cheenamalhotra has mentioned, this is highly related to the existing TLS protocol version issue.
I will try other methods to see whether I can reproduce the hanging behavior or not.

And one more question, could you also check the certificate used by the SQL Server 2012 you are using? Is that SHA-1 certificate?

[zanfranceschi]

Hi @karinazhou,

Thanks for you troubleshooting.

I haven't tried the master branch ─ I have used what comes in the nuget package.

As for the hash version I'll contact the database team as my access to the database is limited.

Do you think it makes sense trying to connect to an instance without encryption or is it too stupid to suggest?

[karinazhou]

@zanfranceschi
According to the other related issues reported before, it may not only depend on the TLS version but also the SHA-1 self-signed certificate used on the server for connection encryption.

One known workaround is to enable TLSv1.2 and use SHA256 certificate on the server.

BTW, I am still trying to reproduce the hanging so far......

[ErikEJ]

What Windows version does your SQL Server run?

[zanfranceschi]

@ErikEJ ─ it's a Windows 2012 - R2

@karinazhou ─ Is there a way to privately message you with a connection string for you to reproduce (maybe at your email on your github profile?)? I can confirm that even without valid credentials, the hanging happens. I've set a public proxy to this database. You could reproduce even without valid credentials.

[karinazhou]

@zanfranceschi Sure. You can find me on Gitter.

[zanfranceschi]

@karinazhou sorry, I don't have a gitter account. I've emailed you at your github public email address.

[karinazhou]

@karinazhou sorry, I don't have a gitter account. I've emailed you at your github public email address.

@zanfranceschi Have you sent something? I haven't received any email yet...

[zanfranceschi]

I have to v-jizho2 at microsoft dot com

My email is [my github account username] at gmail... maybe has gone to spam.

[zanfranceschi]

@karinazhou ─ Just created a gitter account and sent you a note.

[karinazhou]

@zanfranceschi
I test with your connection string and I am able to reproduce the hanging behavior on Ubuntu 20.04 64-bit vm. Here is the dotnet version I have on this test machine:

dotnet --list-sdks
3.1.300
dotnet --list-runtimes
Microsoft.NETCore.App 3.1.4

Driver version : v2.0.0-preview4.20142.4

The same connection string doesn't go into an infinite loop on Ubuntu 18.10 64-bit vm. I can get the error for invalid Login UID.

I add the event listener to my test application and find it got stuck at DbConnectionFactory.PruneConnectionPoolGroups. I will need more investigation on this.

......
 Event Trace (<sc.TdsParser.Connect|SEC> Consuming prelogin handshake).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event Trace (<prov.DbConnectionPoolGroup.ClearInternal|RES|INFO|CPOOL> 1, Idle).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event Trace (<prov.DbConnectionPoolGroup.ReadyToRemove|RES|INFO|CPOOL> 1, Disabled).
 Event Trace (<prov.DbConnectionFactory.QueuePoolGroupForRelease|RES|INFO|CPOOL> 1, poolGroup=1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1, ReleasePoolGroup=1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).

[zanfranceschi]

@zanfranceschi
I test with your connection string and I am able to reproduce the hanging behavior on Ubuntu 20.04 64-bit vm. Here is the dotnet version I have on this test machine:

dotnet --list-sdks
3.1.300
dotnet --list-runtimes
Microsoft.NETCore.App 3.1.4

Driver version : v2.0.0-preview4.20142.4

The same connection string doesn't go into an infinite loop on Ubuntu 18.10 64-bit vm. I can get the error for invalid Login UID.

I add the event listener to my test application and find it got stuck at DbConnectionFactory.PruneConnectionPoolGroups. I will need more investigation on this.

......
 Event Trace (<sc.TdsParser.Connect|SEC> Consuming prelogin handshake).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event Trace (<prov.DbConnectionPoolGroup.ClearInternal|RES|INFO|CPOOL> 1, Idle).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event Trace (<prov.DbConnectionPoolGroup.ReadyToRemove|RES|INFO|CPOOL> 1, Disabled).
 Event Trace (<prov.DbConnectionFactory.QueuePoolGroupForRelease|RES|INFO|CPOOL> 1, poolGroup=1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1, ReleasePoolGroup=1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).
 Event AdvancedTrace (<prov.DbConnectionFactory.PruneConnectionPoolGroups|RES|INFO|CPOOL> 1).

Hi @karinazhou ─ that's good news! Thank you.

Please, let me know if it will result in a fix or it is a compatibility issue that won't be handled.

[karinazhou]

@zanfranceschi

I tried my application with this PR577 and it actually fixed the hanging issue. I add some debugging messages to
src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SNI/SslOverTdsStream.cs: ReadInternal()
where the fix is and I do see it go into the infinite loop there.

The actual exception I get on Ubuntu 20.04 is

Unhandled exception. Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: SSL Provider, error: 31 - Encryption(ssl/tls) handshake failed)

If you test your server with openssl command:
openssl s_client -tls1 -connect <your_server_ip>:<your_server_port>

You can get no protocols available error on ubuntu 20.04. I tested the same openssl command on Ubuntu 18.10 and it doesn't give me the error.

So to summarize it shortly, the hanging issue on Ubuntu 20.04 is due to the following reasons:

1. the server you are using doesn't support TLS1.2
2. Ubuntu 20.04 drops support for TLS 1.0 and TLS 1.1
3. We have a bug in the driver when reading SSL packets.

I will recommend upgrading your server to support TLS 1.2 if you would like to use Ubuntu 20.04 in the future.

[zanfranceschi]

Wow @karinazhou, don't know how to thank you. You've done some serious troubleshooting!

Thank you very much for your terrific work!

[prensing]

Could this issue also affect System.Data.Sqlclient (as opposed to the Microsoft.Data.Sqlclient)? I am using System... and when I switched from Ubuntu 18.04 to Debian 10, I can no longer connect. The behavior is roughly the same: connect() does not return and CPU goes to 100%. I did check the "openssl" command above and I could connect to the server.

[cheenamalhotra]

Hi @prensing

Yes the issue also applies to S.D.S, we will try to backport the fix from PR #577 to System.Data.SqlClient as well.

[cheenamalhotra]

Hi @zanfranceschi

We will close the issue as PR #577 is now merged and is planned to release with v2.0.0.
Please feel free to re-open if you face problems again.
We will continue to work on backporting fix to System.Data.SqlClient.

[mburakeker]

We created the ubuntu bionic images for .NET 5.0 and shared the Dockerfiles if anyone needs them.
https://github.com/dotnet/dotnet-docker/issues/2415#issuecomment-733941423

[prensing]

This issue is still affecting System.Data.SqlClient.

[karinazhou]

Hi @prensing , are you using System.Data.SqlClient v4.8.2? The fix should be in that version.

[prensing]

No, sorry. I missed updating the NuGet package. However, now I get an SSL handshare error, although at least it does not just use 100% CPU and do nothing.
If you have a quick known fix, I would appreciate it. Otherwise will investigate it next week.

Thanks!

[ErikEJ]

@prensing just read the comments above re TLS 1.2 support...