SqlClient 🚀 - The MARS TDS header contained errors using ASP.NET Core and EF Core connecting to ...

Regarding the stackoverflow issue addiotional questions asked by Afsaneh:

Unfortunately can't post the csproj file contents...

I'm using EF Core 2.1.4 with Linq2Db provider (https://github.com/linq2db/linq2db.EntityFrameworkCore) for some of the queries that are poorly translated by the EF Core.., I'm however also referencing the System.Data.SqlClient 4.5.1 package in on of the solution's projects
I'm hosting on Azure Websites, so I assume it's some sort of IIS
I'll have to log this more explicitly. Is logging the "State" field sufficient for you ?

brunolau on 11 Oct 2018

Here's the Dump of the SqlError object:

Class:
16

Errors:
System.Data.SqlClient.SqlError: The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.

Error code:
-2146232060

Line number:
1

Number:
4011

SQL Procedure:
None

Source:
Core .Net SqlClient Data Provider

State:
3

brunolau on 11 Oct 2018

I had a random spate of these errors appear in my ASP.NET application today. I use ASP.NET Core 2.1, EF Core 2.1 and Azure SQL Database. They occurred in a small time window and once I stopped and started the web app the errors seemed to stop.

I think I queried successfully from SSMS while the errors were happening but couldn't say for sure that it was at the same time. If it happens again I'll query in SSMS to confirm that I can.

Is there anything I can provide that will help with a solution to this? (I have application insights and database logging turned on for my application)

NOTES:
MARS is explicitly set to off in my connection string (I use the one Azure Provides via the portal UI)

stap123 on 3 Dec 2018

@stap123 @brunolau Would you be able to provide us a minimal repro sample application? Thanks.

AfsanehR on 3 Dec 2018

@AfsanehR I wouldn't even know where to start, I've so far only seen it once today and restarting my application made it stop happening. I will keep my eyes open and if it starts happening in a pattern of some kind I'll try and put one together but at the moment it seems random so I've got no starting point for a repro.

I realise that's useless to you as well though haha

I have got about 130x exceptions logged in my App Insights at the moment but I doubt the stack trace would be enough for you?

It also looks like the error occurred on ALL queries no matter where they originated from so it doesn't appear to be a query specific issue.

stap123 on 3 Dec 2018

I confirm the same - errors occurring at random queries, random time. I've tried to stress-test the application by running 30 threads that was making DB queries in parallel, but that didn't reproduce the issue either.

I've tried to use the SSMS query while the error was occurring and it worked, I've also tried to deploy to another azure release slot at the time the issue was occurring and it worked.

What I've temporarily ended up with was writing a middleware that once detects this particular error [based on the err message], automatically restarts the web app. As the issue occurs 1-2 times a day, it's acceptable, but much better would be having the issue solved.

What connects both mine and Adam's issue is the ASP.NET Core 2.1 + EF Core 2.1 + Azure SQL DB combo, but I cannot even say what part of the chain to blame.

brunolau on 4 Dec 2018

ok thanks @brunolau @stap123 . We will investigate this with the configurations you mentioned, although I think it would be highly unlikely we hit the same issue as we need to know what queries are running. Will get back to you on this. Could you also please give us the output of dotnet --info?

AfsanehR on 4 Dec 2018

Here you go, obtained from the Azure console

.NET Core SDK (reflecting any global.json):
Version: 2.2.100
Commit: 51868761f2

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x86
Base Path: D:Program Files (x86)dotnetsdk2.2.100

Host (useful for support):
Version: 2.2.0
Commit: 1249f08fed

.NET Core SDKs installed:
1.1.10 [D:Program Files (x86)dotnetsdk]
2.1.403 [D:Program Files (x86)dotnetsdk]
2.1.500 [D:Program Files (x86)dotnetsdk]
2.2.100 [D:Program Files (x86)dotnetsdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.3 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.6 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.0 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.3 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.6 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.0 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.12 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 1.1.9 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.0.9 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.3 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.6 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.2.0 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

brunolau on 5 Dec 2018

Here's mine 👍

.NET Core SDK (reflecting any global.json):
Version: 2.2.100
Commit: 51868761f2

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x86
Base Path: D:Program Files (x86)dotnetsdk2.2.100

Host (useful for support):
Version: 2.2.0
Commit: 1249f08fed

.NET Core SDKs installed:
1.1.10 [D:Program Files (x86)dotnetsdk]
2.1.403 [D:Program Files (x86)dotnetsdk]
2.1.500 [D:Program Files (x86)dotnetsdk]
2.2.100 [D:Program Files (x86)dotnetsdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.3 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.6 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.0 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.3 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.6 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.0 [D:Program Files (x86)dotnetsharedMicrosoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.12 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 1.1.9 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.0.9 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.3 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.6 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.2.0 [D:Program Files (x86)dotnetsharedMicrosoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

stap123 on 5 Dec 2018

Hello, we have experiensed same problem:

System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
at System.Data.SqlClient.SqlCommand.<>c.b__122_0(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Dapper.SqlMapper.QueryAsyncT in C:projectsdapperDapperSqlMapper.Async.cs:line 419

We use

Dapper 1.50.5
.NETCOREAPP 2.1.2
Runtime in docker: FROM microsoft/dotnet:2.1-aspnetcore-runtime AS final

How to reproduce:
When we experienced problem: our database log file was full, and we got exception: the transaction log for database '' is full due to "LOG_BACKUP", after that exception all queries in .net core app was with errors.

When we increased log file, .net 4.7 apps starts to work, but .net core app continue to throw exception. Only restart of application helped

hackfaq on 7 Dec 2018

I got this again this morning between 07:38:09 and 07:39:22.

Not sure if that's useful to help you track anything down or not. (I did nothing to any of my services to stop it happening but I then have successful requests being processed at 07:40:06)

stap123 on 10 Dec 2018

I managed to temporarily auto-mitigate the issue by creating custom Entity framework execution strategy that once encounters the MARS error, closes the connection, clears the pool and reopens. Since I've introduced this workaround, I've been running errorless for 3 days.

To sum it up. Once the error is encountered, closing & reopening the error connection + clearing the connection pool seems to re-establish working order

brunolau on 4 Jan 2019

We are getting the same problem.
Errors starts occuring in same moment on different servers. Application pool restart helps,

Application Server
Microsoft Windows Server 2012 R2 Standard 6.3.9600 Build 9600
dotnet host Version: 2.1.5 Commit: 290303f510
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.5 [C:Program FilesdotnetsharedMicrosoft.AspNetCore
Microsoft.AspNetCore.App 2.1.5 [C:Program FilesdotnetsharedMicrosoft.AspNetCore
Microsoft.NETCore.App 2.1.5 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]

Database
Microsoft SQL Server 2017 (RTM-CU11) (KB4462262) - 14.0.3038.14 (X64) Sep 14 2018 13:53:44 Copyright (C) 2017 Microsoft Corporation Enterprise Edition (64-bit) on Windows Server 2012 R2 Standard 6.3 (Build 9600: )

Application
Dapper 1.50.5
Microsoft.AspNetCore.App 2.1.5

PolitovArtyom on 5 Jan 2019

We're having the same issue. Any solutions?

sepehr1014 on 7 Feb 2019

@AfsanehR
Do you need some additional data for investigation?
I can try to get application and tcp dump while errors occuring

PolitovArtyom on 8 Feb 2019

@PolitovArtyom Yes. That would be helpful.
@sepehr1014 @PolitovArtyom do you have the MARS enabled in your connection? Also if possible a repro would be useful.

AfsanehR on 8 Feb 2019

MARS is not enabled on client and disabled on server

PolitovArtyom on 9 Feb 2019

We're also having the same issue. With NetCore, EFCore and an Azure SQL db

wboevink on 18 Feb 2019

@brunolau could you maybe create a gist with the workaround.

mshenoy83 on 27 Feb 2019

Sure, I will post it in a couple of days, I'm away from the PC for a couple of days and as soon as I get back, I'll do it

brunolau on 27 Feb 2019

Just 1 more thing. I've been facing this error for a few days now. I have a durable function which calls my Sql Server VM. The version of Sql Server configured was Sql Server 2017
(Microsoft SQL Server 2017 (RTM-CU13) (KB4466404) - 14.0.3048.4 (X64) Nov 30 2018 12:57:58 Copyright (C) 2017 Microsoft Corporation Standard Edition (64-bit) on Windows Server 2016 Datacenter 10.0 (Build 14393: ) (Hypervisor)

On a hunch I fired up a new VM, this time with Sql Server 2016. And so far my durable function seems to be working fine. I really throttled it because I was worried maybe the number of connections was causing the problem. So the function is running very slowly now but still it has no errors, so far.

mshenoy83 on 27 Feb 2019

@mshenoy83 Sorry it took a bit longer, here goes my dirty hack. In order to use it, please check the "Custom execution strategy" section of this documentation https://docs.microsoft.com/en-us/ef/core/miscellaneous/connection-resiliency

class CustomExecutionStrategy:

using Brulasoft.Inviton.Data.Internal.SQL;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Storage;
using System;
using System.Data.SqlClient;

namespace Brulasoft.Inviton.Data.Internal
{
    /// <summary>
    /// Custom SQL execution
    /// </summary>
    internal class CustomExecutionStrategy : SqlServerRetryingExecutionStrategy
    {
        public CustomExecutionStrategy(ExecutionStrategyDependencies dependencies) : base(dependencies)
        {
        }

        protected override bool ShouldRetryOn(Exception exception)
        {
            if (base.ShouldRetryOn(exception))
            {
                return true;
            }

            if (exception is SqlException sqlException)
            {
                if (SqlRecoveryAttemptor.ShouldTryRecover(exception, ExceptionsEncountered.Count) && Dependencies != null && Dependencies.CurrentDbContext != null)
                {
                    var context = Dependencies.CurrentDbContext.Context;
                    if (context != null)
                    {
                        var sqlConn = context.Database.GetDbConnection() as SqlConnection;
                        if (sqlConn != null)
                        {
                            SqlRecoveryAttemptor.TryRecover(sqlConn, ExceptionsEncountered.Count);
                            return true;
                        }
                    }
                }
            }

            return false;
        }
    }
}

class SqlRecoveryAttemptor

using System;
using System.Collections.Generic;
using System.Data.Common;
using System.Data.SqlClient;
using System.Text;

namespace Brulasoft.Inviton.Data.Internal.SQL
{
    public class SqlRecoveryAttemptor
    {
        public const int RetryCount = 4;


        public static bool ShouldTryRecover(Exception ex, int retryAttemptsCount)
        {
            return retryAttemptsCount < RetryCount && IsMARSError(ex);
        }

        public static bool IsMARSError(Exception ex)
        {
            return ex.Message.Contains("MARS TDS header");
        }

        public static void TryRecover(SqlConnection sqlConn, int retryAttemptsCount)
        {
            if (retryAttemptsCount > 2)
            {
                System.Threading.Thread.Sleep(1500);
            }

            if (sqlConn.State != System.Data.ConnectionState.Closed)
            {
                TryClose(sqlConn);
                TryClearPool(sqlConn);
                TryOpen(sqlConn);
            }
            else
            {
                TryOpen(sqlConn);
                TryClose(sqlConn);
                TryClearPool(sqlConn);
                TryOpen(sqlConn);
                TryClose(sqlConn);
            }
        }

        private static void TryClearPool(SqlConnection conn)
        {
            try
            {
                SqlConnection.ClearPool(conn);
            }
            catch (Exception)
            {
            }
        }

        private static void TryClose(DbConnection conn)
        {
            try
            {
                conn.Close();
            }
            catch (Exception)
            {
            }
        }

        private static void TryOpen(DbConnection conn)
        {
            try
            {
                conn.Open();
            }
            catch (Exception)
            {
            }
        }
    }
}

brunolau on 4 Mar 2019

👍9 👀1 🚀1

+1 Am experiencing the same problem, not sure how to fix it besides repeatedly restarting (based on comments above).

rcvink on 7 Mar 2019

Same issue here since migrating to EF Core from EF6 in Dec 2018. We see this happen randomly on a single server (among many that are hitting the DB). We have gone 10-15 days without seeing it, then it just pops up and doesn't recover until the server is restarted.

Currently using EF Core 2.2.3 on Azure SQL DB

mscrivo on 22 Mar 2019

I second to that - for us it started once we migrated from ASP.NET to ASP.NET Core

brunolau on 22 Mar 2019

Just to chime in, I just experienced this issue in Azure Functions v2 (.Net Core 2.1). I am using EFCore v2.2.3 to connect to Azure SQL. The call I am making to the DB that was spawning the error was as basic as it gets, essentially a SELECT * into a table with 2 rows and 8 columns.

Since my deployment is a little different from the above, maybe this will help:
For this app, I have 3 different deployments that use the same shared library, that contains my EF Core Data Model.

"External" deployment calls the DB very frequently to get basic info and input data. It errored 1,433 times over the past 12 hours (every call, so far as I can tell) on the first, most basic call, stopping the rest. A restart fixed the error, have not seen it in 20 minutes.
"Data Manipulation" deployment calls the DB fairly frequently (100's per hour), ~~but only errored 5 times in 12 hours, no need to restart~~ (I looked closer, there were zero errors (but a separate error that contained the word "mars" - definitely unrelated)).
"API" deployment never errored. It calls the DB on a sporadic, on demand basis.

Up until recently, this app lived on a Sql Server Web Deployment on a Ubuntu VM. I had never seen this particular error (but had another odd SSL error instead that exhibited similar behavior, with restart fixing).

I will also say that at that time I was using EF Core 2.1.4 as well, and only recently upgraded to 2.2.3.

Connection string is the same for all 3 deployments, the MARS section is: MultipleActiveResultSets=False;

It appears to have started during a period of high DTU use (about 40 min at 90%+). This is not an infrequent occurrence, however, and has actually happened far less over the past day.

Hopefully that helps.

dkershner6 on 30 Mar 2019

As recently announced in the .NET Blog, focus on new SqlClient features an improvements is moving to the new Microsoft.Data.SqlClient package. For this reason, we are moving this issue to the new repo at https://github.com/dotnet/SqlClient. We will still use https://github.com/dotnet/corefx to track issues on other providers like System.Data.Odbc and System.Data.OleDB, and general ADO.NET and .NET data access issues.

divega on 16 May 2019

We have same problem for the following run time env, currently solved the problem solution provided by @brunolau

Application Server
Microsoft Windows Server 2016
.NET Core runtimes installed:
Dotnet core runtime 2.1.5
Database
Microsoft SQL Server 2017

Application
Microsoft.EntityFrameworkCore 2.1.4
Microsoft.AspNetCore.App 2.1.5

nurhat on 21 May 2019

@AfsanehR, @David-Engel we are now hitting this exception with an equivalent stack trace with Microsoft.Data.SqlClient while running our EF Core tests: https://github.com/aspnet/EntityFrameworkCore/issues/16218.

It is hard to isolate a repro (e.g. my understanding is that it only occurs in a complete test run, but not if we run specific tests individually) but it is possible that what we have is enough to help make progress with the investigation.

Any chance that this can be prioritized for 1.0? It seems many customers have reported it already.

cc @smitpatel, @ajcvickers

divega on 24 Jun 2019

👍2

@divega I agree that it would be good to figure this out for 1.0. We'll see if we can fit it in.

David-Engel on 28 Jun 2019

@David-Engel thanks. Let us know if we can somehow help with the repro (even if it is not isolated).

divega on 28 Jun 2019

We ran into this issue using Entity Framework Core 2.2.4. Also using Azure SQL.

Our code that produced this problem was roughly as followed:

var queryable = dbContext.Set<SomeEntity>()
  .Where(e => e.SomeBit = false);

if (command.Filter != null)
    queryable = queryable.Where(command.Filter);

var subQuery = queryable
  .GroupBy(e => e.ExternalReference, (key, elements) => new
  {
    ExternalReference = key,
    Created = elements.Max(e => e.Created)
  })
  .OrderByDescending(q => q.Created)
  .Skip(skip)
  .Take(query.Size);

queryable = dbContext.Set<SomeEntity>()
  .Join(subQuery,
    e => new { e.ExternalReference, e.Created },
    e => new { e.ExternalReference, e.Created },
    (e, _) => e)
  .OrderByDescending(q => q.Created);

var result = await queryable.ToListAsync();
````

The query that is produced by EF Core is:
```sql
SELECT [r].[Id], [r].[Created], [r].[Deleted], [r].[ExternalReference], [r].[SomeId]
FROM [SomeEntity] AS [r]
INNER JOIN (
    SELECT [r0].[ExternalReference], MAX([r0].[Created]) AS [Created]
    FROM [SomeEntity] AS [r0]
    WHERE ([r0].[SomeBit] = 0) AND ([r0].[SomeId] = @__someIdId_0)
    GROUP BY [r0].[ExternalReference]
    ORDER BY [Created]
    OFFSET @__p_1 ROWS FETCH NEXT @__p_2 ROWS ONLY
) AS [t] ON ([r].[ExternalReference] = [t].[ExternalReference]) AND ([r].[Created] = [t].[Created])

The idea here is that we take the newest item for every external reference and page these results.

This works without any problems UNTIL I hit the last page with incomplete page (fetch 20 get 19).
Then I either get a query timeout or the "MARS TDS header contained errors" error.

The solution somewhat puzzled me, but it works...

Where we join our dbset with the subquery we get the DbSet again:

...
queryable = dbContext.Set<SomeEntity>()
  .Join(subQuery,
    e => new { e.ExternalReference, e.Created },
...

When we reuse the DbSet that is acquired on line 1 this works without any issues:

...
queryable = queryable
  .Join(subQuery,
    e => new { e.ExternalReference, e.Created },
...

In the end this is now our working code:
```csharp
var baseQueryable = dbContext.Set();
var queryable = baseQueryable.Where(e => e.SomeBit = false);

if (command.Filter != null)
queryable = queryable.Where(command.Filter);

var subQuery = queryable
.GroupBy(e => e.ExternalReference, (key, elements) => new
{
ExternalReference = key,
Created = elements.Max(e => e.Created)
})
.OrderByDescending(q => q.Created)
.Skip(skip)
.Take(query.Size);

queryable = baseQueryable
.Join(subQuery,
e => new { e.ExternalReference, e.Created },
e => new { e.ExternalReference, e.Created },
(e, _) => e)
.OrderByDescending(q => q.Created);

var result = await queryable.ToListAsync();
````

What I find weird here is that the same query is produced and that it only causes problems on the last page of data.

Hope this helps in finding the problem.

avaneerd on 16 Jul 2019

👍1

We had the same problem as well. We recorded thousands of these errors with no real solution. To point out, we aren't on Azure SQL Server and we aren't using EF. As it turns out the issue to our problem was actually the web server.

We were able to correlate this to traffic because around around 10:30 - 11:00 am (on random days, omitting weekends because our traffic dramatically drops off during the weekend) we'd get these errors. As it turns our, we reviewed our GA reports and compared them to our error handling data and low and behold they matched fairly well (not perfectly though).

Our production web server was running about 8GB of RAM but this wasn't enough to compensate for the amount of traffic we were getting.

We doubled the memory and boom, all the issues went away no more MARS errors.

Again, this may not have any relation to anyone whose posted so far but I found this thread via a Google search so it might help someone whose going through this right now.

Hope it helps someone,

~~Cheers.~~

See new post for clarification.

c4rbon-c0py on 30 Jul 2019

👍1

This has become a major issue in my application. Using 2.2.6 of EntityFrameworkCore. When this error occurs, it is catastrophic and requires a full application domain reset and/or process killing.

Hosting in IIS on a Windows Server and backend is dedicated Sql Server 2016 running on separate VM.

This error occurs during lower traffic times as @c4rbon-c0py has mentioned.

This error can not be reproduce reliably, it is intermittent and apparently random, does not occur in one place, and affects all command paths from what I can see.

Turning on/off MARS on connection string does not change the occurrence of this exception.

I have NO IDEA WHAT TO DO ABOUT THIS. Please help.

CALL STACK:

System.Data.SqlClient.SqlException (0x80131904): 
The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.     
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)     
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)     
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)     
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)     
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()     at System.Data.SqlClient.SqlDataReader.get_MetaData()     
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)     
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)     
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite, String method)     
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)     
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)     
at System.Data.Common.DbCommand.ExecuteReader()     at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.Execute(IRelationalConnection connection, DbCommandMethod executeMethod, IReadOnlyDictionary`2 parameterValues)     
at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.ExecuteReader(IRelationalConnection connection, IReadOnlyDictionary`2 parameterValues)     
at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.BufferlessMoveNext(DbContext _, Boolean buffer)     
at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)     
at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()     
at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)     
at lambda_method(Closure )     
at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ResultEnumerable`1.GetEnumerator()     
at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider._TrackEntities[TOut,TIn](IEnumerable`1 results, QueryContext queryContext, IList`1 entityTrackingInfos, IList`1 entityAccessors)+MoveNext()     
at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()     
at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)     at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)     
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass15_1`1.<CompileQueryCore>b__0(QueryContext qc)     
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)     
at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.Execute[TResult](Expression expression)     
at System.Linq.Queryable.FirstOrDefault[TSource](IQueryable`1 source, Expression`1 predicate)     
at [Application Call Stack - Varies and isn't consistent as to the source]  
ClientConnectionId:323d9c41-0c45-4ab1-9dc9-b2c201c7546e  Error Number:4011,State:3,Class:16

EdonGar on 5 Aug 2019

@avaneerd Can you package your sample code into a complete solution and share it? Not being terribly familiar with EF, it would take me some time to assemble that into something I could debug.

CC @divega

David-Engel on 6 Aug 2019

Here you go: https://github.com/avaneerd/mars-problem-repro
Unfortunately I cannot seem to reproduce the problem in isolation...
This was tested against an Azure SQL S2 database.

This matches our real-world scenario and I'm not sure why I can't reproduce it.
I hope it contributes to finding the problem at least :-/

avaneerd on 6 Aug 2019

I have experienced this issue twice in production over the last 6-8 months.

Our workaround is to perform an App Pool Recycle (restart does not work).

It seems like this occurs a day after server patching.

EDIT:
Database VM: SQL Server 2014 SP3
Windows Server VM: ASP.NET Core 2.2.6 hosted within IIS out of process

coolhome on 26 Aug 2019

I wanted to follow up with some additional information and to correct some bad/false advice I had originally given on my previous post.

Turns out that yes, throwing more resources at our server did help "hide" the problem but shortly after we added those resources, the MARS errors returned (given, with less frequency).

After some continued research on other threads I found the solution to our problem in our "async/await" code. Unfortunately I don't have the links to share but a few of them led me to this belief.

Now, I can't say for 100% fact that this fixed our problem since our web application is seasonal and at the moment our traffic is fairly low but I refactored all of our APIs and DAL to remove any reference to async/await code.

Just before our seasonally traffic dropped off I made these changes and it seemed to have help but again, I'm not truly 100% sure.

Since my original post I made this refactor about a week or 2 after and from that point until today we've had 0 MARS errors tracked.

So again, my apologies for throwing any off the trail on this.

Good luck,

Cheers!

c4rbon-c0py on 4 Sep 2019

We are hitting this on our functional tests pretty consistently now. @David-Engel, @cheenamalhotra maybe you can run the EF Core functional tests as a way to diagnose this.

cc @Wraith2

divega on 16 Sep 2019

Any particular test or is it pseudo random?

Wraith2 on 16 Sep 2019

How do we run tests against an Azure SQL DB?

ErikEJ on 16 Sep 2019

Just use a connection string that points at an azure database and enable encryption. there's nothing special about azure at a protocol level (there is some extra connection resilience stuff for it but that's at socket level).

Wraith2 on 16 Sep 2019

Yes, I know that :smile: - I meant running the EF Core tests against Azure SQL DB

ErikEJ on 16 Sep 2019

@Wraith2 - It is one particular test which always fails when running whole assembly for tests.

smitpatel on 16 Sep 2019

can you point me at it?

Wraith2 on 16 Sep 2019

https://github.com/aspnet/EntityFrameworkCore
Branch: master (where the test is not skipped yet)
Building steps https://github.com/aspnet/EntityFrameworkCore/wiki/getting-and-building-the-code
Test name: Average_on_nav_subquery_in_projection
Stacktrace & details: https://github.com/aspnet/EntityFrameworkCore/issues/17775
Running all of SqlServer.FunctionalTests causes error. Running just the test does not cause it. I have seen error in VS test runner and command line (build.cmd -test in repo root or dotnet test) both.
It is pretty consistent that I have seen it 4 times in a row in a PR validation build.

smitpatel on 16 Sep 2019

I can't get the repository building, lots of errors about duplicate attributes. I'm also in the same position as others here that don't really work with EF, I'm going to need to strip away the abstraction layer and get it into an sql client only reproduction. EF can't be doing anything at the protocol level to cause the issue so it seems safe to say if you can get me an sql only repro I can loop until it fails I'll investigate as best I can.

Do your test run and fail on a specific platform? The managed network implementation is entirely different to the windows one and a failure in one may not occur on the other. If it's windows only it might be an issue inside sni.dll that we can't fix here. if it's shared or unix only we can address it.

I also suspect that it's a failure of outgoing messages, we're sending malformed data at the server which is interpreting it as mars because that's what it looks like. The specific error message we're seeing is coming back from the server in the run loop so we're likely being sent the error and passing it back to the caller not having an error occur in the client network layer (I'd expect TdsParser or the state object to be on the stack in that case)

Wraith2 on 17 Sep 2019

It happens on windows for sure. We are not sure about other platforms as we don't have testing for SqlServer on other platforms.

I can imagine wanting to strip out EF from picture since it won't be issue in EF. But at the same time, the EFCore runs more than 15,000 tests against SqlServer with variety of scenario. It would be hard to create such simulation without EF easily. So honestly I cannot say how to proceed with that. We would be willing to help out working with EF needed.
cc: @divega

smitpatel on 17 Sep 2019

Knowing it's on windows cuts out a lot of potential problem areas. Since it's one test that's failing intermittently can you try looping that one test and see if it fails reliably with just that test?

Wraith2 on 17 Sep 2019

@Wraith2 Re

I can't get the repository building, lots of errors about duplicate attributes. I'm also in the same position as others here that don't really work with EF, I'm going to need to strip away the abstraction layer and get it into an sql client only reproduction.

Maybe we can help you with that. Building EF Core can't be that hard. We do it every day :smile:

divega on 17 Sep 2019

sure, but it's hard to type the word build wrong and they look like real errors to me...

E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\Microsoft.Data.Sqlite.Core\Debug\netstandard2.0\Microsoft.Data.Sqlite.Core.AssemblyInfo.cs(10,12): error CS0579: Duplicate 'System.Reflection.AssemblyCompanyAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\Microsoft.Data.Sqlite.Core.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(11,12): error CS0579: Duplicate 'System.Reflection.AssemblyCompanyAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Analyzers\Debug\netstandard1.3\EFCore.Analyzers.AssemblyInfo.cs(14,12): error CS0579: Duplicate 'System.Reflection.AssemblyCompanyAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Analyzers\EFCore.Analyzers.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(12,12): error CS0579: Duplicate 'System.Reflection.AssemblyConfigurationAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\Microsoft.Data.Sqlite.Core\Debug\netstandard2.0\Microsoft.Data.Sqlite.Core.AssemblyInfo.cs(11,12): error CS0579: Duplicate 'System.Reflection.AssemblyConfigurationAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\Microsoft.Data.Sqlite.Core.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Analyzers\Debug\netstandard1.3\EFCore.Analyzers.AssemblyInfo.cs(15,12): error CS0579: Duplicate 'System.Reflection.AssemblyConfigurationAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Analyzers\EFCore.Analyzers.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(13,12): error CS0579: Duplicate 'System.Reflection.AssemblyCopyrightAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\Microsoft.Data.Sqlite.Core\Debug\netstandard2.0\Microsoft.Data.Sqlite.Core.AssemblyInfo.cs(12,12): error CS0579: Duplicate 'System.Reflection.AssemblyCopyrightAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\Microsoft.Data.Sqlite.Core.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Analyzers\Debug\netstandard1.3\EFCore.Analyzers.AssemblyInfo.cs(16,12): error CS0579: Duplicate 'System.Reflection.AssemblyCopyrightAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Analyzers\EFCore.Analyzers.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(14,12): error CS0579: Duplicate 'System.Reflection.AssemblyDescriptionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\Microsoft.Data.Sqlite.Core\Debug\netstandard2.0\Microsoft.Data.Sqlite.Core.AssemblyInfo.cs(13,12): error CS0579: Duplicate 'System.Reflection.AssemblyDescriptionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\Microsoft.Data.Sqlite.Core.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Analyzers\Debug\netstandard1.3\EFCore.Analyzers.AssemblyInfo.cs(17,12): error CS0579: Duplicate 'System.Reflection.AssemblyDescriptionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Analyzers\EFCore.Analyzers.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(15,12): error CS0579: Duplicate 'System.Reflection.AssemblyFileVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Analyzers\Debug\netstandard1.3\EFCore.Analyzers.AssemblyInfo.cs(18,12): error CS0579: Duplicate 'System.Reflection.AssemblyFileVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Analyzers\EFCore.Analyzers.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\Microsoft.Data.Sqlite.Core\Debug\netstandard2.0\Microsoft.Data.Sqlite.Core.AssemblyInfo.cs(24,12): error CS0579: Duplicate 'System.Reflection.AssemblyFileVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\Microsoft.Data.Sqlite.Core.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Analyzers\Debug\netstandard1.3\EFCore.Analyzers.AssemblyInfo.cs(19,12): error CS0579: Duplicate 'System.Reflection.AssemblyInformationalVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Analyzers\EFCore.Analyzers.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(16,12): error CS0579: Duplicate 'System.Reflection.AssemblyInformationalVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(11,12): error CS0579: Duplicate 'System.Reflection.AssemblyCompanyAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(17,12): error CS0579: Duplicate 'System.Reflection.AssemblyProductAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\Microsoft.Data.Sqlite.Core\Debug\netstandard2.0\Microsoft.Data.Sqlite.Core.AssemblyInfo.cs(25,12): error CS0579: Duplicate 'System.Reflection.AssemblyInformationalVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\Microsoft.Data.Sqlite.Core.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Analyzers\Debug\netstandard1.3\EFCore.Analyzers.AssemblyInfo.cs(20,12): error CS0579: Duplicate 'System.Reflection.AssemblyProductAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Analyzers\EFCore.Analyzers.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(12,12): error CS0579: Duplicate 'System.Reflection.AssemblyConfigurationAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\Microsoft.Data.Sqlite.Core\Debug\netstandard2.0\Microsoft.Data.Sqlite.Core.AssemblyInfo.cs(26,12): error CS0579: Duplicate 'System.Reflection.AssemblyProductAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\Microsoft.Data.Sqlite.Core.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Analyzers\Debug\netstandard1.3\EFCore.Analyzers.AssemblyInfo.cs(21,12): error CS0579: Duplicate 'System.Reflection.AssemblyTitleAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Analyzers\EFCore.Analyzers.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(13,12): error CS0579: Duplicate 'System.Reflection.AssemblyCopyrightAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(18,12): error CS0579: Duplicate 'System.Reflection.AssemblyTitleAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\Microsoft.Data.Sqlite.Core\Debug\netstandard2.0\Microsoft.Data.Sqlite.Core.AssemblyInfo.cs(27,12): error CS0579: Duplicate 'System.Reflection.AssemblyTitleAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\Microsoft.Data.Sqlite.Core.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Analyzers\Debug\netstandard1.3\EFCore.Analyzers.AssemblyInfo.cs(22,12): error CS0579: Duplicate 'System.Reflection.AssemblyVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Analyzers\EFCore.Analyzers.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(14,12): error CS0579: Duplicate 'System.Reflection.AssemblyDescriptionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\Microsoft.Data.Sqlite.Core\Debug\netstandard2.0\Microsoft.Data.Sqlite.Core.AssemblyInfo.cs(28,12): error CS0579: Duplicate 'System.Reflection.AssemblyVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\Microsoft.Data.Sqlite.Core.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Analyzers\Debug\netstandard1.3\EFCore.Analyzers.AssemblyInfo.cs(23,12): error CS0579: Duplicate 'System.Resources.NeutralResourcesLanguageAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Analyzers\EFCore.Analyzers.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(19,12): error CS0579: Duplicate 'System.Reflection.AssemblyVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(15,12): error CS0579: Duplicate 'System.Reflection.AssemblyFileVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\Microsoft.Data.Sqlite.Core\Debug\netstandard2.0\Microsoft.Data.Sqlite.Core.AssemblyInfo.cs(29,12): error CS0579: Duplicate 'System.Resources.NeutralResourcesLanguageAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\Microsoft.Data.Sqlite.Core.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(20,12): error CS0579: Duplicate 'System.Resources.NeutralResourcesLanguageAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(16,12): error CS0579: Duplicate 'System.Reflection.AssemblyInformationalVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(11,12): error CS0579: Duplicate 'System.Reflection.AssemblyCompanyAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(17,12): error CS0579: Duplicate 'System.Reflection.AssemblyProductAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(18,12): error CS0579: Duplicate 'System.Reflection.AssemblyTitleAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(12,12): error CS0579: Duplicate 'System.Reflection.AssemblyConfigurationAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(19,12): error CS0579: Duplicate 'System.Reflection.AssemblyVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(13,12): error CS0579: Duplicate 'System.Reflection.AssemblyCopyrightAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(20,12): error CS0579: Duplicate 'System.Resources.NeutralResourcesLanguageAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(14,12): error CS0579: Duplicate 'System.Reflection.AssemblyDescriptionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(11,12): error CS0579: Duplicate 'System.Reflection.AssemblyCompanyAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(15,12): error CS0579: Duplicate 'System.Reflection.AssemblyFileVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(12,12): error CS0579: Duplicate 'System.Reflection.AssemblyConfigurationAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(16,12): error CS0579: Duplicate 'System.Reflection.AssemblyInformationalVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(13,12): error CS0579: Duplicate 'System.Reflection.AssemblyCopyrightAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(17,12): error CS0579: Duplicate 'System.Reflection.AssemblyProductAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(14,12): error CS0579: Duplicate 'System.Reflection.AssemblyDescriptionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(18,12): error CS0579: Duplicate 'System.Reflection.AssemblyTitleAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(15,12): error CS0579: Duplicate 'System.Reflection.AssemblyFileVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(19,12): error CS0579: Duplicate 'System.Reflection.AssemblyVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(16,12): error CS0579: Duplicate 'System.Reflection.AssemblyInformationalVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(20,12): error CS0579: Duplicate 'System.Resources.NeutralResourcesLanguageAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(17,12): error CS0579: Duplicate 'System.Reflection.AssemblyProductAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\net461\ef.AssemblyInfo.cs(10,12): error CS0579: Duplicate 'System.Reflection.AssemblyCompanyAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(18,12): error CS0579: Duplicate 'System.Reflection.AssemblyTitleAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\net461\ef.AssemblyInfo.cs(11,12): error CS0579: Duplicate 'System.Reflection.AssemblyConfigurationAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(19,12): error CS0579: Duplicate 'System.Reflection.AssemblyVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\net461\ef.AssemblyInfo.cs(12,12): error CS0579: Duplicate 'System.Reflection.AssemblyCopyrightAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
obj\x86\Debug\net461\ef.AssemblyInfo.cs(20,12): error CS0579: Duplicate 'System.Resources.NeutralResourcesLanguageAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\net461\ef.AssemblyInfo.cs(13,12): error CS0579: Duplicate 'System.Reflection.AssemblyDescriptionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(10,12): error CS0579: Duplicate 'System.Reflection.AssemblyCompanyAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\net461\ef.AssemblyInfo.cs(14,12): error CS0579: Duplicate 'System.Reflection.AssemblyFileVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(11,12): error CS0579: Duplicate 'System.Reflection.AssemblyConfigurationAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\net461\ef.AssemblyInfo.cs(15,12): error CS0579: Duplicate 'System.Reflection.AssemblyInformationalVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(12,12): error CS0579: Duplicate 'System.Reflection.AssemblyCopyrightAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\net461\ef.AssemblyInfo.cs(16,12): error CS0579: Duplicate 'System.Reflection.AssemblyProductAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(13,12): error CS0579: Duplicate 'System.Reflection.AssemblyDescriptionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\net461\ef.AssemblyInfo.cs(17,12): error CS0579: Duplicate 'System.Reflection.AssemblyTitleAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(14,12): error CS0579: Duplicate 'System.Reflection.AssemblyFileVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\net461\ef.AssemblyInfo.cs(18,12): error CS0579: Duplicate 'System.Reflection.AssemblyVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(15,12): error CS0579: Duplicate 'System.Reflection.AssemblyInformationalVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\net461\ef.AssemblyInfo.cs(19,12): error CS0579: Duplicate 'System.Resources.NeutralResourcesLanguageAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(16,12): error CS0579: Duplicate 'System.Reflection.AssemblyProductAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(17,12): error CS0579: Duplicate 'System.Reflection.AssemblyTitleAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(18,12): error CS0579: Duplicate 'System.Reflection.AssemblyVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\ef\Debug\netcoreapp2.0\ef.AssemblyInfo.cs(19,12): error CS0579: Duplicate 'System.Resources.NeutralResourcesLanguageAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\ef\ef.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Abstractions\Debug\netstandard2.1\EFCore.Abstractions.AssemblyInfo.cs(14,12): error CS0579: Duplicate 'System.Reflection.AssemblyCompanyAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Abstractions\EFCore.Abstractions.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Abstractions\Debug\netstandard2.1\EFCore.Abstractions.AssemblyInfo.cs(15,12): error CS0579: Duplicate 'System.Reflection.AssemblyConfigurationAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Abstractions\EFCore.Abstractions.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Abstractions\Debug\netstandard2.1\EFCore.Abstractions.AssemblyInfo.cs(16,12): error CS0579: Duplicate 'System.Reflection.AssemblyCopyrightAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Abstractions\EFCore.Abstractions.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Abstractions\Debug\netstandard2.1\EFCore.Abstractions.AssemblyInfo.cs(17,12): error CS0579: Duplicate 'System.Reflection.AssemblyDescriptionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Abstractions\EFCore.Abstractions.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Abstractions\Debug\netstandard2.1\EFCore.Abstractions.AssemblyInfo.cs(19,12): error CS0579: Duplicate 'System.Reflection.AssemblyFileVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Abstractions\EFCore.Abstractions.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Abstractions\Debug\netstandard2.1\EFCore.Abstractions.AssemblyInfo.cs(20,12): error CS0579: Duplicate 'System.Reflection.AssemblyInformationalVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Abstractions\EFCore.Abstractions.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Abstractions\Debug\netstandard2.1\EFCore.Abstractions.AssemblyInfo.cs(21,12): error CS0579: Duplicate 'System.Reflection.AssemblyProductAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Abstractions\EFCore.Abstractions.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Abstractions\Debug\netstandard2.1\EFCore.Abstractions.AssemblyInfo.cs(22,12): error CS0579: Duplicate 'System.Reflection.AssemblyTitleAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Abstractions\EFCore.Abstractions.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Abstractions\Debug\netstandard2.1\EFCore.Abstractions.AssemblyInfo.cs(23,12): error CS0579: Duplicate 'System.Reflection.AssemblyVersionAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Abstractions\EFCore.Abstractions.csproj]
E:\Programming\csharp7\EntityFrameworkCore\artifacts\obj\EFCore.Abstractions\Debug\netstandard2.1\EFCore.Abstractions.AssemblyInfo.cs(24,12): error CS0579: Duplicate 'System.Resources.NeutralResourcesLanguageAttribute' attribute [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.Abstractions\EFCore.Abstractions.csproj]

Wraith2 on 17 Sep 2019

@Wraith2 - Please do git clean before running build. It seems that you are using existing repo from past and VS had issues like that if you don't clean solution. I would recommend starting afresh.

smitpatel on 17 Sep 2019

ok, clean did nothing so I razed the directory with fire and started again. test run now (without me specifying a connection string which is very neat) so how do I test only the sql server tests?

Wraith2 on 17 Sep 2019

Make sure that test is enabled.
https://github.com/aspnet/EntityFrameworkCore/blob/edd48434948dc97a6f5b53efe388405e14339c9f/test/EFCore.Specification.Tests/Query/AsyncSimpleQueryTestBase.cs#L138-L150

run build.cmd on command line on repo root.
then cd into testEFCore.SqlServer.FunctionalTests
run ..\..\.dotnet\dotnet test

smitpatel on 17 Sep 2019

@smitpatel how do we configure the tests to run against SQL Azure DB?

ErikEJ on 17 Sep 2019

Found some info: https://github.com/aspnet/EntityFrameworkCore/pull/6625

cheenamalhotra on 17 Sep 2019

The errors we are seeing now are on localdb, so SQL Azure is _not_ a requirement for this repro anymore.

To configure connectionstring in EF tests (Sql Azure or not) just update connectionstring in this file https://github.com/aspnet/EntityFrameworkCore/blob/release/3.1/test/EFCore.SqlServer.FunctionalTests/config.json

smitpatel on 17 Sep 2019

You know what isn't fun? trying to pick apart a complex build system. I can replicate the error but trying to get rid of the 14000 tests I don't care about is annoying me. I can't open the project because the solution is pinned to a dotnet version I don't have available so I just keep having to try and remove a file then do a full rebuild to see if anything breaks. This doesn't feel even remotely productive.

Wraith2 on 17 Sep 2019

@Wraith2 - If you followed building wiki guide for EF Core and still facing any issue then let us know. We have a wiki page describing steps for specific reason that we use pre-release dependency to build a pre-release product. I can understand frustration of having to pick apart 14,000 tests but at the same time, if the repro does not occur otherwise what choice do we have.

smitpatel on 17 Sep 2019

How do I turn off the strong name requirement for the references so I can use my local SqlClient build? I've got it narrowed to 2 tests and now I want to debug into what's going on. I've added my local sqlclient package directory to the nuget config and changed the EFCore.SqlServer.csproj file to have an explicit version of my package, but my package isn't signed.

E:\Programming\csharp7\EntityFrameworkCore\test\EFCore.SqlServer.FunctionalTests>..\..\.dotnet\dotnet.exe test
CSC : error CS8002: Referenced assembly 'Microsoft.Data.SqlClient, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' does not have a strong name. [E:\Programming\csharp7\EntityFrameworkCore\src\EFCore.SqlServer\EFCore.SqlServer.csproj]

Wraith2 on 17 Sep 2019

@bricelam - Can you advice on strong naming issue above?

smitpatel on 17 Sep 2019

The cause of the problem is in the test ToListAsync_can_be_canceled, it breaks something which then shows the next time you run a query. The Average_on_nav_subquery_in_projection is just a victim because it happens to be next to the other test in the definition order and presumably the test runner walks backwards in vtable order, not that It matters.

A single breaking test is this:

[ConditionalFact]
public async Task BreakTDSStream()
{
    for (var i = 0; i < 10; i++)
    {
        // without fix, this usually throws within 2 or three iterations

        using (var context = CreateContext())
        {
            var tokenSource = new CancellationTokenSource();
            var query = context.Employees.AsNoTracking().ToListAsync(tokenSource.Token);
            tokenSource.Cancel();
            try
            {
                await query;
            }
            catch (Exception)
            {
            }
        }
    }
    using (var context = CreateContext())
    {
        var results2
            = await context.Customers.Select(
                    c => new { Ave = c.Orders.Average(o => o.OrderID) })
                .ToListAsync();
    }
}

Which it seems pretty likely the cancellation is causing a problem somehow and the query outside the loop pulls an invalid connection from the pool and tries to use it.

Observations:

if you try to run ExecuteNonQueryAsync with a cancelled token it doesn't even go as far as checking if there's a valid open connection ( I forgot a connection string in a test project and was surprised it didn't throw) so I think it's unlikely that it's the read, I think it's the OpenAsync with cancelled token but I can't find where the connection is actually created/opened in entity framework.
- it is speed/timing related because if you attempt to step through it sometimes it won't occur. This suggests that an unfinished connection is returned to the pool somehow to me.
- We're absolutely going to need the EF team to help by removing all these layers of abstraction to pare it down to a sequence of operations directly on SqlClient. Async lifecycle management is bad enough without all the extra baggage on top of it.
I believe the query being run is irrelevant to the problem

Wraith2 on 17 Sep 2019

👍7

@Wraith2 - Just to confirm, the test you posted would generate the error when running alone? I can help out removing EF from the picture for that.

smitpatel on 17 Sep 2019

Yes. That test alone reproduced the tds error we're looking for 100% of the time against my local sql instance.

Wraith2 on 18 Sep 2019

❤4

The simplest way to address the strong name issue is to strong name your local build of Microsoft.Data.SqlClient (any key will do). You could also try setting <SignAssembly>False</SignAssembly> in EF Core's root Directory.Build.props, but you may have to change a few other things too (e.g. InternalsVisibleTo attributes)

bricelam on 18 Sep 2019

I can help out removing EF from the picture for that.

Any movement on this?

Wraith2 on 19 Sep 2019

It will be a while before I can get back to this. Give me about a week.

smitpatel on 20 Sep 2019

Can any body help me finding the resolution for this issue?

RamBoddapati on 1 Oct 2019

We're experiencing this same issue with one of our .NET Core APIs as well. We've had two incidents in the last 5 days. As others have previously noted, this puts the application in an unrecoverable state and the only solution is to recycle its app pool.

Host (useful for support):
Version: 2.2.5
Commit: 0a3c9209c0

.NET Core SDKs installed:
No SDKs were found.

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.2 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.1 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.5 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.2 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.1 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.5 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.2 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.2.1 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.2.5 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]

Nick-Madden on 3 Oct 2019

@smitpatel do you have any updates on the standalone repro?

cheenamalhotra on 8 Oct 2019

@cheenamalhotra - I was able to get the error. But after it errored once, it was corrupting my localdb instance that I could not run any other query again. I had to delete and recreate the instance. I will try again soon.

smitpatel on 8 Oct 2019

That's scary!
You may try to wrap up a Dockerfile with sample app to avoid corrupting SQL instances.

cheenamalhotra on 8 Oct 2019

That's scary!

+1 Perhaps we should continue this conversation in private until we fully understand the impact of this bug...

bricelam on 8 Oct 2019

👍1

I was able to get the error. But after it errored once, it was corrupting my localdb instance

I ran my repo several dozen times and I've seen no adverse effect on my instance. So perhaps it's localdb only or can cause problems with very fast repro. Probably worth treating carefully as @bricelam suggests though.

Wraith2 on 8 Oct 2019

@Wraith2 do you know if this is still an issue on .NET Core 3.0? Will upgrading solve the problem?

michaeltnguyen on 11 Oct 2019

@Wraith2 do you know if this is still an issue on .NET Core 3.0? Will upgrading solve the problem?

It is. I just finished upgrading our entire codebase to .net core 3 and EF core 3 and pushed it out to QA, and low and behold, got the error in the new Microsoft.Data.SqlClient. Was rather disappointed.

EDIT: After a few days of observing the EF Core 3/Microsoft.Data.SqlClient changes on our QA systems, we're seeing this error a lot. Averaging twice a day so far. Previously, we never saw it on QA system (because load was so low I presume) and only saw it in production maybe once every 2 months or so. The only saving grace with the new code is that it seems to be able to self recover without having to restart the app/reboot.

To give some more details about our environment:

This is a single console app targetting netcoreapp3.0 running on Windows Server 2019 that makes heavy use of async/await to run many "worker" type jobs. ie. grabbing messages from a message queue and processing them in parallel, which often involves work against our Azure SQL DB instance.
- So far we have not yet seen this happen on our web heads which are running IIS on Windows Server 2019 proxying requests to an asp.net core 3 app.

EDIT 2: After deploying EF Core 3 upgrade to production, we started getting 10's of thousands of these exceptions per hour. Luckily, it was contained to our worker tier, which is pretty resilient and we didn't have to do much to mitigate, it just made our monitoring a mess. I had to deploy @brunolau's wonderful solution and now we're only seeing 10's per hour before it resets the connection pool.

It's really curious that this became a much bigger problem with Microsoft.Data.SqlClient and EF Core 3.

mscrivo on 12 Oct 2019

😕5

Does it make any difference whether MultipleActiveResultSets is set to true or false?

JohnnyFun on 29 Oct 2019

Does it make any difference whether MultipleActiveResultSets is set to true or false?

It does not appear to make a difference if MARS is enabled or not.

David-Engel on 30 Oct 2019

I'm really sad😔 that the issue happened 2 times at our production environment, at today and Sep. 29.

I'm afraid that it will happen again. So I will give up EF Core, and use other technology to access DB, until this issue is resolved. Actually EFCore is convient, I want to use it at work.
Please resolve this issue ASAP.

yayayahei on 8 Nov 2019

@yaeldekel the issue isn't in EF, it's the database driver (which is why it's in the repository) so you'll see it whether you use EF or not. If EF is helpful to your work it would be pointless for you to drop it in favour of direct access.

Wraith2 on 8 Nov 2019

I just got this for the first time with EFCore 3 and WebAPI on the most innocuous query - to load a single user with Identity. Restarting my app fixed it but it is quite worrying.

I am using connection pooling and wonder if that has anything to do with this.

Failed executing DbCommand (79ms) [Parameters=[@__loginProvider_0='?' (Size = 450), @__providerKey_1='?' (Size = 450)], CommandType='Text', CommandTimeout='30'] SELECT TOP(2) [a].[LoginProvider], [a].[ProviderKey], [a].[ProviderDisplayName], [a].[UserId] FROM [AspNetUserLogins] AS [a] WHERE (([a].[LoginProvider] = @__loginProvider_0) AND @__loginProvider_0 IS NOT NULL) AND (([a].[ProviderKey] = @__providerKey_1) AND @__providerKey_1 IS NOT NULL) fail: Microsoft.EntityFrameworkCore.Query[10100] An exception occurred while iterating over the results of a query for context type 'RR.EF.Identity.ApplicationDbContext'. Microsoft.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.

simeyla on 25 Nov 2019

These errors have completely disappeared for us after upgrading to EF Core 3.1/Microsoft.Data.SqlClient 1.1.0.

mscrivo on 6 Dec 2019

👀5 👍5 🎉1

That is surprising and a happy news! 🎉

I'll close this issue for now, if anyone experiences this issue again, an upgrade to Microsoft.Data.SqlClient 1.1.0 would be helpful. If still issue occurs, please let us know with a reproducible app/code.

cheenamalhotra on 9 Dec 2019

Sounds good to me, but to be fair, it may not be completely gone. This is only 5 days of logs now where we haven't seen the issue, whereas it was occurring thousands of times per hour on 3.0/1.0, so definitely a positive sign.

However, when were on EF Core 2.2 and the old SqlDataClient, we saw this issue maybe once every month or two. So that may still be the case, who knows, only time will tell.

mscrivo on 9 Dec 2019

👍4

@cheenamalhotra I agree with @mscrivo. Closing this seems premature. Not observing it for a few days is not the same as proving that the issue is resolved. (especially because the issue is intermittent and hard to reproduce). When we were on SqlClient 1.0.0, we didn't observe it for 3 months until it suddenly became an issue.

michaeltnguyen on 9 Dec 2019

👍2

No problem!

cheenamalhotra on 9 Dec 2019

👍3

Reporting back that since updating to EFCore 3.1 and SqlClient 1.1, we haven't had any more occurrence as well. But still probably worth investigating it to confirm that is is fixed and not just less likely to happen with the risk to regress with future unrelated changes.

alefranz on 17 Dec 2019

👍2

This issue started appearing to us yesterday on .Net Core 3.0 and Microsoft.Data.SqlClient 1.0.19269.1
After upgrading to .Net Core 3.1 and Microsoft.Data.SqlClient 1.1.0 it is still there.
It is deployed to Azure App Service connecting to Azure Sql Database.
We use Dapper to run queries
The service works fine for some time then starts throwing errors and only restart helps
The error I see in Application Insights is:

Microsoft.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at Microsoft.Data.SqlClient.SqlDataReader.get_MetaData()
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at Microsoft.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader(Boolean isInternal, Boolean forDescribeParameterEncryption)
   at Microsoft.Data.SqlClient.SqlCommand.InternalEndExecuteReader(IAsyncResult asyncResult, Boolean isInternal, String endMethod)
   at Microsoft.Data.SqlClient.SqlCommand.EndExecuteReaderInternal(IAsyncResult asyncResult)
   at Microsoft.Data.SqlClient.SqlCommand.EndExecuteReaderAsync(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)

IvanAlekseev on 18 Dec 2019

@IvanAlekseev

Could you provide a repro if this gets reproduced consistently? We still haven't got a standalone repro that reproduces the error.

cheenamalhotra on 20 Dec 2019

@cheenamalhotra @Wraith2 wrote a test that was able to reproduce the issue 100% of the time in this comment

https://github.com/dotnet/SqlClient/issues/85#issuecomment-532403666

michaeltnguyen on 20 Dec 2019

@michaeltnguyen

As commented by Wraith:

We're absolutely going to need the EF team to help by removing all these layers of abstraction to pare it down to a sequence of operations directly on SqlClient. Async lifecycle management is bad enough without all the extra baggage on top of it.

I believe we have been waiting for a repro without any abstraction layers to understand the flow of APIs that cause this issue. There is not enough clarity with the test case from EF suite.

cheenamalhotra on 20 Dec 2019

gotcha. Does the EF team know that you're waiting for their help? (I missed that in Wraith's earlier comment.) Or should one of us try to create a repro using purely SqlClient?

michaeltnguyen on 20 Dec 2019

Does the EF team know that you're waiting for their help? (I missed that in Wraith's earlier comment.)

I would hope so, but I'm not sure if this is in top of their priorities.

Should one of us try to create a repro using purely SqlClient?

That will be awesome! Highly appreciate it!

cheenamalhotra on 20 Dec 2019

I ran @Wraith2 's test in #85 on the master branch and it no longer reproduces this issue. Unfortunately, it appears that the issue is still happening in the wild, so we're back at square one. =/

michaeltnguyen on 3 Jan 2020

😕1

@michaeltnguyen

Can you upload your app in zip as it is which I can test with different source references of Microsoft.Data.SqlClient? I'm not very used to working with EF Core, and it shall save time if you already have a workable code.

I'd like to give it a try if I can find something.

cheenamalhotra on 3 Jan 2020

@cheenamalhotra Sure, I've uploaded it to my google drive. (~600 MB)

https://drive.google.com/open?id=1n3D9RzSPv1nfLYfyPhXHXcVyBq_3tXP1

It's basically just this repository with Wraith2's test added, and Average_on_nav_subquery_in_projection no longer skipped. The new test is in NorthwindAsyncSimpleQueryTestBase.cs

I'm running it via Visual Studio 2019.

michaeltnguyen on 3 Jan 2020

Oh I guess you've not pulled out the test then.. I thought you've pulled out abstractions.
nvm in that case I'll just clone EF Core and add this test there.

cheenamalhotra on 3 Jan 2020

I first tried to reproduce #85 using just SqlClient and SqlCommand, but I couldn't. So I figured the next step is to try to reproduce using wraith's code verbatim in EF Core. (but the test seems to pass)

Here's my attempt of translating #85 into SqlClient, in case it's useful.

https://gist.github.com/michaeltnguyen/e9710fe6530286ab6363ae5f5b245a5c

michaeltnguyen on 3 Jan 2020

I can reproduce this issue in release/3.0 branch of EF Core with the test from Wraith, but not in master branch. I'm wondering what changed in EF Core that this error can no longer be reproduced.
(I guess I found a repro I can now investigate into, and the issue is reproducible with M.D.S v1.1.0)

https://gist.github.com/michaeltnguyen/e9710fe6530286ab6363ae5f5b245a5c

I somehow cannot reproduce in standalone repro with any driver version.

@smitpatel do you have any updates on this error from EF Core side?

cheenamalhotra on 3 Jan 2020

While debugging this issue, I just happened to change my branch today to @Wraith2's PR https://github.com/dotnet/SqlClient/pull/248 and the PR seems to fix this issue. Reverting my branch to master brings back failures.

The use case that fails gets it's trigger from concurrent runs of CancellationTokenSource().Cancel() where Attention signals are sent to SQL Server, and the PR does seem to work on the same paths by locking Send() to avoid race conditions.

Although the PR still needs some work as some test cases seem to fail, but it's also seems to be making a big difference leading to fix this issue.

@Wraith2 would you like to give it a try and confirm the same? As mentioned above, this error is reproducible with the test posted above and release/3.0 branch of EF Core. Also would appreciate if you can address pipeline failures!

cheenamalhotra on 7 Jan 2020

👍2 🎉1

That's not my PR, its yours so i can't do anything with it.

Wraith2 on 7 Jan 2020

@Wraith2 We got the PR updated, reviewed and merged (was due on us), but on testing with NuGet package I still see the issue with EF Core test, and the error shows up intermittently.

I guess the heavy and slow solution got me confused and accidentally thinking this issue is fixed with the PR, due to it's intermittent nature. But I guess I'll continue with investigations 😞

cheenamalhotra on 8 Jan 2020

It really would be good to get an SqlClient only repro. I'm fairly sure it's a timing issue but without being able to identify where the error starts it's guesswork.

Wraith2 on 8 Jan 2020

@mshenoy83 Sorry it took a bit longer, here goes my dirty hack. In order to use it, please check the "Custom execution strategy" section of this documentation https://docs.microsoft.com/en-us/ef/core/miscellaneous/connection-resiliency

class CustomExecutionStrategy:

using Brulasoft.Inviton.Data.Internal.SQL;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Storage;
using System;
using System.Data.SqlClient;

namespace Brulasoft.Inviton.Data.Internal
{
    /// <summary>
    /// Custom SQL execution
    /// </summary>
    internal class CustomExecutionStrategy : SqlServerRetryingExecutionStrategy
    {
        public CustomExecutionStrategy(ExecutionStrategyDependencies dependencies) : base(dependencies)
        {
        }

        protected override bool ShouldRetryOn(Exception exception)
        {
            if (base.ShouldRetryOn(exception))
            {
                return true;
            }

            if (exception is SqlException sqlException)
            {
                if (SqlRecoveryAttemptor.ShouldTryRecover(exception, ExceptionsEncountered.Count) && Dependencies != null && Dependencies.CurrentDbContext != null)
                {
                    var context = Dependencies.CurrentDbContext.Context;
                    if (context != null)
                    {
                        var sqlConn = context.Database.GetDbConnection() as SqlConnection;
                        if (sqlConn != null)
                        {
                            SqlRecoveryAttemptor.TryRecover(sqlConn, ExceptionsEncountered.Count);
                            return true;
                        }
                    }
                }
            }

            return false;
        }
    }
}

class SqlRecoveryAttemptor

using System;
using System.Collections.Generic;
using System.Data.Common;
using System.Data.SqlClient;
using System.Text;

namespace Brulasoft.Inviton.Data.Internal.SQL
{
    public class SqlRecoveryAttemptor
    {
        public const int RetryCount = 4;


        public static bool ShouldTryRecover(Exception ex, int retryAttemptsCount)
        {
            return retryAttemptsCount < RetryCount && IsMARSError(ex);
        }

        public static bool IsMARSError(Exception ex)
        {
            return ex.Message.Contains("MARS TDS header");
        }

        public static void TryRecover(SqlConnection sqlConn, int retryAttemptsCount)
        {
            if (retryAttemptsCount > 2)
            {
                System.Threading.Thread.Sleep(1500);
            }

            if (sqlConn.State != System.Data.ConnectionState.Closed)
            {
                TryClose(sqlConn);
                TryClearPool(sqlConn);
                TryOpen(sqlConn);
            }
            else
            {
                TryOpen(sqlConn);
                TryClose(sqlConn);
                TryClearPool(sqlConn);
                TryOpen(sqlConn);
                TryClose(sqlConn);
            }
        }

        private static void TryClearPool(SqlConnection conn)
        {
            try
            {
                SqlConnection.ClearPool(conn);
            }
            catch (Exception)
            {
            }
        }

        private static void TryClose(DbConnection conn)
        {
            try
            {
                conn.Close();
            }
            catch (Exception)
            {
            }
        }

        private static void TryOpen(DbConnection conn)
        {
            try
            {
                conn.Open();
            }
            catch (Exception)
            {
            }
        }
    }
}

@brunolau curious why there are the extra Open and Close calls, from the ClearPool documentation it seems it shouldn't matter, i.e. you could just call ClearPool. Did you find you needed those extra calls?

earldean on 6 Feb 2020

@earldean Not at all..they're most likely redundant. As the issue was emerging once in 1-2 days and I was basically doing a "blindfold hack", I've just tried anything that might have had patched the problem for me...to sum it up, no particular reason for the additional Open/Close calls

brunolau on 6 Feb 2020

@brunolau Thanks for the quick reply! I just wanted to double check, I thought you possibly have learned something through trial and error :)

It is quite hard to test, not just the bug of course since it is very non-deterministic, but testing the custom SqlServerRetryingExecutionStrategy. I suppose I should share my test case I wrote for this.
Thanks!

using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Internal;
using Microsoft.EntityFrameworkCore.Storage;
using Moq;
using NUnit.Framework;
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.Reflection;
using System.Data.SqlClient;
using System.Runtime.Serialization;

       [Test]
        public void TestReturnsTrueWhenSqlExceptionHasMarsTdsMessage()
        {
            var sqlConnection = new SqlConnection({SomeLocalSqlConnString});

            var dbContext = new DbContext(new DbContextOptionsBuilder().
                UseSqlServer(sqlConnection).Options);

            var mockedICurrentDbContext = new Mock<ICurrentDbContext>();
            mockedICurrentDbContext.Setup(m => m.Context).Returns(dbContext);

            var dependencies = new ExecutionStrategyDependencies(
                mockedICurrentDbContext.Object,
                new DbContextOptionsBuilder<AlarmLogContext>().Options,
                null);

           // The MarsTdsExceptionSqlRecoverHack class is the class that extends SqlServerRetryingExecutionStrategy
            var po = new PrivateObject(new MarsTdsExceptionSqlRecoverHack(dependencies));

            var shouldBeFalse = (bool)po.Invoke("ShouldRetryOn", new Exception("MARS TDS header"));
            NUnit.Framework.Assert.IsFalse(shouldBeFalse);

            var sqlException = FormatterServices.GetUninitializedObject(typeof(SqlException)) as SqlException;
            var messageField = typeof(SqlException).GetField("_message", BindingFlags.NonPublic | BindingFlags.Instance);
            messageField.SetValue(sqlException, "MARS TDS header");

            // test SQL exception without MARS TDS message
            var shouldBeTrue = (bool)po.Invoke("ShouldRetryOn", sqlException);
            NUnit.Framework.Assert.IsTrue(shouldBeTrue);

            sqlException = FormatterServices.GetUninitializedObject(typeof(SqlException)) as SqlException;
            messageField.SetValue(sqlException, "Banana");

            shouldBeFalse = (bool)po.Invoke("ShouldRetryOn", sqlException);
            NUnit.Framework.Assert.IsFalse(shouldBeFalse);
        }

earldean on 7 Feb 2020

@ajcvickers any chance of getting someone on the EF side to see if they can assist with disentangling what EF is doing to the repro we've got so far so we can investigate this as the driver issue it is?

Wraith2 on 7 Feb 2020

👍3

We are on netcoreapp3.1, EFCore 3.1.2 and experiencing this every other day.

.NET Core SDK (reflecting any global.json):
Version: 3.1.101
Commit: b377529961

Runtime Environment:
OS Name: Windows
OS Version: 10.0.18363
OS Platform: Windows
RID: win10-x64
Base Path: C:Program Filesdotnetsdk3.1.101

Host (useful for support):
Version: 3.1.1
Commit: a1388f194c

.NET Core SDKs installed:
2.1.2 [C:Program Filesdotnetsdk]
2.1.201 [C:Program Filesdotnetsdk]
2.1.202 [C:Program Filesdotnetsdk]
2.1.300-rc1-008673 [C:Program Filesdotnetsdk]
2.1.300 [C:Program Filesdotnetsdk]
2.1.301 [C:Program Filesdotnetsdk]
2.1.400 [C:Program Filesdotnetsdk]
2.1.401 [C:Program Filesdotnetsdk]
2.1.502 [C:Program Filesdotnetsdk]
2.1.504 [C:Program Filesdotnetsdk]
2.1.508 [C:Program Filesdotnetsdk]
2.1.602 [C:Program Filesdotnetsdk]
2.1.801 [C:Program Filesdotnetsdk]
2.2.104 [C:Program Filesdotnetsdk]
2.2.203 [C:Program Filesdotnetsdk]
2.2.401 [C:Program Filesdotnetsdk]
3.1.101 [C:Program Filesdotnetsdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.0-rc1-final [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.0 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.1 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.2 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.3 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.6 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.8 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.9 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.12 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.15 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.2 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.4 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.6 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.0-rc1-final [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.0 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.1 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.2 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.3 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.6 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.8 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.9 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.12 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.15 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.2 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.4 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.6 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.1 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.NETCore.App 2.0.3 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.0.7 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.0.9 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.0-rc1 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.0 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.1 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.2 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.3-servicing-26724-03 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.3 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.6 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.8 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.9 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.12 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.1.15 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.2.2 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.2.4 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.2.6 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 3.1.1 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.1 [C:Program FilesdotnetsharedMicrosoft.WindowsDesktop.App]

max-favilli on 5 Mar 2020

We started observing this problem in production sporadically. After digging in our telemetry, we noticed that the exceptions listed ☝️ were always proceeded by exceptions in other requests (that likely shared the same Sql Connection Pool) that indicated that they had called .Cancel() on their SqlCommand in response to their request being canceled. After some experimentation we able to create a sample Console application that appears to reproduce the problem very constantly. The example application can be found here https://github.com/boxofyellow/MarsHeader

boxofyellow on 20 Mar 2020

👍5 🎉3

Thanks @boxofyellow

We've been eagerly waiting for a repro to address this issue, I'll give this a try soon!

cheenamalhotra on 20 Mar 2020

👍2

I've been digging into this and wanted to summarize my finding so far, in case someone else is looking, too.

The error occurs because _nonTransactedOpenResultCount in TdsParser.cs goes below 0 in certain situations. _nonTransactedOpenResultCount gets sent in the MARS header data and values < 0 are not valid.

This error was introduced to System.Data.SqlClient on .NET Core in this PR: https://github.com/dotnet/corefx/pull/26595
Which removes stateObj._syncOverAsync = true; from TryProcessDone() in TdsParser.cs

The issue also exists against .NET Framework if you set this context switch to false (it's true by default): Switch.Microsoft.Data.SqlClient.MakeReadAsyncBlocking

The comment associated with the line above ("// Can't retry TryProcessDone) tells me that TryProcessDone might be completing more than once when it should not be, since at the end of that method is where _nonTransactedOpenResultCount is getting decremented.

We could simply revert the change from the above PR, but that would re-introduce what it was trying to fix. We could also make sure _nonTransactedOpenResultCount simply never gets decremented below 0, but that doesn't seem right, off hand. Unless we can figure out a better solution, we may just have to revert the PR.

David-Engel on 24 Mar 2020

👍5 🎉1

How is TryProcesDone being reached multiple times with the same data? It is an async packet list replay? If so the counters may need moving into the snapshot so a replay will only affect take effect once no matter how many times it's replayed.

Wraith2 on 24 Mar 2020

How is TryProcesDone being reached multiple times with the same data? It is an async packet list replay? If so the counters may need moving into the snapshot so a replay will only affect take effect once no matter how many times it's replayed.

@Wraith2 As I was thinking about it last night, after I commented, it might be that TryProcessDone is only running once. It might be that IncrementAndObtainOpenResultCount is not being called when the driver thinks it has (maybe due to a "perfectly timed" cancel?), thus later trying to decrement the count when it should not. I still need to investigate further.

David-Engel on 24 Mar 2020

I had a look through and found that even putting the counter in the snapshot wouldn't help because the transacted version lives on the SqlInternalTransaction object which isn't snapped.

Wraith2 on 24 Mar 2020

Hey there, we're facing the same situation here:
ef core 2.1.3

System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.

The production app is currently on a heavy traffic situation and the issue appear.
It has never happen before...
We have added CPU's and RAM, we are on a single production line.

Is there any chance to see a fix on this issue, it appears that it was reported in 2018...

golayp on 27 Mar 2020

It also started happening for us a few hours ago on our Azure Sql Business Critical.
The app is also under heavy load.
We are getting hundreds of errors:

Exception: Microsoft.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.

asp.net core 3.1 / windows.
Assembly Microsoft.Data.SqlClient, Version=1.0.19269.1

clement911 on 27 Mar 2020

Actually what is the best option to avoid this blocking issue ?
Because we're really in a bad situation here....

golayp on 27 Mar 2020

👍2

Will it help if we upgrade to the latest EF Core / Microsoft.Data.SqlClient?
We are on ef core 3.1.0 but I see there is a 3.1.3 now,,,

clement911 on 27 Mar 2020

We have done an emergency production restart and it fixed the issue.

We conclude that the process can get into a irreversible failure state (likely caused heavy load) where it has to be restarted. Is that possible?

clement911 on 27 Mar 2020

Not the process, this isn't a corruption bug. If all connections in a pool enter an unrecoverable state and aren't doomed out of the pool then you could end up with every connection in the pool being unusable. In that case you could add code to manually force clear the pool (pretty sure there's public api surface to do this already) which would force creation of all new connections at a minor occasional perf cost.

Wraith2 on 27 Mar 2020

Shouldn't that be done automatically though?
We've never had to deal with this problem and are definitely not keen on meddling with the db pool.

clement911 on 27 Mar 2020

Shouldn't that be done automatically though?

What? an unusable connection be removed from the pool? Sure, but this is a bug so it doesn't do what it's supposed to, that's what bugs are.

We've never had to deal with this problem and are definitely not keen on meddling with the db pool.

As I see if you have 3 options.

Do nothing and live with the problem
Wait for a fix to be found, implemented, verified, packaged and released
Add code to triage the problem

Your call.

Wraith2 on 27 Mar 2020

@clement911 You can use my nasty execution strategy workaround mentioned in the comments above. It still does its job even in EF core 3 (with some slight modifications)

brunolau on 27 Mar 2020

👍1

@brunolau could you please share modifications for the workaround related to EF core 3.

nurhat on 28 Mar 2020

{"code":"System.Data.SqlClient.SqlException","message":"The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.","template":"","data":{}}
I also encountered this problem, using dapper + NETCORE2.2. Is there any update on this issue?

jaredhshuai on 1 Apr 2020

From that message we can see that you're using System.Data.SqlClient not Microsoft.Data.SqlClient, the system version is deprecated in favour of this library, I suggest switching.

Wraith2 on 1 Apr 2020

@Wraith2 I'll replace it immediately, hoping that the problem can be solved.Thanks.

jaredhshuai on 1 Apr 2020

We were also using the the System.Data.SqlClient package, I updated the example repro to try the Microsoft.Data.SqlClient package and the problem appears to be visible there as well.

boxofyellow on 1 Apr 2020

@jaredhshuai @boxofyellow @nurhat

Our proposed fix in discussion is in PR #490 , would you please give that a try and let us know if that resolves your problem?

cheenamalhotra on 1 Apr 2020

I pulled down the nuget package listed in the PR ☝️, it looks like it solved the problem for the Microsoft package. I'll run a batch of them overnight to see if anything else shakes out. Is there any chance we can expect to see a similar fix for the System package? We might take this as the catalyst to switch over to the Microsoft package, but that will dramatically slow down rolling the change out into production.

boxofyellow on 1 Apr 2020

👍1

We shall consider backporting it to System.Data.SqlClient (netcoreapp3.1) if it proves to be promising and we gather confirmation from users from different domains.

cc @David-Engel for final confirmation.

cheenamalhotra on 1 Apr 2020

👍1

@cheenamalhotra I'm planning to test against wraith's test on release/3.0 and the repro from https://github.com/dotnet/SqlClient/issues/85#issuecomment-601864224. have these tests been run yet? If they pass, would that be enough confidence to merge https://github.com/dotnet/SqlClient/pull/490?

michaeltnguyen on 1 Apr 2020

@michaeltnguyen that would be a nice add-on for sure!

I will also be testing this soon, I'm just occupied with other priorities.

cheenamalhotra on 2 Apr 2020

From that message we can see that you're using System.Data.SqlClient not Microsoft.Data.SqlClient, the system version is deprecated in favour of this library, I suggest switching.

@Wraith2
After switching, the issue is still existing.

{"code":"Microsoft.Data.SqlClient.SqlException","message":"The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.","template":"","data":{}}

jaredhshuai on 2 Apr 2020

@cheenamalhotra
Useing following package, I had finished load test.

"Microsoft.Data.SqlClient" Version="2.0.0-preview1.20021.1"

ThroughInput=6.8/s
When the number of samples is less than 10000, it is stable. After more than 10000, the following error occurred：

{"code":"Microsoft.Data.SqlClient.SqlException","message":"The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.","template":"","data":{}}

The final number of samples is 50000, with an error rate of 45%.

jaredhshuai on 2 Apr 2020

@jaredhshuai is that the package linked to in this comment https://github.com/dotnet/SqlClient/pull/490#issuecomment-604634276 ?

Wraith2 on 2 Apr 2020

@Wraith2
When I use this below version,
<PackageReference Include="Microsoft.Data.SqlClient" Version="2.0.0-dev-pr490" />
Visual Studio said it couldn't be found, and recommended the following version for me.
<PackageReference Include="Microsoft.Data.SqlClient" Version="2.0.0-preview1.20021.1" />

jaredhshuai on 2 Apr 2020

You'll have to download the nuget package from the link in that comment and manually add it to the nuget cache folder so it can be resolved. I use a local nuget food for this.

Wraith2 on 2 Apr 2020

I need deployed to QA environment to test. In that case, the package will not be found in QA environment.

jaredhshuai on 2 Apr 2020

Put the file in a folder and then add a nuget config file with a nuget package source pointing at that folder, https://docs.microsoft.com/en-us/nuget/reference/nuget-config-file#package-source-sections

Wraith2 on 2 Apr 2020

👍2

Repetitively running the sample application over night (about 1500 runs) with the package from #490, did not yield any MARDS TDS header exceptions.

boxofyellow on 2 Apr 2020

👍5

When I'm trying to run the pr build against Wraith2's test on release/3.0, I'm getting the following error:

NU1202 Package Microsoft.Data.SqlClient 2.0.0-preview2.20084.1 is not compatible with netstandard1.3

I also tried ONLY updating the version of SqlClient on EfCore.SqlServer.FunctionalTests, but the BreakTdsStream test was still failing. (I'm unsure if it's sufficient to only update SqlClient on that single project, though)

michaeltnguyen on 2 Apr 2020

You also need to update versions.props

ErikEJ on 3 Apr 2020

Hi @ErikEJ , are you saying that I need to update version.props somewhere or that the PR author needs to update version.props in the nuget package? If it's me, could you point me to some basic instructions of what I'd need to do? (I tried some basic googling, but hit diminishing returns very quickly)

michaeltnguyen on 3 Apr 2020

I think maybe I misunderstood. You are not using the EF Core source code?

ErikEJ on 3 Apr 2020

@ErikEJ I am using the EF Core source code. I added Wraith2's breaking test on the release/3.0, then I'm using the PR build of Microsoft.Data.SqlClient to see if it fixes the breaking test.

michaeltnguyen on 3 Apr 2020

Then have a look at the versions.props file in the eng folder

ErikEJ on 3 Apr 2020

When I use the nuget from the PR build, I'm now unable to resolve Microsoft.Data.SqlClient @cheenamalhotra any thoughts? am I just doing something wrong here?

michaeltnguyen on 3 Apr 2020

Have you set up s local feed and added it to NuGet.config ?

ErikEJ on 3 Apr 2020

When I use the nuget from the PR build, I'm now unable to resolve Microsoft.Data.SqlClient @cheenamalhotra any thoughts? am I just doing something wrong here?

Are you not able to restore the package at all or you get an exception? Check your NuGet.config and logs if the location is being considered when restoring.

cheenamalhotra on 3 Apr 2020

@ErikEJ yep, created a local nuget feed and added it to nuget.config. I created a blank project and am unable to resolve Microsoft.Data.SqlClient. I can also PM my email address if we want to take this offline.

@cheenamalhotra I am able to restore. In my blank project, Microsoft.IdentityModel.JsonWebTokens is brought in as a transitive dependency, and I'm able to resolve it.

Might be easier for one of y'all to test... I've spent a day and a half on this and gotten nowhere. I'm still willing to try if I'm just missing something obvious.

michaeltnguyen on 3 Apr 2020

I also tried this and failed earlier, as stated.

ErikEJ on 3 Apr 2020

I'm able to build and run release/3.0, just not with the PR nuget package. Here's what the nuget package contains:

Capture

so either the nuget package is missing Microsoft.Data.SqlClient, or I messed something up.

michaeltnguyen on 3 Apr 2020

The fact that it's working with master branch and not this one, is confusing. However, I did make it work in the past. Will give it a try!

cheenamalhotra on 3 Apr 2020

👍1

I completed running all the EF Core tests in "release/3.0" tag (including BreakTDSStream test as mentioned above) by direct project import of Microsoft.Data.SqlClient (netcore) and all tests are passing with the PR #490 changes, also confirmed failure with "master" branch.

We will proceed with PR merge for next release!

cheenamalhotra on 6 Apr 2020

🎉2 👍1

@cheenamalhotra This is awesome! Do you have a rough estimate on when this will be GA? Are you going to do a hotfix release, or wait until 2.0.0? I guess for those of us that are more eager for the fix, is there a nightly build that we can use in the meanwhile?

michaeltnguyen on 7 Apr 2020

We have considered to backport this fix to v1.1.x as well due to it's high demand, this version will be available sooner than GA v2.0.

cheenamalhotra on 7 Apr 2020

👍4

@cheenamalhotra After the release, please let me know that I've been longing to use it. Thanks.

jaredhshuai on 8 Apr 2020

👍3

@jaredhshuai @michaeltnguyen @ErikEJ and everyone else following this thread:

Thanks for all the support you've provided in diagnosing and getting the fix ready for this issue.
We released v1.1.2 today that contains the fix, please give that a try and let us know if that fixed your problem by upvoting!

We will keep the issue open for accepting feedback!

cheenamalhotra on 16 Apr 2020

👍1

@cheenamalhotra Under the 300 rpm load test, I ran for 30 minutes, but I didn't find any error. It's great.

jaredhshuai on 16 Apr 2020

👍1

I ran the test app all day yesterday (+2500 runs) no errors found.

boxofyellow on 17 Apr 2020

I ran the test app all day yesterday (+2500 runs) errors found.

No errors or many errors? You missed out an important word 😁

Wraith2 on 17 Apr 2020

ooo my @Wraith2 you are right, the missing 'no' makes all the differences... I apologize for the confusion. So to be clear 2500 _clean_ runs.

boxofyellow on 17 Apr 2020

👍1

This sounds great--will the fix be backported to https://www.nuget.org/packages/System.Data.SqlClient? Otherwise, I can look into switching over to Microsoft.Data.SqlClient.

JohnnyFun on 17 Apr 2020

👍1

Even if it's backported to System it's a good idea to look into moving to Microsoft version as the system version will only updated for severe reasons while the version in this repo will get perf updates features etc.

Wraith2 on 17 Apr 2020

👍2

@cheenamalhotra sorry for my ignorance, but I don't explicitly include the package Microsoft.Data.SqlClient in my project, but I do use EFCore, I've seen the same problem occurring in prod a few times now, will this be ported to EFCore?

rogerfar on 17 Apr 2020

The package referenced by EF Core is not updated to latest Microsoft.Data.SqlClient version, hence you will see the issue. There's an open PR to address the same: https://github.com/dotnet/efcore/pull/20377, currently on hold.

You may do a direct reference of this NuGet in the meantime!

cheenamalhotra on 17 Apr 2020

👍2

@rogerfar ser my blog post here: https://erikej.github.io/efcore/sqlclient/2020/03/22/update_mds.html

ErikEJ on 17 Apr 2020

👍4

Just to clarify on the blog a bit from what I've gathered from issues and code.
Managed SNI was unstable for quite a while and lags in performance compared to native. While it would be better in general to use it everywhere until it's proven stable with little to no speed penalty. The windows build debug only switch used to be present on release but was disabled because using it as the default caused people problems. I think we're now getting to the point where it's reliable and fast enough for heavy use. Perf is steadily increasing as you've seen in https://github.com/dotnet/SqlClient/issues/375

Wraith2 on 17 Apr 2020

@rogerfar ser my blog post here: https://erikej.github.io/efcore/sqlclient/2020/03/22/update_mds.html

We have been dealing with this is issue for a couple of months, but we are on Windows platform.

Does it mean the fix won't apply to who is experiencing this on Windows?

max-favilli on 21 Apr 2020

The fix applies to all networking strategies, native and managed, equally.

Wraith2 on 21 Apr 2020

👍1

Closing the issue as latest release v2.0.0-preview3 contains the fix.

cheenamalhotra on 6 May 2020

@David-Engel @Wraith2 I read this thread and understood the fix you applied. But, can you please elaborate more on the reason this bug causes an unrecoverable state which requires an instance restart?

lioa123 on 24 May 2020

If you encounter this problem you cane end up with a connection pool which contains only poisoned connections and that effectively prevents you connecting to the database again. The way to triage that is to clear the pool and there is sample code provided further up in this thread which does that. The nuclear option to clear the pool is to tear down the entire client process.

To be clear It doesn't require an sql instance restart and the error is entirely in the client protocol handling and does not cause a problem on the server side.

Wraith2 on 25 May 2020

👍3

I agree with @Wraith2 .

The problem was entirely client side, and did not cause necessity to perform instance restart. Cleaning up connection pool and/or restarting application should be enough to start afresh.

cheenamalhotra on 25 May 2020

@Wraith2 @cheenamalhotra Thank you for your answers! another two quick questions - as I understand from reading about this issue, is there a relation between heavy load and the appearance of the MARS header error? I mean, is it more probable this issue will appear during heavy load on the application server, which preforms unusual amount of DB calls? And the second question - does this error occur only when using EF, or it can also occur when using ADO .net?

lioa123 on 26 May 2020

@lioa123

Is it more probable this issue will appear during heavy load on the application server, which preforms unusual amount of DB calls?

No this is not related to heavy work load as it was reproducible with a test case from EF Core tests, which later on was adopted in SqlClient tests too.

does this error occur only when using EF, or it can also occur when using ADO .net

It was reproducible with ADO.NET or SqlClient as well, so it can occur in any application irrespective of EF Core in use.

cheenamalhotra on 26 May 2020

👍1

@cheenamalhotra
I'm not sure if it's related to this problem but I get similar exception:

Microsoft.Data.SqlClient.SqlException : The incoming tabular data stream (TDS) protocol stream is incorrect. The TDS headers contained errors.
   at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__164_0(Task`1 result)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)

And fix for me was change code from:

var timeout = TimeSpan.FromSeconds(_bulkCopyTimeout + 1);
var cancellationTokenSource = new CancellationTokenSource(timeout);
cancellationTokenSource.CancelAfter(timeout);

await bulkCopy.WriteToServerAsync(table, cancellationTokenSource.Token);

to:

await bulkCopy.WriteToServerAsync(table);

FYI...
I used Microsoft.Data.SqlClient Version="2.0.0-preview4.20142.4"
my project is .NET Core 3.1
and also in my connection string have MultipleActiveResultSets=True

KondzioSSJ4 on 2 Jun 2020

var cancellationTokenSource = new CancellationTokenSource(timeout);
cancellationTokenSource.CancelAfter(timeout);

I don't think you should need to do both of those things. you initializer the CTS with a timeout and then set that timeout again, that's just going to set two timers as far as I know.

If you can produce that as an isolated repro against the sample Northwind database it'd be helpful, as you can imagine it's hard to identify behaviour based on only a stacktrace.

Wraith2 on 2 Jun 2020

👍3

@Wraith2
I had 3 places :)
because in some version any "timeout" wasn't throw timeout after expected time
but after I update package then no longer required

var bulkCopy = new SqlBulkCopy(connection, SqlBulkCopyOptions.CheckConstraints | SqlBulkCopyOptions.FireTriggers, null);
bulkCopy.BulkCopyTimeout = bulkCopyTimeout;

KondzioSSJ4 on 2 Jun 2020

@Wraith2 @cheenamalhotra

Here you go...
https://github.com/KondzioSSJ4/Repro-for-SQLClient-Issue-85
Sample of how to reproduce this bug (The incoming tabular data stream (TDS) protocol stream is incorrect. The TDS headers contained errors.)
in test after 100 tries i get 51 fails

P.S. I run it on Windows 10 Pro v. 1909

KondzioSSJ4 on 2 Jun 2020

👍3

Excellent, thanks.

Wraith2 on 2 Jun 2020

Thanks @KondzioSSJ4
Reopening to continue investigations.

cheenamalhotra on 2 Jun 2020

We are on .net framework 4.8 and also saw this when we moved to EntityFramework 6.4.0

System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details. ---> System.Data.SqlClient.SqlException: The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.    at System.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__180_0(Task`1 result)    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()    at System.Threading.Tasks.Task.Execute()
--

Our system was fine for a few days, then all of a sudden we got a huge influx of these errors. We remediated the issue with an IIS/appPool restart... we're looking to move back to EF6.2 :(

AirEssY on 5 Jun 2020

Hi @AirEssY

We haven't encountered failing case in .NET Framework apps yet.. can you put up a small repro app and upload here so we can look more into it?

I'd also suggest testing with latest EF 6.4.4 and confirming it also exists with their latest version.

cheenamalhotra on 5 Jun 2020

I can see

""The type initializer for 'Microsoft.Data.SqlClient.TdsParser' threw an exception." error in my Azure function App on netcoreapp3.1
when I'm targeting win-x86.

https://github.com/Azure/azure-functions-host/issues/6088

vitalybibikov on 17 Jun 2020

@cheenamalhotra

Is there any fix for this matter?
Microsoft.AspNetCore.Identity.EntityFrameworkCore V3.1.6

This error occurs spontaneously:
Microsoft.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.

meirib on 12 Aug 2020

I saw this support article pop up today
https://support.microsoft.com/en-au/help/970519/you-receive-a-the-incoming-tabular-data-stream-tds-remote-procedure-ca

Seems very related?

AirEssY on 24 Aug 2020

I saw this support article pop up today
https://support.microsoft.com/en-au/help/970519/you-receive-a-the-incoming-tabular-data-stream-tds-remote-procedure-ca

Seems very related?

@AirEssY Thanks.

Apparently it's a known issue:
https://github.com/dotnet/SqlClient/issues/687

meirib on 24 Aug 2020

We are getting these pretty constantly now. So far 546 instances of this error today. I have recycled app pools across our server farm multiple times. It stops happening for a little while but eventually starts again. We are using EF 6.4.4 in a .NETCore 3.1 app.

System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior) at System.Data.Entity.Infrastructure.Interception.InternalDispatcher1.DispatchTTarget,TInterceptionContext,TResult
at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext)
at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
ClientConnectionId:ef9dcc0a-791a-4fcc-b3c6-6d6fde3c3ddf
Error Number:4011,State:3,Class:16

ItWorksOnMyMachine on 31 Aug 2020

@ItWorksOnMyMachine

Please upgrade version of System.Data.SqlClient driver referenced in your application to v4.8.2 that contains this fix (for .NET Core).

cheenamalhotra on 31 Aug 2020

❤1

@cheenamalhotra Rockstar! Thank you, this solved it!

ItWorksOnMyMachine on 2 Sep 2020

❤1

@cheenamalhotra Our app is .NET Core 2.1. web app. It is using ....nugetpackagessystem.data.sqlclient4.4.3refnetstandard2.0System.Data.SqlClient.dll

Can we update a single nuget package or do we need to upgrade to newer version of .NET core?

osudude on 5 Sep 2020

Hi @osudude

Yes, you may only update the System.Data.SqlClient NuGet package reference.

cheenamalhotra on 8 Sep 2020

How to check that the updated version has the proper version ?

System.Reflection.Assembly::LoadFrom("C:Usersxxx\.nugetpackagessystem.data.sqlclient4.8.2runtimeswinlibnetcoreapp2.1System.Data.SqlClient.dll").GetName().Version

returns :
Major Minor Build Revision
4 6 1 2

Yet,

System.Reflection.Assembly::LoadFrom("C:Usersxxx\.nugetpackagessystem.data.sqlclient4.6.1libnetcoreapp2.1System.Data.SqlClient.dll").GetName().Version

returns :
Major Minor Build Revision
4 5 0 1

So the new version 4.8.2 has an internal version 4.6.1.2 and the old 4.6.1 has an internal version 4.5.0.1 ?

Thanks for the clarification on version numbers.

Waebs on 17 Sep 2020

Hi @Waebs

Yes these versions are correct. They are old semantics for System.Data.SqlClient, and are maintained by dotnet/runtime teams, you may reach out to them if you have any concerns. Thanks!

cheenamalhotra on 18 Sep 2020

We still experience transient TDS MARS errors after upgrading to System.Data.SqlClient 4.8.2 on our Azure Sql database + app service (serving our .net Core 2.1 Auth service with OpenIddict).

Even after setting MultipleActiveResultSets=False ou in Connection string we get arround 150-200 errors =>

_System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader()
at System.Data.SqlClient.SqlCommand.InternalEndExecuteReader(IAsyncResult asyncResult, String endMethod)
at System.Data.SqlClient.SqlCommand.EndExecuteReaderInternal(IAsyncResult asyncResult)
at System.Data.SqlClient.SqlCommand.EndExecuteReader(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)_

Thanks in advance for your help.

Waebs on 18 Sep 2020

Hi @Waebs

Could you provide us a repro app with table schema that reproduces this error, we can investigate into it.

cheenamalhotra on 18 Sep 2020

We confirm that the updrade to System.Data.SqlClient 4.8.2 solve (for us at least) the TDS/MARS errors.

Waebs on 21 Sep 2020

👍1

@cheenamalhotra I am using Microsoft.AspNetCore.All meta nuget package in .NET core 2.2 How do I get the update?

osudude on 24 Sep 2020

@osudude

You can check what is the version of System.Data.SqlClient or Microsoft.Data.SqlClient referenced in your application in your dependency tree, and override that by directly referencing the same package with latest stable version.

cheenamalhotra on 24 Sep 2020

Hi,

We are running into the same issue from .NET Framework 4.6.1 using Serilog.Sinks.MSSqlServer
when running this on my local machine it runs perfectly but when running it on our server it throws the following error:
Unable to write 4 log events to the database due to following error: The type initializer for 'Microsoft.Data.SqlClient.TdsParser' threw an exception.

jean-paul-scorito on 16 Nov 2020

Upgrading Microsoft.Data.SqlClient.SNI from versie 1.1.0 to 2.1.1 (including all its dependencies) fixed it for us some how, no clue how that was related but I found it thank to this github issue, hope it helps someone else!

jean-paul-scorito on 16 Nov 2020

Sqlclient: The MARS TDS header contained errors using ASP.NET Core and EF Core connecting to Azure SQL Server

Most helpful comment

All 203 comments

Related issues