Efcore: NullReferenceException hides actual exception

[lauxjpn]

For some MySQL database server implementations (e.g. MariaDB or MySQL < 8.0), unsupported expressions like OUTER APPLY cannot be translated. We throw late in the query pipeline (in MySqlParameterBasedSqlProcessor), because EF Core might still decide to use some unsupported expressions after a RelationalQueryTranslationPostprocessor implementation has been called.

The following exception e.g. is raised by the NorthwindSplitIncludeQueryMySqlTest.Include_collection_with_outer_apply_with_filter test. The exception itself is expected and we expect it to be propagated all the way to the user:

System.InvalidOperationException: The LINQ expression 'OUTER APPLY Projection Mapping:
(
    SELECT TOP(5) o.OrderID
    FROM Orders AS o
    WHERE c.CustomerID == o.CustomerID
    ORDER BY c.CustomerID ASC
) AS t' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync(). See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
   at Pomelo.EntityFrameworkCore.MySql.Query.ExpressionVisitors.Internal.MySqlCompatibilityExpressionVisitor.CheckTranslated(Expression translated, Expression original) in E:\Sources\Pomelo.EFCore.MySql-5.0\src\EFCore.MySql\Query\ExpressionVisitors\Internal\MySqlCompatibilityExpressionVisitor.cs:line 61
   at Pomelo.EntityFrameworkCore.MySql.Query.ExpressionVisitors.Internal.MySqlCompatibilityExpressionVisitor.CheckSupport(Expression expression, Boolean isSupported) in E:\Sources\Pomelo.EFCore.MySql-5.0\src\EFCore.MySql\Query\ExpressionVisitors\Internal\MySqlCompatibilityExpressionVisitor.cs:line 51
   at Pomelo.EntityFrameworkCore.MySql.Query.ExpressionVisitors.Internal.MySqlCompatibilityExpressionVisitor.VisitOuterApply(OuterApplyExpression outerApplyExpression) in E:\Sources\Pomelo.EFCore.MySql-5.0\src\EFCore.MySql\Query\ExpressionVisitors\Internal\MySqlCompatibilityExpressionVisitor.cs:line 42
   at Pomelo.EntityFrameworkCore.MySql.Query.ExpressionVisitors.Internal.MySqlCompatibilityExpressionVisitor.VisitExtension(Expression extensionExpression) in E:\Sources\Pomelo.EFCore.MySql-5.0\src\EFCore.MySql\Query\ExpressionVisitors\Internal\MySqlCompatibilityExpressionVisitor.cs:line 28
   at System.Linq.Expressions.Expression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Microsoft.EntityFrameworkCore.Query.SqlExpressions.SelectExpression.VisitChildren(ExpressionVisitor visitor) in E:\Sources\EFCore-5.0\src\EFCore.Relational\Query\SqlExpressions\SelectExpression.cs:line 2878
   at System.Linq.Expressions.ExpressionVisitor.VisitExtension(Expression node)
   at Pomelo.EntityFrameworkCore.MySql.Query.ExpressionVisitors.Internal.MySqlCompatibilityExpressionVisitor.VisitExtension(Expression extensionExpression) in E:\Sources\Pomelo.EFCore.MySql-5.0\src\EFCore.MySql\Query\ExpressionVisitors\Internal\MySqlCompatibilityExpressionVisitor.cs:line 32
   at System.Linq.Expressions.Expression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Pomelo.EntityFrameworkCore.MySql.Query.Internal.MySqlParameterBasedSqlProcessor.ProcessSqlNullability(SelectExpression selectExpression, IReadOnlyDictionary`2 parametersValues, Boolean& canCache) in E:\Sources\Pomelo.EFCore.MySql-5.0\src\EFCore.MySql\Query\Internal\MySqlParameterBasedSqlProcessor.cs:line 44
   at Microsoft.EntityFrameworkCore.Query.RelationalParameterBasedSqlProcessor.Optimize(SelectExpression selectExpression, IReadOnlyDictionary`2 parametersValues, Boolean& canCache) in E:\Sources\EFCore-5.0\src\EFCore.Relational\Query\RelationalParameterBasedSqlProcessor.cs:line 64
   at Microsoft.EntityFrameworkCore.Query.Internal.RelationalCommandCache.GetRelationalCommand(IReadOnlyDictionary`2 parameters) in E:\Sources\EFCore-5.0\src\EFCore.Relational\Query\Internal\RelationalCommandCache.cs:line 86
   at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.Enumerator.InitializeReader(DbContext _, Boolean result) in E:\Sources\EFCore-5.0\src\EFCore.Relational\Query\Internal\SplitQueryingEnumerable.cs:line 198
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementation[TState,TResult](Func`3 operation, Func`3 verifySucceeded, TState state) in E:\Sources\EFCore-5.0\src\EFCore\Storage\ExecutionStrategy.cs:line 173

However, it results in a NullReferenceException for _resultCoordinator.DataReaders here, which is then propagated to the user instead of our actual exception:

https://github.com/dotnet/efcore/blob/551c58af27fb94d4a1c05f9837ab9bd1bdaa0562/src/EFCore.Relational/Query/Internal/SplitQueryingEnumerable.cs#L219

System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.Enumerator.Dispose() in E:\Sources\EFCore-5.0\src\EFCore.Relational\Query\Internal\SplitQueryingEnumerable.cs:line 219
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Microsoft.EntityFrameworkCore.TestUtilities.QueryAsserter.AssertQuery[TResult](Func`2 actualQuery, Func`2 expectedQuery, Func`2 elementSorter, Action`2 elementAsserter, Boolean assertOrder, Int32 entryCount, Boolean async, String testMethodName) in E:\Sources\EFCore-5.0\test\EFCore.Specification.Tests\TestUtilities\QueryAsserter.cs:line 103

Include provider and version information

EF Core version: 5.0.0-rc.1

[smitpatel]

Fixed in #23185

@lauxjpn - Let us know if you still hit this error after upgrading to fixed code so I can investigate into it.
Also afaik EF wouldn't change a join type after TranslationPostProcessor. Is this being violated at some point?

[lauxjpn]

Fixed in #23185

Thanks!

Also afaik EF wouldn't change a join type after TranslationPostProcessor. Is this being violated at some point?

I believe so. This seems to be the case for some OUTER APPLY or CROSS APPLY statements. I can investigate later today to find out again, which tests failed for us with a MySqlException instead of the expected translation exception when using MariaDB (that does not have LATERAL support, our translation for those expressions).

[smitpatel]

CollectionJoinApplyingExpressionVisitor should be the last one changing tables. So if base is called beforehand by providers, it should work.

[lauxjpn]

@smitpatel Thanks to bringing my attention back to the issue. As it turns out, we just had a bug in our MySqlCompatibilityExpressionVisitor code:

```c#
public class MySqlCompatibilityExpressionVisitor : ExpressionVisitor
{
protected override Expression VisitExtension(Expression extensionExpression)
=> extensionExpression switch
{
RowNumberExpression rowNumberExpression => VisitRowNumber(rowNumberExpression),
CrossApplyExpression crossApplyExpression => VisitCrossApply(crossApplyExpression),
OuterApplyExpression outerApplyExpression => VisitOuterApply(outerApplyExpression),
ExceptExpression exceptExpression => VisitExcept(exceptExpression),
IntersectExpression intersectExpression => VisitIntercept(intersectExpression),
ShapedQueryExpression shapedQueryExpression => shapedQueryExpression.Update(
Visit(shapedQueryExpression.QueryExpression),
shapedQueryExpression.ShaperExpression), // <-- missing the Visit() call
_ => base.VisitExtension(extensionExpression)
};

    protected virtual Expression VisitCrossApply(CrossApplyExpression crossApplyExpression)
        => CheckSupport(crossApplyExpression, _options.ServerVersion.SupportsCrossApply);

}
```

This was not an issue when processing the expression visitor at the very end of the pipeline, because ShapedQueryExpressions would not exist anymore.