Webapi: InvalidOperationException when filtering some DateTimeOffset field with another field

Created on 7 Feb 2020 · 23Comments · Source: OData/WebApi

I am using the EnableQuery attribute to filter my IQueryable returned by EF Core 3.1.1
When I do a filter like this: organizerId eq 8 and startDate eq now() the OData generates a wrong expression

DbSet<Event>
    .Where(e => e.OrganizerId == __TypedProperty_0 && $it.StartDate == DateTimeOffset.UtcNow == True)

The properly query should be

DbSet<Event>
    .Where(e => e.OrganizerId == __TypedProperty_0 && $it.StartDate == DateTimeOffset.UtcNow)

There is == True at the end

Assemblies affected

<PackageReference Include="Microsoft.AspNetCore.OData" Version="7.3.0" />

Reproduce steps

I've created a repository with this scenario
https://github.com/AlbertoMonteiro/ReproProject

You can run and open the swagger page and in $filter try: organizerId eq 8 and startDate eq now()

I am using ASP.NET Core 3.1 with EntityFrameworkCore.SqlServer 3.1.1
My model is

public sealed class Event
{
    public long Id { get; set; }
    public string Organizer { get; set; }
    public long? OrganizerId { get; set; }
    public DateTimeOffset StartDate { get; set; }
    //other properties omitted for simplify
}

Expected result

The exception should not happen and the query should be generated without the last == True

Actual result

Receiving this exception

System.InvalidOperationException: The LINQ expression 'DbSet<Event>
    .Where(e => e.OrganizerId == __TypedProperty_0 && $it.StartDate == DateTimeOffset.UtcNow == True)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync(). See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.<VisitMethodCall>g__CheckTranslated|8_0(ShapedQueryExpression translated, <>c__DisplayClass8_0& )
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass9_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.Execute[TResult](Expression expression)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetEnumerator()
   at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
   at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at AutoMapper.Extensions.ExpressionMapping.Impl.SourceSourceInjectedQuery`2.GetEnumerator()
   at AutoMapper.Extensions.ExpressionMapping.Impl.SourceSourceInjectedQuery`2.System.Collections.IEnumerable.GetEnumerator()
   at System.Text.Json.JsonSerializer.HandleEnumerable(JsonClassInfo elementClassInfo, JsonSerializerOptions options, Utf8JsonWriter writer, WriteStack& state)
   at System.Text.Json.JsonSerializer.Write(Utf8JsonWriter writer, Int32 originalWriterDepth, Int32 flushThreshold, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.JsonSerializer.WriteAsyncCore(Stream utf8Json, Object value, Type inputType, JsonSerializerOptions options, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)
   at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|29_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

P2 bug

Source

AlbertoMonteiro

All 23 comments

I downloaded the source code and replaced the nuget package for direct project reference
I am on commit b92f2dfcaa10cb00d0e91a611c06deba262774a5

I added a breakpoint in FilterBinder on line 492
https://github.com/OData/WebApi/blob/b92f2dfcaa10cb00d0e91a611c06deba262774a5/src/Microsoft.AspNet.OData.Shared/Query/Expressions/FilterBinder.cs#L485-L492
I noticed that the line if (isNullPropagationRequired) is called 3 times(for this scenario)

the first time it handles the organizerId eq 8 expression and since organizedId is nullable long the variable isNullPropagationRequired become true
the second time it handles the startDate eq now() but this time isNullPropagationRequired is false
the last time it handles the full expression organizerId eq 8 and startDate eq now() but now, since the organizerId side is nullable, the isNullPropagationRequired become true then all sides become nullable and the final query result is

.Lambda #Lambda1<System.Func`2[ReproProject.ViewModels.EventViewModel,System.Boolean]>(ReproProject.ViewModels.EventViewModel $$it)
{
    ($$it.OrganizerId == (System.Nullable`1[System.Int64]).Constant<Microsoft.AspNet.OData.Query.Expressions.LinqParameterContainer+TypedLinqParameterContainer`1[System.Int64]>(Microsoft.AspNet.OData.Query.Expressions.LinqParameterContainer+TypedLinqParameterContainer`1[System.Int64]).TypedProperty
    && 
      (System.Nullable`1[System.Boolean])($$it.StartDate == System.DateTimeOffset.UtcNow)) 
        == .Constant<System.Nullable`1[System.Boolean]>(True)
}

As we can see there is the == .Constant<System.Nullable``1[System.Boolean]>(True) part that was added because all sides of the binary expression were changed to nullable even without the needed.

BindBinaryOperatorNode calls CreateBinaryExpression from (Microsoft.AspNet.OData.Query.Expressions)ExpressionBinderBase

In the follow lines it checks if boths sides have same type.
But for the last call of this method boths sides are not equal, one is nullable boolean and the other is just boolean

https://github.com/OData/WebApi/blob/b92f2dfcaa10cb00d0e91a611c06deba262774a5/src/Microsoft.AspNet.OData.Shared/Query/Expressions/ExpressionBinderBase.cs#L156-L161

I've also noticed that the method CreateBinaryExpression and BindBinaryOperatorNode does not care about the BinaryOperatorKind value, if it is And, maybe they should not try to convert both sides to nullable

I hope it helps to solve this issue

AlbertoMonteiro on 9 Feb 2020

@AlbertoMonteiro Thank your for creating the issue and for providing repro and detailed investigation analysis of the bug.

We're looking into this.

habbes on 13 Feb 2020

@AlbertoMonteiro it does not appear that this is related to DateTimeOffset, the same error occurs when your filter contains an operation on another fields as well, e.g. organizerId eq 8 and id eq 1. So it just seems to be caused by one operation having nullable operands while the other does not.

habbes on 13 Feb 2020

When I use the same nullable field in both operations: organizerId eq 8 or organizerId eq 1, I don't get an error.

However, when I make the Id field nullable and try running a query with the filter organizerId eq 8 and if eq 1, I get the following error. This might not be related to the initial error, but I'm just adding it here for completeness:

System.InvalidOperationException: Rewriting child expression from type 'System.Nullable`1[System.Boolean]' to type 'System.Boolean' is not allowed, because it would change the meaning of the operation. If this is intentional, override 'VisitUnary' and change it to allow this rewrite.

   at System.Linq.Expressions.ExpressionVisitor.ValidateChildType(Type before, Type after, String methodName)

   at System.Linq.Expressions.ExpressionVisitor.ValidateUnary(UnaryExpression before, UnaryExpression after)

   at AutoMapper.Mappers.ExpressionMapper.MappingVisitor.VisitBinary(BinaryExpression node)

   at AutoMapper.Mappers.ExpressionMapper.MappingVisitor.VisitBinary(BinaryExpression node)

   at AutoMapper.Mappers.ExpressionMapper.MappingVisitor.VisitLambdaExpression[T](Expression`1 expression)

   at AutoMapper.Mappers.ExpressionMapper.MappingVisitor.VisitLambda[T](Expression`1 node)

   at AutoMapper.Mappers.ExpressionMapper.MappingVisitor.<>c__23`1.<VisitAllParametersExpression>b__23_9(Expression e, ExpressionVisitor v)

   at System.Linq.Enumerable.Aggregate[TSource,TAccumulate](IEnumerable`1 source, TAccumulate seed, Func`3 func)

   at AutoMapper.Mappers.ExpressionMapper.MappingVisitor.VisitAllParametersExpression[T](Expression`1 expression)

   at AutoMapper.Mappers.ExpressionMapper.MappingVisitor.VisitLambda[T](Expression`1 node)

   at System.Linq.Expressions.ExpressionVisitor.VisitUnary(UnaryExpression node)

   at System.Linq.Expressions.ExpressionVisitor.Visit(ReadOnlyCollection`1 nodes)

   at AutoMapper.Mappers.ExpressionMapper.MappingVisitor.GetConvertedMethodCall(MethodCallExpression node)

   at AutoMapper.Mappers.ExpressionMapper.MappingVisitor.VisitMethodCall(MethodCallExpression node)

   at AutoMapper.Extensions.ExpressionMapping.Impl.SourceInjectedQueryProvider`2.ConvertDestinationExpressionToSourceExpression(Expression expression)

   at AutoMapper.Extensions.ExpressionMapping.Impl.SourceInjectedQueryProvider`2.Execute[TResult](Expression expression)

   at AutoMapper.Extensions.ExpressionMapping.Impl.SourceSourceInjectedQuery`2.GetEnumerator()

   at AutoMapper.Extensions.ExpressionMapping.Impl.SourceSourceInjectedQuery`2.System.Collections.IEnumerable.GetEnumerator()

   at System.Text.Json.JsonSerializer.HandleEnumerable(JsonClassInfo elementClassInfo, JsonSerializerOptions options, Utf8JsonWriter writer, WriteStack& state)

   at System.Text.Json.JsonSerializer.Write(Utf8JsonWriter writer, Int32 originalWriterDepth, Int32 flushThreshold, JsonSerializerOptions options, WriteStack& state)

   at System.Text.Json.JsonSerializer.WriteAsyncCore(Stream utf8Json, Object value, Type inputType, JsonSerializerOptions options, CancellationToken cancellationToken)

   at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)

   at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultAsync>g__Logged|21_0(ResourceInvoker invoker, IActionResult result)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|29_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()

--- End of stack trace from previous location where exception was thrown ---

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()

--- End of stack trace from previous location where exception was thrown ---

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)

   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)

   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)

   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)

   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

habbes on 13 Feb 2020

Yeah, for now, I have disabled the HandleNullPropagation property and the errors do not happen, and all my filters are working in the same way when I was using ASP.NET Core 2.2.

I've tried to change the line 156 if (left.Type != right.Type) adding the condition to enter in if only if the operator is And/Or but it crashed the app in another line when it tries to create a BinaryExpression, because the LINQ says that sides of AndAlso have different type, but if I write the lambda normally it is allowed. I checked some values and I found that IsLifted and IsLiftedToNull are different

Expression<Func<EventViewModel, bool>> exp = eventVm => eventVm.OrganizerId == 8 && eventVm.Id == 1;

> (exp.Body as BinaryExpression).Left
[(eventVm.OrganizerId == Convert(Convert(8)))] 
{ CanReduce=false, Conversion=null, DebugView="$eventVm.OrganizerId == (System.Nullable`1[System.Int64])((System.Int64)8)", 
IsLifted=true, IsLiftedToNull=false
> (exp.Body as BinaryExpression).Right
[(eventVm.Id == 1)] 
{ CanReduce=false, Conversion=null, DebugView="$eventVm.Id == 1L", 
IsLifted=false, IsLiftedToNull=false

As we can see the left side have IsLifted=true, IsLiftedToNull=false
but the expression generated by odata set those 2 properties to true

AlbertoMonteiro on 13 Feb 2020

@AlbertoMonteiro

I've tried to change the line 156 if (left.Type != right.Type) adding the condition to enter in if only if the operator is And/Or but it crashed the app in another line when it tries to create a BinaryExpression, because the LINQ says that sides of AndAlso have different type, but if I write the lambda normally it is allowed. I checked some values and I found that IsLifted and IsLiftedToNull are different

What is the rationale for entering the condition only when the operator is And/Or?

habbes on 13 Feb 2020

@AlbertoMonteiro if one of the operands is nullable, then the other operands is also made nullable, and the operator is "lifted" so that it can operate on nullable types and return a nullable type. This applies for And/Or as well.

habbes on 13 Feb 2020

@AlbertoMonteiro
In the BindFilterClause() method, the expression that has been constructed so far (i.e. $it => $it.OrganizerId == 8 && $it.StartDate == DateTimeOffset.UtcNow) is passed through the ApplyNullPropagationForFilterBody() method.

This applies the following transformation to the body
```c#
body = Expression.Equal(body, Expression.Constant(true, typeof(bool?)), liftToNull: false, method: null);

When the body evaluates to a nullable boolean, it's explicitly set to `body == true` presumably to force the return type to be boolean.

This expression effectively turns the filter to:
```c#
($it.OrganizerId == 8 && $it.StartDate == DateTimeOffset.UtcNow) == true

Ideally this should work, but it appears that it gets translated to
c# $it.OrganizerId == 8 && $it.StartDate == DateTimeOffset.UtcNow == true
The parentheses are removed: i.e. the == true is applied to $it.StartDate == DateTimeOffset.UtcNow instead of the result of the && operation. And this is what causes the error.

The expression generated by OData WebApi, so next I'm going to investigate whether this is an issue with EntityFrameworkCore incorrectly handling the expression. I will try using EntityFramework 6 and see whether it works.

habbes on 13 Feb 2020

👍1

Awesome, thanks for the detailed report @habbes

AlbertoMonteiro on 13 Feb 2020

@AlbertoMonteiro I modified your sample project to use EF6 instead of EFCore and tried to run the query. I got a different error:

System.InvalidOperationException: The parameter '$it' was not bound in the specified LINQ to Entities query expression.

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.ParameterTranslator.TypedTranslate(ExpressionConverter parent, ParameterExpression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TypedTranslator`1.Translate(ExpressionConverter parent, Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TranslateExpression(Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MemberAccessTranslator.TypedTranslate(ExpressionConverter parent, MemberExpression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TypedTranslator`1.Translate(ExpressionConverter parent, Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TranslateExpression(Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.EqualsTranslator.TypedTranslate(ExpressionConverter parent, BinaryExpression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TypedTranslator`1.Translate(ExpressionConverter parent, Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TranslateExpression(Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.BinaryTranslator.TypedTranslate(ExpressionConverter parent, BinaryExpression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TypedTranslator`1.Translate(ExpressionConverter parent, Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TranslateExpression(Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.EqualsTranslator.TypedTranslate(ExpressionConverter parent, BinaryExpression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TypedTranslator`1.Translate(ExpressionConverter parent, Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TranslateExpression(Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TranslateLambda(LambdaExpression lambda, DbExpression input)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TranslateLambda(LambdaExpression lambda, DbExpression input, DbExpressionBinding& binding)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MethodCallTranslator.OneLambdaTranslator.Translate(ExpressionConverter parent, MethodCallExpression call, DbExpression& source, DbExpressionBinding& sourceBinding, DbExpression& lambda)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MethodCallTranslator.OneLambdaTranslator.Translate(ExpressionConverter parent, MethodCallExpression call)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MethodCallTranslator.SequenceMethodTranslator.Translate(ExpressionConverter parent, MethodCallExpression call, SequenceMethod sequenceMethod)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MethodCallTranslator.TypedTranslate(ExpressionConverter parent, MethodCallExpression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TypedTranslator`1.Translate(ExpressionConverter parent, Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TranslateExpression(Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MethodCallTranslator.OneLambdaTranslator.Translate(ExpressionConverter parent, MethodCallExpression call, DbExpression& source, DbExpressionBinding& sourceBinding, DbExpression& lambda)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MethodCallTranslator.SelectTranslator.Translate(ExpressionConverter parent, MethodCallExpression call)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MethodCallTranslator.SequenceMethodTranslator.Translate(ExpressionConverter parent, MethodCallExpression call, SequenceMethod sequenceMethod)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.MethodCallTranslator.TypedTranslate(ExpressionConverter parent, MethodCallExpression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TypedTranslator`1.Translate(ExpressionConverter parent, Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.TranslateExpression(Expression linq)

   at System.Data.Entity.Core.Objects.ELinq.ExpressionConverter.Convert()

   at System.Data.Entity.Core.Objects.ELinq.ELinqQueryState.GetExecutionPlan(Nullable`1 forMergeOption)

   at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass41_0.<GetResults>b__1()

   at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)

   at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass41_0.<GetResults>b__0()

   at System.Data.Entity.Infrastructure.DbExecutionStrategy.Execute[TResult](Func`1 operation)

   at System.Data.Entity.Core.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)

   at System.Data.Entity.Core.Objects.ObjectQuery`1.<System.Collections.Generic.IEnumerable<T>.GetEnumerator>b__31_0()

   at System.Data.Entity.Internal.LazyEnumerator`1.MoveNext()

   at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)

   at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)

   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)

   at AutoMapper.Extensions.ExpressionMapping.Impl.SourceSourceInjectedQuery`2.GetEnumerator()

   at AutoMapper.Extensions.ExpressionMapping.Impl.SourceSourceInjectedQuery`2.System.Collections.IEnumerable.GetEnumerator()

   at System.Text.Json.JsonSerializer.HandleEnumerable(JsonClassInfo elementClassInfo, JsonSerializerOptions options, Utf8JsonWriter writer, WriteStack& state)

   at System.Text.Json.JsonSerializer.Write(Utf8JsonWriter writer, Int32 originalWriterDepth, Int32 flushThreshold, JsonSerializerOptions options, WriteStack& state)

   at System.Text.Json.JsonSerializer.WriteAsyncCore(Stream utf8Json, Object value, Type inputType, JsonSerializerOptions options, CancellationToken cancellationToken)

   at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)

   at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultAsync>g__Logged|21_0(ResourceInvoker invoker, IActionResult result)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|29_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()

--- End of stack trace from previous location where exception was thrown ---

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()

--- End of stack trace from previous location where exception was thrown ---

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)

   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)

   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)

   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)

   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

habbes on 14 Feb 2020

😕1

@AlbertoMonteiro I attempted to create a lambda expression and pass it directly to the efcore query:

Expression<Func<EventViewModel, bool>> filter = (it) =>
        (
            ((it.OrganizerId == (long?)8))
            &&
            ((it.StartDate == DateTimeOffset.UtcNow))
        ) == true;

This seems to get interpreted correctly and does not return an error.

habbes on 14 Feb 2020

👍1

@AlbertoMonteiro However, I also tried to construct an expression tree manually, similar to what OData constructs from the filter query, this does reproduce the error:
c# var leftEq = Expression.Equal( Expression.Property(itParam, typeof(EventViewModel).GetProperty("OrganizerId")), Expression.Constant((long?)8, typeof(long?)), true, null ); var rightEq = Expression.Convert( Expression.Equal( Expression.Property(itParam, "Id"), Expression.Constant(0L, typeof(long)), true, null ), typeof(Nullable<bool>) ); Expression body = Expression.MakeBinary( ExpressionType.AndAlso, leftEq, rightEq ); body = Expression.Equal(body, Expression.Constant(true, typeof(bool?)), liftToNull: false, method: null); var filter = Expression.Lambda(body, new[] { itParam });
It also reproduces the error observed in EF6. I'm not sure what's wrong with this expression, it looks fine to me. I also don't understand how different this tree is from the previous lambda expression. I assume that the conversions to nullable types and/or lifting of the operators to null.

habbes on 14 Feb 2020

👍1

When I remove the conversions to nullable, and disable lifting the operator expressions to null, EFCore returns a successful response:
c# var itParam = Expression.Parameter(typeof(EventViewModel), "$it"); var leftEq = Expression.Equal( Expression.Property(itParam, typeof(EventViewModel).GetProperty("OrganizerId")), Expression.Constant((long?)8, typeof(long?)), false, null ); var rightEq = Expression.Equal( Expression.Property(itParam, "Id"), Expression.Constant(0L, typeof(long)), false, null ); Expression body = Expression.MakeBinary( ExpressionType.AndAlso, leftEq, rightEq ); body = Expression.Equal(body, Expression.Constant(true, typeof(bool)), liftToNull: false, method: null); var filter = Expression.Lambda(body, new[] { itParam });
This corroborates the suspicion that the nullable conversion and lifting may have something to do with it. But I don't know why yet. But it's like to be an issue on the EF6/EFCore end as well.

habbes on 14 Feb 2020

👍1

Nice, at least for now we can still use this if we disable the HandleNullPropagation, I didn't faced no problem with this property disabled.

AlbertoMonteiro on 15 Feb 2020

I've changed the hardcoded expression a little bit, and from debug we can see that only the left side

DebugView
IsLifted
IsLiftedToNull

The Body of lambda still with IsLifted and IsLiftedToNull equals to false

Even without those changes, using your version see bellow the behavior is the same

Expression<Func<EventViewModel, bool>> filter = (it) =>
        (
            ((it.OrganizerId == (long?)8))
            &&
            ((it.StartDate == DateTimeOffset.UtcNow))
        ) == true;

The Lift values still the same

Using the version builded this is what we get

In this case we can see that all expression trees levels have IsLifted and IsLiftedToNull equals to true, except the root

I have one question what the reason of the OData querys generate expressions with the IsLiftedToNull to be true(when there is something nullable)?

AlbertoMonteiro on 15 Feb 2020

I tried to debug again, and in the first time of BindBinaryOperatorNode I trikied the execution with the debugger.

Skipped this line left = ToNullable(left);
Then changed liftToNull to false

Then I left the code runs, and it generated an executable expression.

I don't know anything about LiftToNull, I just discovered this when was debugging this issue 😛, but for the OData code, is this really necessary? Because in itself options do allow to disable this behavior, so since I lack knowledge about it and I can disable it I think that could be skipped.

Should I create an issue on EF Core aswell? Since the input expression ($it.organizerId == 8 && $it.id ==1) == true but when EF Core throws the exception it change ignore the parenthesis as you said in earlier comment.

AlbertoMonteiro on 15 Feb 2020

I looked for the unit tests and I found this one FilterQueryOptionTest.ApplyTo_Returns_Correct_Queryable, then I changed the test data to create a similar scenario that we face here, so I changed the line 35 to this { "SharePrice eq 1 and CustomerId eq 1", new int[0] }, since Customer class does have the SharePrice property that is a decimal?.

But when I ran the test with this data, the expression that is generated does have the LiftToNull to true as the expression in ReproProject also produces, but in the unit tests, EF handles the query properly.

😕

Just realized that the test for this method does not call EF ;D sorry

AlbertoMonteiro on 15 Feb 2020

Thanks @AlbertoMonteiro for looking into taking a closer look.

If there are tests that call EF, those would probably be in E2E tests.

As I understand it, a IsLiftedToNull means that the operator can operate on nullable types. I'm not sure of the actual difference between IsLifted and IsLiftedToNull since IsLifted can be true while IsLiftedToNull is false. I'm not sure in what other contexts an operator could be lifted.

If you lift the left Equals expression to null, then it will return a nullable boolean, and then the AndAlso expression will complain unless you also convert the right expression to return a nullable boolean. Maybe it's the conversion to nullable, and not the actual lifting that causes the error. In any case, if I set liftToNull:false across all the expressions, and avoid the conversion of the right operand to nullable, then it works just fine.

habbes on 17 Feb 2020

👍1

If I set the type of Id property to nullable (long?), and then I rewrite the expression tree to remove the explicit conversion to nullable, but still lift the expressions to nullable, i.e.:
c# var itParam = Expression.Parameter(typeof(EventViewModel), "$it"); var leftEq = Expression.Equal( Expression.Property(itParam, typeof(EventViewModel).GetProperty("OrganizerId")), Expression.Constant((long?)8, typeof(long?)), true, null ); var rightEq = Expression.Equal( Expression.Property(itParam, "Id"), Expression.Constant(0L, typeof(long?)), true, null ); Expression body = Expression.MakeBinary( ExpressionType.AndAlso, leftEq, rightEq, liftToNull: true, method: null ); body = Expression.Equal(body, Expression.Constant(true, typeof(bool?)), liftToNull: false, method: null); var filter = Expression.Lambda(body, new[] { itParam });

I don't get an error.

habbes on 17 Feb 2020

👍1

Interestingly, keeping Id as long (not nullable) and adding a gratuitous conversion to bool (not nullable) and setting litfToNull as false throughout does not throw an error:
c# var itParam = Expression.Parameter(typeof(EventViewModel), "$it"); var leftEq = Expression.Equal( Expression.Property(itParam, typeof(EventViewModel).GetProperty("OrganizerId")), Expression.Constant((long?)8, typeof(long?)), false, null ); var rightEq = Expression.Convert( Expression.Equal( Expression.Property(itParam, "Id"), Expression.Constant(0L, typeof(long)), false, null ), typeof(bool) ); Expression body = Expression.MakeBinary( ExpressionType.AndAlso, leftEq, rightEq, liftToNull: false, method: null ); body = Expression.Equal(body, Expression.Constant(true, typeof(bool)), liftToNull: false, method: null); var filter = Expression.Lambda(body, new[] { itParam });

habbes on 17 Feb 2020

👍1

So now I suspect that it's the conversion to a nullable that throws EFCore off.

I think IsLifted indicates that the operator is lifted so that it can operate on nullable types but does not necessarily return a nullable a type, and that IsLiftedToNull means that it also returns a nullable type, i.e. the result of the comparison can be null.

In the case of a filter operation, the final output is a boolean, not a nullable of boolean. The final step of the filterbinder seems to be the ApplyNullPropagationForFilterBody() method, which forces the return value to be a non-nullable boolean if the generated expression was boolean. I would therefore conclude that it's safe to set HandleNullPropagation to false for filter operations. But I can't call this a fix, seems like hiding the problem somewhat. What do you think @xuzhg

habbes on 17 Feb 2020

👍1

Yeah, so without knowing a complete context to know if the IsLiftedToNull is really needed I guess only in the EF query handler to fix this issue. I am anxious to see if @xuzhg does have something about that.
Thank you very much @habbes to get deep down on this issue, I've learned a lot.

What I discovered about Lift was in this stackoverflow question https://stackoverflow.com/a/58082623/815590

AlbertoMonteiro on 18 Feb 2020

@AlbertoMonteiro

Should I create an issue on EF Core aswell? Since the input expression ($it.organizerId == 8 && $it.id ==1) == true but when EF Core throws the exception it change ignore the parenthesis as you said in earlier comment.

Yeah I think it's a good idea to file the issue on EF Core as well, and maybe to include the expression trees that cause the error.

habbes on 18 Feb 2020

Was this page helpful?

0 / 5 - 0 ratings