Efcore: Query source has already been associated with an expression

[joshmouch]

I just upgraded from version 2.0.2 to 2.1.0-preview2-final, and I started getting an exception. The exception is:

System.InvalidOperationException
  HResult=0x80131509
  Message=Query source (from EntityOne <generated>_1 in value(Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1[My.EntityOne])) has already been associated with an expression.
  Source=Remotion.Linq
  StackTrace:
   at Remotion.Linq.Clauses.QuerySourceMapping.AddMapping(IQuerySource querySource, Expression expression)
   at Remotion.Linq.Clauses.MainFromClause.Clone(CloneContext cloneContext)
   at Remotion.Linq.QueryModel.Clone(QuerySourceMapping querySourceMapping)
   at Remotion.Linq.Clauses.ExpressionVisitors.CloningExpressionVisitor.VisitSubQuery(SubQueryExpression expression)
   at Remotion.Linq.Clauses.Expressions.SubQueryExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitUnary(UnaryExpression node)
   at System.Linq.Expressions.UnaryExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitBinary(BinaryExpression node)
   at System.Linq.Expressions.BinaryExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitBinary(BinaryExpression node)
   at System.Linq.Expressions.BinaryExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitUnary(UnaryExpression node)
   at System.Linq.Expressions.UnaryExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Remotion.Linq.Clauses.ExpressionVisitors.CloningExpressionVisitor.AdjustExpressionAfterCloning(Expression expression, QuerySourceMapping querySourceMapping)
   at Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.CorrelatedCollectionOptimizingVisitor.<>c__DisplayClass18_0.<Rewrite>b__1(Ordering o)
   at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()
   at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
   at Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.CorrelatedCollectionOptimizingVisitor.Rewrite(Int32 correlatedCollectionIndex, QueryModel collectionQueryModel, INavigation navigation, Boolean trackingQuery, QuerySourceReferenceExpression originQuerySource, Boolean forceListResult)
   at Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.CorrelatedCollectionOptimizingVisitor.TryRewrite(SubQueryExpression subQueryExpression, Boolean forceToListResult, Expression& result)
   at Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.CorrelatedCollectionOptimizingVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitMemberAssignment(MemberAssignment node)
   at System.Linq.Expressions.ExpressionVisitor.VisitMemberBinding(MemberBinding node)
   at System.Linq.Expressions.ExpressionVisitor.Visit[T](ReadOnlyCollection`1 nodes, Func`2 elementVisitor)
   at System.Linq.Expressions.ExpressionVisitor.VisitMemberInit(MemberInitExpression node)
   at System.Linq.Expressions.MemberInitExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor.TryOptimizeCorrelatedCollections(QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor.VisitSelectClause(SelectClause selectClause, QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryModelVisitor.VisitSelectClause(SelectClause selectClause, QueryModel queryModel)
   at Remotion.Linq.Clauses.SelectClause.Accept(IQueryModelVisitor visitor, QueryModel queryModel)
   at Remotion.Linq.QueryModelVisitorBase.VisitQueryModel(QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor.VisitQueryModel(QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryModelVisitor.VisitQueryModel(QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor.CreateAsyncQueryExecutor[TResult](QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Storage.Database.CompileAsyncQuery[TResult](QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileAsyncQueryCore[TResult](Expression query, IQueryModelGenerator queryModelGenerator, IDatabase database)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass22_0`1.<CompileAsyncQuery>b__0()
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddAsyncQuery[TResult](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileAsyncQuery[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync[TResult](Expression expression)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.System.Collections.Generic.IAsyncEnumerable<TResult>.GetEnumerator()
   at System.Linq.AsyncEnumerable.<Aggregate_>d__6`3.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

This occurs after creating a pretty complicated IQueryable. Then using this IQueryable to create a list of projected objects like so

var entityOneModelsQuery = from entityOne in entityOnesQuery 
select new EntityOneModel() {
    ...
    EntityTwos = entityOne.EntityTwos.Select(entityTwo => new EntityTwoModel
    {
        ...
    }).ToList()
};

var entityOneModels = entityOneModelsQuery.ToList();  /// Exception occurs here!

[maumar]

@joshmouch can you share the full query (and relevant entities) that cause the issue? I can't reproduce based on the sample you provided - the broken bit must be happening in entityOnesQuery construction. If you are using let keyword in the query that could be a problem.

Also, to unblock you can remove ToList from the EntityTwos part of the projection - this will revert to the old behavior (i.e. not try to optimize the correlated collections)

[ajcvickers]

EF Team Triage: Closing this issue as the requested additional details have not been provided and we have been unable to reproduce it.

BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.

[ghost]

Hi

I'm seeing the same issue, I've reduced my query to the following (it's basically the typical take the latest row per group pattern)

            var test = _context.Trades
                    .GroupBy(t => t.TradeId).Select(x => x.OrderByDescending(y => y.DateTimeCreated).First())
                    .Where(t => t.TradeDate <= valuationDate)
                    .Include(t => t.Instrument)
                    .Select(t => t)
                    .ToArray()
                    ;

throws 'Query source (from Trade y in [x]) has already been associated with an expression.'

Where Trade is a type which looks like

    [Table(nameof(Trade))]
    public class Trade
     {
        public int Id { get; set; }
        public int TradeId { get; set; }
        public DateTime TradeDate { get; set;}
        public int InstrumentId { get; set; }
        public Instrument Instrument { get; set; }
        public DateTimeOffset DateTimeCreated {get;set;}
    }

Removing any of Select / Include / Where eliminates the error.

[perriert]

I was trying to add two filters on a Count() based on a navigation property. I have a Head and a Details model with one:many relationship and wanted to filter Head records by the count of its Details when this error was thrown. It was doing fine for the first filter but immediately after the second failed.

My very ugly workaround until 2.2.0 is to create a DTO with three different versions of the same Details.Count() value and use filters on those one filter to one property.

The failing code (when both filter.Products.Min and filter.Products.Max had values):

public class RequestHead
{
    [Key, MaxLength(15)]
    public string Id { get; set; }
    public virtual ICollection<RequestDetail> Details { get; set; }
}
public class RequestDetail
{
    [Key]
    public int Id { get; set; }
    public string RequestHeadId { get; set; }
    [ForeignKey("RequestHeadId")]
    public RequestHead RequestHead { get; set; }
}
var requests = (from r in _ctx.RequestHeads
    select new RequestDTO {
        ProductsCount = r.Details != null ? (int)r.Details.Count() : 0
    });
if(filter.Products != null) {
    if (filter.Products.Min != null) {
        requests = requests.Where(x=>x.ProductsCount >= filter.Products.Min.Value);
    }
    if (filter.Products.Max != null) {
        requests = requests.Where(x=>x.ProductsCount <= filter.Products.Max.Value);
    }
}

[ajcvickers]

Note from puntage: this is dependent on Relinq. Revisit in 3.0 timeframe.

[sumgup]

I am hit by same issue where I am taking a latest row and applying a filter. .net 3.0 is still a way to go, how can this issue be workaround?

[ajcvickers]

@smitpatel @maumar Can you look into workarounds?

[sumgup]

Still awaiting response on this.

[smitpatel]

@sumit4palz - Unless you post your query and relevant model classes, there is no way for us to provide you a work-around.

[cyilcode]

We are also getting this exception in our project as well. We have been using(also the POC below);

• netcoreapp2.1
• EF Core 2.1.3

Our goal is to be able to calculate some properties from our models' properties on the database side so that we can be able to apply queries to those calculated properties on the database side as well.

Below, i have prepared a complete one class reproduction that is similar to our scenario and also some notes that i thought might be useful in the commented out sections.

```c#
using Microsoft.EntityFrameworkCore;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;
using System.ComponentModel.DataAnnotations.Schema;
using System.Linq;

namespace EfPOC
{
public class Dbtx : DbContext
{
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseSqlServer("Server=.;Database=EFTest;Trusted_Connection=True;");
base.OnConfiguring(optionsBuilder);
}

    public DbSet<Person> People { get; set; }

    public DbSet<Achievement> Achievements { get; set; }
}

public class Person
{
    [Key]
    public int Id { get; set; }

    public int Age { get; set; }

    public IList<Achievement> Achievements { get; set; }

    [NotMapped] // A property that we want to calculate on the db side and not include in the People table as a column.
    public int Fizz { get; set; }

    [NotMapped] // A property that we want to calculate on the db side and not include in the People table as a column.
    public int Buzz { get; set; }
}

public class Achievement
{
    [Key]
    public int Id { get; set; }

    public string Something { get; set; }
}

class Program
{
    static void Main(string[] args)
    {
        using (var db = new Dbtx())
        {
            var t = db.People.Select(x => new Person
            {
                Id = x.Id,
                Age = x.Age,
                Achievements = x.Achievements, // Removing this line makes it work but we also need this as well.
                Fizz = x.Achievements.Count(),
                Buzz = x.Achievements.Select(xz => xz.Something).Distinct().Count() // Removing .Select here makes it work too.
            }).OrderBy(x => x.Buzz) // Removing either this OrderBy or Where clause also makes it work as well.
            .Where(x => x.Buzz == 4);

            var b = t.ToList(); // Exception here.
        }
    }
}

}

As far as i understand, the query above is not able to handle more than one filter/sort clause for a reason that i could not see.

Exception 

```c#
An unhandled exception of type 'System.InvalidOperationException' occurred in System.Private.CoreLib.dll: 'Query source (from Achievement xz in value(Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1[EfPOC.Achievement])) has already been associated with an expression.'
   at Remotion.Linq.Clauses.QuerySourceMapping.AddMapping(IQuerySource querySource, Expression expression)
   at Remotion.Linq.Clauses.MainFromClause.Clone(CloneContext cloneContext)
   at Remotion.Linq.QueryModel.Clone(QuerySourceMapping querySourceMapping)
   at Remotion.Linq.QueryModel.CloningExpressionVisitor.VisitSubQuery(SubQueryExpression expression)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitBinary(BinaryExpression node)
   at System.Linq.Expressions.BinaryExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Remotion.Linq.Clauses.WhereClause.TransformExpressions(Func`2 transformation)
   at Remotion.Linq.QueryModel.TransformExpressions(Func`2 transformation)
   at Remotion.Linq.QueryModel.Clone(QuerySourceMapping querySourceMapping)
   at Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.CorrelatedCollectionOptimizingVisitor.Rewrite(Int32 correlatedCollectionIndex, QueryModel collectionQueryModel, INavigation navigation, Boolean trackingQuery, QuerySourceReferenceExpression originQuerySource, Boolean forceListResult, Type listResultElementType)
   at Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.CorrelatedCollectionOptimizingVisitor.TryRewrite(SubQueryExpression subQueryExpression, Boolean forceToListResult, Type listResultElementType, Expression& result)
   at Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.CorrelatedCollectionOptimizingVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitUnary(UnaryExpression node)
   at System.Linq.Expressions.UnaryExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitMemberAssignment(MemberAssignment node)
   at System.Linq.Expressions.ExpressionVisitor.VisitMemberBinding(MemberBinding node)
   at System.Linq.Expressions.ExpressionVisitor.Visit[T](ReadOnlyCollection`1 nodes, Func`2 elementVisitor)
   at System.Linq.Expressions.ExpressionVisitor.VisitMemberInit(MemberInitExpression node)
   at System.Linq.Expressions.MemberInitExpression.Accept(ExpressionVisitor visitor)
   at Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor.TryOptimizeCorrelatedCollections(QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor.VisitSelectClause(SelectClause selectClause, QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryModelVisitor.VisitSelectClause(SelectClause selectClause, QueryModel queryModel)
   at Remotion.Linq.QueryModelVisitorBase.VisitQueryModel(QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor.VisitQueryModel(QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryModelVisitor.VisitQueryModel(QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor.CreateQueryExecutor[TResult](QueryModel queryModel)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](Expression query, IQueryModelGenerator queryModelGenerator, IDatabase database, IDiagnosticsLogger`1 logger, Type contextType)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass13_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)
   at Remotion.Linq.QueryableBase`1.GetEnumerator()
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at EfPOC.Program.Main(String[] args) in C:\EfPOC\Program.cs:line 63

I have found a way to make things work like below.

• Make the collection member an IEnumerable and assign a query that would return the whole set in your projection like;

```c#
public class Person
{
// .. other properties
// public IList Achievements { get; set; } This becomes below
public IEnumerable Achievements { get; set; }
// .. other properties
}

and the query becomes
```c#
var t = db.People.Select(x => new Person
                {
                    Id = x.Id,
                    Age = x.Age,
                    Achievements = x.Achievements.Where(x => x != null), // Added .Where here
                    Fizz = x.Achievements.Count(),
                    Buzz = x.Achievements.Select(xz => xz.Something).Distinct().Count()
                }).OrderBy(x => x.Buzz)
                .Where(x => x.Buzz == 4);

However, this requires a lot of code change and not very elegant and the only reason why i am sharing this is because it might suit for someone else for a quick fix or it might give some opinions to the developers of EFCore if it is occuring because of a bug in the EF Code.

I would appreciate if you would share your opinions / solutions for our problem.

Thanks.

[ajcvickers]

@smitpatel @maumar Can you look into workarounds?

[maumar]

@cyilcode slightly less involved workaround would be to change Achievements to something like this:

Achievements = x.Achievements.Skip(0).ToList()

The problem is with the correlated collection feature, which tries to optimize projection of collection navigations. It's caused by bug in external package that we use (https://www.re-motion.org/jira/browse/RMLNQ-111)

The optimzation doesn't work if the collection is using a paging operation, so Skip(0) can effectively disable it without changing the result.

@divega @ajcvickers should we add a way (e.g. via context options) to disable toe correlated collection optimizations for cases like this, where it's not trivial to disable it in the query?

[cyilcode]

@maumar Thank you for the workaround suggestion. Also, it would be really nice to have a configuration as you have suggested.

[cmpcdaly]

So I'm having the same issue, but adding the pagination operation disables eager loading - causing quite a performance hit! Is there another workaround? What's also interesting is that the navigation property causing the exception isn't even projected in the statement throwing the exception.

[maumar]

@cmpcdaly you can try rewriting the query so that collection navigation projection is the last operation - if you have OrderBy, Where, etc afterwards

[AndriySvyryd]

@smitpatel Duplicate of #16752?

[smitpatel]

I would say we should just close this. Looking at comments, there were multiple different cases in which this error happened. Mainly with IncludeCompiler and correlated subquery optimization. We don't have both code paths anymore. It is not black and white if the queries which threw this exception were not supported or bug. Probably should mark as fixed.

[AndriySvyryd]

Please file new issues if the above scenarios are not working in 3.0