Efcore: Query: Allow use of AsQueryable method

[colltoaction]

Update

Modifying the description to track support for AsQueryable in linq queries.
AsQueryable method is required when user wants to pass Expression to linq method which is hanging of an navigation property since navigation is of type IEnumerable. With the relinq fix now we can get parsed query with AsQueryable but we are client eval-ing afterwards. This issue is to track work needed on EF Core side to translate query to server.

Query
var query3 = db.Products.Select(p => p.ProductCategories.AsQueryable().Select(pc => pc.Category).Where(Category.IsGenre)).ToList();

QueryExecution:

dbug: Microsoft.EntityFrameworkCore.Query[10104]
      Optimized query model: 
      'from Product p in DbSet<Product>
      select 
          from ProductCategory pc in 
              (from ProductCategory <generated>_1 in DbSet<ProductCategory>
              where Property([p], "ProductId") ?= Property([<generated>_1], "ProductId")
              select [<generated>_1]).AsQueryable()
          join Category pc.Category in DbSet<Category>
          on Property([pc], "CategoryId") equals Property([pc.Category], "CategoryId")
          where [pc.Category].ParentId == (Nullable<Guid>)__Genre_0
          select [pc.Category]'
warn: Microsoft.EntityFrameworkCore.Query[20500]
      The LINQ expression 'join Category pc.Category in value(Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1[EFSampleApp.Category]) on Property([pc], "CategoryId") equals Property([pc.Category], "CategoryId")' could not be translated and will be evaluated locally.
warn: Microsoft.EntityFrameworkCore.Query[20500]
      The LINQ expression 'where ([pc.Category].ParentId == Convert(__Genre_0, Nullable`1))' could not be translated and will be evaluated locally.
dbug: Microsoft.EntityFrameworkCore.Query[10107]
      (QueryContext queryContext) => IEnumerable<IOrderedQueryable<Category>> _InterceptExceptions(
          source: IEnumerable<IOrderedQueryable<Category>> _ShapedQuery(
              queryContext: queryContext, 
              shaperCommandContext: SelectExpression: 
                  SELECT [p].[ProductId]
                  FROM [Commerce].[Product] AS [p], 
              shaper: TypedProjectionShaper<ValueBufferShaper, ValueBuffer, IOrderedQueryable<Category>>), 
          contextType: EFSampleApp.MyContext, 
          logger: DiagnosticsLogger<Query>, 
          queryContext: queryContext)

Original Issue

Hi!

I'm trying to map an existing database to EF Core. Our project has a terrible model where we have to compare the ParentId to a specific Guid to find out the type of a row. E.g. we have the table Categories and each Guid identifies Genre, Mood, etc. (we have a music app).

So I'm trying to write this property in the Category class, but if I do, I'm unable to use Include because it can't be translated:

public virtual bool IsGenre => ParentId == Genre;
private static Guid Genre = Guid.Parse("3CA9FA61-EB62-4480-B476-867F78A9ADB3")

If I instead do ctx.Categories.Where(c => c.ParentId == Guid.Parse("3CA9FA61-EB62-4480-B476-867F78A9ADB3") it works perfectly.

I'm wondering if there's any way to move this to the Category class so I can avoid copy and pasting for each query I need to write.

I tried to manually create the Expression:

public static Expression<Func<Category, bool>> IsGenre = c => c.ParentId == Genre;

but I get the following error on p.ProductCategories.Select(pc => pc.Category).Where(Category.IsGenre):

Error   CS1929  'IEnumerable<Category>' does not contain a definition for 'Where' and the best extension method overload 'Queryable.Where<Category>(IQueryable<Category>, Expression<Func<Category, bool>>)' requires a receiver of type 'IQueryable<Category>'

With .AsQueryable() I'm able to compile the program but I get This overload of the method 'System.Linq.Queryable.AsQueryable' is currently not supported.

Thanks!

[smitpatel]

I could not reproduce this. Below is the code i used. Can you make modifications in it according to what you are trying to do?

``` C#
public class Program
{
public static void Main(string[] args)
{
using (var ctx = new MyContext())
{
ctx.Database.EnsureDeleted();
ctx.Database.EnsureCreated();

        var query1 =
            ctx.Categories.Where(c => c.ParentId == Guid.Parse("3CA9FA61-EB62-4480-B476-867F78A9ADB3")).ToList();

        var query2 = ctx.ProductCategories.Select(pc => pc.Category).Where(Category.IsGenre).ToList();

    }
}

}

public class MyContext : DbContext
{
public DbSet ProductCategories { get; set; }
public DbSet Categories { get; set; }

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    optionsBuilder.UseSqlServer(@"Server=(localdb)\mssqllocaldb;Database=_ModelApp;Trusted_Connection=True;");
}

}

public class ProductCategory
{
public Guid Id { get; set; }
public Category Category { get; set; }
}

public class Category
{
private static Guid Genre = Guid.Parse("3CA9FA61-EB62-4480-B476-867F78A9ADB3");
public static Expression> IsGenre = c => c.ParentId == Genre;

public Guid Id { get; set; }
public Guid ParentId { get; set; }
public ProductCategory Parent { get; set; }

}
```

[colltoaction]

This is actually a many-to-many relationship. Please try the code below (and let me know if you need seed data):

using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Metadata;

namespace EFIssue
{
    public class Program
    {
        public static void Main(string[] args)
        {
            using (var ctx = new MyContext())
            {
                var query1 =
                    ctx.Categories.Where(c => c.ParentId == Guid.Parse("3CA9FA61-EB62-4480-B476-867F78A9ADB3")).ToList();

                var query2 = ctx.ProductCategories.Select(pc => pc.Category).Where(Category.IsGenre).ToList();

                var query3 = ctx.Products.Select(p =>
                    p.ProductCategories.Select(pc => pc.Category)
                                       .Where(Category.IsGenre))
                    .ToList();
            }
        }
    }

    public class MyContext : DbContext
    {
        public DbSet<Product> Products { get; set; }
        public DbSet<ProductCategory> ProductCategories { get; set; }
        public DbSet<Category> Categories { get; set; }

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder.UseSqlServer(@"Server=(localdb)\mssqllocaldb;Database=_ModelApp;Trusted_Connection=True;");
        }

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            modelBuilder.Entity<Category>(entity =>
            {
                entity.HasKey(e => e.CategoryId)
                    .HasName("PK__Category__19093A0BEC90FC43");

                entity.ToTable("Category", "Commerce");

                entity.Property(e => e.CategoryId).ValueGeneratedNever();

                entity.Property(e => e.Name).HasMaxLength(255);
            });

            modelBuilder.Entity<Product>(entity =>
            {
                entity.ToTable("Product", "Commerce");

                entity.Property(e => e.ProductId).ValueGeneratedNever();

                entity.Property(e => e.Name).HasMaxLength(255);
            });

            modelBuilder.Entity<ProductCategory>(entity =>
            {
                entity.ToTable("ProductCategory", "Commerce");

                entity.HasIndex(e => new { e.ProductId, e.CategoryId })
                    .HasName("nci_wi_ProductCategory_E47C12BC-D5AC-4997-BCF5-AD7A7AF8751B");

                entity.HasIndex(e => new { e.CategoryId, e.Id, e.ProductId })
                    .HasName("nci_wi_ProductCategory_C39551F634191AFB3482");

                entity.Property(e => e.Id).HasColumnName("id");

                entity.HasOne(e => e.Category)
                    .WithMany(c => c.ProductCategories)
                    .HasForeignKey(e => e.CategoryId)
                    .OnDelete(DeleteBehavior.Cascade)
                    .HasConstraintName("FK__ProductCa__Categ__0D44F85C");

                entity.HasOne(e => e.Product)
                    .WithMany(p => p.ProductCategories)
                    .HasForeignKey(e => e.ProductId)
                    .OnDelete(DeleteBehavior.Cascade)
                    .HasConstraintName("FK__ProductCa__Produ__0C50D423");
            });
        }
    }

    public class Product
    {
        public Product()
        {
            ProductCategories = new HashSet<ProductCategory>();
        }

        public Guid ProductId { get; set; }
        public string Name { get; set; }

        public virtual ICollection<ProductCategory> ProductCategories { get; set; }
    }

    public class ProductCategory
    {
        public Guid CategoryId { get; set; }
        public Guid ProductId { get; set; }
        public int Id { get; set; }

        public virtual Category Category { get; set; }
        public virtual Product Product { get; set; }
    }

    public class Category
    {
        public Guid CategoryId { get; set; }
        public Guid? ParentId { get; set; }
        public string Name { get; set; }

        public virtual ICollection<ProductCategory> ProductCategories { get; set; }

        public static Expression<Func<Category, bool>> IsGenre = c => c.ParentId == Genre;
        // public virtual bool IsGenre => ParentId == Genre;
        private static Guid Genre = Guid.Parse("3CA9FA61-EB62-4480-B476-867F78A9ADB3");
    }
}

[smitpatel]

@tinchou - Thanks for working repro code. Can you tell in words what you are trying to query so I can translate it to linq.

[colltoaction]

In my real app I would like to:

1. Find some products based on some criteria (ctx.Products.Where(...))
2. Find the categories related to that product (I have to go through the ProductCategories intermediate table since it's a many-to-many relationship)

[smitpatel]

I tried running different queries. The queries in repro code are based criteria on category (rather than product as you mentioned in previous post) but since relationship is many to many, it can be easily written from other way depending on what you want to query.
Query1: Categories based on condition
Query2: ProductCategories based on condition on Category property
Query3: Categories based on condition with data for ProductCategories and products populated.
Query4: List of Products for all ProductCategories of (Category filtered by condition)
Query5: Similary to query you were writing, with forced client eval.

var query1 =
    ctx.Categories.Where(c => c.ParentId == Guid.Parse("3CA9FA61-EB62-4480-B476-867F78A9ADB3")).ToList();

var query2 = ctx.ProductCategories.Select(pc => pc.Category).Where(Category.IsGenre).ToList();

var query3 = ctx.Categories
    .Where(Category.IsGenre)
    .Include(c => c.ProductCategories)
    .ThenInclude(pc => pc.Category)
    .ToList();

var query4 = ctx.Categories.Where(Category.IsGenre)
    .SelectMany(c => c.ProductCategories)
    .Select(pc => pc.Product)
    .ToList();

var query5 = ctx.Products.ToList()
    .Where(p => p.ProductCategories.Select(pc => pc.Category).AsQueryable().Any(Category.IsGenre))
    .ToList();

As you can see you can still get products for categories even with starting from categories db set.
The issue with the query you were writing,
First you need to use Where instead of Select since you want to filter products and not project something from products. And that still would not work without AsQueryable() call since it is Enumerable. The AsQueryable() call is correct compilation wise. But the issue is EF uses Remotion.Linq to parse the query model which is not able to parse AsQueryable() method call yet therefore it throws runtime exception. Doing ToList() on product would mean that Where clause will be evaluated locally using Linq though it would also load whole table from database.
Query4 should be proper way to achieve what you are trying to query.

@anpete

[rowanmiller]

@smitpatel anything to do on the EF side here?

[smitpatel]

From discussions so far, the way user wants to write query is not possible due to limited from re-linq. And there is alternate way to write the query which gives same results.

[divega]

I was curious about this one because it ringed a bell so I took a look:

In the past I found it very useful to create repositories that take predicates of the form Expression<Func<T, bool>> in their methods and I also hit the need to call AsQueryable() just to be able to use those predicates in the context of collection nav. props.

It was very handy that AsQueryable() worked in the old EF stack to satisfy this scenario.

I agree with @smitpatel that in some cases it should be possible to change your code to get the same data loaded from the other side, but I don't think that it is true in all cases, e.g. it won't help if the reverse nav. prop. isn't present. It might also be harder to figure out what query to write.

If this is a fundamental issue in re-linq with AsQueryable() I think it might be worth passing the bug to them.

[rowanmiller]

@smitpatel could you pass off a request to re-linq to look at removing this limitation, then you can close this as external.

[colltoaction]

Fine with me. Please post here if you open an issue on re-linq so I can keep an eye on it!

[rowanmiller]

BTW re-linq issue tracker is at https://www.re-motion.org/jira/browse/RMLNQ

[smitpatel]

Found the existing bug in re-linq
https://www.re-motion.org/jira/projects/RMLNQ/issues/RMLNQ-28?filter=allopenissues

[divega]

cc @MichaelKetting as heads up that we would appreciate if https://www.re-motion.org/jira/projects/RMLNQ/issues/RMLNQ-28 got looked at to unblock us from supporting AsQueryable() in EF Core. Thanks!

[MichaelKetting]

@divega I've taken a quick look at the solution proposed in our Jira ticket and if the custom AsQueryableNode really is all that's needed, you don't need to wait for re-linq to implement this. In EF's QueryCompiler.CreateNodeTypeProvider you already use the extension point for custom node types.

Mind you, I haven't gotten to actually trying the sample from our ticket, but if this works, this would certainly give you the most effective way forward.

BTW, thanks for pointing out that this isn't implemented yet. I've prioritized it for the next release but if the quick fix works, I'd prefer to treat this as nice-to-have.

Edit: Come to think about it, 'most effective' might not be correct since EF just did a 1.0 release and if I fix this in re-linq, tincho could just upgrade re-linq and needn't wait for the next EF release. Interesting thing, this semantic versioning in combination with NuGet...

[Igoreso]

Is there any update? Or maybe a workaround (that doesn't involve building own EF)?
It doesn't seem like this issue have been worked on on re-linq side in almost a year.

[smitpatel]

Reopening the issue and removing from milestone for triage. We need to decide if we want to do something to translate AsQueryable or wait for fix from relinq. The comment above been stale (though may be similar grounds again given we released 2.0 now).

P.S. - if we decide to add support with custom extension, we should file new issue tracking it.

[smitpatel]

@MichaelKetting - Any estimate when fix for https://www.re-motion.org/jira/projects/RMLNQ/issues/RMLNQ-28?filter=allopenissues will be available or should we go ahead and write code on our end as suggested?

[MichaelKetting]

I've got some vacation days coming up the last week of October so that should work out nicely for me having some time on my hands :)

[techniq]

Any updates on this issue as it is a stopper for https://github.com/OData/WebApi/issues/1154? Thanks

[MichaelKetting]

:( My plans for my October vacation fell through, as you might ahve guessed. There's now the Christmas holidays coming up, so that's two weeks and I've the open linq stuff scheduled in.

[techniq]

No worries, and I appreciate your work.

On Thu, Dec 21, 2017, 11:13 AM Michael Ketting [e-mail]
wrote:

:( My plans for my October vacation fell through, as you might ahve
guessed. There's now the Christmas holidays coming up, so that's two weeks
and I've the open linq stuff scheduled in.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aspnet/EntityFrameworkCore/issues/6132#issuecomment-353390910,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAK1RORRImSIqlMWRz-ra7EGbLBvMMMrks5tCoPEgaJpZM4JSO36
.

[techniq]

@MichaelKetting having any luck (and don't let it interfere with your vacation, just trying to determine our schedule and possible alternative approaches).

[MichaelKetting]

@techniq yeah, finishing some work-stuff right now so I'm already in the zone (yay!) and will get to linq-stuff on Thursday.

[techniq]

@MichaelKetting Awesome! Thank you

[MichaelKetting]

@techniq just letting you know, the implementation is under way (https://github.com/re-motion/Relinq/pull/10) but there is an open question that I need to discuss with a colleague of mine before I can close this one up (https://www.re-motion.org/jira/browse/RMLNQ-28?focusedCommentId=20896&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-20896).

[techniq]

Thanks for your work and update. I'll be keeping an eye on these.

On Sun, Jan 7, 2018, 4:49 PM Michael Ketting [e-mail]
wrote:

@techniq https://github.com/techniq just letting you know, the
implementation is under way (re-motion/Relinq#10
https://github.com/re-motion/Relinq/pull/10) but there is an open
question that I need to discuss with a colleague of mine before I can close
this one up (
https://www.re-motion.org/jira/browse/RMLNQ-28?focusedCommentId=20896&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-20896
).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aspnet/EntityFrameworkCore/issues/6132#issuecomment-355855531,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAK1RBMqBPOGMHIFv7CXI1PYr6tyg6nuks5tITwBgaJpZM4JSO36
.

[techniq]

@MichaelKetting - ultimately my hope is to have a functional OData implementation on ASP.NET Core, including support for OData's aggregate. It appears the error I'm getting when attempting to query with an aggregate will be resolved as a direct result of the work you're doing (adding support for .AsQueryable() for EF Core / Linq, unless there is another issue after this one is resolve). I know there is also a more general Support for Entity Framework Core for OData being tracked.

A few questions:

• When this change lands, I should be able to test it out using the EF Core nightlies right?
• Do you know of any other open EF Core items that might need to be addressed beyond the .AsQueryable(). I haven't been able to test it much yet as support for OData on ASP.NET Core has only been available in nightlies/beta for a few weeks, but plan to start kicking it around pretty hard in the next week or so on.

[MichaelKetting]

@techniq

• Yes, it should be a fully compatible release. (am doing SemVer, so... ;) )
• Issues relevant to EF Core: https://www.re-motion.org/jira/issues/?filter=11440

Regarding the current issue; I should be able to close up the PR tomorrow and will then drop a new alpha-build. The full build will be when it's needed/requested by our downstream customers, i.e. I'll keep trying to get fixes into the next version until there's an actual release dependency :)

[techniq]

@MichaelKetting

Regarding the alpha-build, that is in regards to Relinq, correct? From my understanding, at some point you'll need to cut a new major release (due to breaking changes in Relinq) and EFCore will then depend on this version.

If so, I guess the only way to test your change with regards to OData is to checkout EF Core, update the dependency, build, and use this?

Any idea how quickly this might get into EF Core (with regards to the nightlies) if all of my understandings are correct?

[MichaelKetting]

@techniq Yes, that's re-linq.

So far I have been able to keep things non-breaking but eventually there could very well be a time when this isn't going to work any longer (or takes too much effort for a sensible cost/benefit).

For now, you can just set your own dependency to re-linq v2.2.0-alpha.005 in OData (or even just a test project) and let the binding redirects do the rest. No recompile of EF Core needed. (Disclaimer: I hope all the binding redirect generating works in your scenario, there are some issues in the .NET Core MSBuild files about this.)

Either way, I think it's fairly straight forward to get the new version into EF Core, but you'll have to knock there for this :)

Edit: fixed version number

[MichaelKetting]

@techniq Just to let you know, got some usage feedback and will need to do some additonal feature work regarding the QueryModel, so might be done tonight or during the weekend.

[techniq]

Thanks for the update

On Wed, Jan 10, 2018, 2:32 AM Michael Ketting [e-mail]
wrote:

@techniq https://github.com/techniq Just to let you know, got some
usage feedback and will need to do some additonal feature work regarding
the QueryModel, so might be done tonight or during the weekend.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aspnet/EntityFrameworkCore/issues/6132#issuecomment-356522321,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAK1RCvBDp7UhHyS_wkvixiSNuUDyk6lks5tJGeBgaJpZM4JSO36
.

[techniq]

@MichaelKetting any updates by chance?

[MichaelKetting]

@techniq yeah, I just managed to push the AsQueryable feature in 2.2.0-alpha.5 :)

Edit: fixed version number.

[techniq]

I just got the email on the merge. Thanks much. I'll give it a shot with
binding redirects (hopefully) in the next day or so. I really appreciate
you taking care of this.

On Sun, Jan 21, 2018, 4:58 PM Michael Ketting [e-mail]
wrote:

@techniq https://github.com/techniq yeah, I just managed to push the
AsQueryable feature in 2.0.0-alpha.5 :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aspnet/EntityFrameworkCore/issues/6132#issuecomment-359284743,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAK1RF8kWkRAUy6nwKfjbuqipLD2RCufks5tM7L_gaJpZM4JSO36
.

[techniq]

@MichaelKetting Just a heads up, I added the Remotion.Linq 2.0.0-alpha.5 to my project, which fixed the System.NotSupportedException: Could not parse expression '$it.AsQueryable()': This overload of the method 'System.Linq.Queryable.AsQueryable' is currently not supported. exception, but now the dotnet process exits without any stack trace:

I'm not sure if the problem lies within EFCore, OData / ASP.NET Core, or Remotion.Linq. Any thoughts?

[MichaelKetting]

@techniq Hmm...I've usually seen this with StackOverflowExceptions. You could try to install Remotion.Linq v2.2.0-alpha.4 to check if the problem is related to the AsQueryable-feature. That was the only change in alpha.5, so can do an A-B test very easily. Then there's the option of attaching a debugger and debugging the exceptions. A StackOverflow excpetion should show up in the debugger.

Erm...I just noticed, I may have written "v2.0.0-alpha.5" instead of "v2.2.0-alpha.5" in my previous reply. Could you check that one first?

Edit: we never had a v2.0.0-alpha.5, so at least you couldn't have downloaded the wrong version. Maybe the wrong binding redirect, though.

[techniq]

@MichaelKetting As you determined, I definitely have the v2.2.0-alpha.5 version. If I remove Remotion.Linq from my project I returned to the System.NotSupportedException exception about AsQueryable not being implemented so it is definitely using your changes (and should indicate the binding redirects are kicking in, at least I think so).

I was using dotnet on Mac but decided to open up Visual Studio on my Windows VM to see if I could get any more insight into the problem and see this unhandled exception mesage at the bottom:

An unhandled exception of type 'System.AccessViolationException' occurred in Microsoft.AspNetCore.OData.dll
Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

With that, it appears to be a problem on the ASP.NET Core OData side and I'll log an issue over there (although I'm not for certain if it's not the result of the interplay between EFCore, OData, and Remotion.Linq).

[MichaelKetting]

Thanks for the update. The AccessViolationException sounds definitely strange. My best guesses would either be an undetected API change in the alpha.5 (which would really surprise me if I missed that) or some issue when working with unmanged code during the query execution. But I'll adopt a wait-and-see approch for now :)

[smitpatel]

@MichaelKetting - Thanks for the fix. We are now able to get parsed query model. Can you also take a look at https://github.com/aspnet/EntityFrameworkCore/issues/9311. We are at present in preview2 and would soon need RTM version of re-linq to use in our RTM version. (else we would have to revert back to 2.1.1) Can you give an estimate when can RTM version be available?

Removing milestone of this issue to track work needed to get translation to server.

[MichaelKetting]

@smitpatel I can do an RTM build as needed, the question is, should I try to take a shot at the two issues in https://www.re-motion.org/jira/issues/?filter=11440 over the weekend and drop a beta version for you to try first and then an RTM build sometime next week? Or just go with what's already in the bag?

[smitpatel]

@MichaelKetting - Sorry for late response. Sure go ahead with more bug fixes if you want. One week wait is no issue for us. Let me know if you drop another beta version, I can certainly help validate it against EF tests.

[ajcvickers]

Fix it if it's easy in 2.1, else punt.

[anpete]

Confirmed fixed after we take the new version of relinq.

[MichaelKetting]

Okay, didn't manage the OrderBy with Custom Comparer and the QueryModel Bug for v2.2. Sorry about that, but on the plus side the v2.2 rtm build's just gotten sent off to the build server.

[smitpatel]

Thanks @MichaelKetting

[anpete]

Added test in e2a2b9b06a