Efcore: Support partial loading (i.e. not eagerly loading all scalar properties)

[lvasiliou]

Can you please add partial loading and by that I mean in addition to lazy and eager loading of relationships, the third option - lets call it partial for now - loads the related entity but only containing the primary keys and no other data. If needed I could resolve the rest of the data manually.
For example I have a Contact entity with one to many relationship with Address. With partial loading, I will get Contact.Address.AddressId only. I could then resolve the rest of the properties manually. Maybe all maybe the ones I specify via DbContext.Entry(address).Resolve(a=>a.Line1;....

[ajcvickers]

See #9385 for some more ideas in this area.

[ajcvickers]

See #8138 for additional ideas in this area--specifically around lazy loading of scalar properties.

[faibistes]

As a workaround (and that's good enough for me), I do something like this:
Let's assume we have a "estado" table with the field Id as PK and a big text column called "observaciones". This is how you define the entities:

public partial class Estado
{
    public Guid Id { get; set; }
    public DateTime Fecha { get; set; }
    public Observaciones Observaciones { get; set; }
}
public class Observaciones
{
    public Guid Estado { get; set; }
    public string Texto { get; set; }
    public Estado Estado { get; set; }
}
modelBuilder.Entity<Observaciones>(entity =>
{
    entity.HasKey(e => e.Id)
        .HasName("PK_estado");
    entity.HasOne(e => e.Estado).WithOne(o => o.Observaciones).HasForeignKey<Estado>(e => e.Id);
    entity.Property(e => e.Texto)
        .HasColumnName("observaciones")
        .HasColumnType("text");
    entity.ToTable("estado", "reactivos");
});
modelBuilder.Entity<Estado>(entity =>
{
    entity.HasKey(e =>  e.Id)
        .HasName("PK_estado");
    entity.ToTable("estado", "reactivos");
    entity.Property(e => e.Id).HasColumnName("id");
    entity.Property(e => e.Fecha).HasColumnName("fecha");
    entity.HasOne(e => e.Observaciones).WithOne(o => o.Estado).HasForeignKey<Observaciones>(e => e.Id);
});

This works with EF 2.0 (you need to Include() to load observaciones column) I guess that this will lazy load if you define Observaciones and Estado as virtual in EF 2.1, but I haven't tested it.

[bbrandt]

@faibistes Does this produce one query or 2 queries against the same table? If one query, is the table joining with itself on the Id? Any issues from this?

[hmdhasani]

I have tested faibistes solutions and it worked for EF Core 2.1.
also I tired to set Observaciones as Owned Entity but I figured out that Owned Entities are included by default.
If we had an option for owned Entities to not be included by default and be in select part only when they are included, that would be a cleaner solution.

[AndriySvyryd]

@hmdhasani The solution @faibistes showed is Table splitting, that's the recommended way to lazy load scalar properties.

Owned entities will always be eagerly loaded.

[hmdhasani]

Ok, anyway the downside of using table split is adding an unnecessary complexity to data model that can be a high cost for big data models.
It would be more simple if we can do something like these:

var images = _context.Set<Image>()
                           .Ignore(i => i.Content)
                           .ToList();

And

var documents = _context.Set<Document>()
                                    .Include(d => d.Images)
                                        .Ignore(i => i.Content)
                                    .ToList();

I wonder is there currently any way to do this? ( maybe by somehow interposition or manipulation of query generation process )

[hmdhasani]

After trying some approaches I ended up with this workaround:
The base idea is to turn for example this query:

SELECT [f].[Id], [f].[Report], [f].[CreationDate]
FROM [File] AS [f]

into this:

SELECT [f].[Id], '' as [Report], [f].[CreationDate]
FROM [File] AS [f]

by overriding DefaultQuerySqlGenerator.GenerateList()
when the query is like this:

var files = context.Files.AsNoTracking()
                         .IgnoreProperty(f => f.Report)
                         .ToList();

I added AsNoTracking() to become sure the value of ignored column will not change.

here is the full code: ( most parts were taken from https://www.chasingdevops.com/sql-generation-ef-core/ )

public static class IQueryableExtensions
    {
        internal static readonly MethodInfo _ignorePropertyMethodInfo
          = typeof(IQueryableExtensions).GetTypeInfo().GetDeclaredMethod(nameof(IgnoreProperty));

        public static IQueryable<TEntity> IgnoreProperty<TEntity, TProperty>(this IQueryable<TEntity> source, Expression<Func<TEntity, TProperty>> propertyPath) where TEntity : class
            => source.Provider is EntityQueryProvider
                ? source.Provider.CreateQuery<TEntity>(
                  Expression.Call(
                    instance: null,
                    method: _ignorePropertyMethodInfo.MakeGenericMethod(typeof(TEntity), typeof(TProperty)),
                    arguments: new Expression[] { source.Expression, propertyPath }))
                : source;
    }

    internal class IgnorePropertyResultOperator : SequenceTypePreservingResultOperatorBase, IQueryAnnotation
    {
        public IQuerySource QuerySource { get; set; }
        public QueryModel QueryModel { get; set; }

        public LambdaExpression PropertyPathLambda { get; set; }

        public override ResultOperatorBase Clone(CloneContext cloneContext)
          => new IgnorePropertyResultOperator();

        public override StreamedSequence ExecuteInMemory<T>(StreamedSequence input) => input;

        public override void TransformExpressions(Func<Expression, Expression> transformation)
        {
        }
    }

    internal class IgnorePropertyExpressionNode : ResultOperatorExpressionNodeBase
    {
        public static readonly IReadOnlyCollection<MethodInfo> SupportedMethods = new[]
        {
            IQueryableExtensions._ignorePropertyMethodInfo
        };

        private readonly LambdaExpression _propertyPathLambda;

        public IgnorePropertyExpressionNode(MethodCallExpressionParseInfo parseInfo, LambdaExpression propertyPathLambda)
          : base(parseInfo, null, null) => _propertyPathLambda = propertyPathLambda;

        protected override ResultOperatorBase CreateResultOperator(ClauseGenerationContext clauseGenerationContext)
          => new IgnorePropertyResultOperator()
          {
              PropertyPathLambda = _propertyPathLambda,
          };

        public override Expression Resolve(
              ParameterExpression inputParameter,
              Expression expressionToBeResolved,
              ClauseGenerationContext clauseGenerationContext)
                => Source.Resolve(inputParameter, expressionToBeResolved, clauseGenerationContext);
    }

    internal class CustomMethodInfoBasedNodeTypeRegistryFactory : DefaultMethodInfoBasedNodeTypeRegistryFactory
    {
        public override INodeTypeProvider Create()
        {
            RegisterMethods(IgnorePropertyExpressionNode.SupportedMethods, typeof(IgnorePropertyExpressionNode));
            return base.Create();
        }
    }

    public class PropertyIgnorableSelectExpression : SelectExpression
    {
        public List<PropertyInfo> IgnoredProperties { get; } = new List<PropertyInfo>();

        public PropertyIgnorableSelectExpression(
          SelectExpressionDependencies dependencies,
          RelationalQueryCompilationContext queryCompilationContext) : base(dependencies, queryCompilationContext) => SetCustomSelectExpressionProperties(queryCompilationContext);

        public PropertyIgnorableSelectExpression(
          SelectExpressionDependencies dependencies,
          RelationalQueryCompilationContext queryCompilationContext,
          string alias) : base(dependencies, queryCompilationContext, alias) => SetCustomSelectExpressionProperties(queryCompilationContext);

        private void SetCustomSelectExpressionProperties(RelationalQueryCompilationContext queryCompilationContext)
        {
            var lastTrackingModifier
                = queryCompilationContext.QueryAnnotations
                    .OfType<TrackingResultOperator>()
                    .LastOrDefault();

            if (lastTrackingModifier?.IsTracking == false)
            {
                foreach (var ignorePropertyResultOperator in queryCompilationContext.QueryAnnotations.OfType<IgnorePropertyResultOperator>())
                {
                    if (ignorePropertyResultOperator.PropertyPathLambda.Body is MemberExpression memberExpression)
                    {
                        IgnoredProperties.Add((PropertyInfo) memberExpression.Member);
                    }
                }
            }
        }
    }

    internal class PropertyIgnorableSelectExpressionFactory : SelectExpressionFactory
    {
        public PropertyIgnorableSelectExpressionFactory(SelectExpressionDependencies dependencies)
          : base(dependencies)
        {
        }

        public override SelectExpression Create(RelationalQueryCompilationContext queryCompilationContext)
          => new PropertyIgnorableSelectExpression(Dependencies, queryCompilationContext);

        public override SelectExpression Create(RelationalQueryCompilationContext queryCompilationContext, string alias)
          => new PropertyIgnorableSelectExpression(Dependencies, queryCompilationContext, alias);
    }

    internal class CustomSqlServerQuerySqlGeneratorFactory : QuerySqlGeneratorFactoryBase
    {
        public CustomSqlServerQuerySqlGeneratorFactory(
          QuerySqlGeneratorDependencies dependencies,
          ISqlServerOptions sqlServerOptions) : base(dependencies) { }

        public override IQuerySqlGenerator CreateDefault(SelectExpression selectExpression)
          => new CustomQuerySqlGenerator(
            Dependencies,
            selectExpression);
    }

    public class CustomQuerySqlGenerator : DefaultQuerySqlGenerator
    {
        public CustomQuerySqlGenerator(QuerySqlGeneratorDependencies dependencies, SelectExpression selectExpression) : base(dependencies, selectExpression)
        {
        }

        protected override void GenerateList<t>(
             IReadOnlyList<t> items,
             Action<t> generationAction,
             Action<IRelationalCommandBuilder> joinAction = null)
        {
            if (typeof(t) == typeof(Expression) && SelectExpression is PropertyIgnorableSelectExpression expression && expression.IgnoredProperties.Any())
            {
                GenerateListExpression(items, generationAction, joinAction, expression);
                return;
            }

            base.GenerateList(items, generationAction, joinAction);
        }

        protected void GenerateListExpression<t>(
            IReadOnlyList<t> items,
            Action<t> generationAction,
            Action<IRelationalCommandBuilder> joinAction,
            PropertyIgnorableSelectExpression selectExpression)
        {
            NotNull(items, nameof(items));
            NotNull(generationAction, nameof(generationAction));

            joinAction = joinAction ?? (isb => isb.Append(", "));

            for (var i = 0; i < items.Count; i++)
            {
                if (i > 0)
                {
                    joinAction(Sql);
                }

                var item = items[i];
                if (item is ColumnExpression column && column.Property?.PropertyInfo != null
                                && selectExpression.IgnoredProperties.Any(ip =>
                                       ip.PropertyType == column.Property.PropertyInfo.PropertyType
                                    && ip.DeclaringType == column.Property.PropertyInfo.DeclaringType
                                    && ip.Name == column.Property.PropertyInfo.Name)
                                    )
                {
                    string defaultValue;
                    if (column.Property.IsNullable)
                    {
                        defaultValue = "null";
                    }
                    else
                    {
                        //TODO: add more types here

                        if (column.Property.PropertyInfo.PropertyType == typeof(string))
                        {
                            defaultValue = "''";
                        }
                        else
                        {
                            throw new NotSupportedException($"Ignoring {column.Property.PropertyInfo.PropertyType} not supported by {nameof(IQueryableExtensions.IgnoreProperty)}");
                        }
                    }

                    base.Sql.Append($"{defaultValue} AS [{column.Name}]");
                    continue;
                }

                generationAction(item);
            }
        }

        public static T NotNull<T>(T value, string parameterName)
        {
#pragma warning disable IDE0041 // Use 'is null' check
            if (ReferenceEquals(value, null))
#pragma warning restore IDE0041 // Use 'is null' check
            {
                throw new ArgumentNullException(parameterName);
            }

            return value;
        }
    }

it requires this in DbContext:

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder
              .ReplaceService<INodeTypeProviderFactory, CustomMethodInfoBasedNodeTypeRegistryFactory>()
              .ReplaceService<ISelectExpressionFactory, PropertyIgnorableSelectExpressionFactory>()
              .ReplaceService<IQuerySqlGeneratorFactory, CustomSqlServerQuerySqlGeneratorFactory>();

            base.OnConfiguring(optionsBuilder);
        }

[QuanTran91]

Are there any approaches that are less complicated than @hmdhasani ?
I need to ignore some audit properties (createdDate, userId,...) when querying from db (reduce extra step to encrypt/decrypt) but I also want to add or update them.

[AndriySvyryd]

See https://github.com/dotnet/efcore/issues/21251 for ideas on how to combine with Include

[QuanTran91]

Thanks @AndriySvyryd but it would be better if it has Exclude :)