Elasticsearch-net: IndexMany method does not take under consideration the ignored properties

[KrisPetkov]

NEST/Elasticsearch.Net 7.6.1:

Elasticsearch 7.6.2:

When I use the .IndexMany() method all the .Ignore() properties from the DefaultMappingFor are not applied. But If I use the .IndexDocument() method my .Ignore() properties are applied.

Steps to reproduce:

1. Ignore few properties from your class in the DefaultMappingFor
2. Use the .IndexMany() method to insert a list of objects
3. Execute the following command and see that the ignored properties were inserted although they were ignored:

Command:
GET //_search
{
"query": {
"match_all": {}
}
}

Expected behavior
When I use the .IndexMany() method the predefined ignored properties in the DefaultMappingFor should not appear/insert in the Elasticsearch

[russcam]

Hi @KrisPetkov, I can't replicate this. Here is the test that I'm running https://github.com/elastic/elasticsearch-net/commit/193e1c678e4d12b5098c7ae7b721b85df0151e64

```c#
public void Main()
{
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(pool, new InMemoryConnection())
.DisableDirectStreaming()
.DefaultMappingFor(m => m
.Ignore(p => p.Baz)
.IndexName("foos")
);

var client = new ElasticClient(settings);
var docs = new[] { new Foo { Bar = "bar", Baz = "baz" }, };
var bulkResponse = client.IndexMany(docs);
var actual = Encoding.UTF8.GetString(bulkResponse.ApiCall.RequestBodyInBytes);
    var expected =  "{\"index\":{\"_id\":null,\"_index\":\"foos\"}}\n{\"bar\":\"bar\"}\n";

if (actual != expected)
    {
        throw new Exception($"Expected {expected}, Actual: {actual}")
    }

}

public class Foo
{
public string Bar { get; set; }
public string Baz { get; set; }
}
```

which passes (i.e. doesn't throw the exception).

Can you provide a small but complete example that demonstrates what you're seeing?

[KrisPetkov]

Hi @russcam

Thank you for your answer and the provided example.
I have tested it and it does work as it suppose to.
My problem still exists nevertheless.

It seems that the problem occurs when I am using objects that were acquired from the database by using EF Core. If I create a list of this objects manually in the code and pass the list to the .IndexMany method it works. The properties are ignored. But if I pass to the method a list which was formed from the EF Core the properties which should be ignored - are not.

Here's what I'm using:
Note: This is a simplified version of my project

Base Entity:

public class BaseEntity
{
        public int Id { get; set; }
}

Entity:

public class Item : BaseEntity
{
        public Item()
        {
            Guid = Guid.NewGuid();
        }

        public Guid Guid { get; private set; }

        public string Name { get; set; }

        public decimal Weight { get; set; }

        public decimal Length { get; set; }

        public decimal Width { get; set; }

        public decimal Height { get; set; }

        public byte Size { get; set; }

        // Navigation properties

        public virtual ICollection<Discount> AppliedDiscounts { get; set; }
}

Service Constructor:

public Service()
{
var pool = new SingleNodeConnectionPool(new Uri("my_elk_cloud_url"));
            var settings = new ConnectionSettings(pool)
                .BasicAuthentication("my_user", "my_password")
                    .DefaultMappingFor<Item>(m => m
                    .Ignore(p => p.Height)
                    .Ignore(p => p.Length)
                    .Ignore(p => p.Weight)
                    .Ignore(p => p.Width)
                    .IndexName("my_index_name")
                );
            //Private variable for the class
           client = new ElasticClient(settings);
}

My method (Does not ignore the properties):
Note: the list with objects comes from EF Core call to the DB:

public void ReIndex(IList<Item> newItemsToIndex)
{
          client.IndexMany(newItemsToIndex);
}

My method (Works and ignores the properties if I create the objects manually):

public void ReIndex(IList<Item> newItemsToIndex)
{
var listOfObjects = new List<Item>()
                {
                new Item()
                    {
                        Id = 1,
                        Name = "Test item 1",
                        Height = 150,
                        Length = 10,
                        Width = 70,
                        Weight = 5
                        AppliedDiscounts = new List<Discount>()
                        {
                             new Discount() {Id=1, Value=10}
                        }
                    },
                    new Item()
                    {
                        Id = 2,
                        Name = "Another test item",
                        Size = 10,
                        Weight = 0m,
                        Width = 0m,
                        Height = 33,
                        Length = 50
                    }
            };

          client.IndexMany(listOfObjects );
}

[KrisPetkov]

@russcam

I have just made another test and it seems that this problem occurs when I'm using the .UseLazyLoadingProxies(); option of the EF Core. Do you know something about this?

It is strange because properties like Height, Length, etc. are from the main entity and they are not lazy loaded but still the are not ignored.

[russcam]

I have just made another test and it seems that this problem occurs when I'm using the .UseLazyLoadingProxies(); option of the EF Core. Do you know something about this?

Typically, lazy loading with ORMs like EF and NHibernate are achieved by deriving a _dynamic proxy_ type from the entity type, and overriding virtual properties with a lazy loading implementation.

I think you're hitting the same issue as https://github.com/elastic/elasticsearch-net/issues/3929. What makes your scenario trickier however is that you don't know the type of the dynamic proxy that EF generates, therefore the current Ignore property implementation can't be worked around, at least with DefaultMappingFor.

For the time being, would using [Nest.Ignore] attribute on properties work?

[KrisPetkov]

@russcam Thank you for your answer!

I will try with the attribute in an hour or so and will let you know.
But my entities are in an anemic core (core is related to the architecture) project independent of all kind of external dependencies and this will introduce a dependency in the project. But for the sake of test I will try and see if it works.

As you said the ORM generates a dynamic proxy type for the virtual properties and overrides them but the problem is that also non virtual properties are not ignored. I mean that if it was not ignoring the AppliedDiscounts property from my example above it is understandable but it does not work for the Size property, for example, which is a simple byte variable.

And one last question. Are there any plans for a workaround or a solution (if there is a solution at all ) for such a scenario for the future releases? I'm still in development and it is not so urgent but if there are no such plans I can start thinking of some applicable solution for my project.

[KrisPetkov]

@russcam Thank you for your answer!

I will try with the attribute in an hour or so and will let you know.

So, I have made the test and if I use the [Nest.Igonre] attribute everything works (with the lazy loading proxies activated). So maybe the .Ignore() method is the problem and cannot cope with the proxies.
In regards to my previous comment. If there are no plans for fixing this problem in a foreseeable future I will have to make some entity mappings (like AutoMapper or something like this) and to dynamically map my entities to/from a wrapper entity.

[russcam]

So maybe the .Ignore() method is the problem

It's related to how members are reflected to build a type specific serializer, and how those members are compared to mappings on members of a base type. In the attribute case, it is retrieved with the inherit: true, so will pick it up from the base type:

https://github.com/elastic/elasticsearch-net/blob/b669afd8f7716bdd49711feb50af4262de9addb6/src/Nest/CommonAbstractions/SerializationBehavior/PropertyMapping.cs#L64

If there are no plans for fixing this problem in a foreseeable future

I think it would be good to address this behaviour in the near future.

[KrisPetkov]

Thank you! For now I will make a workaround for my project.
Should I leave the issue open for future references or to close it for now?

[russcam]

Should I leave the issue open for future references or to close it for now?

I think it might be good to close this one for now, as the issue is captured in #3929, which is still open. Closing.