Elasticsearch-net: Bug: JsonSerializationException: Self referencing loop detected with type 'Nest.Field'. Path '_source.includes'.

[JulianRooze]

Calling SearchAsync in a concurrent manner while specifying a source filter on the search descriptor causes a JSON serialization exception on the Nest.Field type. This only happens when running concurrent requests, and only when doing >= 8 concurrent tasks in my .NET Core project, or when doing >= 2 concurrent tasks in my LINQPad reproduction script (so .NET Framework). It also only happens when using a source filter.

The message:

JsonSerializationException: Self referencing loop detected with type 'Nest.Field'. Path '_source.includes'.

However, sometimes the exception type is UnexpectedElasticsearchClientException instead of JsonSerializationException.

The stacktrace:

> at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.CheckForCircularReference(JsonWriter writer, Object value, JsonProperty property, JsonContract contract, JsonContainerContract containerContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)   at Nest.FieldsJsonConverter.WriteJson(JsonWriter writer, Object value, JsonSerializer serializer)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)   at Nest.UnionJsonConverter`2.<>c__DisplayClass0_0.b__1(TSecond second)   at Nest.UnionJsonConverter.WriteJson(JsonWriter writer, Object v, JsonSerializer serializer)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)   at Elastic.Internal.JsonNet.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)   at Nest.InternalSerializer.SerializeT

A complete LINQPad reproduction script:

async Task Main()
{
    var c = new ElasticClient();

    while (true)
    {
        var tasks = new List<Task>();

        for (var i = 0; i < 10; i++)
        {
            tasks.Add(Task.Run(async () =>
            {
                var response = await c.SearchAsync<MyObject>(s =>
                {
                    s
                    .Index("my_index")
                    .Source(src => src.Includes(f => f.Field("any_field")));

                    Console.WriteLine(Serialize(s, c));

                    return s;
                });
            }));
        }

        await Task.WhenAll(tasks);
    }
}

public class MyObject
{

}

public static string Serialize(ISearchRequest request, IElasticClient client)
{
    string serialized;
    using (var stream = new MemoryStream())
    using (var streamReader = new StreamReader(stream))
    {
        client.RequestResponseSerializer.Serialize(request, stream);
        stream.Position = 0;
        serialized = streamReader.ReadToEnd();
        return serialized;
    }
}

If this somehow doesn't fail, try increasing the concurrency from 10 to something higher. You only need NEST 6.5.1 and a working Elasticsearch on the default port with a my_index index.

This is using the NEST 6.5.1 NuGet package (also reproduces for 6.5.0).

This behavior is a real head scratcher to me. I'm correct when saying that ElasticClient is meant to be thread-safe, right? That's how we've always used it.

Possibly a regression of #706?

[JulianRooze]

And to clarify: the serialization of the request like in the above example is not necessary for reproduction. If I remove the Console.WriteLine and Serialize method, it still reproduces.

So the minimal code to reproduce it is this:

async Task Main()
{
    var c = new ElasticClient();

    while (true)
    {
        var tasks = new List<Task>();

        for (var i = 0; i < 32; i++)
        {
            tasks.Add(Task.Run(async () =>
            {
                await c.SearchAsync<string>(s => s.Index("my_index")
                    .Source(src => src.Includes(f => f.Field("any_field"))));
            }));
        }

        await Task.WhenAll(tasks);
    }
}

[russcam]

This looks suspiciously like https://github.com/elastic/elasticsearch-net/blob/c6652ac3886a664970b815425d4ca0015a900391/src/Nest/CommonAbstractions/Infer/Fields/FieldsJsonConverter.cs#L21-L24

which is triggered from Fields, when Fields contains only a single instance of Field. That the ReferenceLoopHandling is set on serializer (which I suspect is shared), does not look correct.

Thanks for reporting @JulianRooze, and for the repro 👍 Will need to investigate further

[JulianRooze]

@russcam Ah that makes sense, I hadn't considered checking if it still reproduces when specifying more than one field, but I just tried it and it does not reproduce, confirming your finding.

So I guess the workaround until a fix is published is to always request two fields.

Thanks for looking into it!

[russcam]

I've opened #3618 to address.

We're looking to put out 6.6.0 very shortly, which will contain this fix

[russcam]

Closing this as it's now in NEST 6.6.0

[himadrinath]

@russcam still having this problem in NEST 6.8.0
Elastic.Internal.JsonNet.JsonSerializationException at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.CheckForCircularReference

[russcam]

@himadrinath Please could you

1. Open a new issue, and link to this one with

   Relates: #3617
   

   at the top of the description

2. Provide a small but succinct example the reproduces what you're seeing, in the new issue.

Thanks!