Elasticsearch-net: Elasticsearch.Net.UnexpectedElasticsearchClientException: Exception of type 'System.OutOfMemoryException' was thrown.

[AviramYo33]

NEST/Elasticsearch.Net version: 6.1

Elasticsearch version: 6.1

Steps to reproduce:

I can't reproduce it manually!

From time to time we get this exception:

Elasticsearch.Net.UnexpectedElasticsearchClientException: Exception of type 'System.OutOfMemoryException' was thrown.
---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Nest.Json.Linq.JContainer.ReadContentFrom(JsonReader r, JsonLoadSettings settings)
at Nest.Json.Linq.JContainer.ReadTokenFrom(JsonReader reader, JsonLoadSettings options)
at Nest.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings)
at Nest.DictionaryResponseJsonConverterHelpers.ReadServerErrorFirst(JsonReader reader, Error& error, Nullable1& statusCode) at Nest.ResolvableDictionaryResponseJsonConverter3.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
at Nest.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
at Nest.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Nest.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Nest.Json.JsonSerializer.DeserializeT
at Nest.InternalSerializer.d__201.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Elasticsearch.Net.ResponseBuilder.<SetBodyAsync>d__51.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Elasticsearch.Net.ResponseBuilder.d__21.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Elasticsearch.Net.HttpConnection.<RequestAsync>d__141.MoveNext() in C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Connection\HttpConnection.cs:line 259
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Elasticsearch.Net.RequestPipeline.d__741.MoveNext() in C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:line 478 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Elasticsearch.Net.Transport1.d__151.MoveNext() in C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 120 --- End of inner exception stack trace --- at Elasticsearch.Net.Transport1.d__151.MoveNext() in C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 138 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Nest.ElasticClient.<Nest-IHighLevelToLowLevelDispatcher-DispatchAsync>d__2734.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Nest.IndicesPointingToAliasExtensions.d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Rm.Backend.Rpm.JobPerformanceRepository.ElasticHelper.d__7.MoveNext() in C:\Development
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

[russcam]

First thing I suggest is updating NEST and Elasticsearch.Net 6.1 to the latest 6.x package. Currently, this is 6.8.6. Please see if you continue to get OutOfMemoryException.

If you continue to see the exception, please can you provide code for how the client is configured and typical usage pattern of the client. If you are also able to capture the size of the response, that would also help.

[russcam]

@AviramYo33 can you provide additional information about how the client is configured and the request/response made when an OutOfMemoryException is seen?

[russcam]

We’re closing this issue due to inactivity; if you’re still in need of resolution, feel free to re-open with the additional details and we'd be glad to help you out 👍

[vonbv]

@russcam this happens to me with ElasticSearch.Net 7.6.1 and NEST 7.6.1. I've checked the Release history over at Github and did not find any fix or enhancement that could help with our issue.

Our client is configured in a class that is singleton. This is how we create our client:

       if (_client == null)
            {
                IEnumerable<Uri> uris = _dbSettings.ElasticSearch.Urls.Select(x => new Uri(x));

                //----------------------------
                //  Mapping for 7.x
                //----------------------------
                var settings = new ConnectionSettings(uris.First())                  
                  .DefaultMappingFor<TheDoc>(m => m.IndexName("theIndexName").RelationName("parent"))                  
              ;

                if (!_dbSettings.ElasticSearch.Urls.Contains("localhost"))
                {
                    settings.BasicAuthentication(_dbSettings.ElasticSearch.Username, _dbSettings.ElasticSearch.Password);
                }

                _client = new ElasticClient(settings);


                var indexResponse = _client.Indices.Create("theIndexName", c => c
                    .Map<TheDoc>(m => m.AutoMap())
                    .Settings(s => s
                        // ToDo: settings for dev only.
                        .NumberOfReplicas(_dbSettings.ElasticSearch.DoNotCreateReplicas ? 0 : 1)
                        )
                    );


            }

            return _client;

This is how we build our query, it might help you troubleshoot and see where the issue is. Perhaps we're doing something wrong in the query?

            if (pageSize > 1000) pageSize = 1000;

            var response = await _elasticManager.GetClient().SearchAsync<TheDoc>(s =>
            {
                var sd = s.From(pageNumber)
                .Size(pageSize)
                .Query(q =>
                {
                    QueryContainer qc = null;

                    if (ids != null && ids.Any())
                    {
                        // Serch by ids
                        qc = q.Terms(t => t.Field(f => f.Id).Terms(ids));
                    }                  
                    else if (string.IsNullOrWhiteSpace(keyword))
                    {
                        // Search by the (default) keyword.
                        qc = q.MatchPhrasePrefix(m => m.Field(f => f.Name).Query(keyword).Boost(1.5));
                    }                    
                    else {
                        // Search by the keyword.
                        qc = q.QueryString(qs => qs.Name("TheDoc_query")
                            .Boost(1.1)
                            .Fields(fs => fs.Field(f => f.fldX).Field(f => f.fldY).Field(f => f.fldZ).Field(f => f.fldA))
                            .Query($"{keyword} {keyword.Replace(",", " ")}")
                            .DefaultOperator(Operator.Or)
                            .MaximumDeterminizedStates(3)
                            .Escape()
                            .Fuzziness(Fuzziness.Auto)
                            .FuzzyPrefixLength(2)
                            .FuzzyMaxExpansions(3)
                            .MinimumShouldMatch(2)
                            .Lenient()
                        //.AutoGenerateSynonymsPhraseQuery(false)
                        );
                    }

                    qc = qc && q.Match(m => m.Field(f => f.fld_1).Query(filterA);


                    if (!string.IsNullOrWhiteSpace(filter_1))
                    {
                        qc = qc && q.MatchPhrasePrefix(m => m.Field(f => f.fldX).Query(filter_1).Boost(0.9));
                    }
                    if (!string.IsNullOrWhiteSpace(filter_2))
                    {
                        qc = qc && q.MatchPhrasePrefix(m => m.Field(f => f.fldY).Query(filter_2).Boost(0.9));
                    }

                    return qc;
                }); // end Query


                return sd;
            });

[stevejgordon]

@vonbv We identified an issue with EventCounters which in certain circumstances seemed to be causing an OOM exception. This was addressed in #5088 and released in 7.10.0 of NEST, As a starting point, could you try your code using the latest release of the client and see if that helps?

[vonbv]

@stevejgordon thanks for the quick response! Good job on finding the issue and fixing it. And thank you for pointing out which issue was that.

[vonbv]

It seems to work. I stress-tested our searches and the server has not thrown an OOM exception yet. Thank you Elasticsearch.Net team!

[stevejgordon]

Glad to hear that, thanks for confirming @vonbv

[vonbv]

Hello ElasticSearch.Net Team, we are experiencing the OOM issue again. We are still on 7.10.1 and we have the same configuration and implementation as I shared earlier in this thread. Below is the stack trace we got.

An error has occured: Elasticsearch.Net.UnexpectedElasticsearchClientException: Exception of type 'System.OutOfMemoryException' was thrown. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1.Rent(Int32 minimumLength)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.DeserializeAsync[T](Stream stream, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.ResponseBuilder.SetBodyAsync[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType, CancellationToken cancellationToken)
   at Elasticsearch.Net.ResponseBuilder.ToResponseAsync[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType, CancellationToken cancellationToken)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   at Nest.ElasticClient.SearchAsync[TDocument](ISearchRequest request, CancellationToken ct)
   at Nest.ElasticClient.SearchAsync[TDocument](Func`2 selector, CancellationToken ct)
   at OUR_SEARCH_METHOD

[vonbv]

Hi @stevejgordon just wondering if you guys are able to replicate my issue.

[stevejgordon]

@vonbv Apologies but the notification got lost as this issue is marked closed. I'll reopen so we can investigate this with you. Is the OOM exception happening often and repeatable in a small repro? Are you running on a memory constrained environment / container perhaps? Are you able to grab a profile or any memory dumps we could potentially investigate?

[vonbv]

repeatable in a small repro

You mean if I can take out the part of our code that throws the exception and put it in small dedicated project?

Are you running on a memory constrained environment / container perhaps?

1 GB memory and 30 GB storage

Are you able to grab a profile or any memory dumps we could potentially investigate?
I'll collect the information needed.

[stevejgordon]

You mean if I can take out the part of our code that throws the exception and put it in small dedicated project?

I'm interested to know if that query specifically seems to trigger the OOM. If so, does it occur if the same query is used in a simpler app? Is the exception always thrown in the same place? How long is the app typically running before you experience the exception?

1 GB memory and 30 GB storage

How intensive is the application? Currently it sounds like there's a memory leak somewhere in the app. We'd need to narrow down if it's specifically related to NEST.

I'll collect the information needed.

Thanks! I think this will give us a starting point. Also, what environment is this running on specifically?

[Mpdreamz]

@vonbv

In addition to @stevejgordon's query the following seems to be worth a try.

By default we use a recyclable memory stream implementation that tries to reuse buffers reasonably aggressively.

Since you are on a memory constraint environment perhaps the following works better more predictable by configuring the client with the following.

settings
    .MemoryStreamFactory(Elasticsearch.Net.MemoryStreamFactory.Default)

This implementation of IMemoryStreamFactory creates new MemoryStream() all the time. The overhead in terms of performance should be limited.

[vonbv]

@stevejgordon

does it occur if the same query is used in a simpler app?

The query is in a service used by our API. It will take time to reference that service in a new simple API and deploy it and generate the data. I will do this over the weekend though as it'll be interesting to know how it performs in that environment.

Is the exception always thrown in the same place?

Yes. It's always in the same API that implements the service that runs the specific query, which I shared in this thread. No other API runs the query. However, our other queries are not used as much as the one with issue.

How long is the app typically running before you experience the exception?

I will have to monitor this but based on previous instances, after a few hours of restarting the server, the OOM happens.

How intensive is the application? Currently it sounds like there's a memory leak somewhere in the app. We'd need to narrow down if it's specifically related to NEST

We are still in the development phase so it's not intensive. If there's a memory leak somewhere then I'm thinking why are the other searches works. So what I do is when this OOM occurs on the specified query, I call our other queries and those work just fine. The other queries that are running fine are using other type of documents.

Also, what environment is this running on specifically?

Running on Azure S2

• 200 total ACU
• 3.5 GB memory
  Running on Elastic.co
• 1 GB memory
• 30 GB storage
• version 7.6.2
  Built with:
• .Net Core 2.1
• ElasticSearch.Net 7.10.1
• NEST 7.10.1

@Mpdreamz I will try your suggestion

[stevejgordon]

Thanks @vonbv. It may also relate to this issue in .NET runtime. I know Microsoft did a lot of work on the GC to optimise it for container and constrained memory environments in 3.0 and 3.1. If possible, it may be worth migrating this to the 3.1 runtime to see if that helps?

Do you have App Insights enabled to get a view on the memory usage over time?

[vonbv]

We're at the point that upgrading will take a considerable amount of time especially that several components will be possibly hit. But with that issue you linked to, it seems that's our best option to fix our issue.

Here's a screenshot of the memory usage covering the times when the OOM exception from ES.NET occurs. By digging through all the OOM exceptions, it shows that it resolves after sometime and then come back again.

[stevejgordon]

Hi @vonbv. I wanted to check in and see if you managed to try upgrading and whether that helped?