Elasticsearch-net: Getting empty objects in json after serialzing DateHistogram Aggregation object from SearchTemplateAsync response

[anuragdewangan20]

NEST Version : 6.0.1
Elastic Version : 6.X

Description : I am gettting response from Elastic for my search template. However, after serialzing those response to json, I am getting empty objects for hits and aggregation (DateHistogram).

Steps to reproduce :

Getting response from elastic for below query but there are empty object in json after serialzing the response-

    {
      "size": 100,
      "query": {
        "constant_score": {
          "filter": {
            "bool": {
              "must": [
                {
                  "term": {
                    "id": "2"
                  }
                }
              ]
            }
          }
        }
      },
      "aggs": {
        "CountHistory": {
          "date_histogram": {
            "field": "abc",
            "interval": "60m",
            "min_doc_count": 0,
            "extended_bounds": {
              "min": "2018-02-03",
              "max": "2018-02-012"
            }
          }
        }
      }
    }

NOTE : Serialzation is working fine with NEST 5.X versions.

[russcam]

@anuragdewangan20 could you please show the debug information that includes the JSON response (set DisableDirectStreaming() on ConnectionSettings)

[anuragdewangan20]

DisableDirectStreaming() is already set. Please refer below details for debug information -

Valid NEST response built from a successful low level call on POST: /kvn-v4-customer-unknown-2018.02.12/_search/template?pretty=true&error_trace=true&typed_keys=true&ignore_unavailable=true&allow_no_indices=true

Audit trail of this API call:

• [1] PingSuccess: Node: http://localhost:9200/ Took: 00:00:00.1570475
• [2] HealthyResponse: Node: http://localhost:9200/ Took: 00:00:00.3547501
  

  Request:

{
"source": "{ \"size\": 0, \"query\": { \"constant_score\": { \"filter\": { \"bool\": { \"must\": [ { \"term\": { \"tenant_id\": \"2\" } }, { \"match\": { \"external_id\": \"{{coreinstance_id}}\" } }, { \"term\": { \"logsource.origin\": \"collector\" } }, { \"range\": { \"received_timestamp\": { \"gte\": \"{{startDate}}\", \"lte\": \"{{endDate}}\" } } } ] } } } }, \"aggs\": { \"CountHistory\": { \"date_histogram\": { \"field\": \"abc\", \"interval\": \"60m\", \"min_doc_count\": 0, \"extended_bounds\": { \"min\": \"{{startDate}}\", \"max\": \"{{endDate}}\" } } } } }",
"params": {
"startDate": "2018-02-12T00:01:00Z",
"endDate": "2018-02-12T02:01:00Z }
}

Response:

{
"took" : 7,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.0,
"hits" : [ ]
},
"aggregations" : {
"date_histogram#CountHistory" : {
"buckets" : [
{
"key_as_string" : "2018-02-12T00:00:00.000Z",
"key" : 1518393600000,
"doc_count" : 1
},
{
"key_as_string" : "2018-02-12T01:00:00.000Z",
"key" : 1518397200000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-12T02:00:00.000Z",
"key" : 1518400800000,
"doc_count" : 0
}
]
}
}
}

[jonyadamit]

Hey @anuragdewangan20 that response seems fine, what is exactly missing?

[jhellemann]

Hi,

NEST Version : 6.0.1
Elastic Version : 5.6.5

I'm also having this issue. From NEST client.SearchAsync method I get:

{
"aggs": { 
  "sterms#timeOfDay" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "1_MORNING",
          "doc_count" : 48
        },
        {
          "key" : "2_LATE_MORNING",
          "doc_count" : 134
        },
        {
          "key" : "3_AFTERNOON",
          "doc_count" : 7
        },
        {
          "key" : "4_LATE_AFTERNOON",
          "doc_count" : 52
        },
        {
          "key" : "5_EVENING",
          "doc_count" : 159
        }
      ]
    }
}

While serilizing back to JSON in my API, I get:

{
"aggs": {
  "timeOfDay": {
            "items": [
                {},
                {},
                {},
                {},
                {}
            ],
            "docCountErrorUpperBound": 0,
            "sumOtherDocCount": 0,
            "meta": null,
            "docCount": 0,
            "bgCount": 0
        }
  }
}

With NEST 5.X this looks like:

{
  "aggs": {
      "timeOfDay": {
            "items": [
                {
                    "key": "1_MORNING",
                    "keyAsString": null,
                    "docCount": 199,
                    "docCountErrorUpperBound": null,
                    "aggregations": null
                },
                {
                    "key": "2_LATE_MORNING",
                    "keyAsString": null,
                    "docCount": 273,
                    "docCountErrorUpperBound": null,
                    "aggregations": null
                },
                {
                    "key": "3_AFTERNOON",
                    "keyAsString": null,
                    "docCount": 338,
                    "docCountErrorUpperBound": null,
                    "aggregations": null
                },
                {
                    "key": "4_LATE_AFTERNOON",
                    "keyAsString": null,
                    "docCount": 164,
                    "docCountErrorUpperBound": null,
                    "aggregations": null
                },
                {
                    "key": "5_EVENING",
                    "keyAsString": null,
                    "docCount": 786,
                    "docCountErrorUpperBound": null,
                    "aggregations": null
                }
            ],
            "docCountErrorUpperBound": 0,
            "sumOtherDocCount": 0,
            "meta": null,
            "docCount": 0,
            "bgCount": null
        }
    }
}

[anuragdewangan20]

@jonyadamit, After serializing response from NEST, there are some empty objects as mentioned above by @jhellemann.

Thanks.

[russcam]

I'm taking a look at this

[russcam]

@jhellemann if you're using Elasticsearch 5.x, you should use NEST 5.x; Per the compatibility matrix, NEST 6.x is not supported or expected to be compatible with Elasticsearch 5.x

[russcam]

@anuragdewangan20 Just so I understand the issue correctly, could you confirm that this is what you're doing:

1. Making a request to Elasticsearch that includes a Date Histogram aggregation
2. Receiving the response from Elasticsearch, with JSON being correctly deserialized into the MultiBucketAggregate<DateHistogramBucket> for the Date Histogram aggregation
3. Reserializing the ISearchResponse<T> back into JSON within the application for another purpose.

Is that correct?

[anuragdewangan20]

@russcam That's Correct.

[russcam]

The change has come about in 6.x because the aggregate types implement IReadOnlyDictionary<string, IAggregate> as opposed to exposing IReadOnlyDictionary<string, IAggregate> as a property on the type. Because the type now implements IEnumerable<T> through IReadOnlyDictionary<string, IAggregate>, the internal Json.NET serialization treats the type as a collection, ignoring other properties that it may have. This is why items array comes through as a collection of {}.

Now, an issue here is that the aggregate types only have read (deserialization) implementations, meaning they have not purposely supported writing (serialization) again to JSON.

If needing to reserialize to a JSON string, it may be better to do one of these two things:

1. make the request with the low level client, using the high level search request type, and return a string response. Then, if needing a strongly typed response from the JSON string, perform the deserialization to SearchResponse<T> in the application

   or

2. set .DisableDirectStreaming() in the request configuration for this request only, so that the response bytes are buffered on the ISearchResponse<T>. Then get the JSON string response from these bytes with Encoding.UTF8.GetString(response.ApiCall.ResponseBodyInBytes);

Taking one of these approaches also means that the JSON string represents exactly what has come back from Elasticsearch.

Would either approach fit your needs?

[jhellemann]

@russcam Yes, you are absolutely correct and I'm aware of this. I upgraded to NEST 6.x in my solution and fixed all breaking changes and going to upgrade to ES 6.x next.

One of the first differences I saw in response was this with empty objects in items so I check issues and found @anuragdewangan20 issue that seems to be the same as mine.

I guess the issue with NEST 6.x with both ES 5.x and 6.x is this: _Now, an issue here is that the aggregate types only have read (deserialization) implementations, meaning they have not purposely supported writing (serialization) again to JSON._

Thank you for a good answer!

[russcam]

@jhellemann no worries! Does 1. Or 2. work for you?

Would need to investigate how feasible implementing write methods for aggregates would be. I think we could quite easily support better serialization just by attributing AggregateDictionary with JsonObjectAttribute and setting Write to false on the custom JsonConverter; it wouldn't be quite the same output as 5.x though.

[jhellemann]

@russcam I need to investigate a bit more, but I'm thinking of making my own simpler Dictionary JSON format for aggregations as the clients in my API really don't need (nor understands :) ) the complex ES aggregations format.

[anuragdewangan20]

@russcam Both solution will not work for us because,
1st solution requires lot of rework and 2nd solution doesn't give exact json what we see in Sense response.

[russcam]

@anuragdewangan20

1st solution requires lot of rework and 2nd solution doesn't give exact json what we see in Sense response.

Both solutions will provide you exactly what is returned from Elasticsearch, and what you see in Sense / Kibana Console.

The first approach requires a small change to use the low level client; You can still use a SearchRequest<T> or SearchDescriptor<T> with the low level client

```c#
// build your search request using NEST's request types
var descriptor = new SearchDescriptor()
.Query(q => q
.Match(m => m
.Field(f => f.Title)
.Query("Elasticsearch Kibana")
) && +q
.Term("type", "question")
);

var response = client.LowLevel.Search("posts", PostData.Serializable(descriptor));

// the JSON string response from Elasticsearch
var jsonString = response.Body;

ISearchResponse searchResponse = null;
using (var stream = new MemoryStream(Encoding.UTF8.GetBytes(jsonString)))
searchResponse = client.RequestResponseSerializer.Deserialize>(stream);

// do something with the documents
var documents = searchResponse.Documents;

The second approach will provide the JSON returned from Elasticsearch on the response

```c#
var searchResponse = client.Search<Question>(s => s
    .Query(q => q
        .Match(m => m
            .Field(f => f.Title)
            .Query("Elasticsearch Kibana")
        ) && +q
        .Term("type", "question")
    )
    .RequestConfiguration(r => r
        // capture the request and response bytes for this call only
        .DisableDirectStreaming()
    )
);

// do something with the documents
var documents = searchResponse.Documents;

// the JSON string response from Elasticsearch
var jsonString = Encoding.UTF8.GetString(searchResponse.ApiCall.ResponseBodyInBytes);

[anuragdewangan20]

@russcam I am using SearchTemplateAsync method. I can see different json in Encoding.UTF8.GetString(searchResponse.ApiCall.ResponseBodyInBytes) as compare to sense response.

[russcam]

@anuragdewangan20 please provide a complete but succinct example that demonstrates the difference you see

[anuragdewangan20]

Response in Sense:
{
"took": 14,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0,
"hits": []
},
"aggregations": {
"LogCollectorCount": {
"buckets": [
{
"key_as_string": "2018-02-11T00:00:00.000Z",
"key": 1518307200000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T01:00:00.000Z",
"key": 1518310800000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T02:00:00.000Z",
"key": 1518314400000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T03:00:00.000Z",
"key": 1518318000000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T04:00:00.000Z",
"key": 1518321600000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T05:00:00.000Z",
"key": 1518325200000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T06:00:00.000Z",
"key": 1518328800000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T07:00:00.000Z",
"key": 1518332400000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T08:00:00.000Z",
"key": 1518336000000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T09:00:00.000Z",
"key": 1518339600000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T10:00:00.000Z",
"key": 1518343200000,
"doc_count": 1
},
{
"key_as_string": "2018-02-11T11:00:00.000Z",
"key": 1518346800000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T12:00:00.000Z",
"key": 1518350400000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T13:00:00.000Z",
"key": 1518354000000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T14:00:00.000Z",
"key": 1518357600000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T15:00:00.000Z",
"key": 1518361200000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T16:00:00.000Z",
"key": 1518364800000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T17:00:00.000Z",
"key": 1518368400000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T18:00:00.000Z",
"key": 1518372000000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T19:00:00.000Z",
"key": 1518375600000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T20:00:00.000Z",
"key": 1518379200000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T21:00:00.000Z",
"key": 1518382800000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T22:00:00.000Z",
"key": 1518386400000,
"doc_count": 0
},
{
"key_as_string": "2018-02-11T23:00:00.000Z",
"key": 1518390000000,
"doc_count": 0
},
{
"key_as_string": "2018-02-12T00:00:00.000Z",
"key": 1518393600000,
"doc_count": 0
}
]
}
}
}

Response in searchResponse.ApiCall.ResponseBodyInBytes

{
"took" : 127,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 4,
"max_score" : 0.0,
"hits" : [ ]
},
"aggregations" : {
"date_histogram#LogCollectorCount" : {
"buckets" : [
{
"key_as_string" : "2018-02-11T00:00:00.000Z",
"key" : 1518307200000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T01:00:00.000Z",
"key" : 1518310800000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T02:00:00.000Z",
"key" : 1518314400000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T03:00:00.000Z",
"key" : 1518318000000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T04:00:00.000Z",
"key" : 1518321600000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T05:00:00.000Z",
"key" : 1518325200000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T06:00:00.000Z",
"key" : 1518328800000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T07:00:00.000Z",
"key" : 1518332400000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T08:00:00.000Z",
"key" : 1518336000000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T09:00:00.000Z",
"key" : 1518339600000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T10:00:00.000Z",
"key" : 1518343200000,
"doc_count" : 4
},
{
"key_as_string" : "2018-02-11T11:00:00.000Z",
"key" : 1518346800000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T12:00:00.000Z",
"key" : 1518350400000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T13:00:00.000Z",
"key" : 1518354000000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T14:00:00.000Z",
"key" : 1518357600000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T15:00:00.000Z",
"key" : 1518361200000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T16:00:00.000Z",
"key" : 1518364800000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T17:00:00.000Z",
"key" : 1518368400000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T18:00:00.000Z",
"key" : 1518372000000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T19:00:00.000Z",
"key" : 1518375600000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T20:00:00.000Z",
"key" : 1518379200000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T21:00:00.000Z",
"key" : 1518382800000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T22:00:00.000Z",
"key" : 1518386400000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-11T23:00:00.000Z",
"key" : 1518390000000,
"doc_count" : 0
},
{
"key_as_string" : "2018-02-12T00:00:00.000Z",
"key" : 1518393600000,
"doc_count" : 0
}
]
}
}
}

[russcam]

@anuragdewangan20 picking this back up.

The difference you see is related to the client sending typed_keys by default in 6.x; you can disable this with

c# client.Search<MyDocument>(s => s .TypedKeys(false) );

and then the response will be identicial to Kibana Console / Sense.

[russcam]

@jyothi530 Will either approach in https://github.com/elastic/elasticsearch-net/issues/3093#issuecomment-366158401 work in your case?

[rdcm]

@russcam
I am also getting something similar.
request:

aggs: {    
   group_by_name: {      
      terms: {        
         field: grade.name      
      }    
   }  
}

aggregation section in response(no "buckets", instead array of empty objects named "items"):

{
  "aggregations":{
    "group_by_name":{
      "items":[
        {  },
        {  },
        {  },
        {  },
        {  },
        {  },
        {  }
      ],
      "docCountErrorUpperBound":0,
      "sumOtherDocCount":0,
      "meta":null,
      "docCount":0,
      "bgCount":0
    }
  }
}

C# code:

.Aggregations(aggs => aggs
   .Terms(
      "group_by_name",
      t => t.Field(x => x.Grade.Name)
    )
)

Is the same issue?

[russcam]

@rdcm Please take a look at one of the approaches in https://github.com/elastic/elasticsearch-net/issues/3093#issuecomment-366158401

[rdcm]

@russcam
With first approach in your comment, ISearchResponse<T> searchResponse has invalid state after deserialization from string,
"debugInformation": "Invalid NEST response built from a null ApiCall which is highly exceptional, please open a bug if you see this\n"
second solution has same error after deserialization to SearchResponse<T>

Do you have plan for more usefull solutions?

[mmichaels01]

Having the same issue @rdcm

[vasan123]

I am also having the same issue, like when I query data using json in console or browser I get the proper response, but using elastic search api in python I got empty objects. Any idea if this has fixed? also any alternate solution?

[russcam]

@rdcm

With first approach in your comment, ISearchResponse searchResponse has invalid state after deserialization from string,
"debugInformation": "Invalid NEST response built from a null ApiCall which is highly exceptional, please open a bug if you see this\n"

DebugInformation, Audit trail, etc. are built from a lot of information that is not in the _response_ body e.g. HTTP response status code. In the case of option 1, You can check the response from the low level client.

second solution has same error after deserialization to SearchResponse

Can you demonstrate with an example what you mean?

[Mpdreamz]

Closing this due to inactivity and the suggested solutions are good workarounds in my opinion.