Elasticsearch: Including inner hits drastically slows down query results

[clexmond]

Running the following query takes > 30 sec with ~13 million property records:

{
  "query": {
    "filtered": {
      "query": {
        "has_child": {
          "type": "transaction",
          "score_mode": "max",
          "inner_hits": {},
          "query": {
            "filtered": {
              "query": {
                "function_score": {
                  "functions": [
                    {
                      "gauss": {
                        "dateSold": {
                          "scale": "12w"
                        }
                      }
                    }
                  ]
                }
              },
              "filter": {
                "bool": {
                  "must": [
                    {
                      "has_parent": {
                        "type": "property",
                        "filter": {
                          "geo_bounding_box": {
                            "type": "indexed",
                            "location": {
                              "bottom_left": {
                                "lat": 33.676925635929535,
                                "lon": -84.57515716552734
                              },
                              "top_right": {
                                "lat": 33.86271855861567,
                                "lon": -84.15252685546875
                              }
                            }
                          }
                        }
                      }
                    },
                    {
                      "terms": {
                        "clientId": [
                          0,
                          1
                        ]
                      }
                    }
                  ]
                }
              }
            }
          }
        }
      }
    }
  },
  "size": 500
}

Removing the "inner_hits": {} drops the execution time to < 1 sec. Is this expected just by including inner hits?

[clexmond]

Updating "size": 1 with inner_hits on takes roughly the same amount of time as without inner_hits altogether. Increasing the size of the return set increases the query time linearly by roughly 200ms per inner hit.

[clintongormley]

It sounds like this query is being rerun for each hit. I wonder if this could be improved?

@martijnvg what do you think?

[martijnvg]

@clintongormley Yes, we run a small search for inner hits (local on the shard during the fetch phase) for each hit we return to include the inner hits per hit. This is okay if only 10 hits (default) are returned, but if the size gets increased more time needs to be spent on executing this mini searches to gather the top inner hits per hit.

This can be improved. In theory we can execute one search in the fetch phase that collects all inner hits for all hits being returned. This reduces the overhead of all these small searches. The fetch phase does have infrastructure to achieve this, so we should look into this.

[clexmond]

@martijnvg It sounds like moving in that direction might also allow us to limit the total number of inner hits returned rather than just on a per hit basis. Is that correct?

[martijnvg]

@clexmond I think that would require an additional setting too in the query dsl? The number of inner hits being returned is based on: size * number_of_inner_hits_definition * size_in_inner_hits. A global limit can be added and would just stop adding inner hits to search hits in the response if more than the specified time is time or more then the specified inner hits have been added.

But the idea behind this enhancement is to just remove the overhead that comes with the many small searches (which is noticeable when size is set to a higher value) by just executing on search for the all inner hits per shard during the fetch phase. This will should speed things up and hopefully listing to the total number of inner hits isn't needed then.

[SanZhiYuan]

is this fixed with the latest release? It takes us 10 minutes when querying 50,000+ results -.-

[martijnvg]

@SanZhiYuan No, this hasn't been fixed. So you set size to 50000 in the search api?

[SanZhiYuan]

@martijnvg yep, I was wondering this may work but not working well when querying with size = 50000, which actually too bad, -.-. I have switched to query with 500 results per request, within 10 threads at the same time, and now it is better. It doesn't take much more time than a single request with size = 500, so I guess it is OK for ES to parallel when running the "small search", and in my case, our qps is pretty low, but with more than 100 million docs. ^.^

[ljcollins25]

+1.

[rpedela]

+1

I am having similar performance issues with inner_hits with a large-ish index and grandchildren. I have a query that takes about 750 ms without inner hits but takes 7-8 seconds with them using a size of 10.

[ljcollins25]

What I actually want is the nested objects grouped by their parent document. In theory, this shouldn't even require a separate query during the fetch phase because the original query contains the results that I want. It just needs to find the parent document for each nested document (which is already happening during the fetch phase) and group that into the inner hits.

Should I consider a different approach than inner hits? Nested Aggregations? I was worried about the performance of nested aggregations here since my documents have a very large (> 10000) nested documents per parent document.

Why are nested objects hidden? In theory one could query the nested objects separately and join/group on the client?

[martijnvg]

The time it takes to fetch inner hits during the fetch phase depends on the amount documents being fetched. This is directly based on the regular size option and size option inside an inner hit definition and the number of inner hit definitions in a search request.

@rpedela How many inner hit definitions are specified in your query? and what is the specified size in each of this inner hit definition?

Should I consider a different approach than inner hits? Nested Aggregations?

You could try to use a nested agg with a top_hits as metric agg.

Why are nested objects hidden? In theory one could query the nested objects separately and join/group on the client?

Because nested docs are inlined with the main document and don't have an id like normal documents have.

[rpedela]

I have one inner_hits definition for children and one for grandchildren. The inner_hits size is set to 1 for both. I just want the top inner child and grandchild for my particular use case.

I originally tried aggregations but the data set has high cardinality and that was much slower.

[akasper]

@martijnvg: I've started working on this issue. Can we discuss some of the details at your earliest convenience?

[martijnvg]

Hey @akasper, the best way to discuss this is by opening a PR with the changes you like to make.

[mitsuh]

+1

[martijnvg]

Inner hits does no magic, it just executes extra fetch operation (during fetch phase) in order to inline inner hits into the regular hits. If inner hits is enabled on queries to retrieve many hits (lets say more than 100) then inner hits adds a significant performance tax to search request. If size=100 and a single inner hits is enabled then in total upto 400 hits are retrieved (100 for main and 300 for inner hits (default size for inner hit is 3) ). And if more inner hits are enabled and more regular hits are requested then this only gets worse.

Inner hits has been designed to give insight in the top matching nested document or child / parent documents. It wasn't build to retrieve many hits.

I'm leaning towards adding soft limits here. So that when inner hits is enabled and the regular size is higher than 100 we report an request error. Also when on the individual inner hits more than 10 inner hits are requested then we should report a request error as well. And finally the inner hits default size should be 1 instead of 3.

The idea that I shared initially to improve the fetching of inner hits during fetch phase would improve things a bit, but when many hits and inner hits are requested would resolve in similar bad performance.

[martijnvg]

Update: An improvement has made to inner hits (#24571) that will improve the slowness that is being reported here when inner hits has been enabled. This improvement will be part of the 5.5 release.

I expect a good improvement with the query shared in the description. The rewrite of the has_parent query is expensive and now this will be executed only once per shard whereas before it would happen upto 500 times.

[clexmond]

Thanks @martijnvg exciting news! I've largely refactored away from nested / inner hits, but would love to take advantage of it again with these improvements.

[rpedela]

Excellent! I look forward to future performance improvements discussed in the PR as well.

[dimfeld]

If you're having inner_hits performance problems with nested objects, it's may also be worth trying stored fields on the nested object, as the documentation suggests. I saw a performance improvement of more than 10x when I turned off _source and started using stored fields, in a case where my search was fetching a hundred objects and extracting about 50 inner hits from each one.

[martijnvg]

To complement @dimfeld suggestion, instead of using stored fields, doc values fields can also be used. The upside is the doc values are usually already enabled (due to the defaults).

[ianomad]

@clexmond great, what version is this going to be part of? also is there any workaround to achieve better performance without this fix?

[martijnvg]

great, what version is this going to be part of?

Version 5.5. Also the upcoming 5.5.2 will contain a fix for a performance bug (#25864)

also is there any workaround to achieve better performance without this fix?

No

[byronvoorbach]

Hey @martijnvg!
Do you guys have an idea on when you're releasing 5.5.2? At one of my clients, I was just about to upgrade the whole street to 5.5.1 (thinking it contained #25864).
Would prefer to take the 5.5.2 if release is not too far in the future

[martijnvg]

@byronvoorbach If you're on a version older than 5.5.0 then it makes sense to upgrade to get a performance boost with inner hits, otherwise waiting for 5.5.2 to be released makes sense. I suspect it will be released in a week or two.

[byronvoorbach]

@martijnvg Thanks for the quick reply, upgrading now it is 👍

[ilusharulkov]

Hello!
I have ES 5.6.4 and inner_hits query is extremly slow. It takes more then 10 sec. with 200k items in index.

[martijnvg]

@ilusharulkov Can you ask the question on the forum instead? I'm happy to help there.

[ilusharulkov]

Of course!

[martijnvg]

Two improvements have been made to inner hits to improve its performance. #24571 #25864

Also the a performance tip was documented that explains why fetching _source of nested documents is slow and that fetching doc values fields for nested documents is faster alternative: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-inner-hits.html#nested-inner-hits-source

Closing this issue, if any other inner hits specific performance issue is found then this should be handled in a new issue.