Searchkick: Result window is too large

[theonegri]

I'm using this gem and Kaminari for the pagination to search through roughly 7 millions documents.

When clicking on "Last »" the pagination I get the error: "Result window is too large, from + size must be less than or equal to: [10000] but was [6745575]"

[ankane]

This is a limit Elasticsearch places on pagination. You can change the limit with index.max_result_window in Elasticsearch, or set max_pages in Kaminari.

[milushov]

Hello, @ankane, can you tell how to set index.max_result_window through searchkick? I didn't found any references in the readme.

[ankane]

It's a setting on the Elasticsearch server, so you aren't able to set it through Searchkick.

[milushov]

@ankane thanks for info

for those who will encounter same issue, here is a quick workaround:

# items_staging - name of the index
# localhost:9200 - default host and port
curl -XPUT "http://localhost:9200/items_staging/_settings" -d '{ "index" : { "max_result_window" : 100000 } }'
# ckecking that setting saved ok
curl -s -XGET "http://localhost:9200/items_staging/_settings?pretty" | grep window

[ankane]

Didn't realize you could set it with _settings. In that case, you should be able to do:

class Item < ActiveRecord::Base
  searchkick settings: {index: {max_result_window: 100000}}
end

[Lewis-Clayton]

Just in case anyone else is copy paste happy like me. The above should be max_result_window

class Item < ActiveRecord::Base
  searchkick settings: {index: {max_result_window: 100000}}
end

[ankane]

Nice catch, updated :)

[ankane]

Also, be sure you understand the implications of increasing it.

---

To understand why deep paging is problematic, let’s imagine that we are searching within a single index with five primary shards. When we request the first page of results (results 1 to 10), each shard produces its own top 10 results and returns them to the coordinating node, which then sorts all 50 results in order to select the overall top 10.

Now imagine that we ask for page 1,000—results 10,001 to 10,010. Everything works in the same way except that each shard has to produce its top 10,010 results. The coordinating node then sorts through all 50,050 results and discards 50,040 of them!

You can see that, in a distributed system, the cost of sorting results grows exponentially the deeper we page. There is a good reason that web search engines don’t return more than 1,000 results for any query.

https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html

[chamnap]

@ankane, do you have any plan to support the scroll api of elasticsearch?

[ankane]

@chamnap No plans at the moment.

[chamnap]

@ankane I need to paginate the search results, which could have 10 millions records. Based on the above, I could not just increase max_result_window. What is the most efficient way of doing this currently beside using the scroll api?

[ankane]

As far as I know, those are your choices.

[maxdbn]

@ankane searchkick settings: {index: {max_result_window: 100000}} did not work for me and I'm still getting this error. There is still no way to paginate through more than 10K results?
Thank you for this wonderful gem btw

# UPDATE: forgot to reindex! thank you.

[kathgironpe]

OMG. I am paginating millions of records. Should have seen this earlier. Thanks guys!

[maxdbn]

@katgironpe If you actually have millions of results it would be better for you to use scroll api

[kathgironpe]

@MaxDBN We're not returning a large number of results per page. So simply increasing the limit to 7m or 10m will work. But definitely there may be cases where we will need the scroll API. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html Thank you!

[monteirobrena]

I have a similar problem when I tried to access my last page:

search?page=1944

I received the error below. But pay attention that my page is 1944 and the error show as 19440. Any ideas? Thank you.

[500] {"error":{"root_cause":[{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [19440]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"blah_production_201103105241514","node":"sdfasdf-g","reason":{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [19440]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter."}}]},"status":500}

PS.: I'm using 2.4.6 version of ElasticSearch and searchkick 2.3

[ankane]

Hey @monteirobrena, the solution is above.

[monteirobrena]

Thank you for the quick answer, @ankane. But I think that I don't need to increase the limit max_result_window because my value 1944 is so much lower than 10000. I need to identify/fix why the value is 19440 when I receive the error.

[ankane]

Are there 10 items per page?

[monteirobrena]

Yes.

But I still can't understand what is the need to increase this value when it's already paginated and I don't need to request all of this in the same request.

[ankane]

I'm not sure I can explain it any better than excerpt from the Elasticsearch docs above. Maybe you can ask on Stack Overflow.

[monteirobrena]

Sure! Thank you, @ankane.

[wcpaez]

Instead of using kaminari max_pages, you can use the total_pages option on kaminari paginate helper:

<%= paginate(@search, total_pages: @total_pages) %>

This depends on how many you allow per page, in my case @total_pages is 333 to not break the 10000 limit:

@total_pages = @search.total_count > 10000 ? 333 : @search.total_pages