Elasticsearch: Allow post_filter access to the _score

[dmitry]

I found it's allowed to access _score inside the aggs, but not allowed to access this field in the post_filter queries, though it's executed after the aggs. Is there a reason for that or it could be added?

An example query:

{
  "body": {
    "post_filter": {
      "script": {
        "script": "_score > 100 && _score < 200"
      }
    },
    "query": {
      "function_score": {
        "functions": [{
          "script_score": {
            "script": "doc['price'].value * 100"
          }
        }]
      }
    }
  },
  "index": ["items"],
  "type": []
}

script_score could be much more complex, it's just an example.

More than that I cannot find a solution to that problem: http://stackoverflow.com/questions/39109511/how-to-filter-by-script-field-in-elasticsearch?noredirect=1&lq=1

[s1monw]

just use script query https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-query.html#query-dsl-script-query

[dmitry]

@s1monw do you mean in my example I have to use script with doc['price'].value * 100 in the post_filter? What about performance? It's definitely not easy to control script for both, as I would like to use _score inside the aggs afterwards. So search query will be slower at least 2x times, or even more...

[dmitry]

Hey guys, @uboness and @jpountz, please comment.

[dmitry]

Really need this feature, as score calculation in each filter slows down our search. What can I do to push it forward?

[clintongormley]

@jpountz is there any reason why the _score is not available in a post_filter?

[jpountz]

Agreed it'd be nice for scores to be available to post filters but there is no way that I can think of that we can propagate the information today.

[dmitry]

@jpountz does it mean it cannot be implemented in the nearest future?

[gm42]

Is it not possible to propagate the information because there is no "formal" channel of communication between scoring and post_filter in current ES?

[jpountz]

Yes. In short, the script query returns an implementation of the Lucene Query API that uses a script internally, and the Query API does not give ways to propagate scores so that scripts can be made aware of them. We could hack it in the case that the post filter only contains a script query (if the query is an instance of that class, then get back the wrapped script and pass the scorer object to it so that it can be aware of the scores), but that would not work in the general case, for instance if the query in the post filter is a compound query (eg. bool) with script queries on the leaves.

[gm42]

@jpountz thanks for the reply. Since situation is like that, shouldn't this feature request be abstracted to a more general "access computed fields in post_filter"?

In an ideal scenario (not really an ES expert here, please bear with me) one could compute via scripts multiple "virtual" script fields and then use them in the post_filter. I know that this possibility would have an impact on the memory usage since reduction would require more buffers, but just throwing it there as it seems a common use-case...

[dmitry]

I still hope the _score could be accessed in a post_filter this year :)

For more details also look to the https://github.com/elastic/elasticsearch/issues/10767 (use-case is quite often)

[izaakschroeder]

Any update on this issue? Using query points for pagination and score sorting really depends on having this! 😄

[dmitry]

@izaakschroeder I also had such usecase, would be great if anyone from the ES team can look into that soon.

[cbuescher]

cc @elastic/es-search-aggs