Elasticsearch: random_score doesn't work if passed filter at query

[hbakhtiyor]

if remove the filter from the query, it works well

e.g.

  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": [{ "term": { "slug": "awe2ome" } }],
          "must_not": [{ "term": { "slug": "not_awe2ome" } }],
          "should": [],
          "filter": [{ "term": { "is_deleted": false } }],
          "minimum_should_match": 1
        }
      },
      "random_score": {},
      "score_mode": "sum"
    }
  }
}

official docker image elasticsearch:2.1.1

[jpountz]

Can you be more specific about what does not work?

[hbakhtiyor]

surely, randomization documents doesn't work

[clintongormley]

Random scoring works just fine:

POST t/t/_bulk
{"index":{"_id": 1}}
{ "is_deleted": false, "slug": "awe2ome"}
{"index":{"_id": 2}}
{ "is_deleted": false, "slug": "awe2ome"}
{"index":{"_id": 3}}
{ "is_deleted": false, "slug": "awe2ome"}
{"index":{"_id": 4}}
{ "is_deleted": false, "slug": "awe2ome"}
{"index":{"_id": 5}}
{ "is_deleted": false, "slug": "awe2ome"}

GET t/_search?_source=false
{
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": [
            {
              "term": {
                "slug": "awe2ome"
              }
            }
          ],
          "must_not": [
            {
              "term": {
                "slug": "not_awe2ome"
              }
            }
          ],
          "should": [],
          "filter": [
            {
              "term": {
                "is_deleted": false
              }
            }
          ],
          "minimum_should_match": 1
        }
      },
      "random_score": {},
      "score_mode": "sum"
    }
  }
}

Returns:

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 5,
    "max_score": 0.30669054,
    "hits": [
      {
        "_index": "t",
        "_type": "t",
        "_id": "4",
        "_score": 0.30669054
      },
      {
        "_index": "t",
        "_type": "t",
        "_id": "3",
        "_score": 0.28026438
      },
      {
        "_index": "t",
        "_type": "t",
        "_id": "2",
        "_score": 0.2672884
      },
      {
        "_index": "t",
        "_type": "t",
        "_id": "5",
        "_score": 0.19387394
      },
      {
        "_index": "t",
        "_type": "t",
        "_id": "1",
        "_score": 0.10168079
      }
    ]
  }
}

The scores (and order) change on every run.

[hbakhtiyor]

result:

{
  "took": 62,
  "timed_out": false,
  "_shards": {
    "total": 15,
    "successful": 15,
    "failed": 0
  },
  "hits": {
    "total": 461426,
    "max_score": 0,
    "hits": [
      {
        "_index": "my-index",
        "_type": "category",
        "_id": "56585bae66104e7300fde480",
        "_score": 0
      },
      {
        "_index": "my-index",
        "_type": "category",
        "_id": "56585bae66104e7300fdea62",
        "_score": 0
      },
      {
        "_index": "my-index",
        "_type": "category",
        "_id": "56585bae66104e7300fdee41",
        "_score": 0
      },
      {
        "_index": "my-index",
        "_type": "category",
        "_id": "56585bad66104e7300fdbb28",
        "_score": 0
      },
      {
        "_index": "my-index",
        "_type": "category",
        "_id": "56585bad66104e7300fdc968",
        "_score": 0
      },
      {
        "_index": "my-index",
        "_type": "category",
        "_id": "56585bae66104e7300fdd7db",
        "_score": 0
      },
      {
        "_index": "my-index",
        "_type": "category",
        "_id": "56585bae66104e7300fdf286",
        "_score": 0
      },
      {
        "_index": "my-index",
        "_type": "category",
        "_id": "56585bae66104e7300fdf4b5",
        "_score": 0
      },
      {
        "_index": "my-index",
        "_type": "category",
        "_id": "56585bae66104e7300fdf4c3",
        "_score": 0
      },
      {
        "_index": "my-index",
        "_type": "category",
        "_id": "56585bae66104e7300fdde9e",
        "_score": 0
      }
    ]
  }
}

the scores always are zeros and order not change on every run

[clintongormley]

@hbakhtiyor you're only providing half the information? how is your mapping/documents/query different from the one where i demonstrate that this is working correctly?

[hbakhtiyor]

@clintongormley the query is the same, your examples works well too, but with my documents it doesn't work

{

    "state": "open",
    "settings": {
        "index": {
            "creation_date": "1462118131951",
            "number_of_shards": "5",
            "number_of_replicas": "1",
            "uuid": "erwjFwrPTcGbbRZZqcR9mw",
            "version": {
                "created": "2010199"
            }
        }
    },
    "mappings": {
        "organization": {
            "properties": {
                "is_deleted": {
                    "type": "boolean"
                },
                "city": {
                    "type": "string"
                },
                "country": {
                    "type": "string"
                },
                "zip_code": {
                    "type": "string"
                },
                "street": {
                    "type": "string"
                },
                "area": {
                    "type": "string"
                },
                "id": {
                    "type": "string"
                },
                "slug": {
                    "type": "string"
                },
                "name": {
                    "type": "string"
                },
                "legal_name": {
                    "type": "string"
                },
                "location": {
                    "type": "geo_point",
                    "lat_lon": true
                }
            }
        }
    },
    "aliases": [ ]

}

[clintongormley]

@hbakhtiyor Please can you provide a full curl recreation demonstrating the problem (in the same way I did to demonstrate that it works)

[hbakhtiyor]

@clintongormley

now recognized that your example doesn't work too, if only leave filter at bool

GET t/_search?_source=false
{
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "filter": [
            {
              "term": {
                "is_deleted": false
              }
            }
          ],
          "minimum_should_match": 1
        }
      },
      "random_score": {},
      "score_mode": "sum"
    }
  }
}

[clintongormley]

That's because the filter clause returns scores of 0, which then get multiplied with any functions (because boost_mode defaults to multiply) which results in a final score of 0.

Change boost_mode to replace or sum, or use a constant_score query instead of bool.filter

[hbakhtiyor]

@clintongormley thanks for your explanation