Elasticsearch: "score_mode": "max" is not working for nested Query and function score

[MaKuehn]

Elasticsearch version: 5.4.0

Plugins installed: [X-Pack]

JVM version (java -version):java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

OS version (uname -a if on a Unix-like system): macOS Sierra Version 10.12.4 (16E195)

Description of the problem including expected versus actual behavior:
The minimum score of the nested document is applied to the parent, although "score_mode": "max" is specified.

Steps to reproduce:

Create Index

PUT tests
{
  "mappings": {
    "test": {
      "properties": {
        "nestedDoc": {
          "type": "nested"
        }
      }
    }
  }
}

Index a test document

PUT tests/test/1
{
  "topVal" : 1,
  "nestedDoc": [
    {
      "nestedVal": 2
    },
    {
      "nestedVal": 3
    }
  ]
}

Verify the existence of the test document

GET tests/_search

Do a nested query with function scoring on the nestedVal. The expected score of the parent document should be 3, the maximum. However, 2 is returned. "avg" as score_mode works.

GET tests/test/_search
{
  "explain": true, 
  "query": {
    "nested" : {
      "query" : {
        "function_score" : {
          "query" : {
            "match_all": {}
          },
          "functions" : [
            {
              "filter" : {
                "match_all" : {
                  "boost" : 1.0
                }
              },
              "field_value_factor" : {
                "field" : "nestedDoc.nestedVal",
                "factor" : 1.0,
                "missing" : 0.0,
                "modifier" : "none"
              }
            }
          ],
          "score_mode" : "sum",
          "boost_mode" : "replace"
        }
      },
      "path" : "nestedDoc",
      "score_mode" : "max",
      "inner_hits" : {
        "name" : "nestedDoc",
        "ignore_unmapped" : true,
        "from" : 0,
        "size" : 30,
        "version" : false,
        "explain" : true,
        "track_scores" : true,
        "_source" : false
      }
    }
  }
}

[MaKuehn]

Tested under 5.1.2 and 5.3.0 and it worked as expected.

[michaelsiebers]

+1

[baltendo]

+1

[baltendo]

I found a problem in Lucene which might be related ...
https://issues.apache.org/jira/browse/LUCENE-7833

[baltendo]

We from Holidu (@MaKuehn, @michaelsiebers and me) can confirm that the scoring bug originates from the mentioned bug in Lucene. We built our own lucene-join-6.5.0 jar and replaced it on our test cluster and the test query above returned "max_score": 3.

[jpountz]

Lucene 6.6 is now out and has the fix, so we just need to upgrade to have the fix in 5.5. Separately, I'll try to see how we can get the fix in 5.4.x as well.

[jpountz]

This will be fixed in 5.4.2 and 5.5.0.

[elasticsearcher]

Thanks for fixing this! In the meantime, version 5.3.3 works as well.