Elasticsearch: "must_not exists" query inside a nested query does not match documents but "must exists" does

[elasticsearcher]

Elasticsearch version (bin/elasticsearch --version): Version: 5.3.3, Build: 65e26e6/2017-05-22T12:03:12.318Z, JVM: 1.8.0_131

Plugins installed: [x-pack]

JVM version (java -version): JVM: 1.8.0_131

OS version (uname -a if on a Unix-like system): Darwin Kernel Version 15.6.0: Thu Jun 23 18:25:34 PDT 2016; root:xnu-3248.60.10~1/RELEASE_X86_64 x86_64

Description of the problem including expected versus actual behavior:

I have a mapping type that has a nested property that can be empty for some documents. I tried running the following query to find out which documents don't have a value for the nested field but it returned no hits even though such documents do in fact exist:

POST test_index/test_type/_search
{
  "query": {
    "nested": {
      "path": "nested_field",
      "query": {
        "bool": {
          "must_not": {
            "exists": {
              "field": "nested_field.name"
            }
          }
        }
      }
    }
  }
}

However, the same query but with "must" instead of "must_not" works fine and returns correct results:

POST test_index/test_type/_search
{
  "query": {
    "nested": {
      "path": "nested_field",
      "query": {
        "bool": {
          "must": {
            "exists": {
              "field": "nested_field.name"
            }
          }
        }
      }
    }
  }
}

In the end, I was able to get what I needed by inverting the "nested" and "must_not" queries:

POST test_index/test_type/_search
{
  "query": {
    "bool": {
      "must_not": {
        "nested": {
          "path": "nested_field",
          "query": {
            "exists": {
              "field": "nested_field.name"
            }
          }
        }
      }
    }
  }
}

Steps to reproduce:

1. Create an index, a mapping with a nested field, and one document that sets the nested field and one that doesn't:

PUT test_index/

PUT test_index/_mapping/test_type
{
  "dynamic": "strict",
  "properties": {
    "nested_field": {
      "type": "nested",
      "properties": {
        "name": {
          "type": "text"
        }
      }
    }
  }
}

PUT test_index/test_type/1
{
  "nested_field": {
    "name": "test"
  }
}

PUT test_index/test_type/2
{
}

1. Run the following query:

POST test_index/test_type/_search
{
  "query": {
    "nested": {
      "path": "nested_field",
      "query": {
        "bool": {
          "must_not": {
            "exists": {
              "field": "nested_field.name"
            }
          }
        }
      }
    }
  }
}

1. Notice that it returns no hits

[jpountz]

This is the expected behaviour:

• Query 1 finds all documents that have one nested document or more that do not have a value for nested_field.name
• Query 2 finds all documents that have one nested document or more that have a value for nested_field.name
• Query 3 finds all documents whose none of the nested documents have a value for nested_field.name

The inverse of query 2 is indeed query 3, not query 1.

[elasticsearcher]

@jpountz Thanks for the explanation, this makes sense. I overlooked the fact that a document must have a nested document in the first place for the "must_not" clause to apply.

[zhugw]

Solved a similar problem of mine