Elasticsearch: must_not queries on nested types should also return parents without children

[mayya-sharipova]

This came from: https://discuss.elastic.co/t/range-query-has-weird-results/149359

This query is supposed to return documents where there is not any nonAvailability as well as those that do not fall in the range. However, this query only returns documents that have nonAvailability values. Any documents with '[]' values (unlimited availability) do not return.

"query": {
  "nested" : {
   "path" : "nonAvailability",
   "query" :  {
    "bool" : {
     "must_not" : [{
      "range" : {
       "nonAvailability.date" : {
        "gte" : "2018-07-01",
        "lt" : "2018-07-03",
        "relation" : "within"
       }
      }
     }]
    }
   }
  }
}

[mayya-sharipova]

One thing we can do is:
First get all parents are they are now returned and join them with parents without any nested docs.

We can get all parents without any nested docs with the following query:

GET _search
{
  "query": {
    "must_not": {
      "nested": {
        "path": "nonAvailability",
        "query": {
          "exists": {
            "field": "nonAvailability.date"
          }
        }
      }
    }
  }
}

[elasticmachine]

Pinging @elastic/es-search-aggs

[cbismuth]

Looks interesting! I’d like to work on this one.

[cbismuth]

cURL recreation script created here.

[cbismuth]

Hi @mayya-sharipova, I'm hacking around in the NestedQueryBuilder to tweak the innerQuery and make it include parents without nested docs, but I'm not sure I'm going in the right direction.

Could you please elaborate a little bit more about where you think we can join with parents without any nested docs? Thank you.

[mayya-sharipova]

@cbismuth Thank you for working on this. Now I am thinking about this, and it is a little more complex than I thought. We need to add parents without any children docs only in the case of must_not query on children. Also the query to find all parents without any children is not simple.

I would like to ask @jimczi opinion on this. Jim is it really worth doing this considering complexity?

[jimczi]

The must_not clause is in a nested context so it can only match children documents. The nested query only sees children document in the innerQuery and I don't think we should change that.
One way to circumvent this limitation is to flip the bool and nested query:

{
    "query": {
        "bool": {
            "must_not": [
                {
                    "nested" : {
                        "path" : "nonAvailability",
                        "query": {
                            "range" : {
                                "nonAvailability.date" : {
                                    "gte" : "2018-07-01",
                                    "lt" : "2018-07-03",
                                    "relation" : "within"
                                }
                            }
                        }
                    }
                }
            ]
        }
    }
}

This way the must_not clause is executed at the parent level and parent without children can match.

[mayya-sharipova]

@jimczi Thank you for suggestion, Jim . Do you suggest to run your query in addition to the original query?

Because looks like your query by itself doesn't address the user's problem. He wanted to find both with a single request: 1) parent docs without any children + 2) parent docs with children that don't satisfy certain criteria. Your query can find parents without any children well, but I think it may miss some parents that should be returned. For example, for parents that have both children that satisfy and don't satisfy criteria:

• "nested -> must_not CRITERIA " will return them , because they have children that don't satisfy criteria,
• "must_not ->nested CRITERIA " will miss them, because they also have children that satisfy criteria and we are reversing those parents ids.

I was thinking can we run your query as a part of nested->must_not request inside elasticsearch, or should we leave it to users?

[mayya-sharipova]

After investigation with @jimczi, we have decided not to implement anything on the elasticsearch side, as it doesn't worth the complexity. Instead the user can address his problem with one of the following ways depending on his expectations regarding parents with mixed children (that satisfy and don't satisfy must_not criterion).
First reversing nested and must. This will return parents without any children and parents that have children satisfying must_not criterion, but will miss some parents that have both children that satisfy and don't satisfy must_not criterion:

{
  "query": {
    "bool": {
      "must_not": [
        {
          "nested": {
            "path": "nonAvailability",
            "query": {
              "range": {
                "nonAvailability.date": {
                  "gte": "2018-07-01",
                  "lt": "2018-07-03",
                  "relation": "within"
                }
              }
            }
          }
        }
      ]
    }
  }
}

Second is to combine the original query with another bool query that will give us parents without children. The query will be more complex but it should return all necessary parents.

{
  "query": {
    "bool": {
      "should": [
        {
          "bool": {
            "must_not": {
              "nested": {
                "path": "nonAvailability",
                "query": {
                  "exists": {
                    "field": "nonAvailability.date"
                  }
                }
              }
            }
          }
        },
        {
          "nested": {
            "path": "nonAvailability",
            "query": {
              "bool": {
                "must_not": [
                  {
                    "range": {
                      "nonAvailability.date" : {
                        "gte" : "2018-07-01",
                        "lt" : "2018-07-03",
                        "relation" : "within"
                      }
                    }
                  }
                ]
              }
            }
          }
        }
      ]
    }
  }
}