Elasticsearch: Enriching context suggester to work with nested model

[nilabhsagar]

The nested model presents great capability, however when it comes to context suggestion on nested model, presently there is no way to provide nested fields as context. Because of this limitation user has to manually construct and index context values. Also, To retain the hierarchical relation these context values need to be indexed as a separate type. This increases complexity and maintenance effort and is relatively slow. The below example shows the above problem:

In the example user types in "Dennis" with a pre selected context(ProdName) value as "Health Plan One". We can see the problem in the current vs expected output.

This feature aims to solve above problem.

Addresses this: https://discuss.elastic.co/t/context-suggestor-in-elastic-search-applied-on-the-nested-fields/44946

Features

• Flexible Filtering: Suggestions should be filtered with fields specified as filters in the mapping.
• Read Input from other Fields: This enables the suggester field values to be picked from other input fields in the documents, however the old behavior of inline input values being specified in the suggest field will be supported as well.
• Nested Support: Both the suggest field and the filters can be nested to any level.

Mapping, Query and Response interfaces will follow/extend the interfaces of context suggester.

Proposed Mapping

For this functionality to work a field of type "filteredsuggest" can be defined at mapping time.
Format:

{
       ...
      "suggest_field_name": {
      "type": "filteredsuggest",
      "parent_path": <Nested path for this suggestion field>,
      "input_fields": "Read suggestion value from field/(s),
      "filters": {
        <Category filters>
      }
    }
}

Filters
Filters suggestions by criteria considering the nested hierarchy. At present the supported filter type is "category". This will be defined as below:

"filters": {
    "Name-of-filter": {
      "type": "category",
      "path": "Field path reference to read the filter value"
    },
    ...
}

Example

Following shows a field mapping for a filteredsuggest field named "name_suggest" defined at root level:

PUT {INDEX_NAME}  
{
  "mappings": {
    "Provider": {
      "properties": {
        "name": {
          "type": "text"
        },
        "Products": {
          "type": "nested",
          "properties": {
            "ProdName": {
              "type": "text"
            },
            "ProdType": {
              "type": "text"
            },
            "Hospitals": {
              "type": "nested",
              "properties": {
                "HospName": {
                  "type": "text"
                },
                "HospType": {
                  "type": "text"
                }
              }
            }
          }
        },
        "name_suggest": {
          "type": "filteredsuggest",
          "parent_path": "",
          "input_fields": "name",
          "filters": {
            "ProdType": {
              "type": "category",
              "path": "Products.ProdType"
            },
            "HospType": {
              "type": "category",
              "path": "Products.Hospitals.HospType"
            }
          }
        }
      }
    }
  }
}

Following shows a field mapping for a filteredsuggest field named "hosp_name_suggest" defined at Hospital level:

PUT {INDEX_NAME} 
{
  "mappings": {
    "Provider": {
      "properties": {
        "name": {
          "type": "text"
        },
        "Products": {
          "type": "nested",
          "properties": {
            "ProdName": {
              "type": "text"
            },
            "ProdType": {
              "type": "text"
            },
            "Hospitals": {
              "type": "nested",
              "properties": {
                "HospName": {
                  "type": "text"
                },
                "HospType": {
                  "type": "text"
                },
                "hosp_name_suggest": {
                  "type": "filteredsuggest",
                  "parent_path": "Products.Hospitals",
                  "input_fields": "Products.Hospitals.HospName",
                  "filters": {
                    "ProdType": {
                      "type": "category",
                      "path": "Products.ProdType"
                    },
                    "HospType": {
                      "type": "category",
                      "path": "Products.Hospitals.HospType"
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Proposed Indexing

At index time a document may or may not have suggest field instance. By default the suggest field input and filter values will be picked from the referenced fields as per mapping. This may be overridden by explicitly specifying the suggest field at document level and providing values for input.

Index Document with explicit input values for name_suggest mapping - document 1

POST {INDEX_NAME}/{TYPE_NAME}
{
  "name": "Simon",
  "Products": [
    {
      "ProdName": "Health Plan One",
      "ProdType": "TypeA",
      "Hospitals": [
        {
          "HospName": "Sakara hospital",
          "HospType": "MultiSpec"
        }
      ]
    },
    {
      "ProdName": "Health Plan One EEE",
      "ProdType": "TypeB",
      "Hospitals": [
        {
          "HospName": "Sagar hospital",
          "HospType": "Specialist-Heart"
        }
      ]
    }
  ],
  "name_suggest": {
    "input": ["Simon","Simon Robinson","Samuel"]
  }
}

Suggest Field Input Derived from other fields for name_suggest, hosp_name_suggest mapping- document 2

POST {INDEX_NAME}/{TYPE_NAME}
{
  "name": "Simon",
  "Products": [
    {
      "ProdName": "Health Plan One",
      "ProdType": "TypeA",
      "Hospitals": [
        {
          "HospName": "Sakara hospital",
          "HospType": "MultiSpec"
        }
      ]
    },
    {
      "ProdName": "Health Plan One EEE",
      "ProdType": "TypeB",
      "Hospitals": [
        {
          "HospName": "Sagar hospital",
          "HospType": "Specialist-Heart"
        }
      ]
    }
  ]
}

Proposed Query

Without filters - name_suggest; with document 1 indexed

POST {INDEX}/_search
{
  "suggest": {
    "name_suggestion": {
      "prefix": "sim",
      "filteredsuggest": {
        "field": "name_suggest",
        "size": 10
      }
    }
  }
}

The response is as below:

{
    ...
    "suggest" : {
    "name_suggestion" : [
      {
        "text" : "sim",
        "offset" : 0,
        "length" : 3,
        "options" : [
          {
            "text" : "Simon",
            "score" : 1.0
          },
          {
            "text" : "Simon Robinson",
            "score" : 1.0
          }
        ]
      }
    ]
  }
}

With filters - name_suggest; with document 2 indexed

POST {INDEX}/_search
{
  "suggest": {
    "name_suggestion": {
      "prefix": "sim",
      "filteredsuggest": {
        "field": "name_suggest",
        "size": 10,
        "filters": {
          "ProdType": "TypeB",
          "HospType": "Specialist-Heart"
        }
      }
    }
  }
}

The response is as below:

{
    ...
    "suggest" : {
    "name_suggestion" : [
      {
        "text" : "sim",
        "offset" : 0,
        "length" : 3,
        "options" : [
          {
            "text" : "Simon",
            "score" : 1.0
          }
        ]
      }
    ]
  }
}

With filters and fuzziness - name_suggest; with document 1 indexed

POST {INDEX}/_search
{
  "suggest": {
    "name_suggestion": {
      "prefix": "sim",
      "filteredsuggest": {
        "field": "name_suggest",
        "size": 10,
        "fuzzy": {
          "fuzziness": 2
        },
        "filters": {
          "ProdType": "TypeB",
          "HospType": "Specialist-Heart"
        }
      }
    }
  }
}

The response is as below:

{
    ...
    "suggest" : {
    "name_suggestion" : [
      {
        "text" : "sim",
        "offset" : 0,
        "length" : 3,
        "options" : [
          {
            "text" : "Samuel",
            "score" : 2.0
          },
          {
            "text" : "Simon",
            "score" : 2.0
          },
          {
            "text" : "Simon Robinson",
            "score" : 2.0
          }
        ]
      }
    ]
  }
}

Without filters - hosp_name_suggest; with document 2 indexed

POST {INDEX}/_search
{
  "suggest": {
    "name_suggestion": {
      "prefix": "sa",
      "filteredsuggest": {
        "field": "Products.Hospitals.hosp_name_suggest",
        "size": 10
      }
    }
  }
}

The response is as below:

{
    ...
    "suggest" : {
    "name_suggestion" : [
      {
        "text" : "sa",
        "offset" : 0,
        "length" : 2,
        "options" : [
          {
            "text" : "Sagar hospital",
            "score" : 1.0
          },
          {
            "text" : "Sakara hospital",
            "score" : 1.0
          }
        ]
      }
    ]
  }
}

With filters - hosp_name_suggest; with document 2 indexed
POST {INDEX}/_search

{
  "suggest": {
    "name_suggestion": {
      "prefix": "sa",
      "filteredsuggest": {
        "field": "Products.Hospitals.hosp_name_suggest",
        "size": 10,
        "filters": {
          "ProdType": "TypeB",
          "HospType": "Specialist-Heart"
        }
      }
    }
  }
}

The response is as below:

{
    ...
    "suggest" : {
    "name_suggestion" : [
      {
        "text" : "sa",
        "offset" : 0,
        "length" : 2,
        "options" : [
          {
            "text" : "Sagar hospital",
            "score" : 1.0
          }
        ]
      }
    ]
  }
}

[nilabhsagar]

Hi, Any update on this feature?

[bleskes]

@areek @clintongormley can you voice an opinion?

[nilabhsagar]

@areek @clintongormley Looking forward to know your opinion.

[nilabhsagar]

Hi, Any update on this feature?

[devikiranr]

Filtered Auto Suggestions on nested model had spawned out of a genuine requirement from a Product in our organization. We are currently using our fork of the Elasticsearch main repo for this and are having to merge with the release branch on every release of Elasticsearch. Hoping to have a good discussion on this and move ahead in the right direction. We have the code, analysis , benchmark numbers ready and are eager to implement the changes/ suggestions from you to take it to the next level.
We strongly believe that this will add value to many Elasticsearch users who are work on nested model and require filtered autosuggestions.

[clintongormley]

Why can't you just use copy_to to copy the nested values up to the top level? As in:

{
  "mappings": {
    "Provider": {
      "properties": {
        "name": {
          "type": "text"
        },
        "Products": {
          "type": "nested",
          "properties": {
            "ProdName": {
              "type": "text",
              "copy_to": "product_name"
            },
            "ProdType": {
              "type": "text",
              "copy_to": "product_type"
            }

Then you can just use product_name and product_type as your context fields.

[nilabhsagar]

By using copy_to we loose the relationship between nested entities field.
Consider below Mapping and a sample document

Mapping

{
  "mappings": {
    "Provider": {
      "properties": {
        "name": {
          "type": "text"
        },
        "ProdName": {
          "type": "text"
        },
        "ProdType": {
          "type": "text"
        },
        "Products": {
          "type": "nested",
          "properties": {
            "ProdName": {
              "type": "text",
              "copy_to": "ProdName"
            },
            "ProdType": {
              "type": "text",
              "copy_to": "ProdType"
            }
          }
        },
        "suggest": {
          "type": "completion",
          "contexts": [
            {
              "name": "ProdName",
              "type": "category",
              "path": "ProdName"
            },
            {
              "name": "ProdType",
              "type": "category",
              "path": "ProdType"
            }
          ]
        }
      }
    }
  }
}

Docuement-1

{
  "name": "Simon",
  "suggest": {
    "input": ["Simon"]
  },
  "Products": [
    {
      "ProdName": "UHC Choice Plus",
      "ProdType": "TypeA"
    },
    {
      "ProdName": "UHC Choice Plus EPO",
      "ProdType": "TypeB"
    }
  ]
}

Docuement-2

{
  "name": "Simran",
  "suggest": {
    "input": ["Simran"]
  },
  "Products": [
    {
      "ProdName": "UHC Choice Plus",
      "ProdType": "TypeB"
    },
    {
      "ProdName": "UHC Choice Plus EPO",
      "ProdType": "TypeA"
    }
  ]
}

Once the document is indexed, the suggestion context values will be as below
Docuement-1

"ProdType": ["TypeA","TypeB"]
"ProdName": ["UHC Choice Plus","UHC Choice Plus EPO"]

Docuement-2

"ProdType": ["TypeB","TypeA"]
"ProdName": ["UHC Choice Plus","UHC Choice Plus EPO"]

Here we can see that the relationship between the ProdName and ProdType is lost; Now if some one is looking for suggestions having prefix as "sim" and with the contexts as ["ProdName": UHC Choice Plus", "ProdType": "TypeA"], both "Simon" and "Simran" will surface as suggestions. However, going by the document structure only "Simon" should have surfaced.

[clintongormley]

Hi @nilabhsagar

We discussed this in our FixItFriday session today. This is a problem that could be worked around by combining the various nested fields into a single top-level context (eg combining ProdType and ProdName into a single context).

While this may not be as convenient as your suggestion, we are not keen to expand the completion suggester in this direction. It is already a complex memory hungry beast, and implementing your feature request would make it more so, not least because the addition of so many contexts will cause an automaton complexity explosion.

If this code works for you, then fine, I suggest that you convert it into a plugin that will make it easier for you to use than having to fork ES on every release. But we don't want to take the suggester in this direction.

Actually, we'd like to rethink how our suggesters are implemented and come up with an easier, simpler approach than we have currently but that is going to be a major project.

thanks anyway

[nilabhsagar]

Hi @clintongormley,

The suggestion for combining the fields into a single top-level context inherently has a complexity which we have addressed with this feature request. By using the solution provided by us, a user of ES nested model will be:

• Able to use this functionality by just using simple mapping and query.
• No coding effort will be required to create and copy context fields.
• Able to cope up with the data model changes.
• Will be free from analyzing the data model required for auto suggestion feature.
• Will get Nested model as a complete package for their problem at hand.

Regarding your memory usage concern, we do have some benchmark numbers to share which is performed on a real data set with 5 level of nesting. Since, we are dealing with nested model it is quite obvious that there will be lot of duplication and FST will provide a really good compression to minimize the memory footprint. I am attaching below the benchmark results for your reference.

Benchmark system configuration

Using XAnalysingSuggester

Using NRTSuggester

There are few points about the NRTSuggester based implementation, where we can see that the memory usage is too high. This is due to the fact that every context term is getting appended with doc id and hence FST is poorly compressed. This can certainly be addressed.

We see this will be a nice feature addition to Nested model or rather a foundation step for filtered auto suggestion on nested model. Looking forward to know your opinion.

[devikiranr]

@clintongormley
At present there is no work around for this in elasticsearch. Combining the filter fields into a top level context field would let go of the relationship between them.
To explain from the document structure above;

for " Simon" , 
    "UHC Choice Plus" was a "TypeA" product  
        "UHC Choice Plus EPO" was a "Type B" product```

```
for "Simran"
"UHC Choice Plus" was a "TypeB" product
"UHC Choice Plus EPO" was a "Type A" product

for the above scenario, the "top-level context" field values getting indexed against both the documents would be;
`"ProdType_ProdName": ["TypeA","TypeB","UHC Choice Plus","UHC Choice Plus EPO"]`

to reiterate;
 _The requirement at our end was to have autosuggestion on the top level "name" field which is filtered by the "ProdType" and "ProdName" fields._ 

Now consider the below query:
```prefix: "sim"
filter :{
    "ProdType":"Type A",
    "ProdName": "UHC Choice Plus"
} 

The expectation was to get those names as suggestions which had _"UHC Choice Plus" as a "Type A"_ product. (more of an AND ) i.e. _only "Simon" should have surfaced_.

While the top level context, would not be able to handle this as it would not know if for a given "name" the document had a nested product with the specified "ProdName" and "ProdType". Hence _both "Simran" & "Simon" would surface_.