Elasticsearch: Feature loss in Elasticsearch 6.x : template reference in a percolate query

[fbaligand]

With ES 5.X, I can index a document with a percolator field which references a stored template query.
Example :

{
  "category": "category1",
  "percolator_query":  {
    "template": { "id": "mytemplate-query", "params": { "param1": "value1" } }
  }
}

But since Elasticsearch 6.0, template queries are removed.
And to my known, I can't call search template api (the replacement) from a percolator field.

=> Is there a way in Elasticsearch 6.X to call a stored template query from a percolator field ?
=> If not, could you add a feature which answers this need ? I need it to migrate to ES 6.X

[martijnvg]

@fbaligand Yes, the template query has been removed. What is your use case here exactly and could you use the script query instead?

Note that the template query if used as a percolator query had some trickiness. The template is evaluated when the percolator is indexed, so if the template changes all the percolator using that template will still need to be reindexed in order for the template change to take effect.

[fbaligand]

Hi @martijnvg

script query seems to be a good replacement, but when I look script query documentation, it seems we cannot reference a pre-registered script (with an id option, for example).
=> Is this possible but just not currently documented ?

My use case is :

• I try to categorize a input document, using percolate API
• all my "percolate" documents use the same query template, but with one different parameter
• my query template is quite complex and I want to easily update it when I want
• and yes, I know that each time I change my query template, I have to reindex "percolate" documents, but it's quite easy calling simply : POST /my-index/_update_by_query

[fbaligand]

@martijnvg
I tried the following test to try script as you suggest :

POST index1/doc/1
{
  "title": "my title"
}

POST _scripts/template1
{
    "script": {
        "lang": "mustache",
        "source": {
            "query": {
                "match": {
                    "title": "{{query_string}}"
                }
            }
        }
    }
}

POST index1/_search
{
  "query": {
    "script": {
      "script": {
        "id": "template1",
        "params": {
          "query_string": "title"
        }
      }
    }
  }
}

But it raises this error :

{
  "error": {
    "root_cause": [
      {
        "type": "general_script_exception",
        "reason": "Failed to compile stored script [template1] using lang [mustache]"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "index1",
        "node": "HmS7EuebTPWLQGoXeciEyw",
        "reason": {
          "type": "query_shard_exception",
          "reason": "failed to create query: {\n  \"script\" : {\n    \"script\" : {\n      \"id\" : \"template1\",\n      \"params\" : {\n        \"query_string\" : \"title\"\n      }\n    },\n    \"boost\" : 1.0\n  }\n}",
          "index_uuid": "gjZJhSJgRSOvFjVF32Hlvg",
          "index": "index1",
          "caused_by": {
            "type": "general_script_exception",
            "reason": "Failed to compile stored script [template1] using lang [mustache]",
            "caused_by": {
              "type": "illegal_argument_exception",
              "reason": "mustache engine does not know how to handle context [filter]"
            }
          }
        }
      }
    ]
  },
  "status": 400
}

However, if I test to call the script by search template API, it works fine :

GET index1/_search/template
{
    "id": "template1", 
    "params": {
        "query_string": "title"
    }
}

=> Maybe I configured something wrong ? Any idea ?

[martijnvg]

script query seems to be a good replacement, but when I look script query documentation, it seems we cannot reference a pre-registered script (with an id option, for example).
=> Is this possible but just not currently documented ?

Referring to a stored script via its id is possible, like in other ES apis. Just replace the source field with the id field. As is mentioned here: https://www.elastic.co/guide/en/elasticsearch/reference/6.1/modules-scripting-using.html

and yes, I know that each time I change my query template, I have to reindex "percolate" documents, but it's quite easy calling simply : POST /my-index/_update_by_query

I see, that is from an api perspective a reasonable workaround. However on the ES side it still results in the same amount of work that needs to be performed.

[martijnvg]

=> Maybe I configured something wrong ? Any idea ?

That is because only script languages (painless by default) are supported inside script queries.

[martijnvg]

@fbaligand Based on what you've described, I think it makes sense to add template support just to percolate queries, by adding a template element that just the percolator field mapper understands. I think it should be limited to only allow a stored template to be specified, because specifying the template source in a percolator query itself doesn't make a lot of sense. The only downside is that manually the update by query api needs to be invoked after the template has changed, but considering that there isn't a clean way to solve this internally, this is an acceptable workaround.

[fbaligand]

thank you for this enhancement suggestion !

[jpountz]

cc @elastic/es-search-aggs

[ddadon10]

Hi everyone,

I hope you are doing well,

I am glad to see a conversation about the template reference because I have the similaire issues,

I got other use cases for the template reference outside of the Percolate Query scope:

1) Kibana Filter

I use template to create complexe filter and update them easily :

I did not found how to do that with the _search/template (I Got an error in the Kibana Filter)

2) Call Template inside other Templates

I actually don't know if this a good practice but I found this very useful:

With the template reference I can call template inside other template, it help me to stay DRY and have a better abstraction.

For example I have the weather template that call the standard template with some args.

weather.mustache

{
    "template": {
      "file": "standard",
      "params": {
        "domain": "weather.com",
        "name": "weather"
      }
    }
}

standard.mustache

{
  "constant_score": {
    "filter": {
      "bool": {
        "should": [{
            "term": {
              "request.url.domain": "{{domain}}"
            }
          },
          {
            "term": {
              "initiator.first.url.domain": "{{domain}}"
            }
          },
          {
            "term": {
              "initiator.last.url.domain": "{{domain}}"
            }
          }
        ],
        "_name": "{{name}}"
      }
    }
  }
}

I did not find other way do to that without the template reference.

It would be great to have these features still availables in ES 6.X :)

Best,

[idmanish]

I am using ES version 5.6 and trying to get percolator work using template query mustache, Even I tried example mention above with ES 5.X, it's not working.
Do I need to do something additional to get template query with the percolator.
It's working for normal search but not with percolator

[fbaligand]

@idmanish
Could you push here the REST commands that you have executed ?