Elasticsearch: Exclude system indices from matching index templates

[bczifra]

Describe the feature:
A template with an index pattern like "index_patterns" : [ "*"],... will match system indices like .kibana or .watcher*. It would be helpful to either be able to provide a way to additionally _exclude_ matches based on a given pattern or to somehow specifically exclude system indices from matching a template.

A concrete example of where this can present problems:

1. The .tasks index is no presented in the index.
2. An index template is created that defines a default _doc mapping with its index pattern set to *. The user intends this to apply to all of the user-generated indices.
3. Elasticsearch tries to create the .tasks index when a new task needs to be persisted, but cannot because it tries to add a task type mapping in addition to the default _doc mapping from the user-defined template. This results in the following error:

[2019-05-23T12:32:43,301][WARN ][o.e.t.LoggingTaskListener] [syseslog5-dk1] 52102862 failed with exception
org.elasticsearch.transport.RemoteTransportException: [syseslog3-dk1][10.194.1.139:9300][indices:admin/create]
Caused by: java.lang.IllegalArgumentException: Rejecting mapping update to [.tasks] as the final mapping would have more than 1 type: [task, doc]

[elasticmachine]

Pinging @elastic/es-core-features

[howardhuanghua]

Hi @bczifra , do we have the plan to implement this enhancement?

[bczifra]

@howardhuanghua I believe https://github.com/elastic/elasticsearch/issues/50251 will help address this.

[rgarcia89]

Where are we with this request?

[thenom]

I made the assumption that using the well documented exclude when providing a multi index pattern would work:

    "index_patterns" : [
      "*",
      "-.*"
    ],

But it seems this doesn't follow the same. This seems like a good method as it complies with other areas of Elasticsearch while still giving you full control, including over the .indices if required.

[lennehendrickx]

@thenom This does not seem to work. As per the documentation the index patterns array is evaluated as "or".

https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-templates.html
Defines a template named template_1, with a template pattern of te* or bar*. The settings and mappings will be applied to any index name that matches the te* or bar* pattern.

[thenom]

@lennehendrickx Yeh thats the problem, i was hoping it would work. Potentially and'ing the - removals or even offering:

    "index_patterns" : [
      "*,-.*"
    ],

To be honest though i haven't tried it this way

[lennehendrickx]

That's actually also not going to work, since the index_patterns field does not accept characters like ",".
An option is to use aliases with a specific pattern for your indexes.

[maggieghamry]

Trying

    "index_patterns" : [
      "*,-.*"
    ],

is still including system indices. Is there a recommended way to specify "index_patterns" : ["*"] would exclude any system indices?