Elasticsearch: Index Lifecycle Policies effecting system indices in version 7.7

[rgarcia89]

Elasticsearch version (bin/elasticsearch --version): 7.7

Plugins installed: []

JVM version (java -version):

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:
In elasticsearch version 7.7 system indicies are also being affected by Index Lifecycle Policies. This causes some serious issues when it comes to freezing, changing to read-only or even deleting indicies using policies.

Steps to reproduce:
Update to version 7.7 and list the managed vs unmanaged indicies

Screenshots version 7.6.2 vs 7.7.0

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[joegallo]

@rgarcia89 would it be possible for you to share the relevant parts of the details for denic-index-policy? I'm curious about the exact scenario by which, for example, your .kibana_1 index ended up being managed by ILM.

[rgarcia89]

@joegallo sure please fine them below:
If I create a template as shown below, hidden indicies will also get assigned the policy.

PUT _template/denic-index-template?include_type_name
{
  "order": 100,
  "index_patterns": [
    "*"
  ],
  "settings": {
    "index": {
      "format": "1",
      "lifecycle": {
        "name": "denic-index-policy"
      },
...
}

In order to not run into this issue anymore, I had to use a pretty ugly workaround, which at least works:

PUT _template/denic-index-template?include_type_name
{
  "order": 100,
  "index_patterns": [
    "a*",
    "b*",
    "c*",
    "d*",
    "e*",
...

[dakrone]

@joegallo * templates should be ignored for hidden indices, as per:

https://github.com/elastic/elasticsearch/blob/81a1e9eda8e6183f5237786246f6dced26a10eaf/server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateService.java#L347-L353

So either the .kibana_1 index isn't appropriately marked as hidden, or else there is a bug elsewhere.

[rgarcia89]

@dakrone in that case there must be a bug, because I can 100% confirm that they are not being ignored

[joegallo]

I've spun up a 7.6.2 and a 7.7.0 cluster, and created a test-template on both. It looks like this:

GET _template/test-template
{
  "test-template": {
    "index_patterns": [
      "*"
    ],
    "mappings": {
      "properties": {},
      "_meta": {},
      "_source": {}
    },
    "aliases": {},
    "order": 0,
    "settings": {
      "index": {
        "number_of_replicas": "3"
      }
    }
  }
}

Then I created a dot-prefixed index on both versions using PUT .myowndotindex and then fetched the settings:

GET .myowndotindex/_settings
{
  ".myowndotindex": {
    "settings": {
      "index": {
        "provided_name": ".myowndotindex",
        "number_of_replicas": "3", # <---<<<
        "uuid": "5WrPU2erSJSIBJRGn4z5-Q",
        "number_of_shards": "1",
        "creation_date": "1607641206788",
        "version": {
          "created": "7070099"
        }
      }
    }
  }
}

Crucially, both versions ended up with the number_of_replicas that I'd have expected if they were created via the template. And that makes some amount of sense to me, in that I don't see any special code around ignoring dot-prefixed indices when looking up templates (i.e. AFAICT "*" has always meant "all indices", at least w.r.t. dot-prefixes).

@rgarcia89 are you perhaps seeing an order of operations artifact? If you have a cluster with a .kibana_1 and other system indices already in existence, and you add a "*" template to it, it doesn't retroactively apply that template to the already created indices, BUT OTOH if you created the "*" template before those indices were created, then it would.

[rgarcia89]

@joegallo thats correct, templates are not being applied retroactively, which is totally fine for me. However, of course I created the template before the next hidden index has been created. The "*" template is being used as kind of catch all for indicies that do not match any other template.

From my perspective the documentations says that hidden indicies will not be addressed by the "*" so I don't understand why it is acting differently. Thinking about the fact, that these indicies are necessary for running kibana I would expect some kind protection that ensures that hidden or at least system related indicies cannot be touched by an lifecycle policy

[rgarcia89]

@joegallo any update on the request?

[joegallo]

From my perspective the documentations says that hidden indicies will not be addressed by the "*"

Can you link me to where the docs indicate this? I'm guessing there's a difference between the docs and the behavior, or perhaps the expected behavior, and I'd like to narrow in on that.

[rgarcia89]

Can you link me to where the docs indicate this? I'm guessing there's a difference between the docs and the behavior, or perhaps the expected behavior, and I'd like to narrow in on that.

This is what I understand from here: https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-index.html#hidden-indices