Elasticsearch: Should hidden indices be exempt from auto-create rules

[mayya-sharipova]

After a user sets cluster setting action.auto_create_index to false, hidden indices can not be created which results in errors.
For example, watcher fails to update watch records, as it can't create corresponding .watcher-history* indices.

Should hidden indices be exempt from auto-create rules?
Another possibility is for a watcher to create these indices explicitly but that seems wrong for time-based indices ?

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Watcher)

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[jakelandis]

Specific to Watcher history, we will likely convert those over to data streams. Under the covers it will still use hidden indices, but they would be explicitly created via the data stream.

However, the question "Should hidden indices be exempt from auto-create rules?" is still a good question.

[droberts195]

The system/hidden index meta issue, #50251, has an item:

• [ ] Implement ability for system indices to always be created even with action.auto_create_index. Possibly means implementing the solution described in #20640

It does not currently state that hidden indices will be immune as well as system indices, but if the solution _is_ #20640 then that would cover it.

If the solution turns out _not_ to be #20640 then I am wondering if the requirement should be that all _internal_ indices should be immune to action.auto_create_index whether they are hidden or system indices. But hidden indices that are not internal, i.e. names not beginning with . could be affected by action.auto_create_index.

[gwbrown]

Thanks for raising the issue!

You're right @droberts195, it's been planned for a while that system indices will be immune to auto_create_index, what's proposed in this issue is new: that hidden indices also be exempt from that setting.

As it stands today, I prefer https://github.com/elastic/elasticsearch/issues/20640 - that we allow overriding of auto-creation in an index template. That solution allows us to be sure that there is an index template (so we don't accidentally get default mappings) and allows applications that manage index templates to manage whether auto-creation should be allowed or not, rather than having a cluster-level setting.

Regarding this proposal, my biggest concern is overloading hidden indices with too much meaning. Currently, the hidden/visible status of an index only impacts wildcard resolution (at wildcard expansion time and at index template matching time, though in different ways), and I think keeping it that way would probably be for the best. That said, I'm potentially open to expanding the definition if we would gain a large enough benefit from doing so.

I guess my core question is this: What would doing this get us that https://github.com/elastic/elasticsearch/issues/20640 doesn't?

[gwbrown]

We discussed this in today's Fix-It meeting. The conclusion we reached was that this approach has only minor benefits over the approach of a flag on an index template as in https://github.com/elastic/elasticsearch/issues/20640, and allowing auto-creation based on the hidden-ness of an index mixes together two separate ideas that we would rather keep separate. Especially since hidden-ness is determined by an index setting that may be changed at any time. For that reason, we decided to not implement this approach.

I'm going to close this issue, with continuing discussion on https://github.com/elastic/elasticsearch/issues/20640. While there was further discussion on what should be implemented, I'll summarize that over on https://github.com/elastic/elasticsearch/issues/20640 to avoid repeating myself.

[colings86]

@gwbrown given the above comment, should this issue be closed? or did we later decide to keep it open?

[gwbrown]

@colings86 Whoops - I definitely meant to close this. I think I just clicked the wrong button. Thanks!