Elasticsearch: Index name resolver should not match '.*' indices unless pattern starts with '.'

[dimitris-athanasiou]

Internal indices should behave as non-existent for the user unless explicitly requested. Basically, like linux, we change the index name resolver to always exclude indices starting with '.' unless the index expression explicitly starts with a '.'. This would avoid expressions matching internal indices, which most of the time would lead to unwanted surprises.

[elasticmachine]

Pinging @elastic/es-core-infra

[javanna]

This is going to affect also whoever creates indices starting with ., besides our own products. In fact with such a change we would be introducing the notion of hidden indices. Not saying it's a bad idea though. We have tried something similar before (see #16904), but I am not sure why we haven't implemented it after all.

[s1monw]

We spoke about this in fixit friday and we came up with an idea. I tried to make indices hidden in the past and it had some issues and tired to do more. I think I can take another look at it with these requirements:

• indices need to be marked as hidden with a setting then they won't be resolved.
• indices that are marked as hidden will be resolved through aliases
• aliases can be marked as hidden as well (think of .security)
• the . prefix should not be taken into account here otherwise BWC is difficult for users

@javanna @clintongormley WDYT

[clintongormley]

Sounds good to me.

I presume we'd have an expand indices option which would allow us to resolve wildcards to hidden indices as well? (Otherwise we'd have no way of figuring out if such indices exist other than trying each one by name).

aliases can be marked as hidden as well (think of .security)

Why would we need to mark aliases as hidden? Surely the fact that the alias resolves to hidden indices would be sufficient?

[droberts195]

I presume we'd have an expand indices option which would allow us to resolve wildcards to hidden indices as well?

This option would be very useful for ML, as ML can have many internal indices. Even if we somehow managed to get away without it for production, it would make life hard for developers if we cannot search .ml-anomalies* when debugging. So it would be nice if the default was to exclude hidden indices, but there was an option to include them.

[rjernst]

@droberts195 I think this is already handled in the issue description. Your example starts with . and therefore would work. The idea here is to omit dot indices when matching *.

[droberts195]

This will likely be superseded by whatever is decided in #38678.

[jaymode]

Hidden indices was implemented in #51164 and as part of the resolution for another issue, #52304, we plan to make aliases hidden. Once that is done, I believe we can close this issue. cc @gwbrown

[jaymode]

Hidden aliases have been implemented in #52547