Elasticsearch: Introduce the notion of system composable index templates

[martijnvg]

Currently many index template and composable index template exist for Elastic (internal) features, these
templates are installed by the elected master node when the cluster has booted. If these templates are
removed then these templates are added back and there is logic that optionally updates these templates if after
an Elasticsearch upgrade.

Besides what is described above, there isn't anything special about these template. While these templates should
be treated in a special way. For example these templates shouldn't be returned by default in the get template api. Nor should it be possible to even attempt to delete or modify those templates.

I think it would be great if system composable index templates and system component templates could be defined in the same way as system indices (Either via SystemIndexPlugin interface or another plugin interface). This should replace IndexTemplateRegistry and subclasses.

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[gwbrown]

It sounds like this overlaps substantially with https://github.com/elastic/elasticsearch/pull/65604, we should make sure we're coordinating efforts.

/cc @pugnascotia, @jaymode

[martijnvg]

Thanks @gwbrown for referencing that pr. I wasn't sure whether this falls under system indices, but the goal for opening this issue was for improving how we today manage templates of internal data streams / time based index patterns, which currently aren't a system indices, but are just hidden, which seems to fall outside the scope of the current system index development effort.

[pugnascotia]

There are obvious parallels between what I'm doing for system indices and the index template registry. However as the moment, I don't believe any system indices strictly require a template, because we can embed the settings and mappings in the system index descriptor. There's no support right now if a system index definitely requires a template - for example, if we ever wanted a system data stream. I had thought that a system data stream was unlikely.

At the moment, I think any implementation for this issue would be similar to #65604, but wouldn't actually overlap _per se_.

[martijnvg]

@pugnascotia I agree with your assessment that system indices have no dependency on this issue which purpose is to replace template registry infrastructure with system index templates.