Cloud-on-k8s: Be able to configure index settings / ILM with CRD

[mtparet]

Proposal

Be able to configure ILM (and generally index settings) through a CRD.

Why is it so important?

This makes possible to natively industrialize (reducing a lot the cost of creating new clusters) creating ElasticSearch cluster ready for typical use cases with the right configuration.

Also it enables gitops for what is possible to do with these new CRDs.

[andreykaipov]

I actually spoke with @sebgl during Kubecon about the possibility of this feature. It's crazy you had the same thought around the same time! Our use case would definitely fall under the ability to do GitOps, e.g. maintaining a set of ILM CRDs that our pipelines can deploy to K8s, resolved by this operator.

[sebgl]

A few thoughts:

• We should investigate whether this should be a stack (Elasticsearch) feature instead of an ECK feature.
• In general we want to avoid having multiple ways of doing things: ILM is a good example of a feature already available through the Elasticsearch API that we may not want to reimplement.
• ECK would probably reconcile/apply the ILM settings forever. If the user then patches the settings through Elasticsearch API but ECK reverts the user changes to set its own version of ILM policies, things can become confusing.
• The need for configuration-as-code/GitOps probably extends to a larger scope and include eg. the automatic creation of indices with a predefined mapping.

[pebrc]

Related #2155

[easyCZ]

What's the state of this feature? Are you accepting PRs for it?

Improbable has a use-case to use the operator to create and bootstrap a cluster, including users, roles, ILMs, aliases, index-tempaltes and index-patterns for Kibana. I'd be happy to contribute this.

[agup006]

Hey folks, we are still taking a look into this

[weeco]

Adding this feature in the upstream Elasticsearch has been rejected. How can we proceed now?