Helm-charts: elasticsearch curator?

Created on 27 Jul 2019 · 6Comments · Source: elastic/helm-charts

Is this still the recommended way to delete old indices?
If so any plans to add a curator chart, let alone official docker image?

question triagstale

Source

naseemkullah

👍1

All 6 comments

Is this still the recommended way to delete old indices?

Nope, the recommended way is to use Index Lifecycle Management. You can configure this via the Elasticsearch API, in the Kibana UI or as part of the filebeat/metricbeat configuration (what I personally prefer). The relevant sections for one of the beats charts looks like this:

filebeatConfig:
  filebeat.yml: |
    setup.ilm:
      enabled: true
      overwrite: true
      policy_file: /usr/share/filebeat/ilm.json

  ilm.json: |
    {
      "policy": {
        "phases": {
          "hot": {
            "actions": {
              "rollover": {
                "max_age": "1d"
              }
            }
          },
          "warm": {
            "min_age": "1h",
            "actions": {}
          },
          "delete": {
            "min_age": "14d",
            "actions": {
              "delete": {}
            }
          }
        }
      }
    }

If so any plans to add a curator chart, let alone official docker image?

No plans that I know of. The main other feature that curator has which is not yet part of the stack is Snapshot Lifecycle Management which is currently being worked on in https://github.com/elastic/elasticsearch/issues/38461

Crazybus on 29 Jul 2019

👍1

Great thanks for the tip! Seeing that many users today swap out filebeat for fluent(d or -bit), is there a way to declare the policy.phases directly in either ES or kibana chart values?

Or perhaps it is something to configure in the es output of fluent(d or -bit) ...

naseemkullah on 29 Jul 2019

@Crazybus Adding a policy to an index template can be be done via API to automatically applied to indexes matching the templates index pattern like you mentioned above but looks like the Kibana dashboard does not have an option to create the index template.

dcvtruong on 29 Jul 2019

is there a way to declare the policy.phases directly in either ES or kibana chart values?

Since this is a cluster wide setting my recommendation would be to have whatever automation that is orchestrating the installs/upgrades of your helm charts to also hit the Elasticsearch API to set this up. It's important to do it here so that you can properly detect if the request fails or not and do something about it.

If you really insist on having it in the values.yaml itself I think it might work to set is as a lifecycle.postStart hook. With the main downside being that you won't have any error handling and it will be applied multiple times. The tricky part will be making sure that it properly waits for the cluster to be ready when bootstrapping. The postStart hooks unfortunately fire as soon as the container is running instead of waiting for them to be ready.

@Crazybus Adding a policy to an index template can be be done via API to automatically applied to indexes matching the templates index pattern like you mentioned above but looks like the Kibana dashboard does not have an option to create the index template.

Correct. The current docs point to the Elasticsearch documentation for how to create the index template. It is possible to create the index template in the dev tools app in Kibana but I don't believe there is a GUI for it.

Crazybus on 30 Jul 2019

👍1

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.