Elasticsearch: Use ILM to manage internal indices

[gwbrown]

Now that we have a system for curating indices built-in to Elasticsearch in Index Lifecycle Management, we should use this for internal indices which either already roll over through another mechanism or would benefit from automatically rolling over.

Some of this work has already been completed, see https://github.com/elastic/elasticsearch/pull/37443 for an example.

Internal indices which we may want to manage this way include:

• [x] Watcher
  
  
  
  • [x] .watch-history-*
  
  
• [ ] Monitoring
  
  
  
  • [ ] .monitoring-alerts-*
  
  
  • [ ] .monitoring-beats-*
  
  
  • [ ] .monitoring-es-*
  
  
  • [ ] .monitoring-kibana-*
  
  
  • [ ] .monitoring-logstash-*
  
  
• [ ] ML (maybe?)
  
  
  
  • [ ] .ml-anomalies-*
  
  
  • [x] .ml-state-*
  
  

The above list is preliminary and may be missing indices or contain indices we may choose to not manage with ILM.

[elasticmachine]

Pinging @elastic/es-core-features

[elasticmachine]

Pinging @elastic/ml-core

[ppf2]

.reporting-* from Kibana reporting is another potential candidate.

[benwtrent]

@gwbrown we have talked about doing this for .ml-anomalies* and .ml-state* indices.

Related issues:

• .ml-anomalies* https://github.com/elastic/elasticsearch/issues/29946
• .ml-state* https://github.com/elastic/elasticsearch/issues/29938

[hendrikmuhs]

related: #38805

Because ILM is a module that can be turned off, any dependency introduced must also add checks to prevent combinations like "ILM:off" + "Dependent Module:on"

[Leaf-Lin]

Adding apm-* indices as another option?

[Aqualie]

Is this still being worked on? I'm currently working through cleaning up my own indices as I have ILM setup for all my indexes and cut down my total count to just 41. Looking all the numerous system indices created it's currently at 81... 2x the amount of my own indices... It'd be great if we could use the same ILM logic and perhaps just set a total shard size of 50GB rather than these daily indices...

.monitoring-logstash-7-2020.04.30
.apm-agent-configuration
.kibana_1
.kibana_2
.kibana_3
.kibana_4
.kibana_5
.kibana_task_manager_1
.kibana_task_manager_2
.logstash
.management-beats
.ml-annotations-6
.ml-anomalies-custom-linux_anomalous_network_activity_ecs
.ml-anomalies-custom-linux_anomalous_network_port_activity_ecs
.ml-anomalies-custom-linux_anomalous_network_service
.ml-anomalies-custom-linux_anomalous_network_url_activity_ecs
.ml-anomalies-custom-linux_anomalous_process_all_hosts_ecs
.ml-anomalies-custom-linux_anomalous_user_name_ecs
.ml-anomalies-custom-packetbeat_dns_tunneling
.ml-anomalies-custom-packetbeat_rare_dns_question
.ml-anomalies-custom-packetbeat_rare_server_domain
.ml-anomalies-custom-packetbeat_rare_urls
.ml-anomalies-custom-packetbeat_rare_user_agent
.ml-anomalies-custom-rare_process_by_host_linux_ecs
.ml-anomalies-custom-rare_process_by_host_windows_ecs
.ml-anomalies-custom-suspicious_login_activity_ecs
.ml-anomalies-custom-windows_anomalous_network_activity_ecs
.ml-anomalies-custom-windows_anomalous_path_activity_ecs
.ml-anomalies-custom-windows_anomalous_process_all_hosts_ecs
.ml-anomalies-custom-windows_anomalous_process_creation
.ml-anomalies-custom-windows_anomalous_script
.ml-anomalies-custom-windows_anomalous_service
.ml-anomalies-custom-windows_anomalous_user_name_ecs
.ml-anomalies-custom-windows_rare_user_runas_event
.ml-anomalies-custom-windows_rare_user_type10_remote_login
.ml-anomalies-shared
.ml-config
.ml-meta
.ml-notifications
.ml-notifications-000001
.ml-state
.monitoring-alerts-7
.monitoring-beats-7-2020.04.28
.monitoring-beats-7-2020.04.29
.monitoring-beats-7-2020.04.30
.monitoring-beats-7-2020.05.01
.monitoring-beats-7-2020.05.02
.monitoring-beats-7-2020.05.03
.monitoring-beats-7-2020.05.04
.monitoring-beats-7-2020.05.05
.monitoring-es-7-2020.04.29
.monitoring-es-7-2020.04.30
.monitoring-es-7-2020.05.01
.monitoring-es-7-2020.05.02
.monitoring-es-7-2020.05.03
.monitoring-es-7-2020.05.04
.monitoring-es-7-2020.05.05
.monitoring-kibana-7-2020.04.29
.monitoring-kibana-7-2020.04.30
.monitoring-kibana-7-2020.05.01
.monitoring-kibana-7-2020.05.02
.monitoring-kibana-7-2020.05.03
.monitoring-kibana-7-2020.05.04
.monitoring-kibana-7-2020.05.05
.monitoring-logstash-7-2020.04.29
.monitoring-logstash-7-2020.05.01
.monitoring-logstash-7-2020.05.02
.monitoring-logstash-7-2020.05.03
.monitoring-logstash-7-2020.05.04
.monitoring-logstash-7-2020.05.05
.security-7
.tasks
.triggered_watches
.watcher-history-10-2020.04.29
.watcher-history-10-2020.04.30
.watcher-history-10-2020.05.01
.watcher-history-10-2020.05.02
.watcher-history-10-2020.05.03
.watcher-history-10-2020.05.04
.watcher-history-10-2020.05.05
.watches

[dakrone]

@Aqualie yes this is still being worked on, currently we're working on the ILM bootstrap issue that requires setting up an alias, that's part of what #53100 and #53101 are part of, once those are released it will be easier to transition these indices to use ILM.

[skmizuho]

Following...
Until this is done, I was wondering what is the best practice on autodeleting the .monitoring indices after, say 30 days?