Kibana: [Ingest Manager] Support custom ILM policies

[neptunian]

Currently all index templates point to the corresponding type of data ILM policy which comes installed in ES calls logs or metrics. We need to support a custom ILM policies, either per package or per dataset or both. @ruflin recently made a change to a dataset manifest file to override index template mappings or settings: https://github.com/elastic/package-registry/issues/497. But I think the ILM policy file can be added per dataset in the ilm_policy directory before the index template is created and referenced there. The index template should reference the most specific policy. Most specific being the dataset and least the global). Policies will need to be tracked and removed like other installation assets.

Note: There is a stack monitoring package that @ycombinator will be introducing in the future that will need ILM policies per dataset package.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ycombinator]

There is a stack monitoring package that @ycombinator will be introducing in the future that will need ILM policies per dataset.

Strictly speaking it will be per _package_ but we could define the same policy per dataset too, if that's all we decide to support.

[ph]

@mostlyjason I don't think we need that with data tier? correct?

[mostlyjason]

Yes we are planning a data tiers feature that will allow users to set retention and other settings. @mukeshelastic is helping to lead that