Kibana: Add Searchable Snapshot configuration to Hot and Cold phases of ILM UI

[jethr0null]

Summary

A user needs to be able to configure searchable snapshots within the Hot and Cold phases of the ILM UI.

Changes to make

• Detect license level and gray out option for searchable snapshots and display a message with guidance if not Enterprise license
• Add toggle option to enable/disable searchable snapshots in Hot phase
• Add toggle option to enable/disable searchable snapshots in Cold phase
• If the cluster is running in cloud, default repository to the found-snapshots repository
• If the cluster not running in cloud, allow a user to select an existing repository (or direct them to create one if none are available)
• Set replicas to zero by default when searchable snapshots are enabled in Cold (e.g. populate form with 0)

Outstanding Questions

• Should we be setting a default index priority (or even exposing it in the UI) when searchable snapshots is enabled in Hot? If so, what should the priority be?
• Should we be setting a default index priority (or even exposing it in the UI) when searchable snapshots is enabled in Cold? If so, what should the priority be?
• Should we be defaulting to/suggesting a particular number of segments when force merge is enabled in Hot?
• Should we be defaulting to/suggesting a particular number of segments when force merge is enabled in Cold?
• Should force merge default to true in Hot?
• Should force merge default to true in Cold?
• Should the force merge action be available in both Hot and Cold? If not, which phase should it be enabled in?
• Should the freeze action be disabled when searchable snapshots are enabled in Cold?
• Are there any other actions/settings which should be considered for either the Hot or the Cold phase?
• Are there any other actions/settings which should be disabled when searchable snapshots are enabled?

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[jloleysens]

Thanks for creating this issue @jethr0null ! I think you've answered most of my questions regarding the ask for the UI. I would just wanted to make sure I understand the Cloud case correctly:

If the cluster is running in cloud, default repository to found repository

• On cloud there will always be at least one snapshot repository configured
• It seems like defaulting to the first found snapshot is something that could benefit on-prem users too? If not I assume there is a specific concern on doing this?
• "default"ing to the first found snapshot I assume still means we will enable cloud to select another snapshot repository?

[jloleysens]

FYI I am linking to the ascii doc written for this action in ES: https://github.com/elastic/elasticsearch/blob/master/docs/reference/ilm/actions/ilm-searchable-snapshot.asciidoc

[jethr0null]

I realized that I may not have been explicit enough in my original comment. When I said found repo, I was referring to the found-snapshots repository that cloud provisions for all clusters. I'll update the original comment as well.

Can you elaborate a bit on your question about defaulting to the first found snapshot? Where in the UI are you thinking we would surface a snapshot specifically? Or earlier does my clarification answer your question?

[jethr0null]

@andreidan @jasontedor could you help to provide guidance around the outstanding questions here? Or could you please point us in the right direction of any folks who might be able to if not?

cc @sajjadwahmed @dakrone

[dakrone]

Here are my opinions:

Set replicas to zero by default when searchable snapshots are enabled in Cold (e.g. populate form with 0)

They shouldn't have to do this, it happens automatically.

Should we be setting a default index priority (or even exposing it in the UI) when searchable snapshots is enabled in Hot? If so, what should the priority be?

No, searchable snapshots inherit the priority of the original index, so it should not be changed.

Should we be setting a default index priority (or even exposing it in the UI) when searchable snapshots is enabled in Cold? If so, what should the priority be?

No, searchable snapshots inherit the priority of the original index, so it should not be changed.

Should we be defaulting to/suggesting a particular number of segments when force merge is enabled in Hot?

If it's enabled it hot (which it should not be by default), then defaulting to 1 is likely closest to what our users do.

Should we be defaulting to/suggesting a particular number of segments when force merge is enabled in Cold?

Same about defaulting to 1 segment.

Should force merge default to true in Hot?

No, this is an advanced case.

Should force merge default to true in Cold?

Force merge is not available in cold, only in hot or warm. I don't think we should default it to "on" yet.

Should the force merge action be available in both Hot and Cold? If not, which phase should it be enabled in?

It's available, I don't see a reason why we need to disallow a user from specifying both in the UI, since it's possible in the API, however, it might be useful to warn them if they double it up in case it's unintentional.

Should the freeze action be disabled when searchable snapshots are enabled in Cold?

Technically they can do this, but it's unlikely they really want to, so maybe dissuade them from doing it somehow?

Are there any other actions/settings which should be considered for either the Hot or the Cold phase?

Within the scope of searchable snapshots? I can't think of any others immediately.

Are there any other actions/settings which should be disabled when searchable snapshots are enabled?

If searchable snapshots are enabled in the hot phase, the rest of the policy cannot have a shrink, force merge, freeze, or additional searchable snapshot action configured.

[jloleysens]

Hey @dakrone @andreidan @jethr0null, I had some follow up questions on your responses to @jethr0null . I have started working on the new field and form changes here: https://github.com/elastic/kibana/pull/83783.

I'd appreciate a bit more guidance in translating some these requirements to UI. Please see the attached screenshot of the current form for reference of what all is visible (pre searchable snapshot field):

Current form

Set replicas to zero by default when searchable snapshots are enabled in Cold (e.g. populate form with 0)

They shouldn't have to do this, it happens automatically.

Does this mean in the UI we should disallow users from configuring replicas when searchable snapshots are enabled? If that is the case we should inform the user at the time of turning on searchable snapshot that replica settings are automatically set to some value and disable that field for all phases following the phase in which we are snapshotting?

Should we be setting a default index priority (or even exposing it in the UI) when searchable snapshots is enabled in Hot? If so, what should the priority be?

No, searchable snapshots inherit the priority of the original index, so it should not be changed.

Similarly, does this mean if searchable snapshot is on in hot we hide all "index priority" configuration fields or do we still show it only in hot? If we still show it in hot when snapshotting in hot, do we still show it in cold if snapshotting in cold?

Should we be defaulting to/suggesting a particular number of segments when force merge is enabled in Hot?

If it's enabled it hot (which it should not be by default), then defaulting to 1 is likely closest to what our users do.

How does this interact with the ability to turn force merge off in searchable snapshots via force_merge_index? Do we need to hide the forcemerge action in hot if snapshotting in hot?

Cheers!

[andreidan]

Does this mean in the UI we should disallow users from configuring replicas when searchable snapshots are enabled?

The searchable snapshot action sets the number of replicas to 0, in the phase where it's configured (ie. if configured in the cold phase, _after_ the searchable snapshot action is executed the index will have 0 replicas). The user can still configure the number of replicas for the index afterwards (ie. in the warm phase) should they decide it's something they want (for example, to distribute the search load)

Similarly, does this mean if searchable snapshot is on in hot we hide all "index priority" configuration fields or do we still show it only in hot?

I don't believe we should hide it. In the hot phase the set_priority action executes _before_ the searchable_snapshot - the mounted index will inherit the configured priority.

If we still show it in hot when snapshotting in hot, do we still show it in cold if snapshotting in cold?
Yes

How does this interact with the ability to turn force merge off in searchable snapshots via force_merge_index? Do we need to hide the forcemerge action in hot if snapshotting in hot?

No. We made the default true for the searchable_snapshot as it's important for the searchable snapshots performance for the index to be force merged before being converted. I don't think this configuration on the searchable_snapshot action should influence the force_merge action (if the index is force merged using the action, the force-merge step that's part of the searchable_snapshot action will be a no-op).

EDIT:
Thinking about it some more - maybe configuring the searchable_snapshotforce merge step should not be available at all in the UI (only via the API).
With regards to hiding or not the force_merge action when searchable_action is configured, I'd still say we'd have to then explain why it disappeared, which might add complexity and cognitive load (as opposed to leaving it as it won't hurt performance)