Elasticsearch: Unable to override index setting soft_delete when restoring snapshot

[NickFane]

Background
My team is working on implementing CCR to replicate data to another cluster of ours. The index we want to replicate is around 110gb, and was created about a week back on version 7.9. We are unable to replicate the index due to missing operations, but upon later investigation, we found that soft_delete had defaulted to false (which we are unsure if it is due to a bug, or bad config on our end).

We contacted Elastic support, and they mentioned to try restoring a snapshot, but overriding the index settings to set soft_delete to "true". The assumption here is that with soft_deletes now set to true, the index will retain the operations used to restore the snapshot. If this is the case, we can replicate the newly restored snapshot (which now contains all the required operations).

When we tried this, we found that we couldn't override this setting with snapshots.

Request
The feature request is for soft_deletes to be able to be overridden during the restoration of a snapshot. In order to be able to use features such as CCR that rely on these internal operations, on large indices which no longer contain them (or perhaps never did).

Also for extra info, we've also tried simply reindexing the data, however, we've found that with some recommended settings (such as [slices == shardCount]), that it takes hours to finish the task. The added load this has on our cluster, causes us to not be able to run this in a production environment.

[dnhatn]

@NickFane Thanks for reporting the issue. The current validation is too restricted. We will make a fix for this.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Engine)

[dnhatn]

@NickFane In the meantime, you can use clone API to create a new index with soft-deletes enabled.

[dnhatn]

I've opened https://github.com/elastic/elasticsearch/pull/62018.