Elasticsearch: Atomically Rename Index

[pickypg]

This is a duplicate of #17426, which was closed back in March, 2018. However, things have changed rather significantly where this feature has become more of a usability issue than a convenient way to fix an index's name.

Problem

With the introduction of both Rollover, particularly within ILM, the need to know a preexisting alias "name" is largely core to the usage of the API and using it is incredibly useful.

However, in a lot of very common usage, users may find themselves not having enough knowledge of ES to understand the bootstrapping requirement, and more importantly they may not be able to stop an automated service once the bootstrapping has been setup (e.g., imagine a bunch of Beats pushing to mybeatindex, but you delete the backing indices). Imagine having to stop all of those Beats instances just to create the alias again (noting that they would try to create the index using the alias' name and then you would struggle to create a new index with that alias), or imagine someone deleting all of their indices and not resetting the alias before the next payload arrives.

Solution

Although _much_ easier said than done, it would be incredibly helpful if the aliases API could atomically rename an index as part of setting up aliases.

If we supported that behavior, then any tool talking to ES could blindly assume that an alias exists -- no bootstrapping required -- and features like Rollover could automatically setup the alias in the same step as pointing its alias to the next index. This dodges all of the frustration with handling the alias in a scenario where things are already running.

Alternative Solution

A possible alternative may be to support renaming an index on creation via its template if it matches the alias. This could possibly be akin to kicking off rollover to get its non-colliding index pattern.

Partial Workaround

For users that need to delete all of their indices behind a Rollover alias, the right solution is to pre-create a new index and point the alias to it _before_ deleting the other indices. However, this again comes back to understanding the bootstrapping process.

[elasticmachine]

Pinging @elastic/es-core-features

[pickypg]

I have run into another need for this:

_Weeks_ worth of weekly metricbeat indices were mapped incorrectly (missing the template), which breaks the Infra UI. The solution is pretty simple: reindex and apply an alias for today.

However, I cannot reindex with live indexing in play, thus my options are:

• Wait for the template to take effect for next week's index, then reindex previous indices.
• Reindex data and accept that I will have a small gap when I inevitably delete / add the alias to the reindexed version.

Neither option really gives me a complete solution, but if I could atomically rename the index while applying an alias to its replacement index, then I could divert indexing to the aliased indexing _and_ reindex into it at the same time with zero downtime, zero issues, and an immediately working UI with data being backfilled.

[dakrone]

We discussed this in the core/features meeting and decided that while we would definitely like to do this. It's technically challenging. I spoke with Simon about it and one of the prerequisites for being able to easily do this is to switch all the instances where we use a String to represent an index to the actual Index class.

For now I'm removing the discuss and leaving this as high-hanging fruit.

[jcttrll]

This may be impractical, but one option I looked for in the index templates feature is forcing the index name at creation time, as well as applying an alias. The force-name feature doesn't exist, however.

As a concrete example, imagine LogStash (or anything else) writes to indexes like app-2019-08-20. It takes advantage of automatic index creation. However, what I _really_ want is for LogStash to write to the _alias_ app-2019-08-20, which corresponds to some index (some other name I couldn't care less about--could be randomly generated, for all I care). This enables me to atomically flip that alias to another index, then reindex the real index into the second index. But LogStash--like other things taking advantage of automatic index creation--doesn't have the ability to create index A and then write to index (alias) B forever afterward. It just starts writing to A and always writes to A.

This isn't extremely well thought through, but my idea is something like this in the index template:

{
    "order": 10,
    "index_patterns": [
      "app-*"
    ],
    "rename": "{index}-something",
...
}

Given the index name app-2019-08-20, this would create an index named app-2019-08-20-something with an alias of app-2019-08-20.