Curator: [Feature] Create index command for curator

[ppf2]

Use case:

Current setup is a single daily index in Logstash, but now that index has grown too large, the user is looking to split out its document types into a few separate indices, eg. logstash_a.2015.11.13, logstash_b.2015.11.13 and logstash_c.2015.11.13 for example.

Because there are already a whole bunch of Kibana 4 dashboards built, they don't want to have to change the index pattern in K4 - because K4 doesn't currently make it easy for users to update all the existing visualizations to change the index pattern association.

Which means that they will have to change their LS config to fan out to A, B and C indices each day, and then create logstash-2015.11.13 (for example) as the name of the index alias that covers the 3 new indices, etc.. This will match what their Kibana index pattern expects.

The plan was for them to schedule the curator create alias command right after the new index rolls every day, so that the new alias (Eg. logstash-2015.11.13) is created. This is because ES does not allow creating aliases against non-existing indices (so it has to be run after the index has rolled).

The challenge here is that their admins will have to deal with daylight saving changes when scheduling the curator jobs. Alternatives are to run curator on a separate client machine that is configured to use UTC time. Or pre-create the next set of Logstash daily indices ahead of time.

Note that the 2nd approach above (pre-creating daily indices) will have to be done in a staggered way so that you don't end up with a ton of empty/future shards that will end up sharing the same indexing buffer with the current index (in 2.0, by default, it takes 5 mins for ES to mark a shard to be inactive - a direct sync flush will not prevent these shards from getting a portion of the indexing buffer).

There may be other use cases out there where having the ability to pre-create a bunch of empty/future Logstash indices will be helpful.

[ferki]

I would also like to create a new emtpy index sometimes on a scheduled basis. I might be able to send a PR about it later, but if someone can beat me to it, go ahead ;)

[untergeek]

There are some changes to this for 4.0 that allow this to work, but it's very naive right now. It lacks any magic for creating "future" indices, or at least indices with future names.

I would like to add the ability to specify an offset in seconds in the future for a name to be, and have Curator add the timedelta to the original. That would allow for creation of future indices. It's not as pressing right now, so if someone wants to look at the code and docs and figure out a way to add that, the feature will arrive sooner.

Other ideas for "future" index creation include looking at name pattern and incrementing it, i.e. index1, index2, index3. It could also be index_001, index_002, index_003, or something like that. This, combined with aliasing, could be very useful.

[untergeek]

614 adds some more functionality here.

[dudlo]

Sorry, I think this is a big hole in the feature set. I like Curator and planned on using it, rather than writing custom Bash script that shoots curl commands. I like alias action. However, the lack of support for time-based index creation defeats the purpose of using Curator.

I do not use logstash but I want to follow the pattern of daily index, for example my_index-2017-03-22. I want to run cron-triggered curator script after midnight that will:

• create new index
• run alias to update my "current", "last week",... aliases

Alias action does it nicely. But create_index cannot create date-based index name.

Please consider adding this feature.

[sundarv85]

We are currently having a lot of hacks (by write one day in the future and then running the curator for getting the alias in place), to handle this situation.

If curator can support index creation, it will be very useful

[untergeek]

https://www.elastic.co/guide/en/elasticsearch/reference/5.2/date-math-index-names.html

This is the answer! And I tested it with Curator 4.2.6 against Elasticsearch 5.2.0 (should work in the entire 5.x branch. Haven't tested the 2.x branch yet).

Using this file:

actions:
  1:
    action: create_index
    description: Create logstash index for tomorrow using Elasticsearch date math
    options:
      name: '<logstash-{now/d+1d}>'

I get this DEBUG output:

2017-03-27 14:21:34,074 DEBUG              curator.utils     parse_date_pattern:1034 Fully rendered name: <logstash-{now/d+1d}>
2017-03-27 14:21:34,074 DEBUG                curator.cli         process_action:98   Doing the action here.
2017-03-27 14:21:34,074 INFO      curator.actions.create_index              do_action:391  Creating index "<logstash-{now/d+1d}>" with settings: {}
2017-03-27 14:21:35,088 INFO                 curator.cli                    cli:203  Action ID: 1, "create_index" completed.
2017-03-27 14:21:35,089 INFO                 curator.cli                    cli:204  Job completed.

Note that it says it's 2017-03-27 14:21:35. I'm UTC-6, so in UTC terms, it's still the 27th of March.

So this action file created this output:

curator_cli show_indices --ignore_empty_list --filter_list '{"filtertype": "pattern", "kind": "prefix", "value":"logstash-"}'
...
logstash-2017.03.26
logstash-2017.03.27
logstash-2017.03.28

It created a logstash index for March 28th. This should solve your problems.