Curator: Merge multiple indices in one

[davmrtl]

Hi,

Is it possible to add support for merging multiple old indices in one.

In the case of having an index per day, after a couple of months, we are running into having a lot of openned indices and Elasticsearch becomes slow because of that.

It would be nice if we were able to merge the old indices by weeks, by months or by year.

By example, instead of having 60 openned indices (one per day) for the last 2 months, merging by weeks the indices older than 7 days would results in:

  7 openned indices for the last 7 days
+ 7 openned indices for the other 7 weeks
-----
  14 openned indices (instead of 60 indices)

This would really help for the memory management of Elasticsearch and will help for sure the response time while querying in Kibana.

Thanks,

David

[untergeek]

This is an interesting feature request. Unfortunately, it's not as simple as merely "merging" indices. It involves a full reindexing of all indices into another index.

This is not a lightweight operation. It potentially could be very taxing to clusters, especially if you're already concerned about memory consumption.

You'd consume double the space on your cluster by the time the re-index completed (or more, depending on the number of replicas).

There are many other issues that could arise from this. I'll ponder on it.

[davmrtl]

@untergeek I'll wait for news around that case, thank you.

[bgagnon]

+1

Here is my (very) similar use case: I am a naive ELK user, so I start small and keep all defaults. This gives me one Elasticsearch index per day of logging data, which is reasonable. I get more and more data sources, which in turn yields more and more indices.

Three months later, I realize I have hundreds of indices with very little data in any of them, though a reasonable volume if I look at the sum. I now want to change the logstash config and use an index per month instead of an index per day. With Curator I could back-merge existing indices into month-based aggregates, if only this feature existed.

As for the memory / storage requirements, I don't think it's a big issue: merge indices one at a time, deleting them as soon as they have been merged. At worst, it would be equivalent to storing one extra day of data.

[tuckerpm]

+1 for bgagnon's usecase

[untergeek]

@bgagnon and @tuckerpm, these are still one-off use cases. Curator is meant for index management tasks done on a regular time table. While Curator may be metaphorically likened to a swiss army knife with lots of functionality, the more you add, the less easily used it becomes:

I don't want Curator to be like that knife. The feature you are requesting is not something that would be done on a daily basis, nor is it something likely to be done on a weekly or monthly basis either. In the event that you want to merge indices, it's actually relatively trivial to run a simple logstash instance to do what you just described:

input {
  elasticsearch {
    # ... elasticsearch connection args here...
    index => "logstash-2015.11.*"
  }
}

filter {}

output {
  elasticsearch {
    # ... elasticsearch connection args here...
    # Make Logstash re-index them to monthly indices with this addition
    index => "logstash-%{+YYYY.MM}"
  }
}

This example will ingest all documents from all indices from November 2015 and merge them into a single index. You can tweak the wildcard to get bigger or smaller swaths of indices at one time.

@davmrtl Your use case could actually be solved in a cleaner and less painful way by sending your documents to two indices simultaneously, rather than by using an _ex post facto_ merging process (which is actually just reindexing). If you still want to do _ex post facto_ reindexing, you can use a similar process to the one outlined above. However, unless you're heavily constrained by I/O and disk space, it would likely be easier to index to two destinations at once:

output {
  # Weekly indices
  elasticsearch {
    # ... other config args ...
    index => "logstash-%{+YYYY.ww}"
  }
  elasticsearch {
    # ... other config args ...
    # This is the default index pattern, merely represented here for visibility
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

It is important to note that in either case, you are still indexing twice. You just choose when and how that happens.

In this way you'd already have weekly indices, but would also have the flexibility of daily indices for fast access in Kibana. When they outlive their usefulness, you use Curator to delete them, but your data is still preserved in the weekly indices.

For the present, I don't think I'll be adding reindexing functionality to Curator for the aforementioned reason that it is an uncommon use case, and that there are tools better suited to that use case.

[markwalkom]

https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06 is another example of using LS for this

[referup-tarantegui]

I've tested @untergeek's solution. But the monthly index has less log entries (a lot less) than the sum of all daily indexes of the same month.

[untergeek]

@referup-tarantegui What version of Logstash? There would be a lot of logs in recent versions of Logstash explaining such a failure.

With regards to the original post, the _reindex API will be useful for this, but it is only available in the most recent versions of Elasticsearch (2.1+, I think). It will, at some point, be added to Curator.

[referup-tarantegui]

Hi @untergeek I've used Logstash 5.1.1 with Elasticsearch 2.4

[untergeek]

@referup-tarantegui then you should check the Logstash logs. Which solution did you use? Double-indexing or re-indexing?

[referup-tarantegui]

@untergeek The merge process finishes ok, nothing on the logstash logs. At the end one big index with a lot less entries. From 1 billion entries (30 indexes) to 100 million entries (1 index). At the moment I didn't apply anything because the results are not acceptable in production.

[untergeek]

30 to 1 is a tall order. A _very_ tall order. Are you trying to do that all at once? If you're really, seriously trying to do that in a single pass, I'm kind of not surprised it's not able to effectively query all of that.

Two options that may make this easier would be to make an alias that encompasses all 30 source indices and use that as the source "index" in the input.

The other option is to use the _reindex API.

[referup-tarantegui]

I've applied your snippet:

input {
  elasticsearch {
    # ... elasticsearch connection args here...
    index => "logstash-2015.11.*"
  }
}

filter {}

output {
  elasticsearch {
    # ... elasticsearch connection args here...
    # Make Logstash re-index them to monthly indices with this addition
    index => "logstash-%{+YYYY.MM}"
  }
}

The objective was to squash all month daily indexes into one monthly index. I've tested to to merge 7 daily indexes into a 1 weekly index, but there is data loss too.

The alias solution not applies because I want to reduce the number of old indexes.

I didn't tested the reindex API yet.

[untergeek]

The alias would only be to change index => "logstash-2015.11.*" to index => "alias_name". You can delete the alias afterwards. I think the issue here is that some part of the chain can't keep up with demand.

[referup-tarantegui]

The index solution doesn't fit for me now, because I want to know if the heap memory use is more efficient with one big index versus several indexes. That's the reason because I'm testing the merge index solution and I need to do this laboratory tests before production.

[temporafugiunt]

Little late to the party... But...
We needed to consolidate daily indexes into monthly indexes in an easy command line based way, and we are a .NET shop, so we created a command line application and a nuget package for the client library DLL for this purpose. The source is available here and can be compiled in any Visual Studio 2017 version:

https://github.com/trackitsvalue/indexify

Cheers!