Curator: Daily snapshots

[bobrik]

I want to keep backups of my daily indices forever and I want daily snapshots where each snapshot only contains index for specific day. Like this:

• snapshot sessions-2014-06-10 contains index sessions-2014-06-10
• snapshot sessions-2014-06-11 contains index sessions-2014-06-11

Looks like this isn't supported by curator currently. What is supported:

• Incremental snapshot with fixed name: sessions will contain each index as another "layer".
• Daily snapshots with all indices (?). Not sure how to use that to solve my problem.

My option would allow to automatically fix broken snapshots and skip duplicate snapshots for indices.

Am I missing something? Should I add PR for this?

[untergeek]

Thanks for opening this issue! I really appreciate feedback and suggestions as they drive me to make improvements.

The behavior you're describing more closely resembles the behavior in curator 1.2.0. I changed from this behavior after I came to understand snapshots better.

When I initially coded snapshot behavior, I thought that snapshots would pile up additional copies of indices, somehow, even though I knew they supported incremental backups. What I learned is that snapshots capture at the _segment_ level. As long as the segment is referenced by _any_ other snapshot in the system, deleting a snapshot will not delete the referenced data. Subsequent snapshots of the same indices simply back up any new segments since the last snapshot in that repository (incrementals).

But how does that work in practice?

Thought experiment

For the sake of this thought experiment, let's presume I am capturing daily indices, that my oldest index is 2014.06.01, and that we run daily snapshots on indices older than 1 day.

If I run curator to capture a snapshot on 2014.06.02 and tell it to capture the last 30 days, I will have a snapshot called something like curator-20140601023004 which will contain the index logstash-2014.06.01. The next night I'd have a snapshot called something like curator-20140602023006 which would contain the indices logstash-2014.06.01, & logstash-2014.06.02.

This pattern would continue as long as we let it, incrementally adding indices and snapshot names.

On 2014.07.01, my snapshot will be called curator-20140701023003 and will contain each index for the month of June. On 2014.07.02, curator deletes index logstash-2014.06.01 as it is older than 30 days. The next snapshot would contain all indices (omitting the now deleted logstash-2014.06.01) from June, and logstash-2014.07.01.

Let's leap forward in time to 90 days after 2014.06.02. If I attempt to prune snapshots older than 90 days, then the snapshot _taken_ 2014.06.02 will be deleted. Because index logstash-2014.06.01 is in that snapshot, you would think this would remove the snapshot data for index logstash-2014.06.01, but it doesn't. Why not? Because index logstash-2014.06.01 continued to appear in snapshots until 2014.07.02. You would not actually delete the snapshot _data_ for logstash-2014.06.01 until another 30 days had passed.

No such thing as duplicates

While it may make sense from an operational standpoint to have them separated, snapshots should be approached differently from a tape-backup-like, solid copy per backup way of thinking. Your indices are simply collections of segments. If fully optimized, they are only 1 segment per shard. The snapshot repository simply stores the segments and the metadata needed to reference them as "indices" (or collections of segments) when you restore. It is important to understand that if you take two snapshots of the same, fully-optimized index (and therefore the same segments) with different names (e.g. snapshot-a & snapshot-b), the repository will only have 1 copy of the shards in that index, and both snapshots will reference those segments. If I try to delete snapshot-a, the referenced segments will not be deleted because snapshot-b still references those segments. Until all snapshots referencing the segments are deleted, the segments will persist in the repository.

Restoring snapshots

You can restore individual indices from any snapshot, which implies the ability to restore to a given point in time.

Since the restore operation can pull individual indices from a snapshot (implying the referenced segments at that moment of time in history), there is no longer much reason to have snapshots be one index each. In fact, with incrementals, it's easier to figure out what point in time you want to target by having the rough timestamp in the snapshot name. Of course, even without this, the snapshot metadata indicates the time the snapshot was taken.

Failed snapshots

This is something that perhaps curator could handle better, particularly with warning the user. You will see curator log messages if there is a failure, though, unless you set wait_for_completion to false. You would still see a failure in the Elasticsearch logs, however. If a segment cannot be captured in a snapshot, it may be corrupted (which is another problem entirely), in which case it would not be able to be captured ever, or until it is repaired (if possible). Again, curator currently does not have a way to protect against this case.

In the event of a failure, the optimistic view is that Curator will simply try again the next time you run. In the noted case of a corrupted segment, there is not much curator can do.

My suggested use case for time-series indices

Because optimize can potentially improve search speed and recovery time, it makes sense to back up the indices after they've been optimized to a low number of segments per shard. Typically this is after a day or two, however, meaning there are no backups in the short term.

One solution I can recommend is to have two separate repositories (which need to have separate file system paths, otherwise you'd be writing the data to the same place), one being for short-term data, one for long-term data.

The short term snapshots should perhaps capture _all (or at least the indices you care most about), and could be run cyclicly, even as short an interval as every 5 minutes. No joke! Because only segments that change would be updated, this would keep live data backed up very quickly. You'd perhaps only keep 4 to 7 days of this data, at most because you'd be capturing indices --older-than 2 days to the long-term snapshot repository, after they were optimized. The long-term snapshots would have fully optimized segments and incrementals would not likely happen because new data is not likely to stream in for a date in the past.

Conclusion

If you still want the old behavior, please feel free to make use of curator 1.2.2. There's nothing wrong with this approach if you want to do so, but curator will support the current methodology going forward.

You're also free to write your own script using the create_snapshot() method from the Elasticsearch Curator Python API to create named snapshots the way you want them.

[bobrik]

Wow! I think you should link wiki to this comment to make it clear for new folks.

I think I'll use the next approach:

• snapshot sessions-2014-06-10 contains index sessions-2014-06-10 and older indices
• snapshot sessions-2014-06-11 contains index sessions-2014-06-11 and older indices

This way I'll have consistent view of my data for each day even though old data doesn't change. Not sure if my current backup is doing what I wanted because of elasticsearch/elasticsearch#7859.

# docker run -i -t --rm digitalwonderland/docker-elasticsearch-curator --host web487 -n snapshot --older-than 1 --prefix analytics- --timestring %Y-%m-%d --repository ceph_s3 --snapshot-name 'analytics-'$(date -d 'day ago' +%Y-%m-%d)
2014-09-27 17:16:03,023 INFO      Job starting...
2014-09-27 17:16:03,024 INFO      DRY RUN MODE.  No changes will be made.
2014-09-27 17:16:03,024 INFO      Default timeout of 30 seconds is too low for command SNAPSHOT.  Overriding to 21,600 seconds (6 hours).
2014-09-27 17:16:03,027 INFO      DRY RUN: Capturing snapshots of specified indices...
2014-09-27 17:16:03,082 INFO      analytics-2014-09-27 is within the threshold period (1 days).
2014-09-27 17:16:03,083 INFO      DRY RUN: Snapshots captured for specified indices.
2014-09-27 17:16:03,083 INFO      Done in 0:00:00.069132.

Looks like filter_by_timestamp prints excluded indices, but in context they look like included. Maybe that's just my bad english, though.

Thank you for great explanation!

[untergeek]

--older-than will only work on _full_ day indices. Since it's the 27th, it is therefore an incomplete index, and would be skipped.

[bobrik]

Yeah, but messages come in order:

2014-09-27 17:16:03,027 INFO      DRY RUN: Capturing snapshots of specified indices...
2014-09-27 17:16:03,082 INFO      analytics-2014-09-27 is within the threshold period (1 days).
2014-09-27 17:16:03,083 INFO      DRY RUN: Snapshots captured for specified indices.

And only analytics-2014-09-27 is printed after 'specified indices'. For me it looks like the only index that is going to be snapshotted is analytics-2014-09-27. No other indices are specified (printed) anyway. Maybe it's better to print included indices rather then excluded.

[untergeek]

Right! And because the method snapshot() doesn't call _op_loop(), you'll only see exclusions. I suppose in snapshot() I should add something to show what indices have been selected, if only in the debug output.

[trompx]

Hello,

I got the way elasticsearch store data as segments on the repository but so I have one more question, similar to http://elasticsearch-users.115913.n3.nabble.com/Delete-index-after-backup-td4070371.html but the answer does not appears obvious.

Say I make a snapshot of my cluster : mycluster_2015-08-27 and 10 days later I remove some data from my cluster that the snapshot point to.

How is the segments persisted in S3 with the incremental nature of the backup process ? In the doc at https://www.elastic.co/guide/en/elasticsearch/guide/current/retiring-data.html, the "Achiving Old Indices" is not really detailing what is going on.

So can I be sure that as long as a snapshot exists on the repository, all indexes deleted in the cluster would still be restorable from S3 ?

Thank you for the clarification.

[untergeek]

@trompx

Until all snapshots which reference an index (and therefore by extension, a segment) are deleted, the segment (or index) is not deleted.

So, short answer: Yes. As long as the snapshot exists (and is not corrupted), you can restore.

[trompx]

@untergeek

Thank you very much for the fast answer and precisions.

[syndy1989]

Hi,
Is it possible to take snapshot of only certain data within indices using curator?

[untergeek]

No. Not even Elasticsearch can do that. And since Curator is really only an index selection wrapper around the Elasticsearch API calls, Curator cannot do this.

[syndy1989]

Thank you very much @untergeek

[syndy1989]

Nginx: How do i forward http request to another port?

I have three ES nodes.

Node 1(primary ):http://localhost:9200

Node 2:http://localhost:9201

Node 3:http://localhost:9202

what i want to do is, if primary node fails ES will make node 2 as primary but i want the page should redirect automatically to localhost:9201 and how would i do that using nginx on windows or is there any other way to do that. Thanks in advance

[untergeek]

@syndy1989 please ask usage questions in the Elasticsearch area of https://discuss.elastic.co

That question is totally unrelated to Curator

[ssukumar-conga]

Hi,
will it be possible for curator to take all indices backup once a day and incremental for every 15 minutes