Curator: Unable to delete close indices from action: delete_indices

[umarmir]

Hi,

I am using action: delete_indices with action: close but getting error
curator ~/.curator/deleteclose.yml 2016-10-19 06:34:33,400 INFO Preparing Action ID: 1, "close" 2016-10-19 06:34:33,417 INFO Trying Action ID: 1, "close": Close 7 days older indices via creation_date 2016-10-19 06:34:35,428 ERROR Unable to complete action "close". No actionable items in list: <class 'curator.exceptions.NoIndices'>
while when i checked there were many close indices but still it giving me this. If you plz guide me.

actions:
  1:
    action: close
    description: Close 7 days older indices via creation_date
    options:
      delete_aliases: True
      ignore_empty_list: False
      timeout_override: 300
      continue_if_exception: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: .+-.+-.+-
      exclude:
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
      exclude:
  2:
    action: delete_indices
    description: Delete 10 days older indices via creation_date
    options:
      ignore_empty_list: False
      timeout_override: 300
      continue_if_exception: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: .+-.+-.+-
      exclude:
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 10
      exclude:

curator --version                                          
curator, version 4.1.2   

[untergeek]

What's happening is that when it attempts to do the "close" action, there are no indices that need to be closed that match the filters provided, hence the No actionable items in the list message.

It may be that this is the case, that they're already closed. This would result in an empty list of indices. This raises an exception, internally, because you can't close indices if there aren't any in the list. Curator provides a way to ignore an empty list condition. For example, you are setting up Curator to close indices older than 30 days, but you only have 10 days of indices right now. This would result in an empty list condition, but it is expected. Anytime where an empty list is a non-fatal exception, you can skip the current action and to move on to the next action in the configuration by setting ignore_empty_list: True in the options section.

If you switch to log in DEBUG mode, you'll have a much more complete list of what is going on. If you want me to help continue troubleshooting this, please attach a debug log (sanitize it of sensitive information if needed).

[umarmir]

Output while using curator curator --dry-run ~/.curator/deleteclose.yml and setting ignore_empty_list: True, But issue is that i am unable to delete my old close indices.

2016-10-19 07:47:09,450 INFO      Preparing Action ID: 1, "close"
2016-10-19 07:47:09,466 INFO      Trying Action ID: 1, "close": Close 7 days older indices via creation_date
2016-10-19 07:47:12,113 INFO      DRY-RUN MODE.  No changes will be made.
2016-10-19 07:47:12,113 INFO      (CLOSED) indices may be shown that may not be acted on by action "close".
2016-10-19 07:47:12,114 INFO      DRY-RUN: close: ***-***-***-2016-10-08 (CLOSED) with arguments: {'delete_aliases': True}
2016-10-19 07:47:12,114 INFO      DRY-RUN: close: ***-***-***-2016-10-09 (CLOSED) with arguments: {'delete_aliases': True}
2016-10-19 07:47:12,121 INFO      Action ID: 1, "close" completed.
2016-10-19 07:47:12,121 INFO      Preparing Action ID: 2, "delete_indices"
2016-10-19 07:47:12,125 INFO      Trying Action ID: 2, "delete_indices": Delete 10 days older indices via creation_date
2016-10-19 07:47:13,030 INFO      DRY-RUN MODE.  No changes will be made.
2016-10-19 07:47:13,030 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2016-10-19 07:47:13,031 INFO      Action ID: 2, "delete_indices" completed.
2016-10-19 07:47:13,031 INFO      Job completed.

Thanks for quick response BTW

[untergeek]

Try running with loglevel: DEBUG in your curator.yml file. You should see which indices are found matching your pattern for action id 2.

[umarmir]

It's a huge output.. last lines for action id 2, if it can help...
while running with loglevel: DEBUG

2016-10-19 08:29:19,067 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "***-all-***-2016-10-19" age (1476849551), direction: "older", point of reference, (1476016158)
2016-10-19 08:29:19,068 DEBUG          curator.indexlist       __not_actionable:39   Index ***-all-***-2016-10-18 is not actionable, removing from list.
2016-10-19 08:29:19,068 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "***-all-***-2016-10-18" age (1476763151), direction: "older", point of reference, (1476016158)
2016-10-19 08:29:19,068 DEBUG          curator.indexlist       __not_actionable:39   Index ***-***-***-2016-10-18 is not actionable, removing from list.
2016-10-19 08:29:19,068 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "***-***-***-2016-10-18" age (1476787306), direction: "older", point of reference, (1476016158)
2016-10-19 08:29:19,068 DEBUG          curator.indexlist       __not_actionable:39   Index ***-***-***-2016-10-19 is not actionable, removing from list.
2016-10-19 08:29:19,068 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "***-***-***-2016-10-19" age (1476871167), direction: "older", point of reference, (1476016158)
2016-10-19 08:29:19,068 DEBUG          curator.indexlist       __not_actionable:39   *** ***-***-en-2016-10-15 is not actionable, removing from list.
2016-10-19 08:29:19,069 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "***-***-en-2016-10-15" age (1476523102), direction: "older", point of reference, (1476016158)
2016-10-19 08:29:19,069 DEBUG          curator.indexlist       __not_actionable:39   Index ***-***-en-2016-10-14 is not actionable, removing from list.
2016-10-19 08:29:19,069 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "***-***-en-2016-10-14" age (1476438943), direction: "older", point of reference, (1476016158)
2016-10-19 08:29:19,069 DEBUG              curator.utils        iterate_filters:847  Post-instance: [u'***-***-si-2016-10-09']
2016-10-19 08:29:19,069 DEBUG     curator.actions.delete_indices               __init__:334  master_timeout value: 300s
2016-10-19 08:29:19,069 INFO               curator.utils           show_dry_run:620  DRY-RUN MODE.  No changes will be made.
2016-10-19 08:29:19,069 INFO               curator.utils           show_dry_run:623  (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2016-10-19 08:29:19,070 INFO               curator.utils           show_dry_run:630  DRY-RUN: delete_indices: ***-***-si-2016-10-09 (CLOSED) with arguments: {}
2016-10-19 08:29:19,070 INFO                 curator.cli                    cli:202  Action ID: 2, "delete_indices" completed.
2016-10-19 08:29:19,071 INFO                 curator.cli                    cli:203  Job completed.

[untergeek]

A few more lines might be more instructive. You can also attach the file by dragging & dropping or selecting them (see the message at the bottom of the comment entry area).

It appears, however, that it can only find one index that is older than 10 days that scheduled for deletion.

[umarmir]

All output that i have
Output.txt

[untergeek]

So, here's an example.

2016-10-19 08:29:18,983 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "***-all-***-2016-09-29" age (1476283462), direction: "older", point of reference, (1476016158)

This entry has a date stamp on the index name of 2016-09-29. But the creation date is 1476283462, which is Wed, 12 Oct 2016 14:44:22 GMT. Your point of reference, which is 10 days before the time of execution, is 1476016158, or Sun, 09 Oct 2016 12:29:18 GMT.

I can see by this that your data is older than the index it was put into. If you need to delete this older data, perhaps you should change your age filter to be:

    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 10

Try that with --dry-run and see if that does what you expect. Using source: name changes the source of age to be derived from the index name, rather than creation_date, which is when the index was created in Elasticsearch. It's apparent that you've ingested some older data.

[umarmir]

We know about this issue that we are still getting data in old date named indices, but it's separate thing. Here i am mentioning two entries which were created on 09 Oct 2016 with same date convention currently they are closed. But action: delete_indices unable to delete them, as they are 10 days old and closed indices.

2016-10-19 08:29:18,984 DEBUG curator.indexlist __excludify:58 Removed from actionable list: Index "***-***-en-2016-10-09" age (1476022761), direction: "older", point of reference, (1476016158)
creation date 1476022761 Sun, 09 Oct 2016 14:19:21 GMT

2016-10-19 08:29:19,024 DEBUG curator.indexlist __excludify:58 Removed from actionable list: Index "***-***-en-2016-10-09" age (1476019759), direction: "older", point of reference, (1476016158)
creation date 1476022761 Sun, 09 Oct 2016 13:29:19 GMT

[untergeek]

Curator doesn't consider either to be old enough to be acted on.

Point of reference: 1476016158, which is Sun, 09 Oct 2016 12:29:18 GMT. The point of reference is unit_count * unit * # of seconds in a "unit" subtracted from the epoch time at Curator run time.

The times are measured exactly, and a strict greater than or less than marks the cutoff. By that measure, those indices are not going to be caught by Curator, as is indicated in the log file:

"***-***-en-2016-10-09" age (1476022761) is Sun, 09 Oct 2016 14:19:21 GMT, which is almost 2 hours younger than the cutoff, so it doesn't get deleted.

"***-***-en-2016-10-09" age (1476019759) is Sun, 09 Oct 2016 13:29:19 GMT, which is one hour younger than the cutoff, so it doesn't get deleted.

[untergeek]

As it's been 2 hours since my last comment, if you re-run with no changes, you should see these indices acted on. Is that the case?

[umarmir]

Thanks, Now it make sense.
Now that indices gone.

It's working fine.