Curator: delete_indices action file is not running and doesn't show any errors

[Denagar]

hi,
i installed Elasticsearch curator ver-5.4.1, with ELK 5.6.4 single node cluster.
i have around 60 indices, and would like to remove the old ones.

i am using curator to perform snapshot, and close the specific indices. but when i try to delete them using the curator delete_indices action file. the action is not starting and nothing happens.

this is my action file

actions:
  1:
    action: 'delete_indices'
    description: >-
      Delete system indices older than 1 days (based on index name),
    options:
      continue_if_exception: False
      ignore_empty_list: False
      disable_action: False
     filters:
    - filtertype: pattern
      kind: prefix
      value: '.monitoring-kibana-6-'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 1

the log file show this

2017-12-25 13:56:56,429 DEBUG                curator.cli                    run:108  Client and logging options validated.
2017-12-25 13:56:56,429 DEBUG                curator.cli                    run:112  default_timeout = 300
2017-12-25 13:56:56,429 DEBUG                curator.cli                    run:116  action_file: delete-indices.yml

and stops.

does anyone have any idea as to why ?
appreciate your help.

[vanetix]

Having the same issue here on Ubuntu 16.04, running 5.4.1.

The above action file was running successfully on curator version 5.4.0, but after upgrading seems to just exit with the following output:

root@logstash-dev:~/.curator# curator prune-indices.yml
2018-01-22 17:31:24,782 INFO      Preparing Action ID: 1, "delete_indices"

[untergeek]

@Denagar @vanetix These errors are suggestive of misconfigured yaml files .

@Denagar
For example, in what is pasted above, filters is inappropriately indented an extra space.

@vanetix Can you double-check for properly indented yaml and try again? If that fails, please share your config.

[vanetix]

Thanks for the reply - It seemed to be an issue with the 5.4.1 build in the debian repository, removing the debian package and installed with pip fixed the issue with no alterations to the configuration.

[untergeek]

Debian has a 5.4.1 build? You're using that instead of the Elastic-provided one?

[vanetix]

Sorry for the confusing wording, I was using elasticsearch-curator version 5.4.1 from the elasticsearch debian repository, https://packages.elastic.co/curator/5/debian stable main.

[TeroPihlaja]

I'm seeing similar error on Debian with the 5.4.1 package from the elastic repository.

cli run: curator --config /root/curator-config.yml /root/curator-actions.yml

Logs:

2018-03-02 10:34:20,269 DEBUG                curator.cli                    run:108  Client and logging options validated.
2018-03-02 10:34:20,269 DEBUG                curator.cli                    run:112  default_timeout = 30
2018-03-02 10:34:20,270 DEBUG                curator.cli                    run:116  action_file: /root/curator-actions.yml
2018-03-02 10:34:20,278 DEBUG                curator.cli                    run:118  action_config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete old indices.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'airrep2-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m', 'unit': 'months', 'unit_count': 6, 'exclude': None}]}}}
2018-03-02 10:34:20,278 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'actions': <class 'dict'>}
2018-03-02 10:34:20,279 DEBUG     curator.validators.SchemaCheck               __init__:27   "Actions File" config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete old indices.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'airrep2-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m', 'unit': 'months', 'unit_count': 6, 'exclude': None}]}}}
2018-03-02 10:34:20,279 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any([In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot'])])}
2018-03-02 10:34:20,279 DEBUG     curator.validators.SchemaCheck               __init__:27   "action type" config: {'action': 'delete_indices', 'description': 'Delete old indices.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'airrep2-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m', 'unit': 'months', 'unit_count': 6, 'exclude': None}]}
2018-03-02 10:34:20,280 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any([In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot'])]), 'description': Any([<class 'str'>, <class 'str'>]), 'options': <class 'dict'>, 'filters': <class 'list'>}
2018-03-02 10:34:20,280 DEBUG     curator.validators.SchemaCheck               __init__:27   "structure" config: {'action': 'delete_indices', 'description': 'Delete old indices.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'airrep2-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m', 'unit': 'months', 'unit_count': 6, 'exclude': None}]}
2018-03-02 10:34:20,291 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'continue_if_exception': Any([<class 'bool'>, All(Any([<class 'str'>, <class 'str'>]), <function Boolean at 0x7f05ec837950>, msg=None)]), 'disable_action': Any([<class 'bool'>, All(Any([<class 'str'>, <class 'str'>]), <function Boolean at 0x7f05ec837e18>, msg=None)]), 'ignore_empty_list': Any([<class 'bool'>, All(Any([<class 'str'>, <class 'str'>]), <function Boolean at 0x7f05ec839378>, msg=None)]), 'timeout_override': Any([Coerce(int, msg=None), None])}
2018-03-02 10:34:20,292 DEBUG     curator.validators.SchemaCheck               __init__:27   "options" config: {'ignore_empty_list': True, 'continue_if_exception': False, 'disable_action': False}
2018-03-02 10:34:20,292 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: <function Filters.<locals>.f at 0x7f05ec86e730>
2018-03-02 10:34:20,292 DEBUG     curator.validators.SchemaCheck               __init__:27   "filters" config: [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'airrep2-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m', 'unit': 'months', 'unit_count': 6, 'exclude': None}]
2018-03-02 10:34:20,293 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'filtertype': Any([In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state'])]), 'kind': Any(['prefix', 'suffix', 'timestring', 'regex']), 'value': Any([<class 'str'>, <class 'str'>]), 'exclude': Any([<class 'bool'>, All(Any([<class 'str'>, <class 'str'>]), <function Boolean at 0x7f05ec86e268>, msg=None)])}
2018-03-02 10:34:20,293 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'airrep2-'}
2018-03-02 10:34:20,293 DEBUG     curator.validators.filters                      f:48   Filter #0: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'airrep2-', 'exclude': False}
2018-03-02 10:34:20,294 DEBUG     curator.defaults.filtertypes                    age:56   AGE FILTER = [{'direction': Any(['older', 'younger'])}, {'unit': Any(['seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years'])}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any([<class 'str'>, <class 'str'>])}, {'epoch': Any([Coerce(int, msg=None), None])}, {'exclude': Any([<class 'bool'>, All(Any([<class 'str'>, <class 'str'>]), <function Boolean at 0x7f05ec821bf8>, msg=None)])}, {'source': Any(['name', 'creation_date', 'field_stats'])}, {'stats_result': Any(['min_value', 'max_value'])}, {'timestring': Any([<class 'str'>, <class 'str'>])}]
2018-03-02 10:34:20,295 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'filtertype': Any([In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state'])]), 'direction': Any(['older', 'younger']), 'unit': Any(['seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years']), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any([<class 'str'>, <class 'str'>]), 'epoch': Any([Coerce(int, msg=None), None]), 'exclude': Any([<class 'bool'>, All(Any([<class 'str'>, <class 'str'>]), <function Boolean at 0x7f05ec821bf8>, msg=None)]), 'source': Any(['name', 'creation_date', 'field_stats']), 'stats_result': Any(['min_value', 'max_value']), 'timestring': Any([<class 'str'>, <class 'str'>])}
2018-03-02 10:34:20,295 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m', 'unit': 'months', 'unit_count': 6}
2018-03-02 10:34:20,295 DEBUG     curator.validators.filters                      f:48   Filter #1: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m', 'unit': 'months', 'unit_count': 6, 'stats_result': 'min_value', 'epoch': None, 'exclude': False}
2018-03-02 10:34:20,295 DEBUG                curator.cli                    run:121  Full list of actions: {1: {'action': 'delete_indices', 'description': 'Delete old indices.', 'options': {'ignore_empty_list': True, 'continue_if_exception': False, 'disable_action': False, 'timeout_override': None}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'airrep2-', 'exclude': False}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m', 'unit': 'months', 'unit_count': 6, 'stats_result': 'min_value', 'epoch': None, 'exclude': False}]}}
2018-03-02 10:34:20,296 DEBUG                curator.cli                    run:126  action_disabled = False
2018-03-02 10:34:20,296 DEBUG                curator.cli                    run:130  continue_if_exception = False
2018-03-02 10:34:20,296 DEBUG                curator.cli                    run:132  timeout_override = None
2018-03-02 10:34:20,296 DEBUG                curator.cli                    run:134  ignore_empty_list = True
2018-03-02 10:34:20,296 INFO                 curator.cli                    run:144  Preparing Action ID: 1, "delete_indices"
2018-03-02 10:34:20,296 DEBUG              curator.utils             get_client:803  kwargs = {'hosts': ['localhost'], 'port': 9200, 'use_ssl': False, 'ssl_no_validate': False, 'http_auth': '<removed>', 'master_only': False, 'client_key': None, 'client_cert': None, 'certificate': None, 'aws_key': None, 'aws_sign_request': False, 'url_prefix': '', 'aws_token': None, 'aws_secret_key': None, 'timeout': 30}
2018-03-02 10:34:20,303 DEBUG              curator.utils             get_client:880  Not using "requests_aws4auth" python module to connect.

curator-actions.yml:

actions:
  1:
    action: delete_indices
    description: >-
      Delete old indices.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: airrep2-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m'
      unit: months
      unit_count: 6
      exclude:

curator-config.yml:

client:
  hosts:
    - localhost
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth: <removed>
  timeout: 30
  master_only: False

logging:
  loglevel: DEBUG
  logfile: /var/log/curator
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

[untergeek]

Try setting blacklist: [] and run again. You should see more information about the inability to connect, or what it is actually trying.

[TeroPihlaja]

Looks like our issue was caused by incorrect x-pack permissions.

[untergeek]

Thanks for the follow-up, @TeroPihlaja

[Aketzu]

In @TeroPihlaja case it was just missing permissions. We have X-Pack with security and curator user requires

• Cluster privileges: monitor
• Indices: *
• Index privileges: monitor, delete_index

[untergeek]

Since these seem to be configuration issues, rather than bugs in Curator, I'm going to close this.

[sbellone]

Hi, it's indeed a configuration problem, but having an error message about unauthorized access, SSL failure, etc... could be a good improvement, it's misleading to have it failing silently.

[untergeek]

@sbellone see #1237

[sbellone]

I was not talking about the formatting, but the configuration itself. For example if you make a mistake in the path of the certificate, you'll get no output even in DEBUG, it just fails silently.